--- layout: default title: 'A stateful key/value server in Elixir' --- I wanted to make a simple key-value server in Elixir - json in, json out, GET, POST, with an in-memory map. The point is to [reinvent the wheel](https://blog.codinghorror.com/dont-reinvent-the-wheel-unless-you-plan-on-learning-more-about-wheels/), and learn me some Elixir. My questions were: a) how do I build this without Phoenix and b) how do I persist state between requests in a functional language? Learning new stuff is always painful, so this was frustrating at points and harder than I expected. But I want to emphasize that I did get it working, and do understand a lot more about how Elixir does things - the community posts and extensive documentation were great, and I didn't have to bug anyone on StackOverflow or IRC or anything to figure all this out. Here's what the learning & development process sounded like from inside my head. 1. First, how do I make a simple JSON API without Phoenix? I tried several tutorials using [Plug](https://hexdocs.pm/plug/readme.html) alone. Several of them were out of date / didn't work. Finally [found this one](http://samueldavies.net/2017/04/29/how-to-build-a-lightweight-webhook-endpoint-in-elixir-with-plug-not-phoenix/), which was up-to-date and got me going. `Ferret` was born! 2. How do I reload code when I change things without manually restarting? Poked around and found [the remix app](https://github.com/AgilionApps/remix). 3. Now I can take JSON in, but how do I persist across requests? I think we need a subprocess or something? That's [what CodeShip says](https://blog.codeship.com/statefulness-in-elixir/), anyhow. 4. Okay, I've got an Agent. So, where do I keep the agent PID so it's reusable across requests? 5. Well, where the heck does Plug keep session data? [3] That should be in-memory by default, right? Quickly, [to the source code](https://github.com/elixir-lang/plug)! 6. Hrm, well, [that doesn't tell me a lot](https://github.com/elixir-lang/plug/blob/master/lib/plug/session.ex). Guess it's abstracted out, and in a language I'm still learning. 7. Maybe I'll make a separate plug to initialize the agent, then dump it into the request bag-of-data? Pretty sure `plug MyPlug, agent_thing: MyAgent.start_link` will work. Can store that in my Plug's options, then add it to `Conn` so it's accessible inside requests 8. Does a plug's `init/1` get called on every request, or just once? What about my Router's `init/1` ? Are things there memoized? 9. Guess I'll assume the results are stored and passed in as the 2nd arg to `call/2` in my plug. 10. Wait, what does `start_link` return? ``` 14:15:15.422 [error] Ranch listener Ferret.Router.HTTP had connection process started with :cowboy_protocol:start_link/4 at #PID<0.335.0> exit with reason: \{\{\%MatchError{term: [iix: {:ok, #PID<0.328.0>}]} ``` 11. WHY DO I KEEP GETTING THIS?! ``` ** (MatchError) no match of right hand side value: {:ok, #PID<0.521.0>} (ferret) lib/plug/load_index.ex:10: Ferret.Plug.LoadIndex.init/1 ``` 12. *figures out how to assign arguments* turns out `[:ok, pid]` and `{:ok, pid}` and `%{"ok" => pid}` [are different things](https://elixir-lang.org/getting-started/keywords-and-maps.html) 13. *futzes about trying various things to make that work* 14. How do I log stuff, anyway? Time to [learn Logger](https://hexdocs.pm/logger/Logger.html). 15. THE ROUTE IS RIGHT THERE WHAT THE HELL?! ``` 14:29:45.127 [info] GET /put 14:29:45.129 [error] Ranch listener Ferret.Router.HTTP had connection process started with :cowboy_protocol:start_link/4 at #PID<0.716.0> exit with reason: \{\{\%FunctionClauseError{arity: 4, ``` 16. *half an hour later* - Oh, I'm doing a GET request when I routed it as POST. I'm good at programmering! I swear! I'm smrt! 17. Turns out `Conn.assign/3` and `conn.assigns` are how you put things in a request - not `Conn.put_private/3` [like plug/session uses](https://github.com/elixir-lang/plug/blob/master/lib/plug/session.ex#L59). 18. Okay, I've got my module in the request, and the pid going into my KV calls 19. WTF does this mean?!?! ``` Ranch listener Ferret.Router.HTTP had connection process started with :cowboy_protocol:start_link/4 at #PID<0.298.0> exit with reason: {\{:noproc, {GenServer, :call, [#PID<0.292.0>, ``` 20. *bloody hours pass* 21. The `pid` is _right bloody there!_ `Logger.debug` shows it's passing in the same pid for every request! 22. Maybe it's keeping the `pid` around, but the process is actually dead? How do I figure that out? *tries various things* 23. Know what'd be cool? `Agent.is_alive?` . Things that definitely don't work: 1. `Process.get(pid_of_dead_process)` 2. `Process.get(__MODULE__)` 3. `Process.alive?(__MODULE__)` Which is weird, since an Agent is a GenServer is a Process (as far as I can tell). [This article](http://eddwardo.github.io/elixir/2015/10/22/elixir-pingpong-table/) on "Process ping-pong" was helpful. 24. Finally figured out to use `GenServer.whereis/1` , passing in `__MODULE__` , and that will return `nil` if the proc is dead, and info if it's alive. 25. Turns out I don't need my own plug at all: just init the Agent with the `__MODULE__` name, and I can reference it by that, [just like a GenServer](https://hexdocs.pm/elixir/GenServer.html#module-name-registration). 26. IT'S STILL SAYING `:noproc` ! JEEBUS! 27. Okay, I guess remix doesn't re-run `Ferret.Router.init/1` when it reloads the code for the server. So when my Agent dies due to an ArgumentError or whatever, it never restarts and I get this `:noproc` crap. 28. I'll just manually restart the server - I don't want to figure out supervisors right now. 29. This seems like it should work, why doesn't it work? `Agent.get_and_update __MODULE__, &Map.merge(&1, dict)` 30. Is it doing a javascript async thing? Do I need to tell Plug to wait on the response to `get_and_update` ? 31. Would using `Agent.update` and then `Agent.get` work? Frick, I dunno, how async are we getting here? [All](http://blog.songsaboutsnow.com/elixir/processes/2016/05/20/using-processes-to-hold-state.html). [the](https://hexdocs.pm/elixir/Agent.html#module-examples). [examples](https://elixir-lang.org/getting-started/mix-otp/agent.html#other-agent-actions). use a pid instead of a module name to reference the agent. 32. How would I even tell plug to wait on an async call? 33. Oh, frickin'! `get_and_update/3` has to [return a _tuple_](https://hexdocs.pm/elixir/Agent.html#get_and_update/3) , and there's no function that does single-return-value-equals-new-state. I need a function that takes the new map, merges it with the existing state, then duplicates the new map to return, but `get_and_update/3` 's function argument _only receives the current state_ and doesn't get the arguments. `get_and_update/4` supposedly passes args, but you have to pass a Module & atom instead of a function. I couldn't make that work, either. 34. Does Elixir have closures? I mean, that wouldn't make a lot of sense from a "pure functions only" perspective, but in Ruby it'd be like ```ruby new_params = conn.body_params Agent.get_and_update do |state| new_state = Map.merge(state, new_params) [new_state, new_state] end ``` ...errr, whelp, no, that doesn't work. 35. The [Elixir crash-course guide](https://elixir-lang.org/crash-course.html#partials-and-function-captures-in-elixir) doesn't mention closures, and I'm not getting how to do this from the examples. 36. *hours of fiddling* 37. uuuuugggghhhhhhhh [functional currying monad closure pipe recursions](https://onor.io/2014/03/31/partial-function-application-in-elixir/) are breaking my effing brain. You have to [make your own](http://blog.patrikstorm.com/function-currying-in-elixir) curry, or [use a library](https://github.com/Qqwy/elixir_currying). This seems unnecessary for such a simple dang thing. 38. Is there a difference between `Tuple.duplicate(Map.merge(&1, dict), 2)` and `Map.merge(&1, dict) |> Tuple.duplicate(2)` ? I dunno, neither one of those are working. 39. What's the difference between????? 1. `def myfunc do ... end ; &myfunc` 2. `f = fn args -> stuff end ; &f` 3. `&(do_stuff)` 40. Okay, this is what I want: ​ `&(Map.merge(&1, dict) |> Tuple.duplicate 2)` Why is `dict` available inside this captured function definition? I dunno. 41. **BOOM OMG IT'S WORKING!** Programming is so cool and I'm awesome at it and this is the best! 42. Let's `git commit`! 43. Jeebus, I better write this crap down so I don't forget it. Maybe someone else will find it useful. Wish I coulda Google'd this while I was futzing around. 44. I'm gonna go murder lots of monsters [with my necromancer](http://www.eurogamer.net/articles/2017-06-27-diablo-3s-necromancer-is-great-but-it-cant-raise-diablo-2-from-the-dead) while my brain cools off. Then hopefully come back and figure out: 1. functions and captures 2. pipe operator's inner workings 3. closures??? 4. supervisors ## Links I used: [Elixir Getting-Started Guide](https://elixir-lang.org/getting-started/introduction.html) [Maps: elixir-lang](https://elixir-lang.org/getting-started/keywords-and-maps.html#maps) [Logging with Logger](http://elixir-recipes.github.io/logging/logging-with-logger/) ### Processes & State [Statefulness in a Stateful Language (CodeShip)](https://blog.codeship.com/statefulness-in-elixir/) [Processes to Hold State](http://blog.songsaboutsnow.com/elixir/processes/2016/05/20/using-processes-to-hold-state.html) [When to use Processes in Elixir](https://medium.com/elixirlabs/when-to-use-processes-in-elixir-18287da73d47) [Elixir Process Ping-Pong](http://eddwardo.github.io/elixir/2015/10/22/elixir-pingpong-table/) [Using Agents in Elixir](http://culttt.com/2016/08/17/using-agents-elixir/) [Agent - elixir-lang](https://elixir-lang.org/getting-started/mix-otp/agent.html) [Concurrency Abstractions in Elixir (CodeShip)](https://blog.codeship.com/concurrency-abstractions-in-elixir/) [GenServer name registration (hexdocs)](https://hexdocs.pm/elixir/Agent.html#module-name-registration) [GenServer.whereis](https://hexdocs.pm/elixir/GenServer.html#whereis/1) - for named processes [Agent.get_and_update (hexdocs)](https://hexdocs.pm/elixir/Agent.html#get_and_update/3) - hope you are good with currying: no way to pass args into the update function unless you can pass a module & atom (and that didn't work for me) ### Plug [How to build a lightweight webhook endpoint with Elixir](http://samueldavies.net/2017/04/29/how-to-build-a-lightweight-webhook-endpoint-in-elixir-with-plug-not-phoenix/) [Plug (Elixir School)](https://elixirschool.com/lessons/specifics/plug/#adding-another-plug) - intro / overview [Plug body_params](https://stackoverflow.com/questions/34165697/how-do-i-get-the-request-body-from-a-connection) - StackOverflow [plug/session.ex](https://github.com/elixir-lang/plug/blob/master/lib/plug/session.ex) - how do they get / store session state? [Plug.Conn.assign/3 (hexdocs)](https://hexdocs.pm/plug/Plug.Conn.html#assign/3) [Plug repo on Github](https://github.com/elixir-lang/plug) ### Function Composition [Currying and Partial Application in Elixir](http://stratus3d.com/blog/2014/06/27/currying_and_partial_application_in_elixir/) [Composing Elixir Functions](https://www.bignerdranch.com/blog/composing-elixir-functions/) [Breaking Up is Hard To Do](https://www.bignerdranch.com/blog/breaking-up-is-hard-to-do-how-to-decompose-your-code/) [Function Currying in Elixir](http://blog.patrikstorm.com/function-currying-in-elixir) [Elixir Crash Course](https://elixir-lang.org/crash-course.html#partials-and-function-captures-in-elixir) - partial function applications [Partial Function Application in Elixir](https://onor.io/2014/03/31/partial-function-application-in-elixir/) [Elixir vs Ruby vs JS: closures](https://www.amberbit.com/blog/2015/6/14/closures-elixir-vs-ruby-vs-javascript/)