Introducing new open-source tools for the Elixir community
Steve Cohen | Pinterest engineer, Data
The eighties aren’t typically remembered as a somber and serious decade. Yet, while the masses were listening to Wang Chung’s “Everybody have fun tonight” and hoping for a DeLorean based time machine, several engineers at a Swedish telecom company were accidentally inventing the future. They were solving a problem for telecommunications, but by a strange coincidence, the problem they were solving also turned out to be very similar to the problems faced by engineers that would see that DeLorean time machine after it completed its 30 year jump to the future.
Despite its many advantages, this magical system never really got traction, but spread like an urban legend: Spoken in hushed tones by seasoned engineers, who always heard about it third hand. Sure, you knew a cousin who had a roommate that knew a gal that ran a couple million connections off of two servers, but you never actually believed that stuff anyway.
Well, I’m here to tell you that the stories were real. And there’s a sequel.
I am, of course, talking about Erlang, and while Erlang has had some recent high-profile success at WhatsApp, it also has some pretty nasty legacy due to the fear around “the syntax”. While those intrepid engineers at Ericsson got so much right, they based their creation on Prolog, which never caught on outside of certain niche communities. And because of this, they made it easy for programmers to declare, upon seeing Erlang for the first time, that it “looks weird” or “makes my eyes hurt.” That dismissal might not sound like much, but for years, it has hampered the adoption of Erlang.
Until now.
Enter one José Valim, the Rails core committer who grew tired of making Rails handle concurrency and sought refuge in Erlang’s warm embrace. This wasn’t enough, because he realized how people dismissed with pithy comments about its syntax. After tireless effort, the results are quite amazing: The Elixir Language. Elixir provides a syntax that a Python or Ruby programmer will find acceptable and one that maps 1:1 back to native Erlang. It actually goes further than that and gives you a small, powerful language, which is why we’re so excited about Elixir at Pinterest.
Elixir (and Erlang) are exciting to us because of the way they handle parallelism and errors. One of the surprising things about computers is they haven’t gotten much faster since around 2005. What has happened is that we can now fit more cores on a CPU, turning what was one processor into many. The problem with this is that programming languages haven’t really made taking advantage of these cores particularly easy. You might be able to create a thread, but managing that thread and handling shared state is really, really tough. What Elixir does differently is to run inside of an Actor system. An Actor system is just something that implements a couple of rules, which are:
- An Actor has a mailbox.
- Actors communicate with other Actors by sending them immutable messages.
- Messages are put into the Actor’s mailbox.
- When an Actor’s mailbox has a message, code is run with that message as an argument. This code is called serially.
- When an Actor encounters an error, it dies.
- Actors can supervise other Actors, and if the supervised Actors die, the supervisor is sent a message. If this message isn’t handled, the supervisor dies.
It doesn’t sound like much, but these rules make handling concurrency very simple, and recovering from errors possible. One of the things I didn’t tell you about Elixir is that it’s a functional language and has no mutable data structures. For example, if you have a Map and want to add a key and value to that Map, you call a function which returns a new Map with the key and value.
my_map = HashDict.new
other_map = HashDict.put(my_map, :my_age, 39)
IO.inspect(my_map)
#HashDict<[]> # my_map has not been changed
IO.inspect(other_map)
#HashDict<[my_age: 39]> # other_map has the new valuesYou can see from the above that the original Map is not changed. Because of this property, the Erlang VM can pause execution at any time and location in the code, resuming right where it left off at some time in the future. Most non-functional programming languages cannot do that, and this property enables Elixir and Erlang systems to keep your processors busy all the time.
Another core philosophical underpinning of Elixir is that “most errors are transient and caused by the accumulation of bad state.” This happens all the time in running systems, which is why restarting them fixes many problems. What if this behavior was a part of the system though? Well, then you’d have Elixir, Erlang, and now you understand the beginning of the “let it crash” philosophy.
Because Actors can monitor one another, they can also restart one another, and, in the case of a failure, an actor can be restarted with a known good state. This is really important, because it lets transient errors be truly transient, and it prevents you from writing a lot of error handling code. Heck, it prevents you from writing a lot of code that handles out of bound conditions in your data, preferring instead to code to the “happy path” and letting the Actor crash and burn if something else happens. This is, of course, a gross oversimplification, but when coding in Elixir, you find yourself writing less boilerplate error handling code and more business logic.
There are other reasons to love Elixir, too. It has excellent runtime instrumentation. I once fixed a production system while it was running by logging into a remote console, diagnosing the problem and restarting the logging application, which was overloaded due to a backend restart. Elixir also has a clean and simple standard library that’s very thoughtfully designed. The language makes heavy use of pattern matching, a technique that prevents *value* errors which are much more common than *type* errors. It also has an innovative pipelining operator, which allows data to flow from one function to the next in a clear and easy to read fashion. To get a feel for pipelining, contrast the two bits of code in Python and Elixir:
python
" ".join(map(str.capitalize, "hello_there_world".split("_")))
elixir
"hello_there_world"
|> String.split("_")
|> Enum.map(&String.capitalize\1)
|> Enum.join(" ")Notice how in the python example, everything is backwards and spills out from a nested inner context to an outer context. Now contrast how linear and clear the Elixir code is, processing the data step by step. The output of the previous step is automatically sent to the next function call as the first parameter. This technique is inspired by pipes in Unix. Gorgeous.
So, we like Elixir and have seen some pretty big wins with it. The system that manages rate limits for both the Pinterest API and Ads API is built in Elixir. Its 50 percent response time is around 500 microseconds with a 90 percent response time of 800 microseconds. Yes, microseconds.
We’ve also seen an improvement in code clarity. We’re converting our notifications system from Java to Elixir. The Java version used an Actor system and weighed in at around 10,000 lines of code. The new Elixir system has shrunk this to around 1000 lines. The Elixir based system is also faster and more consistent than the Java one and runs on half the number of servers.
Wait, there’s more.
We didn’t write this blog post just to crow about a new programming language, though that would be kind of awesome. We also love open-source, and we’re going to give some code to the Elixir community. In that light, we’re excited to announce two projects that we’ve been leaning on quite heavily, Elixometer and Riffed.
Elixometer
Elixometer is a wrapper around the already excellent Exometer library, which provides stats and logging. You can’t fix what you can’t measure, and Exometer provides visibility across your code and even into included libraries and the Erlang VM itself. Elixometer enhances Exometer and provides Elixir-style access to its deep logging and stats collecting functionality.
Riffed
We use the Thrift RPC mechanism heavily, and sadly, there’s no Thrift implementation for Elixir. While there is an Erlang implementation, for a variety of reasons, it’s not very friendly for an Elixir developer. Riffed fixes this and makes heavy use of Elixir macros to rewrite the Erlang implementation into something an Elixir developer will love.
We’re pretty jazzed about Elixir and love building highly concurrent, fast and scalable servers in it. We’re also big fans of open-source, and you should watch this space for more Elixir-y goodness. If this sounds like something you’d like to be a part of and you have deep knowledge of the Erlang VM, get in touch with us!
