Patrice

In programming, "speed" is not so simple to measure. There are a few measures you look at: Concurrency: The number of things your system can handle at once. Throughput: The actual number of requests served per second in real world conditions. Latency: How long it takes for any one request under real world conditions to complete. This includes queueing delay. Queueing Delay: How long a request has to wait at the front door before your system even starts processing it. You typically are playing a balancing act between maximizing throughput and minimizing latency on any single server, while increasing number of servers (capacity) to account for actual load. A system under load does not behave the same as a system with spare capacity. Raw speed doesn't save you under contention, because "just do the thing as fast as it can possibly be done" is just insufficient. Instead, you have to write your code in such a way that any one given thing can "yield" to other things. If you don't, the requests you're working on finish quickly, but once all of your threads/processes are fully utilized, you have no way to handle more load without introducing queueing (which eventually becomes failures/timeouts) on callers. This brings us to queueing delay. When the system is processing all of the data that it can, the callers have to all just sit there and either wait, retry or fail. If your system were able to accept more concurrent work, you can lower this queuing delay under heavy load. If you can't do that, you end up trading away too much of your goodput (requests that actually succeed within their deadline) in exchange for consistently low latency for the operations that you do handle. When building systems at scale, the general guiding light is keeping the queue delay low, and keeping goodput as high as possible. To do this, you accept as much work as the internal system can track (increasing memory usage typically), allowing latency to go up, in order to accept more concurrent traffic internally. You still need backpressure and load shedding, since no scheduler saves you from arrival rates that permanently exceed what you can serve, but within those bounds you keep goodput maximized. For this to actually work, your internal schedulers need to be able to do something called "preemption". This is where any given operation cannot take an entire CPU core for longer than some given period of time, or for some amount of operations. A "preemptive scheduler" balances work fairly across all of the operations that are asking for CPU cycles. (Work stealing, where idle cores pull queued work from busy ones, helps too, but plenty of cooperative runtimes have that. Preemption is the key part.) The BEAM is a powerhouse for this. It preempts processes by counting "reductions," so no process can hog a scheduler. Immutability and share-nothing processes make that preemption cheap and harmless: a process is never paused while holding a lock someone else needs, and every process has its own heap and garbage collector, so there are no global stop-the-world pauses. This allows BEAM applications to achieve maximal goodput, with very low failures due to queue delay, while spreading latency across all work fairly. Without that preemption, you would see a huge spread among the latency of your system. i.e some requests happening practically immediately, while other requests take pathologically large amounts of time to complete. I get that people want programming language not to matter these days, but at a very minimum you cannot escape the limitations of your runtime. The BEAM's runtime is built for operating at scale, and will always be smoked on benchmarks for tight loop operations where there is no resource contention. #ElixirLang