I tried to build a load balancer

#Why

So, I was wondering how an application like NGINX actually works. I've always taken for granted that I can put my web apps behind it and have it route thousands of requests to different ports.

So I wanted to open that box a little and understand how things actually work under the hood.

#And why Elixir

No specific reason, really. Elixir has been on my radar since I started studying functional programming, and I like its syntax.

That said, it ended up being a surprisingly good choice for this, for a few reasons I'll get to later.

#Accept incoming HTTP requests

To tell you the general idea. I spin up three simple containers, each one is a backend that my load balancer is gonna redirect requests to, in ports 4001 4002 and 4003

First, I had to set up a socket to listen for incoming requests. The whole idea of a load balancer is request -> LB -> Backend, so before anything else, I need to be able to accept that first request.

Turns out this is trivial in Elixir with listen_socket, and it can be done in about 15 lines of code: I bind port 9090, accept a connection, and print the raw bytes of the HTTP request.

{:ok, listen_socket} =
  :gen_tcp.listen(9090, [
    :binary,
    packet: :raw,
    active: false,
    reuseaddr: true
  ])
IO.puts("listening on 9090 -- waiting for one connection")
{:ok, client} = :gen_tcp.accept(listen_socket)
{:ok, data} = :gen_tcp.recv(client, 0)
IO.puts("\n--- raw bytes from the client ---")
IO.puts(data)
IO.puts("--- end (#{byte_size(data)} bytes) ---")
:gen_tcp.close(client)

Just like a traditional web server, we can now receive HTTP requests, which is the first step toward our load balancer.

#Respond to those requests

In the first step, I was just printing the raw bytes and doing practically nothing with the requests. Let's change that.

{:ok, listen_socket} =
  :gen_tcp.listen(9090, [:binary, packet: :raw, active: false, reuseaddr: true])
IO.puts("listening on 9090")
{:ok, client} = :gen_tcp.accept(listen_socket)
{:ok, request} = :gen_tcp.recv(client, 0)
[request_line | _rest] = String.split(request, "\r\n")
IO.puts("request line: #{request_line}")
body = "hello from a hand-written server\n"
response =
  "HTTP/1.1 200 OK\r\n" <>
  "Content-Type: text/plain\r\n" <>
  "Content-Length: #{byte_size(body)}\r\n" <>
  "\r\n" <>
  body
:gen_tcp.send(client, response)
:gen_tcp.close(client)
IO.puts("responded and closed")

With a few more lines of code, I can now build a response by hand and send it back to the client. Nothing special, just the foundation of our LB.

#A minimal load balancer acting as a reverse proxy

Now let's get to the real deal and build the first version of it.

In the end, a reverse proxy is just an application that "sits" in front of our servers, so clients talk to it instead of talking directly to our servers.

To do that, we're going to use something like this:

defp accept_loop(listen) do
  {:ok, client} = :gen_tcp.accept(listen)
  handle(client)
  accept_loop(listen)
end

The important part of this snippet is the handle function, which is responsible for forwarding our request to a specific port:

  defp handle(client) do
    {:ok, request} = :gen_tcp.recv(client, 0)
    {:ok, backend} =
      :gen_tcp.connect(@backend_host, @backend_port,
        [:binary, packet: :raw, active: false])

    :gen_tcp.send(backend, force_close(request))
    response = read_until_closed(backend)
    :gen_tcp.close(backend)

    IO.puts("#{first_line(request)}  ->  #{served_by(response)}")

    :gen_tcp.send(client, response)
    :gen_tcp.close(client)
  end

The problem is: what if a request takes a long time to respond? Say two requests reach our LB 10ms apart, and the first one takes 10s to answer. That's an obvious problem: the second request has to wait for the first one to finish before accept_loop is called again. The whole thing is synchronous.

#Why Elixir turned out to be a surprisingly good choice

That's when Elixir comes in handy. Take a look at this new version of accept_loop:

  defp accept_loop(listen) do
    {:ok, client} = :gen_tcp.accept(listen)
    pid = spawn(fn ->
      receive do
        :socket_is_yours -> handle(client)
      end
    end)
    :ok = :gen_tcp.controlling_process(client, pid)
    send(pid, :socket_is_yours)

    accept_loop(listen)
  end

In Elixir, I can spawn a new process, hand the connection over to it, and forget about it. Nobody is waiting anymore, so I can handle a bunch of requests at the same time. And if that process crashes, it dies alone.

One interesting detail is the receive block. In Erlang/Elixir, every socket has an owner, called its controlling process, and only the owner can read from it. Right after accept, the owner is accept_loop, not the new process. To transfer ownership, I call :gen_tcp.controlling_process(client, pid), but I can only do that once spawn has returned the pid, and by then the new process may already be running. If it called handle immediately, it could try to read the socket before the transfer happened and crash with {:error, :not_owner}. That's a race condition: it would work most of the time and fail randomly. So the new process waits until accept_loop sends :socket_is_yours, which happens only after the transfer.

That's how my load balancer handles concurrency: one process per connection. This only works because these are BEAM processes, not OS processes. Each one costs a few KB of memory, so spawning thousands of them is cheap.

NGINX gets to the same result a different way. Instead of one process per connection, it runs a few worker processes (usually one per CPU core), and each worker uses an event loop: it asks the kernel which of its thousands of open sockets are ready, and serves those. Different mechanism, same goal: never let one slow request block everyone else.

So far so good: my proxy can handle many requests at once. But notice that it forwards every request to the same backend, port 4001. The other two servers sit idle. That means it isn't balancing any load yet; it's just a reverse proxy. Let's fix that.

defmodule Proxy do
  @backend_host ~c"127.0.0.1"
  @backend_port 4001
  def start(port) do
    {:ok, listen} =
      :gen_tcp.listen(port, [
        :binary,
        packet: :raw,
        active: false,
        reuseaddr: true,
        backlog: 1024
      ])

    IO.puts("proxy on #{port} -> #{@backend_host}:#{@backend_port}")
    accept_loop(listen)
  end

#The balancing part

One beautiful thing about a load balancer is that it can spread a very high workload across multiple servers, so no single one of them has to carry it all.

But to do that, the load balancer needs a rule for choosing which server gets the next request. That rule is called a balancing policy, and the simplest one is round-robin: the servers take turns, in a fixed order, and when you reach the end of the list you start over. With my three backends, that's 4001, 4002, 4003, 4001, 4002, 4003, and so on, like dealing cards around a table.

The whole algorithm fits in two lines:

defp pick_backend(counter) do
  n = :atomics.add_get(counter, 1, 1)
  Enum.at(@backends, rem(n, length(@backends)))
end

Every request bumps a counter by one, and rem (the remainder of a division) turns that ever-growing number into a position in the list. With three backends, rem(n, 3) can only ever be 0, 1 or 2, so as n goes 1, 2, 3, 4, 5, 6..., the position goes 1, 2, 0, 1, 2, 0... That's the "start over" part. When I sent 3000 requests through it, each backend got exactly 1000.

Round-robin has a blind spot, though: it counts requests, not work. If one request takes 10 seconds and the rest take 1ms, round-robin will still send the next request to the busy server when its turn comes, because it has no idea that server is busy. Smarter policies, like least connections, fix this, but they need to keep track of how busy each server is.

Anyway, comparing those policies and their tradeoffs is something I might dig into another time.

One thing that really caught my attention in this step was the use of Erlang's atomics for the counter. Why not just use a normal variable?

Remember that every client connection is handled by its own process, and each of those processes calls pick_backend. So many processes need to read and bump the same counter, often at the same moment. That creates two problems.

First, the counter has to be shared. BEAM processes don't share memory: when a process is spawned, it gets its own copy of every variable it uses. If the counter were a plain variable, every process would start from the same number, bump its own copy, and pick the same backend. Every request would go to the same server, which is exactly what I was trying to fix. :atomics solves this by giving me an integer that lives in memory every process can reach:

counter = :atomics.new(1, signed: false)

Second, bumping it has to be indivisible. "Add 1" looks like one step, but it's really three: read the value, add 1, write it back. If two connections arrive at the same instant, this can happen:

process A reads 5
process B reads 5         <- before A has written
process A writes 6 -> picks rem(6, 3)
process B writes 6 -> picks rem(6, 3)   <- same backend!

Both requests go to the same server, and one increment is simply lost. This is called a lost update, and like the socket race earlier, it works almost all the time and only breaks under load. :atomics.add_get avoids it because the CPU does the read, the add and the write as one indivisible operation. That's what "atomic" means: it can't be split. Two processes calling it at the same instant are guaranteed to get different numbers.

Those two properties together are what make the distribution come out even: 1000/1000/1000 ( in a case of 3000 requests ) only happens if no two connections ever get the same n.

you can check the full code here if you want

This was one hour of an "deep dive" in something that i use daily had some knowledge gap

See ya!