Load Balancing in System Design: Algorithms, Health Checks, and Trade-offs

Why load balancing exists

A single server can only do so much. Give it enough traffic and it runs out of CPU, memory, or connections — and when it dies, your whole product goes with it. The fix is to run many identical servers and put something in front of them that spreads requests across the pool. That something is a load balancer.

A good load balancer buys you two things at once:

• Scalability — add more servers to handle more traffic, without clients knowing or caring how many there are.
• Availability — if one server fails, the balancer stops sending traffic to it and the rest of the pool absorbs the load.

Layer 4 vs Layer 7

Load balancers operate at one of two layers of the network stack, and the difference matters in interviews.

• Layer 4 (transport) balances on IP address and TCP/UDP port. It doesn't look inside the request, so it's fast and cheap — but it can't make decisions based on the content of a request.
• Layer 7 (application) understands HTTP. It can route on path, headers, cookies, or hostname — e.g. send /api/* to one pool and /images/* to another. More flexible, slightly more overhead.

Rule of thumb: reach for L7 when you need content-based routing, TLS termination, or per-route pools; L4 when you just need to spread raw connections as fast as possible.

The routing algorithms

Once traffic arrives, the balancer has to pick a server. The common strategies:

1. Round robin — hand each new request to the next server in order. Simple and fair when servers are equal and requests are cheap.
2. Weighted round robin — give beefier servers a larger share. Useful for a mixed fleet (e.g. old and new hardware).
3. Least connections — send the request to whichever server currently has the fewest open connections. Better when requests vary a lot in duration.
4. Least response time — factor in both active connections and how quickly each server has been responding.
5. IP hash — hash the client's IP to always route them to the same server. A cheap way to get session stickiness (more on that below).

There's no universally "best" one — the right choice depends on whether your requests are uniform or spiky, and whether your servers are identical.

Health checks: the part people forget

A load balancer is only as good as its knowledge of which servers are alive. Health checks are periodic probes — a simple GET /healthz, a TCP connect, or a deeper check that verifies the server can reach its database. When a server fails its check, the balancer pulls it out of rotation; when it recovers, it goes back in.

Two things worth mentioning in an interview:

• Shallow vs deep checks. A shallow check (is the port open?) is cheap but can keep sending traffic to a server whose database connection is dead. A deep check catches that but costs more.
• Draining. When you remove a server for a deploy, you want to stop sending it new connections while letting existing ones finish — "connection draining" — so you don't drop in-flight requests.

Sticky sessions (and why to avoid needing them)

Some apps store session state in server memory, so a user must keep hitting the same server. Load balancers support this with sticky sessions (via a cookie or IP hash). It works, but it undermines the whole point of load balancing: if one server holds a user's state and it dies, that user is logged out, and traffic can no longer be spread freely.

The better pattern is to make your servers stateless — push session state into a shared store like Redis — so any server can handle any request. Then stickiness becomes an optimization, not a requirement.

What to say in the interview

When a system design prompt reaches real scale, adding a load balancer is almost always part of the answer. Score points by going past "add a load balancer":

• Name the algorithm and justify it ("least connections, because our requests have very different durations").
• Mention health checks and draining so failover and deploys are graceful.
• Call out the single point of failure — the balancer itself — and resolve it with a redundant pair (active-passive or active-active) behind a floating IP or DNS.
• Prefer stateless servers so you're not forced into sticky sessions.

Get those four points in and you've shown you understand not just what a load balancer is, but how it behaves when things go wrong — which is what the interview is really testing.