Lesson 10 of 14 · 10 min · HTTP, TLS and traffic management

Load balancing: algorithms, health checks and layers

Covers: nginx 1.24 on Ubuntu 24.04 with two Python backends. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • What a load balancer does and why you need one
  • The main algorithms and when to choose each
  • Health checks, failover and connection draining
  • Layer 4 versus layer 7 load balancing and sticky sessions
  • A tested nginx configuration, including a surprise about multiple workers

In simple words

One server can only handle so many visitors. A load balancer is like the person at a bank who sends each customer to a free counter. It spreads requests over several servers and stops sending people to a counter that has closed. This lesson covers how it chooses and how it knows a server is down.

Words to know

  • Load balancer: a device or service that shares incoming requests among several servers.
  • Health check: a small test request sent regularly to see whether a server is still working.
  • Round robin: the simplest way to share: each server in turn.
  • Sticky session: always sending the same user to the same server.

Why load balance

One server has a limit and is a single point of failure. A load balancer (LB) spreads requests across several identical servers (the pool, target group or upstream) so that:

  • capacity grows by adding servers (horizontal scaling),
  • availability improves: if one server fails, the others carry on,
  • deployments can be rolling: take servers out of the pool one at a time.

For this to work the servers should be stateless (any server can handle any request), keeping sessions and uploads in shared stores such as Redis, a database or object storage.

Algorithms

Algorithm How it chooses Good for Watch out
Round robin each server in turn equal servers, similar requests ignores how busy each server is
Weighted round robin like round robin but servers with a higher weight get proportionally more servers of different sizes, or canary (weight 1 vs 20) weights must be maintained
Least connections the server with the fewest active connections requests of very different length (uploads, long queries) needs the LB to track connections
Least response time fastest and least busy latency-sensitive services commercial / cloud features
IP hash / hash of a key the same client (or key) always goes to the same server simple stickiness, cache locality uneven if few clients; clients behind one NAT look like one
Consistent hashing hash on a key, but adding or removing servers moves few keys caches, sharded backends more complex
Random (power of two choices) pick two at random, use the less busy large pools, Envoy and others

Tested in nginx

Two backends answering backend-1 and backend-2:

upstream app {
    zone app 64k;
    server 127.0.0.1:9001 weight=2;
    server 127.0.0.1:9002;
}
server {
    listen 8088;
    location / { proxy_pass http://app; }
}

Nine requests with weight 2:1 gave 6 backend-1 and 3 backend-2; with equal weights, six requests gave 3 and 3. Switching the line to ip_hash; sent all six requests from the same client to the same backend; least_conn; with idle servers alternated between them.

The surprise: without the zone app 64k; line, our first test sent all requests to backend-1, even with equal weights. nginx runs several worker processes (12 on our test machine), and without a shared-memory zone each worker keeps its own round-robin counters. At low traffic one worker handled all connections, so the split looked wrong. Adding a zone makes the load-balancing state shared. Two lessons: test your load balancer's behaviour rather than assuming it, and know that "per-process state" can distort results at low traffic.

Health checks and failover

A balancer must stop sending traffic to a dead server.

  • Passive checks: the LB notices failures in real traffic. In nginx, max_fails=3 fail_timeout=30s marks a server unavailable for 30 seconds after 3 failures; requests then go to the others. We stopped one backend while sending requests: every request still returned 200, because nginx retried the next server.
  • Active checks: the LB probes a URL (such as /healthz) every few seconds and removes servers that fail, adding them back when they pass. Cloud load balancers and HAProxy do this natively; open-source nginx has only passive checks (active checks are in NGINX Plus).

A good health endpoint is cheap, checks what the instance really needs (can it reach its database?) and does not itself cause an outage when a shared dependency is down, since then every instance would be marked unhealthy at once. Separate liveness ("is the process alive?") from readiness ("can it serve traffic now?"), a distinction Kubernetes uses.

If all backends fail, the LB returns 502/503. nginx backup servers can serve a maintenance page then:

server 127.0.0.1:9999 backup;

Connection draining (deregistration delay) lets in-flight requests finish when a server leaves the pool; essential for rolling deployments.

Layer 4 versus layer 7

  • A layer 4 (transport) load balancer forwards TCP/UDP connections by IP and port without reading the content. Very fast and protocol-agnostic (databases, MQTT, TLS pass-through). Examples: AWS Network Load Balancer, HAProxy in TCP mode, nginx stream.
  • A layer 7 (application) load balancer understands HTTP: it can route by host and path (/api to one pool, /static to another), terminate TLS, rewrite headers, redirect, apply WAF rules and do cookie-based stickiness. Examples: AWS Application Load Balancer, nginx http, Envoy, Traefik.

Choose layer 7 for web traffic; choose layer 4 for non-HTTP protocols, extreme throughput, or end-to-end TLS you must not touch.

Session persistence (sticky sessions)

If the app keeps user state in a server's memory, the same user must reach the same server. Options: ip_hash, or a cookie set by the LB (layer 7). It works but has costs: uneven load, and users lose their session when that server dies. The better design is stateless servers with a shared session store; use stickiness as a stopgap.

Load balancers in the cloud and Kubernetes

  • AWS: Application Load Balancer (L7), Network Load Balancer (L4), Gateway Load Balancer (for appliances). Targets are instances, IPs or containers, with health checks, and listeners with rules.
  • Global load balancing: DNS-based (Route 53 weighted/latency/failover records) or anycast CDNs steer users to the nearest healthy region.
  • Kubernetes: a Service balances across pods; Ingress controllers (nginx, Traefik) do layer 7 routing; Service type: LoadBalancer asks the cloud for an external LB. The Kubernetes course covers these.

Also mind that the LB itself must not be a single point of failure: cloud LBs are redundant by design; self-run ones use pairs with a floating IP (keepalived) or are fronted by DNS.

Common mistakes

  • Stateful servers behind a plain round robin (logins that "randomly" fail).
  • Health checks that are too shallow (always 200) or too deep (one failing dependency drains the whole pool).
  • No draining, so deploys cut off active requests.
  • No capacity headroom: if one of three servers fails, can two carry the load?
  • Trusting algorithm names without testing (the per-worker counters above).
  • Forgetting the LB's own timeouts (idle timeout shorter than the app's keep-alive causes sporadic 502s).

Try it yourself

  1. Reproduce the weighted test with two Python backends and nginx. Count responses with curl in a loop and sort | uniq -c.
  2. Remove the zone line and see whether your split changes at low traffic.
  3. Stop one backend during a loop of requests and observe failover; then stop both and read the 502 in the error log.
  4. Draw a design for a web app with a load balancer, three app servers, shared sessions in Redis and a database, and mark the health checks.

Check yourself

When would you choose least connections over round robin?

When requests take very different amounts of time (uploads, reports, long-lived connections), so some servers would otherwise accumulate many slow requests while others sit idle.

What is the difference between a layer 4 and a layer 7 load balancer?

Layer 4 forwards connections by IP and port without understanding the protocol. Layer 7 reads HTTP and can route by host, path or headers, terminate TLS and modify requests.

What are sticky sessions, and what is the better alternative?

Sending a user always to the same server (by IP hash or cookie) because it holds their session. The better design is stateless servers with sessions in a shared store such as Redis.

What does a health check do, and what makes a good one?

It lets the balancer remove failed servers from rotation. A good one is cheap and tests what the instance needs to serve traffic, but is designed so that one shared failing dependency does not mark every instance unhealthy at once.

Interview questions

How would you deploy a new version with no downtime behind a load balancer?

Rolling update: remove one server (with connection draining), update and health check it, add it back, and repeat; or blue-green / canary by shifting weights between two pools.

Why might round robin give an uneven load?

Requests differ in cost, servers differ in capacity, long-lived connections accumulate on some servers, and per-process counters (as with multiple nginx workers without a shared zone) can skew small samples.

A load balancer returns 503 for everything. What do you check?

Whether any targets are healthy: health check path and port, security group allowing the balancer to reach the targets, the app actually listening, and recent deployments.

Summary

  • A load balancer spreads traffic over a pool for capacity and availability; servers should be stateless.
  • Algorithms: round robin, weighted, least connections, hash-based; pick by request shape and need for stickiness.
  • Health checks, failover and connection draining keep users unaffected by failures and deploys.
  • Layer 7 for HTTP routing and TLS termination, layer 4 for raw speed or non-HTTP protocols; test the actual behaviour.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Networking for DevOps for DevOps Engineer and scores every answer, with what to fix.