Load Balancing in System Design (Interview-Ready Guide)
Explain what a load balancer does, pick an algorithm (Round Robin, Least Connections, etc.), and defend the tradeoffs under realistic traffic and failure modes.
The mental model: a load balancer is a traffic decision point with health, policy, and observability
In system design, a load balancer (LB) is a component that receives requests and routes them to one of many backend instances to improve scalability, availability, and performance. In interviews, the key is to describe (1) where the LB sits (client -> LB -> service instances), (2) what it optimizes (distribution, latency, utilization, resilience), and (3) what signals it uses (instance health, connection counts, latency, weights, locality, and session affinity). Two major types show up in designs: - Layer 4 (L4) load balancing: routes based on transport-level info (IP/port, TCP/UDP). It’s fast and simpler; good for generic TCP services, high throughput, and when you don’t need to inspect HTTP. - Layer 7 (L7) load balancing: understands application-level protocols (HTTP/gRPC). It enables smarter routing (path/header-based routing, canary releases, rate limiting, auth offload), but adds overhead and complexity. Algorithms are policies for “which instance gets the next request.” The right choice depends on whether your requests are uniform, whether connections are long-lived, whether backends have heterogeneous capacity, and what failure/overload looks like. Interviewers look for you to state assumptions (traffic shape, request cost variance, connection behavior) and then pick a simple policy that fits, with health checks and fallback behavior.
A 6-step interview framework to design and justify load balancing
1) Place the load balancer and define the tier
Say where you need balancing: edge (client -> API), service-to-service (internal), or database/read replicas. Name the type: L4 for raw TCP/UDP throughput; L7 for HTTP/gRPC features like routing by path, A/B testing, or auth/rate limits.
2) State the routing goal and what “fair” means
Define what you’re optimizing: throughput, tail latency (p95/p99), instance utilization, availability during failures, or cost. Example: “We want stable p99 latency under bursty traffic, not just even request counts.”
3) Choose an algorithm and connect it to request/connection behavior
Use simple matching logic: - Round Robin: best when instances are identical and requests are roughly uniform. - Least Connections: best when connections are long-lived or request durations vary; spreads load by active connection count. - Weighted Round Robin / Weighted Least Connections: best when instances have different capacity (mixed instance types, gradual scale-outs). - IP Hash / Consistent Hashing: best when you need session affinity or cache locality (but be explicit about why and the tradeoffs). Mention that “Least connections” can still be wrong if one request is CPU-heavy but short-lived; then consider latency-aware or queue-length signals if available.
4) Add health checks, failover, and overload controls
Specify active health checks (periodic probes) and passive checks (e.g., error rate spikes). Define thresholds and removal from rotation. Include retries with bounded attempts, circuit breakers, and timeouts to avoid amplifying outages. If relevant, add rate limiting or load shedding at L7.
5) Address state: sticky sessions vs stateless services
Prefer stateless backends with shared session storage (Redis) or tokens (JWT) so any instance can serve any request. Use sticky sessions only when necessary (legacy state in memory, websocket affinity) and call out its downsides (uneven load, harder failover, cache invalidation patterns).
6) Validate with a quick scenario and call out observability
Walk through a realistic spike/failure: “One AZ goes down; LB shifts traffic; health checks remove instances; autoscaling adds capacity.” Mention key metrics: backend latency, error rate, active connections, request rate, LB retries, and per-target saturation. This shows you can operate the system, not just draw boxes.
Common load balancing mistakes interviewers notice
- •Defaulting to Round Robin without stating assumptions about request uniformity or connection duration
- •Ignoring health checks and failover behavior (what happens when an instance is slow, not just down)
- •Overusing sticky sessions instead of making services stateless (or not acknowledging the downside of stickiness)
- •Not distinguishing L4 vs L7 and choosing L7 “because it’s smarter” without needing its features
- •Forgetting retry/timeout strategy and accidentally designing a retry storm during partial outages
- •Assuming autoscaling fixes poor balancing (it helps capacity, not distribution correctness)
- •Not addressing heterogeneous backends (mixed instance types) where weighted algorithms are required
- •Skipping multi-AZ/region implications: locality routing, cross-zone costs, and how traffic shifts during AZ failure
Interview example: choosing Round Robin vs Least Connections for an API + WebSockets system
Weak Answer
“I’ll put a load balancer in front and use Round Robin. It spreads traffic evenly so it should be fine. We can add more servers if needed.”
Strong Answer
“We’ll use an L7 load balancer for the HTTP API so we can do path-based routing (/api vs /static), rate limiting, and better observability. For the HTTP endpoints with short, fairly uniform requests, Round Robin or Weighted Round Robin is fine. But we also support WebSockets for real-time updates. Those connections are long-lived and can vary a lot per user. For the WebSocket targets, Least Connections (or Weighted Least Connections) is a better default so one instance doesn’t get stuck with too many active sockets. We’ll run active health checks and remove unhealthy targets quickly. To avoid sticky sessions for HTTP, the API will be stateless and store session/user state in Redis. For WebSockets we may need connection affinity; if so we’ll use consistent hashing on a user identifier with a plan for rebalancing when instances scale (acknowledging some reconnect churn). We’ll monitor p95/p99 latency, connection counts per target, and error rates to tune thresholds.”
Why it works
The good answer ties the algorithm to concrete traffic behavior (short HTTP calls vs long-lived WebSockets), distinguishes L7 features from L4 simplicity, and explicitly addresses health checks and state. It also communicates tradeoffs (affinity causing uneven load, reconnect churn on scale events) and shows operational awareness with metrics—exactly what interviewers use to judge senior system design reasoning.
Turn this into an interview skill: practice making the call and defending it
Knowing definitions isn’t enough in a system design interview—you’re evaluated on how quickly you pick a reasonable load balancing approach, articulate assumptions, and handle failure and state. On CrackTheOffer, you can practice prompts that force the key decisions (L4 vs L7, Round Robin vs Least Connections, stickiness, health checks, retries) and get repetition on explaining tradeoffs clearly under time pressure.