Wait, What?
A load balancer is not trying to make every server receive the same number of requests. It is trying to make the system behave well while requests, servers and costs keep changing.
Equal request counts can still produce terrible balance if one request lasts 5 milliseconds and another lasts 5 minutes. Hash-based stickiness can improve cache locality while worsening load skew. Health checks can remove a failing server but suddenly remap traffic. The algorithm therefore has to decide what “balanced” means for the workload.
This makes load balancing an excellent bridge from beginner algorithms to professional distributed systems: counters, randomness, hashing, feedback, partial information and failure all appear in one visible problem.
Quick Answer
Learn load-balancing algorithms through the route server pool → round robin → weighted round robin → random choice → least connections → least request/latency signals → sticky routing → hashing → consistent hashing → virtual nodes → rendezvous and jump hashing → power of two choices → health and capacity changes → locality versus fairness → feedback delay → tail latency → professional production selection. A beginner should be able to simulate request assignment across three servers. A professional should be able to explain the metric being balanced, the information the algorithm needs, how membership changes affect routing and which failure modes appear under skew or delayed feedback.
1. Define the Unit of Work Before Choosing the Algorithm
A request, connection, byte, task, shard and CPU-second are not interchangeable units. If requests differ greatly in cost, balancing request counts can still produce highly unequal load.
2. Round Robin Is the Best First Algorithm
Round robin cycles through available servers in order: A, B, C, A, B, C. It requires little state and produces predictable distribution when servers and requests are similar.
NGINX documents round robin as its default HTTP load-balancing method: NGINX HTTP Load Balancing.
3. Round Robin Balances Assignments, Not Work
If server A receives a 20-second stream while B receives a 20-millisecond request, both received one assignment but not equal work. This simple counterexample teaches why an algorithm’s observable metric matters.
4. Weighted Round Robin Models Unequal Capacity
If one server is twice as capable as another, weights can send it proportionally more traffic. The learner should distinguish static capacity weights from live load measurements: a weight says what a server is expected to handle, not what it is handling now.
5. Least Connections Uses Feedback
Least-connections routing chooses the server with the fewest active connections. It can better handle long-lived connections than simple round robin because it observes current occupancy.
NGINX and HAProxy both document least-connections variants for workloads where connection duration matters. See HAProxy Backends.
6. Feedback Can Be Stale
In a distributed balancer, the view of active connections, queue depth or latency may already be outdated by the time a decision is made. An algorithm that looks more informed can perform worse if its information arrives slowly or noisily.
7. Least Connections Is Not Least Work
Ten idle WebSocket connections may cost less than one CPU-heavy report. Connection count is a proxy. Professionals therefore ask whether the proxy correlates with the actual resource bottleneck.
8. Random Choice Is Better Than It First Appears
Choosing a server uniformly at random needs almost no coordination and avoids deterministic patterns. Over enough requests it can distribute load reasonably when server capacities and request costs are similar.
The existing How to Learn Randomized Algorithms article owns general probabilistic reasoning. Load balancing makes random choice a concrete systems policy.
9. Power of Two Choices Uses Tiny Extra Information for Large Benefit
Instead of selecting one random server, sample two and send the work to the less-loaded one. This small amount of comparison can dramatically reduce the maximum-load gap compared with one random choice.
The “power of two choices” family has a deep theoretical literature, including weighted variants studied by Microsoft Research: Balanced Allocations: The Weighted Case.
10. Two Choices Trade Coordination for Better Tail Balance
Sampling two servers requires one extra observation but can avoid sending work to an already overloaded destination. This is a useful design pattern: a very small information budget can outperform both fully blind routing and expensive global optimisation.
11. Sticky Routing Solves a Different Problem
Sometimes the goal is not simply to spread work. Repeated requests from the same user or key may benefit from reaching the same server because of session state, cache locality or data ownership.
12. Ordinary Hashing Creates a Remapping Problem
A simple mapping such as hash(key) mod N works while the number of servers N stays fixed. Add or remove one server and the modulus changes, causing many keys to move to different destinations.
The existing How to Learn Hash Tables article owns generic hash functions and collision handling. Distributed routing adds the question of how assignments should change when the destination set changes.
13. Consistent Hashing Minimises Disruption When Membership Changes
Consistent hashing maps keys and servers into a common hash space so adding or removing a server remaps only a limited fraction of keys rather than reshuffling nearly everything.
The original consistent-hashing work by Karger and colleagues formalised smoothness, spread and load for distributed caching: Consistent Hashing and Random Trees.
14. The Hash Ring Is a Teaching Model, Not the Only Implementation
A common explanation places server points around a circle and assigns each key to the next server clockwise. The ring makes minimal remapping visible, but production systems may use other consistent-assignment schemes such as rendezvous hashing, jump consistent hash or Maglev-style lookup tables.
15. Virtual Nodes Improve Distribution
One physical server can occupy multiple positions on the ring. These virtual nodes smooth uneven gaps in the hash space and can represent unequal server capacity by assigning different numbers of positions.
16. Consistent Hashing Optimises Movement, Not Live Load
A perfectly stable key assignment can still overload one server if a few keys become extremely hot. Consistency and balance are related but not identical objectives.
17. Rendezvous Hashing Ranks Destinations Per Key
Highest-random-weight or rendezvous hashing computes a score for each key–server pair and selects the highest. It provides deterministic assignment and graceful membership changes without requiring a ring structure.
18. Jump Consistent Hash Removes the Ring’s Storage Cost Under a Constraint
Jump consistent hash maps a key to sequentially numbered buckets with very small state and minimal movement when the bucket count changes. Its limitation is that the destination set must fit that sequential-bucket model.
See Lamping and Veach’s A Fast, Minimal Memory, Consistent Hash Algorithm.
19. Production Proxies Mix Several Policies
Modern proxies expose round robin, random, least-request and consistent-hash families because workloads differ. Envoy Gateway, for example, supports Round Robin, Random, Least Request and Consistent Hash policies: Envoy Gateway Load Balancing.
20. Health Checks Change the Candidate Set
When a server fails health checks, the routing algorithm must stop sending normal traffic to it. For hash-based schemes, removing one server also causes some keys to move. Recovery creates another membership change.
21. Slow Start Prevents a Recovered Server From Being Flooded
A newly added or recovered server may have cold caches and no warmed connections. Immediately assigning its full expected share can create a latency spike. Practical systems often ramp traffic gradually.
22. Queue Length Can Be More Informative Than Connection Count
If requests wait in a server queue, queue depth may reveal saturation earlier than connection count. But obtaining accurate queue information can require extra telemetry and coordination.
23. Latency-Aware Routing Risks Feedback Loops
If every balancer sends traffic away from a temporarily slow server at once, another server may become overloaded, causing the preferred destination to oscillate. Smoothing, hysteresis and bounded updates can make the control loop more stable.
24. Tail Latency Is Often More Important Than the Average
A routing policy can have excellent average latency while producing a painful 99th percentile if a small subset of servers or keys becomes overloaded. Professional evaluation therefore looks at latency distributions, not one mean value.
25. Locality and Fairness Pull in Different Directions
Sticky routing improves cache reuse but can trap a hot customer on one backend. Frequent rebalancing spreads load but destroys locality and may increase cache misses. The correct policy depends on whether computation, network, cache or state movement is the dominant cost.
26. Distributed Load Balancing Has Partial Information
A central balancer can maintain a richer global view but may become a bottleneck or failure domain. Client-side or decentralised balancing scales control decisions but each participant sees only part of the system.
The existing How to Learn Distributed Algorithms article owns general distributed-state and failure reasoning. Load balancing specialises that reasoning around repeated assignment decisions.
27. Common Learning Failure States
- Assuming equal request counts mean equal load.
- Using least connections when connection count poorly predicts work.
- Assuming fresh telemetry when measurements are delayed.
- Using ordinary modulo hashing while frequently changing server count.
- Thinking consistent hashing guarantees balanced hot-key load.
- Confusing session stickiness with fault tolerance.
- Ignoring cache warm-up after failover or scale-out.
- Optimising average latency while missing tail latency.
- Adding more feedback without checking for oscillation.
- Comparing algorithms without defining the balanced resource.
28. A Beginner-to-Professional Learning Ladder
- Level 1: distribute 12 requests across three servers with round robin.
- Level 2: repeat with unequal request durations and explain the imbalance.
- Level 3: simulate weighted round robin and least connections.
- Level 4: compare one random choice with power of two choices.
- Level 5: map keys with modulo hashing, then change N and count remaps.
- Level 6: build a small consistent-hash ring and add one server.
- Level 7: reason about virtual nodes, hot keys and unequal capacity.
- Level 8: model health changes, slow start and stale feedback.
- Level 9: compare locality, fairness and tail latency for a realistic workload.
- Level 10: diagnose production imbalance using load distribution, queue depth, latency percentiles, cache hit rate and membership events together.
29. Teach With Coloured Tokens Before Configuration Files
Represent servers as cups and requests as tokens with different weights. First run round robin while pretending every token costs the same. Then reveal the weights. Repeat with least-load and two-choice policies. The physical imbalance becomes visible before learners encounter proxy syntax.
This prediction-first movement from visible execution to modification aligns with PRIMM: Using PRIMM to teach programming.
30. Fade Worked Routing Traces Into Independent Policy Choice
Begin with a fully annotated sequence showing server loads after every request. Then hide some states and ask learners to reconstruct them. Finally change one assumption—unequal capacity, hot keys, long connections or stale measurements—and ask whether the original policy still fits.
Faded worked examples with metacognitive scaffolding have shown benefits in novice programming problem solving: Shin et al. (2023).
31. Immediate, Delayed and Transfer Checks
- Immediate: trace round robin, least connections and consistent hashing on a tiny pool.
- Counterexample: create equal request counts but unequal server work.
- Failure: remove one server and predict which assignments move.
- Delayed: reconstruct why two random choices can outperform one.
- Transfer: choose policies for stateless APIs, long-lived streams, caches and shard ownership.
- Professional: explain a tail-latency incident using routing policy, load skew, feedback delay, health events and cache locality together.
32. AI Assistance Boundary
AI can generate request traces, simulate hash rings, suggest counterexamples and explain proxy documentation. The learner should still be able to identify what resource is being balanced, trace the routing rule, predict remapping under membership change and verify production configuration against authoritative documentation.
Professional Direction
Advanced study includes rendezvous hashing, Maglev hashing, bounded-load consistent hashing, weighted random selection, EWMA latency, queue-aware routing, locality-aware routing, client-side balancing, outlier detection, retry budgets, hedged requests, slow start, adaptive concurrency, shard rebalancing, replicated-data placement, hot-key splitting, load shedding and control-theoretic stability. Production proxies such as NGINX, HAProxy and Envoy provide a useful bridge from textbook policy to operational trade-offs.
Algorithm-learning rule: when traffic looks balanced, ask what quantity was balanced, how fresh the measurements were, what happened to sticky assignments when membership changed, whether hot keys were hidden by averages and whether the policy improved the tail rather than merely the mean.
eduKate Sengkang 