When an application starts, it often lives on a single server. This server handles everything: web traffic, application logic, and the database. For a while, this works. But as traffic grows, the server starts to struggle. The obvious solution seems to be to make the server more powerful. This is called vertical scaling—adding more CPU, RAM, or faster storage to the existing machine. Think of it as upgrading a delivery van to a massive lorry. It can carry more, but it's still just one vehicle, and there's a limit to how big it can get.

The more robust alternative is horizontal scaling, or scaling out. Instead of making one server bigger, you add more servers to share the workload. Instead of one giant lorry, you build a fleet of delivery vans. This approach is more flexible, resilient, and often more cost-effective. Adding more commodity servers is usually cheaper than buying one top-of-the-line machine. But now you have a new problem: with a whole fleet of servers, how do you decide which one handles the next incoming request?

The answer is a load balancer. This is a dedicated server or service that sits in front of your application servers and acts as a traffic controller. Its primary job is to distribute incoming requests across the available servers, ensuring no single server gets overwhelmed. This distribution improves performance, reliability, and availability.

Load balancers operate at different layers of the network stack, primarily Layer 4 (Transport) and Layer 7 (Application). A Layer 4 load balancer makes routing decisions based on network information from the TCP/IP packets, like the destination IP address and port. It's fast and simple because it doesn't need to inspect the content of the traffic. It just forwards packets.

A Layer 7 load balancer is much smarter. It can inspect the content of the request itself, such as HTTP headers, URLs, or cookies. This allows for more intelligent routing. For instance, it could route requests for /api to a set of powerful application servers and requests for /images to servers optimised for delivering static content.

To decide which server gets the next request, load balancers use different algorithms. The simplest is Round Robin, which just cycles through the list of servers. Least Connections is a bit smarter, sending traffic to the server that currently has the fewest active connections. Another common method is IP Hash, which uses the client's IP address to determine which server receives the request. This ensures a particular user is always sent to the same server.

Directing a user to the same server every time, as IP Hash does, can be useful. But it points to a major challenge in distributed systems: managing state. State is any data that the application remembers from one request to the next, like what's in a user's shopping cart or if they're logged in.

If that information is stored on a single server, you have a problem. What happens if the load balancer sends the user's next request to a different server? That new server has no idea who the user is or what was in their cart. This is why a stateless application design is the "best friend" of horizontal scaling.

In a stateless architecture, each request from a client contains all the information needed to be processed. The server doesn't store any session information between requests. Any server in the fleet can handle any request because the necessary state is either sent with the request or stored in a shared, external location. This shared location could be a distributed cache like Redis or a database that all servers can access.

An older solution to the state problem is using sticky sessions with the load balancer. This configures the load balancer to always send a user's requests to the same server where their session was first created. This works, but it complicates things. It makes the system less resilient; if that specific server fails, the user's session is lost. It also makes scaling harder, as traffic isn't distributed as evenly. A stateless design with a distributed state store is the modern, more robust approach.

This leads to a crucial principle: mitigating any single point of failure (SPOF). In our original single-server setup, the server itself was a SPOF. If it went down, the whole application was offline. Horizontal scaling helps eliminate the application server as a SPOF, but now what about the load balancer? Or the database? A truly resilient architecture requires redundancy at every layer. This often means having a pair of load balancers in an active-passive setup and using database replication strategies. Every component that could bring down the entire system if it fails needs a backup plan.

By moving from a single server to a distributed system with load balancing and a stateless design, applications can handle massive scale while remaining resilient to failure.