When you use a major online service, you're not interacting with a single supercomputer. You're talking to a distributed system, a network of many computers working together to act as one. This approach allows services to handle millions of users and store enormous amounts of data. Spreading the work out makes a system scalable.

But it also introduces a major challenge: what happens when parts of the system fail? A single computer might crash, or a network connection between data centers could break. Distributed systems must be designed to handle these failures gracefully without the user ever noticing. This resilience is called fault tolerance.

When designing a distributed system, engineers face a fundamental dilemma known as the CAP theorem. It states that a distributed data store can only provide two of three guarantees: Consistency, Availability, and Partition Tolerance.

Since network failures (partitions) are a fact of life in distributed systems, partition tolerance is something you must have. This means designers are forced to choose between consistency and availability when a network partition occurs.

• CP (Consistent and Partition-Tolerant): If a network partition happens, the system will shut down the non-consistent part to avoid returning incorrect data. It chooses correctness over being available. Many banking and financial systems favor this model.

• AP (Available and Partition-Tolerant): During a partition, the system remains available but might return stale data from some nodes. It chooses availability over perfect consistency. Social media feeds and content delivery networks often use this model.

How do nodes in a distributed system agree on the state of things, like who holds a lock on a resource or what the value of a piece of data should be? This problem is known as consensus. Achieving consensus is difficult in a world where nodes can crash and messages can be delayed or lost.

Consensus algorithms are protocols that allow a group of computers to agree on a single value. Two of the most well-known algorithms are Paxos and Raft.

Paxos is one of the first and most influential consensus algorithms. It's known for being provably correct but is also notoriously difficult to understand and implement correctly. It uses a multi-phase protocol involving roles like "proposers," "acceptors," and "learners" to propose and agree upon values.

Raft was designed as an easier-to-understand alternative to Paxos. It achieves the same level of fault tolerance but with a simpler logic. Raft works by electing a "leader" from the group of nodes. This leader is then responsible for managing the replicated log and coordinating writes. If the leader fails, the other nodes elect a new one. This leader-based approach makes the system's behavior easier to follow and debug.

To achieve fault tolerance and high availability, data is often duplicated across multiple machines. This is called data replication. If one machine fails, the data is still safe on others. But this introduces the challenge of keeping all the copies, or replicas, consistent.

There are different models for how this consistency is managed:

Many systems use replication strategies to balance performance and consistency. In a leader-follower setup, all writes go to a leader node, which then pushes the update to follower nodes. This simplifies consistency, as there's a single source of truth for all changes. If the leader fails, a follower is promoted to become the new leader.

Let's check your understanding of these core principles.

Understanding these concepts is key to building systems that are both resilient and scalable, forming the backbone of modern internet services.