Distributed Consensus and Saga Patterns
Network Failure Reality
The Unreliable Network
When you build an application on a single computer, things are relatively simple. The parts of your program can talk to each other almost instantly and reliably. But modern systems, like a large-scale ERP, rarely live on just one machine. Instead, they are —a group of separate computers that need to work together over a network.
Imagine you have an application server that needs to tell your PostgreSQL database to update inventory. It sends a command across the network. This is where the problems begin. A network is just a series of physical wires or wireless signals connecting computers. And just like any physical thing, it can fail.
Think of it like the game of 'Broken Telephone.' A message starts clear at one end, but by the time it gets to the other, it might be garbled, delayed, or never arrive at all. In computer networks, the same thing happens. The connection can drop, a router can get overloaded, or a cable can be accidentally unplugged.
What Goes Wrong?
When your application sends a command like, "Subtract 5 from the inventory of 'Product A'," that message doesn't travel as a single unit. It's broken down into tiny pieces of data called and sent across the network. Each packet is like a single word in a very long sentence. For the message to be understood, all the packets have to arrive in the right order.
This process can fail in several ways:
- Packet Loss: A packet might simply get lost along the way. A network device could drop it due to congestion, and it never reaches the database.
- Latency: The packets might take a very long time to arrive. This delay, known as , can happen if the network is busy or the physical distance is great. A command that's too slow can be as bad as one that never arrived.
- Duplication: Sometimes, the network might accidentally deliver the same packet more than once. If your command was "Subtract 5," receiving it twice would be a big problem.
| Failure Mode | Analogy | ERP Example |
|---|---|---|
| Packet Loss | A page of your order form gets lost in the mail. | The database never receives the command to update inventory. |
| Latency | Your order form takes weeks to arrive by boat. | The inventory update happens so late that you've already sold the item to someone else. |
| Duplication | A photocopier error sends two copies of your order form. | The database subtracts 5 from inventory, and then subtracts 5 again. |
Did You Get My Message?
To deal with lost messages, systems use a simple trick: acknowledgments. When the database server receives a command, it sends a tiny message back saying, "I got it!" This is often called an ACK.
If the application server doesn't get an ACK within a certain amount of time, it assumes the original message was lost and sends it again. This sounds like a good solution, but it creates a new problem. What if the original command did arrive and get executed, but the ACK was the message that got lost on the way back?
From the application's perspective, the situation is ambiguous. It doesn't know the difference between these two scenarios:
- Its command was lost, and nothing happened.
- Its command worked, but the confirmation was lost.
Sending the command again is risky. In the first case, it's the right thing to do. In the second case, it could cause the inventory to be updated twice.
This uncertainty is the single biggest challenge in building reliable distributed systems. You can never be 100% sure about the state of another computer.
This is why you can't always rely on a single, simple database transaction to keep data correct when multiple systems are involved. The network's unreliability forces us to design our systems with failure in mind from the very beginning. We have to assume messages will be lost, delayed, or duplicated, and build logic to handle the fallout.