Staff Engineer System Design Mastery for FAANG
Distributed Transaction Patterns
Managing Distributed Transactions
When a business process touches multiple microservices, maintaining data consistency becomes a major challenge. A single user action, like placing an order, might require updating the Order service, debiting the Payment service, and reserving inventory in the Warehouse service. If one of these steps fails, how do you prevent the system from entering an inconsistent state, like taking payment for an item that can't be shipped?
This is where distributed transaction patterns come in. They provide strategies for coordinating actions across services to ensure that a business process either completes fully or is cleanly rolled back, preserving data integrity across the entire system. These patterns move beyond the guarantees of a single database's ACID properties, which are insufficient when data is decentralized.
Implement each business transaction that spans multiple services as a saga.
The Saga Pattern
A saga is a sequence of local transactions where each step is handled by a single service. When a service completes its local transaction, it triggers the next step in the sequence. If any step fails, the saga executes compensating transactions to undo the work of the preceding steps. This approach ensures eventual consistency without requiring all services to be locked, making it highly available and scalable.
Compensating Transaction
noun
An operation that reverses the effect of a previous transaction. For example, if a DebitAccount transaction occurred, its compensating transaction would be CreditAccount.
Sagas can be implemented in two main ways: choreography and orchestration.
Choreography: Services communicate by publishing and subscribing to events. There is no central coordinator. Each service knows which event to listen for and which event to publish upon completing its task.
Orchestration: A central coordinator, or orchestrator, tells each service what to do and when. The orchestrator manages the sequence of operations and is responsible for triggering compensating transactions if something goes wrong.
Let's visualize the difference using our e-commerce order example.
Choreography is simple and decentralized. Services are loosely coupled. However, it can become difficult to track the overall state of a business transaction, as the logic is distributed across many services.
Orchestration centralizes the workflow logic, making it easier to monitor and debug. The orchestrator explicitly calls each service. This reduces the complexity for individual services, as they no longer need to know about the overall workflow. The trade-off is that the orchestrator can become a single point of failure and a bottleneck if not designed carefully.
A crucial companion to the Saga pattern, especially in choreography, is the Outbox Pattern. This pattern ensures that a service reliably publishes events after it has successfully updated its own database. It works by having a service write both its state change and the event to be published into the same local database transaction. A separate process then reads from this "outbox" table and publishes the events to a message broker. This guarantees that an event is never published if the corresponding database update fails, preventing inconsistencies.
Stricter Consistency Patterns
While sagas offer great scalability through eventual consistency, some business processes demand stricter, all-or-nothing guarantees, known as atomicity. For these scenarios, other patterns are more suitable, though they come with performance trade-offs.
Two-Phase Commit (2PC): This pattern introduces a central transaction coordinator. In the first phase ('prepare'), the coordinator asks each participating service if it is ready to commit its part of the transaction. If all services respond 'yes', the coordinator initiates the second phase ('commit'), telling all services to permanently save their changes. If any service says 'no' or fails to respond, the coordinator tells everyone to roll back.
2PC provides strong consistency, but it is a blocking protocol. Services must lock resources while waiting for the coordinator's final decision, which can significantly reduce availability and increase latency.
A key weakness of 2PC is that if the coordinator fails after the 'prepare' phase but before the final 'commit' or 'abort' message is sent, all participating services are left in a blocked state, unsure of the outcome. Three-Phase Commit (3PC) attempts to solve this by adding a 'pre-commit' phase, but it is complex and does not fully eliminate all failure scenarios, so it's rarely used in practice.
The Try-Confirm-Cancel (TCC) pattern offers a middle ground. It's similar to a saga but provides more guarantees. Each step in the transaction has three parts:
- Try: A service tentatively reserves resources or performs a provisional operation. This step should be designed to be unlikely to fail.
- Confirm: If all 'Try' operations succeed, a coordinator calls the 'Confirm' operation on each service, which makes the provisional changes permanent.
- Cancel: If any 'Try' operation fails, the coordinator calls 'Cancel' on all services that completed the 'Try' phase. The 'Cancel' operation is a compensating transaction that reverses the 'Try' operation.
TCC requires more complex logic within each service but avoids the long-lived resource locks of 2PC, offering better performance while still managing transactions at a business level.
| Pattern | Consistency Model | Pros | Cons |
|---|---|---|---|
| Saga | Eventual | High availability, scalable, services are loosely coupled. | Hard to debug (esp. choreography), no global atomicity. |
| 2PC/3PC | Strong (Atomic) | Guarantees all-or-nothing completion. | Low availability, high latency, blocking, coordinator is a single point of failure. |
| TCC | Strong (at business level) | Non-blocking, better performance than 2PC. | Requires complex logic (Try, Confirm, Cancel) in each service. |
Choosing the right pattern involves a crucial trade-off. You must balance the need for strong consistency against the demands of availability and scalability. For most high-throughput systems at FAANG-like companies, eventual consistency using the Saga pattern is the standard, with strong consistency patterns reserved for critical operations like financial transfers where atomicity is a strict business requirement.
What is the primary role of a compensating transaction within the Saga pattern?
An e-commerce system uses event-driven communication. The Order service listens for a 'PaymentProcessed' event, and the Shipping service listens for an 'OrderConfirmed' event. This setup, where services react to each other's events without a central manager, is an example of which pattern?
These patterns form the bedrock of resilient, large-scale system design, enabling independent teams to build services that work together reliably.
