Foundational Problems Networks are unreliable, making 100% agreement between systems theoretically impossible, which forces developers to design for failure and uncertainty from the start.

Core Trade-offs When systems are separated by a network, you must often trade perfect, immediate data consistency for system availability, embracing temporary data discrepancies.

The Saga Solution The Saga pattern manages long-running business processes by breaking them into a sequence of small, independent, and local database transactions that can be individually confirmed.

Saga Implementation Sagas can be implemented decentrally via event-driven Choreography where services react to each other, or centrally via an Orchestrator that explicitly commands each step.

Reliability Patterns To build robust Sagas, patterns like the Outbox ensure messages are sent reliably, while Idempotency prevents duplicate messages from causing incorrect side effects.

Operational Concerns Because Sagas are asynchronous and distributed, you must implement explicit monitoring with correlation IDs to trace a single business process across all its steps and services.

Foundational Problems

1. Networks are Unreliable: Messages can be lost, delayed, or duplicated. Never assume a message sent was a message received.
2. The 'Acknowledge' (ACK) is Key: An ACK is a confirmation that a message was received, but the ACK itself can also get lost.
3. The Two Generals' Problem: This thought experiment proves that 100% certainty of agreement between two systems over an unreliable network is mathematically impossible.
4. Infinite Acknowledgements: The problem leads to an infinite loop where each confirmation needs its own confirmation, with the last message always being uncertain.
5. Design for Uncertainty: The takeaway is to stop trying to achieve perfect certainty and instead design systems that can handle ambiguity.

Core Trade-offs

1. ACID is Local: ACID (Atomicity, Consistency, Isolation, Durability) properties are guarantees for a single database transaction, not across multiple services.
2. Two-Phase Commit (2PC) is Flawed: 2PC tries to extend ACID across services but is slow and 'blocking,' meaning one failed service can freeze the entire system.
3. CAP Theorem is Law: In a network failure (Partition), you must choose between Consistency (erroring out) or Availability (staying online with potentially stale data).
4. 'P' is Not Optional: In any real-world distributed system, you must assume network partitions will happen; the choice is always between C and A.
5. Eventual Consistency: This model accepts that the system will be temporarily inconsistent, but will resolve to a correct state over time.

The Saga Solution

1. Sagas are Sequences: A Saga is a sequence of local transactions where each transaction updates a single service and triggers the next step.
2. No Giant Locks: Unlike 2PC, Sagas do not hold long-lived locks on resources, which improves system performance and availability.
3. Sagas are Business Processes: They are best used for 'long-lived transactions' that map to a multi-step business process (e.g., booking a trip).
4. Failure Requires Compensation: If a step in a Saga fails, there is no automatic rollback. You must execute an explicit 'compensating transaction' to undo the work.
5. Forward Recovery: Sagas don't go backward (rollback); they go forward by executing new transactions that semantically reverse previous steps.

Saga Implementation

1. Choreography is Decentralized: Services communicate by publishing and listening to events without a central coordinator.
2. Choreography's Risk: As more services are added, Choreography can become complex and hard to debug, like a tangled web of 'spaghetti'.
3. Orchestration is Centralized: A single 'Orchestrator' component issues commands to services, telling them what to do and when.
4. Orchestration's Benefit: The business logic is in one place, making it easier to monitor, debug, and change the overall process flow.
5. The Trade-off: Choreography offers simplicity and decoupling for small systems; Orchestration offers control and visibility for complex ones.

Reliability Patterns

1. The Outbox Pattern: Guarantees that a business data update and its corresponding event message are saved atomically in the same local database transaction.
2. How Outbox Works: An event is written to an 'outbox' table. A separate process reads from this table and sends the message to other services.
3. At-Least-Once Delivery: Due to network issues, a message might be delivered more than once. Systems must be prepared for this.
4. Idempotency is the Solution: An operation is idempotent if running it multiple times has the same effect as running it once.
5. Idempotency Keys: Use a unique identifier (like an order_id) to track which operations have already been processed and safely ignore duplicates.

Operational Concerns

1. Distributed Tracing is Essential: You cannot understand a distributed process by looking at the logs of just one service.
2. Correlation IDs: A unique ID is generated at the start of a Saga and passed along with every step and message.
3. Unified Logging: This ID allows you to filter logs from all services to see the end-to-end journey of a single business transaction.
4. State Machines: Model your Saga as a state machine (PENDING, APPROVED, SHIPPED, CANCELLED) to easily track where an order is in the process.
5. Monitor for 'Stuck' Sagas: Set up alerts for Sagas that remain in an intermediate state for too long, indicating a potential failure that needs manual intervention.

Distributed System: A collection of independent computers that appears to its users as a single coherent system.

• Used in context: Our ERP is a distributed system because the inventory service and the shipping service run on different servers but must work together.
• Topic: Foundational Problems

---

Packet Loss: When a 'packet' of data being sent across a network fails to reach its destination.

• Used in context: The command to update inventory was never received due to packet loss, so the stock count is now incorrect.
• Topic: Foundational Problems

---

Two Generals' Problem: A thought experiment showing it's impossible for two parties to agree on a coordinated action over an unreliable communication channel.

• Used in context: We can't guarantee the payment service received our request because of the Two Generals' Problem; we can only design a system that handles the uncertainty.
• Topic: Foundational Problems

---

ACID Properties: A set of properties (Atomicity, Consistency, Isolation, Durability) that guarantee database transactions are processed reliably.

• Used in context: While a single PostgreSQL update is protected by ACID, our cross-service order process is not.
• Topic: Core Trade-offs

---

Two-Phase Commit (2PC): A protocol used to achieve atomic transaction commit across multiple nodes, but it can cause the system to block if any node is unresponsive.

• Used in context: We avoided using 2PC for our ERP because its blocking nature would severely harm system availability.
• Topic: Core Trade-offs

---

CAP Theorem: A principle stating that it is impossible for a distributed data store to simultaneously provide more than two out of three guarantees: Consistency, Availability, and Partition Tolerance.

• Used in context: According to the CAP theorem, we chose Availability during a network split, meaning our ERP might show slightly stale data but will remain online.
• Topic: Core Trade-offs

---

Eventual Consistency: A consistency model which guarantees that, if no new updates are made, all replicas of data will eventually return the same last updated value.

• Used in context: The accounting ledger shows the sale a few seconds after the inventory is updated; the system is eventually consistent.
• Topic: Core Trade-offs

---

Saga Pattern: A design pattern for managing data consistency across microservices in distributed transaction scenarios.

• Used in context: We used the Saga pattern to handle our complex order fulfillment process without locking multiple databases.
• Topic: The Saga Solution

---

Compensating Transaction: An action that semantically reverses the effect of a previous transaction in a Saga.

• Used in context: When the payment failed, a compensating transaction was executed to add the item back to the inventory.
• Topic: The Saga Solution

---

Choreography: A decentralized Saga pattern where services react to events from other services without a central controller.

• Used in context: In our simple system, we use Choreography: the Order service publishes an OrderCreated event, and the Inventory service subscribes to it.
• Topic: Saga Implementation

---

Orchestration: A centralized Saga pattern where a single 'orchestrator' class or service tells the other services what to do.

• Used in context: For our complex return process, we chose Orchestration so the logic for each step is clearly defined in one place.
• Topic: Saga Implementation

---

Outbox Pattern: A reliability pattern that ensures an event is published if, and only if, the state-changing database transaction is successful.

• Used in context: By using the Outbox pattern, we guarantee we'll never send an OrderShipped email if the database record fails to update.
• Topic: Reliability Patterns

---

Idempotency: The property of an operation where it can be applied multiple times without changing the result beyond the initial application.

• Used in context: Our payment processing endpoint is idempotent; if the same charge request arrives twice, the customer is only charged once.
• Topic: Reliability Patterns

---

Correlation ID: A unique identifier passed through a chain of requests or events in a distributed system to trace the entire flow.

• Used in context: By searching for the correlation ID in our logs, we could trace the entire lifecycle of a single customer order across five different services.
• Topic: Operational Concerns

❌ MISTAKE: Assuming that if you don't receive an error from a network call, it must have succeeded.

• Why it happens: Developers are used to in-process function calls that either return a value or throw an exception. Network calls have a third state: silence (timeout), which tells you nothing.
• ✅ Instead: Design for uncertainty. Treat a timeout as an unknown state and have a process to verify the outcome later or build your operations to be safely repeatable (idempotent).
• Topic: Foundational Problems

---

❌ MISTAKE: Confusing a Saga's 'compensating transaction' with a database 'rollback'.

• Why it happens: The term 'undo' sounds like a rollback. But a rollback magically reverts changes, while a compensation is a new, forward-moving action.
• ✅ Instead: Remember that compensation applies a semantic reversal. You don't delete the Order row; you update its status to CANCELLED and run a Refund transaction.
• Topic: The Saga Solution

---

❌ MISTAKE: Updating the database and then, in a separate step, sending an event message.

• Why it happens: It seems like a logical sequence of operations. However, the system could crash between the database commit and sending the message, leaving them out of sync.
• ✅ Instead: Use the Outbox Pattern. Write the event to a special table within the same database transaction as your business data change. This makes the two actions atomic.
• Topic: Reliability Patterns

---

❌ MISTAKE: Believing 'Eventual Consistency' means 'slow and unreliable'.

• Why it happens: The term sounds vague and non-committal. It can be perceived as just accepting broken data.
• ✅ Instead: View eventual consistency as a deliberate engineering trade-off that prioritizes system availability and performance over immediate, perfect consistency, with guarantees that consistency will be achieved.
• Topic: Core Trade-offs

---

❌ MISTAKE: Forgetting to add a Correlation ID to all messages and logs.

• Why it happens: It's an extra piece of data that seems unimportant when building a single service in isolation.
• ✅ Instead: Make the Correlation ID a mandatory, first-class citizen in all communication. Without it, debugging a failed order across three services is nearly impossible.
• Topic: Operational Concerns

Order Processing Saga (with Failure)

Goal: Process a customer order that involves inventory, payment, and shipping services.

Step 1: Reserve Inventory

• What happens: The Order service starts the Saga. It sends a command to the Inventory service to reserve the product.
• Key indicator: Inventory table shows a decrease in available_stock and an increase in reserved_stock.

Step 2: Process Payment

• What happens: Upon inventory confirmation, the Orchestrator (or a responding service in Choreography) commands the Payment service to charge the customer's card.
• Key indicator: A transaction appears in the payment gateway's system.
• Common error: The payment is declined due to insufficient funds.

Step 3 (Failure Path): Execute Compensating Transaction

• What happens: The Payment service reports failure. The Orchestrator now sends a CancelInventoryReservation command to the Inventory service.
• Key indicator: The Inventory service executes a local transaction to move the stock from reserved_stock back to available_stock.

Step 4: Mark Saga as Failed

• What happens: The Orchestrator updates the Order's status to FAILED_PAYMENT.
• Key indicator: The final state of the order is logged, and the process terminates.

Topic: The Saga Solution

---

The Outbox Pattern Process

Goal: Reliably send a message/event after a database transaction commits.

Step 1: Single Atomic Transaction

• What happens: The business logic performs a database operation (e.g., UPDATE products SET stock = 99 WHERE id = 123) and, in the same transaction, inserts a row into an outbox_events table (e.g., { event_type: 'ProductStockUpdated', payload: '...' }).
• Key indicator: The products table and the outbox_events table are both updated together or not at all.

Step 2: The Relay/Publisher Process

• What happens: A separate, independent process periodically scans the outbox_events table for new, unprocessed messages.
• Key indicator: This process runs on a loop (e.g., every 500ms).

Step 3: Publish and Mark as Sent

• What happens: For each new event found, the relay process sends it to the message broker (e.g., Kafka, RabbitMQ). After receiving a successful acknowledgment from the broker, it updates the event's row in the outbox_events table to processed = true.
• Key indicator: The event appears in the message queue, and the corresponding row in the outbox is marked.
• Common error: The relay crashes after sending the message but before marking it as processed. On restart, it will send the same message again, creating a duplicate (which is why consumers must be idempotent).

Topic: Reliability Patterns