Idempotent Webhook Architecture

Reliable Webhook Architecture

In a billing system, processing a webhook twice can lead to double-charging a customer or granting duplicate service credits. Events can be delivered more than once due to network timeouts, retries from the provider, or deployment issues on your end. The only way to guarantee correctness is to design your handler to be idempotent.

Design webhook handlers to be idempotent, ensuring repeated requests with the same event ID do not cause inconsistent data states.

The core architectural pattern is to separate acknowledgment from processing. When a webhook arrives, your first job is to validate its authenticity and log its unique identifier. You must then send a 200 OK response immediately. This tells the provider, like Stripe, that you've received the event and they don't need to retry. The actual business logic happens asynchronously, after you've secured the event.

Verification and Trust

Every webhook request from Stripe includes a Stripe-Signature header. This signature is generated using a signing secret unique to your webhook endpoint and the request payload. Verifying this signature is non-negotiable. It confirms the request is genuinely from Stripe and that the payload hasn't been tampered with in transit. If the signature is invalid, reject the request immediately with a 400 Bad Request status. Do not process it further.

const stripe = require('stripe')(process.env.STRIPE_SECRET_KEY);

// Somewhere in your Express route handler
app.post('/webhook', express.raw({type: 'application/json'}), (request, response) => {
  const sig = request.headers['stripe-signature'];
  const endpointSecret = process.env.STRIPE_WEBHOOK_SECRET;

  let event;

  try {
    event = stripe.webhooks.constructEvent(request.body, sig, endpointSecret);
  } catch (err) {
    // On error, send a 400 error back to Stripe
    response.status(400).send(`Webhook Error: ${err.message}`);
    return;
  }

  // Acknowledge receipt before processing
  response.status(200).send();

  // ... hand off to an async processor
  processBillingEvent(event);
});

Notice that the response is sent before processBillingEvent is called. This respects Stripe's tight timeout window and prevents unnecessary retries for valid, received events.

Idempotency Through Event Logging

Stripe events each have a unique identifier, like evt_1J2x3Y4Z.... This ID is our key to achieving idempotency. The strategy is to log this event ID in a dedicated database table before any processing occurs. When a new event arrives, we first check if its ID already exists in our log. If it does, we can safely ignore it, knowing it's a duplicate delivery.

This check-and-insert operation must be to prevent race conditions. If two identical webhooks arrive at nearly the same instant, both might pass the existence check before either has a chance to perform the insert. The result is a duplicate processing job.

The simplest way to enforce atomicity is to leverage a UNIQUE constraint on the event ID column in your database. This way, any attempt to insert a duplicate ID will fail at the database level, which is the ultimate source of truth.

Column	Type	Constraints
`event_id`	`VARCHAR(255)`	`PRIMARY KEY`
`status`	`ENUM('pending', 'processed', 'failed')`	`NOT NULL`
`created_at`	`TIMESTAMP`	`DEFAULT NOW()`
`payload`	`JSONB`

The table above, let's call it processed_events, serves as our idempotent ledger. The status column helps track the event's lifecycle, which is useful for debugging and manual intervention.

Locking and Processing

With the event ID logged, our background processor can now safely execute the business logic. However, we still need to handle concurrency. If your infrastructure scales horizontally, multiple instances of your processor could potentially grab the same pending event from the table at the same time.

This is where database-level locking is essential. Before processing an event, the worker should acquire a lock on the corresponding row in the processed_events table. In PostgreSQL, this is commonly done using SELECT ... FOR UPDATE. This statement locks the selected rows, preventing any other transaction from modifying or locking them until the current transaction completes.

async function processEvent(eventId) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');

    // Attempt to lock the row for this event
    const res = await client.query(
      'SELECT status FROM processed_events WHERE event_id = $1 FOR UPDATE NOWAIT',
      [eventId]
    );

    if (res.rows.length === 0 || res.rows[0].status !== 'pending') {
      // Event doesn't exist, is already processed, or is locked
      await client.query('COMMIT');
      return;
    }

    // --- Main Business Logic ---
    // e.g., increment seat count, provision service, etc.
    // await updateUserSubscription(event.data.object);
    // --------------------------

    // If successful, mark as processed
    await client.query(
      'UPDATE processed_events SET status = \'processed\' WHERE event_id = $1',
      [eventId]
    );

    await client.query('COMMIT');
  } catch (err) {
    await client.query('ROLLBACK');
    // Optionally update status to 'failed' and handle error
    throw err;
  } finally {
    client.release();
  }
}

The NOWAIT clause is a useful addition. If the row is already locked by another transaction, the query will fail immediately instead of waiting, allowing the worker to move on without being blocked. This entire sequence, wrapped in a transaction, ensures that each event is processed exactly once, even under high concurrency and with network retries.

Quiz Questions 1/5

What is the primary reason for a webhook handler to send a 200 OK response immediately after receiving an event, before executing the main business logic?

Quiz Questions 2/5

To prevent race conditions where two identical webhooks arriving simultaneously are both processed, the operation of checking for and inserting the event ID must be atomic.

Advanced Stripe Orchestration for Nodejs