Processing Stripe payments via a webhook adds new considerations to your system. In particular, your endpoint is publicly reacheable. While Stripe publishes their IP ranges, relying solely on network-level allowlisting is brittle. Your endpoint remains exposed, making cryptographic signature verification the only true defense. Furthermore, as one of the prime initial targets for penetration attacks, it is of interest to keep its responsibilities to a minimum and assume it would be the first system to be breached in an attack. This is why I introduce a new service with the sole responsibility of receiving and acknowledging Stripe payments and forwarding them to our internal services through a private API within a Private Virtual Cloud.

sequenceDiagram
    autonumber
    actor User
    participant Stripe
    participant Payment Webhook (Express)
    participant Firestore (DB)
    participant Quiz API
    participant Reconcile Cron Job

    User->>Stripe: Completes Payment
    Stripe->>Payment Webhook (Express): POST /webhook/stripe
    
    rect rgba(0, 150, 255, 0.15)
        Note over Payment Webhook (Express), Quiz API: Synchronous Fulfillment Phase
        Payment Webhook (Express)->>Firestore (DB): Check Idempotency (COMPLETED/PENDING)
        Payment Webhook (Express)->>Firestore (DB): Mark as PENDING
        Payment Webhook (Express)->>Quiz API: POST /sparks (with RS256 JWT)
        Quiz API-->>Payment Webhook (Express): 200 OK
        Payment Webhook (Express)->>Firestore (DB): Mark as COMPLETED
        Payment Webhook (Express)-->>Stripe: 200 OK (Stops retries)
    end
    
    rect rgba(255, 150, 0, 0.15)
        Note over Reconcile Cron Job, Stripe: Asynchronous Fallback Phase (Every 12h)
        Reconcile Cron Job->>Firestore (DB): Query all PENDING payments
        Reconcile Cron Job->>Quiz API: Retry /sparks fulfillment
        Reconcile Cron Job->>Firestore (DB): Mark as COMPLETED on success
        
        Reconcile Cron Job->>Stripe: Fetch events (last 24h)
        Reconcile Cron Job->>Firestore (DB): Check if Stripe event exists
        opt If event entirely missing from DB
            Reconcile Cron Job->>Quiz API: Fulfill missing payment
            Reconcile Cron Job->>Firestore (DB): Mark as COMPLETED
        end
    end

The Silent Failure: Why Standard Webhooks Drop

Relying solely on a webhook leads to silent failures, among which: Time outs while your serverless function is starting up, time outs while processing the payment, and expected downtime from either the webhook itself or the service’s server. And payments can be 24/7, so all of these can and will happen. When this happens, Stripe will retry a couple times, but exponentially delay, until it’s an unbearable wait for your customer.

As soon as the Stripe request reaches your webhook, the user experience is entirely within our control, and so, this system has the singular aim of fulfilling the customer’s transaction regardless of Stripe’s state. So, as soon as the request reaches our systems, I mark the payment as “PENDING” in our database, and proceed to fulfill the customer’s order. I set up internal retry mechanisms, over which I have more control than Stripe’s. And I set up idempotency so I can eventually let Stripe know all is well. For that, I have this data model:

// Data model stored in Firestore (processed_payments/{stripeSessionId})
{
  userId: string;       // The sanitized UUID or reference ID of the purchaser
  amount: number;       // The amount of internal currency ("sparks") to credit
  status: string;       // Enum: PaymentStatus.PENDING | PaymentStatus.COMPLETED
  timestamp: string;    // ISO Date string of when the webhook was first received
  lockedAt: string;     // The ISO timestamp of when a process claimed this document
  processedAt?: string; // ISO Date string of when the Quiz API successfully processed it
  reconciled?: boolean; // Flag indicating if this was recovered by the cron job (Phase 2)
}

The Real-Time Ingestion Layer (Express & Firestore)

Our Express receiver starts by validating Stripe’s cryptographic signature, and in a cascade style, forwards it into our internal systems to fulfill.

// src/app.ts

// We need raw body for Stripe signature verification
app.post('/webhook/stripe', express.raw({type: 'application/json'}), async (request, response) => {
  const sig = request.headers['stripe-signature'];
  let event;

  try {
    // 1. Validate the Stripe signature using the raw body
    event = stripe.webhooks.constructEvent(request.body, sig as string, endpointSecret);
  } catch (err: any) {
    logger.error(`Webhook Error: ${err.message}`, { error: err.message });
    response.status(400).send(`Webhook Error: ${err.message}`);
    return;
  }

  // ... (Payload sanitization, Idempotency checks, and API calls) ...

  // 2. Acknowledge the request to prevent Stripe from retrying
  response.send();
});

I only return a 200 OK to Stripe after I have a cascading success from our internal systems. Even if our internal service fails, we already recorded the Payment in our internal database, and established a claim lock on it. The newly introduce risk of this is getting a failure as the success cascades back to Stripe, which is why introducing idempotency and locking is so important. This idempotency is ensured by using the Checkout Session ID (cs_…) or Payment Intent ID (pi_…) as the document ID of the processed payment in our payment database. Even if Stripe retries a request that was already fulfilled, it will quickly be seen as fulfilled within the webhook by checking our Firestore, and so, it can quickly return a success response to Stripe without causing duplicate transactions and reengaging our internal systems.

Zero-Trust Internal Communication (RS256 JWTs)

Even though this Payments webhook only communicates to internal servers within a private VPC, zero-trust is maintained by cryptographically signing its requests to internal servers with pre-shared public keys. These pair of keys is managed by Pulumi, and so, separately given the private key to the Payments webhook and the public key to the internal services that need to know. When a payment succeeds, the Payments service creates a JWT containing the userId and the amount of sparks, signing it with its Private Key. And on receipt, the internal server verifies with its public key, in a stateless way. So, even breaches in the internal VPC are not enough to forge payments.

Lastly, as Stripe events can have user generated content, sanitization is done in the Payments webhook before signing, ensuring the data sent to internal services is devoid of SQL Injection and XSS payloads.

The Asynchronous Reconciliation Engine (Cloud Run)

As a safety net, we have a serverless scheduled job, via Cloud Run, with 2 phases. Phase 1 reconciles any payments we have in our internal database in a pending state. And Phase 2 fetches any payments from Stripe it may have missed. This helps fulfill payments in-between Stripe attempts, increasing the overall reliability. To avoid duplicate handling of the same payment, we also keep a Firestore transaction to claim each row being processed. Because of Firestore’s optimistic concurrency control, we handle this by using a short transaction to claim, and establish a lockedAt: our loop will only attempt to claim transactions without lockedAt or where it’s older than 5 minutes, ensuring that if any job crashes, another will take over after 5 minutes.

Currently, this reconciliation runs twice a day, and it attempts the last 24 hours of events, establishing an overlap to ensure no event is missed.