Stripe webhooks in production

Verify the signature, write the raw event to a table with a unique constraint on the Stripe event id, return 200 inside the response budget, process the event asynchronously behind an idempotency guard, and run a scheduled job that re-reads Stripe and repairs drift. That’s the entire architecture. Five steps, and they haven’t changed in years.

Teams get it wrong because each step defends against a different failure and none of them show up in development. They show up at 2am when a customer has been charged twice and your database still says free plan. What follows is each failure mode with its fix underneath, in the order they tend to bite.

What each of the five steps is actually for

Read this as the summary and the rest of the article as the argument. If you change one thing today, change the row whose failure column describes something you’ve already seen.

Failure mode one: the slow handler that becomes a double charge

Stripe documents a response budget measured in seconds, currently 20, and retries a failed or timed-out delivery with exponential backoff for up to three days in live mode. Both numbers are Stripe’s, both are in their webhook docs, and both have moved before, so check them rather than trusting this paragraph.

The chain nobody plans for goes like this. Your handler expands a customer, writes three rows, sends a welcome email through a provider having a slow morning, and takes 22 seconds. Stripe stops waiting and retries. Your first execution is still running. Two copies of the handler are now in the same code path, and the credit-granting write happens twice.

Why the retry is not the bug

At-least-once delivery is the contract rather than a defect. Stripe cannot know whether your 504 meant “I never saw it” or “I did the work and then the load balancer killed the connection”, so it assumes the worse case. Any design that treats a second delivery as anomalous is already broken. Design for the duplicate and the timeout stops mattering.

What “fast” means as a number

Don’t aim for the documented ceiling. Aim for verification, one insert, and a 200: low tens of milliseconds of your own work, plus whatever your platform charges for a cold start. That target is ours, not Stripe’s requirement.

Cold start is what ruins it, and on serverless it’s less under your control than you think. A route that pulls in your ORM, your email client and your analytics SDK pays for all three on first invocation. Keep the import graph brutally small, which is the same discipline behind why your Next.js site is slow and which fixes are worth doing.

Failure mode two: an unverified endpoint is a free subscription

If you parse the JSON body and act on it without verifying the Stripe-Signature header, your billing system has a public write API. Anyone who can guess or observe the URL can POST a hand-written invoice.paid and provision themselves whatever your handler provisions. No rate limit fixes this and no URL is obscure enough to matter.

Verification is an HMAC-SHA256 over the timestamp and the exact raw body, keyed by the endpoint’s signing secret. Stripe’s SDKs also enforce a default timestamp tolerance of five minutes, so a captured payload can’t be replayed against you indefinitely.

The three ways verification breaks in practice

First, a JSON body parser runs before your handler and hands you a re-serialised object. Key order and whitespace change, the HMAC no longer matches, and someone “fixes” it by deleting the verification call. In a Next.js route handler you want the raw text.

Second, the wrong secret. Each endpoint has its own whsec_ value, the CLI generates a different one for local forwarding, and test and live mode are separate again. Four secrets for one integration is normal. Name the environment variables accordingly.

Third, a proxy or edge runtime that mutates the body through recompression, charset normalisation, or a size guard that truncates. Verify before anything else touches the bytes.

// Illustrative only. app/api/stripe/webhook/route.ts
export async function POST(req: Request) {
  const raw = await req.text();                 // raw bytes, not parsed JSON
  const sig = req.headers.get("stripe-signature");

  let event;
  try {
    event = stripe.webhooks.constructEvent(raw, sig, process.env.STRIPE_WEBHOOK_SECRET);
  } catch {
    return new Response("bad signature", { status: 400 });
  }

  // Durable write, then answer. Nothing else happens on this request.
  const inserted = await db.insertStripeEvent({
    id: event.id,                               // UNIQUE constraint lives here
    type: event.type,
    apiVersion: event.api_version,
    payload: raw,
    receivedAt: new Date(),
  });                                            // ON CONFLICT DO NOTHING

  if (inserted) await queue.enqueue({ eventId: event.id });
  return new Response(null, { status: 200 });
}

Failure mode three: no idempotency guard, so retries apply twice

The unique constraint on the Stripe event id is the load-bearing part of the design, and it belongs in the schema rather than in application code. A check-then-insert is a race: two concurrent deliveries both read “not seen” and both proceed. A unique index cannot be raced. One insert wins, the other conflicts, and the conflict is your answer.

Event-level deduplication is necessary and not sufficient, because Stripe emits several events for one commercial fact. One upgrade can produce checkout.session.completed, customer.subscription.updated and invoice.paid. Each has a distinct id, so each passes the dedupe check, and if all three handlers grant a credit pack you’ve granted three.

So guard the effect as well as the event. Every effect derives its own natural key from the Stripe object it came from, meaning the invoice id for a credit grant, or the subscription id plus period start for a renewal, and that key is unique too. The test is blunt: replay one event five times through the real worker and diff the database. If anything differs after the first pass, you don’t have idempotency, you have luck.

Failure mode four: the webhook as source of truth

This is the expensive one, because it fails silently and stays failed. If the only thing that ever writes plan = pro is the webhook handler, one event that never lands leaves that account wrong permanently. Nothing in the system notices. The customer does, weeks later, usually in public.

Events go missing for ordinary reasons. A migration outlasted the three-day retry window. Someone rotated the signing secret and every delivery 400d for an afternoon. Stripe disabled the endpoint after sustained failures and the warning email landed in a shared inbox nobody reads.

The correct model: Stripe is the ledger, your database is a cache of it.Store the customer and subscription ids against your account and treat every entitlement field as a projection you can rebuild from Stripe’s API at any time. That commitment is what makes reconciliation possible, and it’s the same boundary question behind whether to build, buy or self-host authentication: who owns the record, and who merely holds a copy.

Failure mode five: events that arrive out of order

Stripe does not guarantee ordering, and never claimed to. Retries alone break it: an event that failed once and succeeded on the second attempt arrives after events created later. A cancellation lands before the upgrade that preceded it, the naive handler writes the upgrade last, and a cancelled customer looks active.

Two defences, in order of preference. The strong one: don’t trust the payload for current state, re-fetch the object from Stripe’s API and write what Stripe says now. The event becomes a notification that something changed rather than a description of what it changed to. Ordering stops mattering entirely.

The weaker one, for when the extra API call is genuinely too expensive: store the created timestamp of the last event applied per object and discard anything older. Cheap, and it fails on same-second events, which do happen.

Pin the API version on the endpoint while you’re here. Endpoints render events at their own configured version, and an unpinned one can start receiving a different object shape when Stripe ships a new version. Treat a version bump as a deliberate change with its own test pass.

The reconciliation job is what makes all of this survivable

Everything above reduces the chance of drift. Reconciliation detects the drift that happens anyway, and it’s the part almost nobody builds. A scheduled job, hourly for a subscription product and nightly if volume is low, pages through subscriptions modified since the last run, compares status and price against your database, and repairs differences.

Two rules keep it from becoming its own source of bugs. The repair path calls the exact same idempotent apply function the webhook worker calls, so there’s one definition of “active pro subscription” rather than two that slowly disagree. And every repair emits a metric. Drift repaired silently is drift you’ll never debug; if the job fixes forty accounts a night, you have a webhook defect and that number is how you find out.

The transitions that quietly lose money

These four leak revenue in both directions, billing someone who cancelled and serving someone who stopped paying, and generic webhook advice skips all of them.

Trial to paid

The subscription is already active-adjacent during a trial, so an entitlement check written as “status is trialing or active” passes throughout. The moment of truth is the first invoice after the trial, which can fail. Handle customer.subscription.trial_will_end to prompt for a payment method, then treat the first successful invoice as the real conversion event.

Cancel at period end

When a customer cancels for the end of the cycle, Stripe sends customer.subscription.updated with a cancellation flag set while the status stays active. Handlers that revoke access on any cancellation signal cut off a customer who has paid through to a date weeks away. Revoke on the terminal deletion event, or on the period end date, never on the flag.

Failed payments and dunning

A failed renewal moves the subscription into a delinquent state and starts Stripe’s retry schedule, which your dashboard controls and your code must not second-guess. Degrade immediately or keep serving through the retry window. Either is defensible, but it has to be one decision in one place. The common bug is an entitlement check that treats every non-active status as expired, killing accounts Stripe is about to successfully re-charge.

Proration and mid-cycle changes

An upgrade halfway through a month changes the subscription immediately and settles the money later. Entitlement follows the subscription item. Revenue reporting follows the invoice. Teams that read both from the same event get one of them wrong, usually the one finance looks at.

Replaying after an outage, and testing before one

Stripe keeps events queryable for a limited retention window, 30 days at time of writing, and exposes both a list endpoint and a resend action. That’s your recovery path: list events for the affected period, filter to the types you care about, push them back through your own worker. Because every step above is idempotent, a replay that overlaps already-processed events is harmless. That’s the payoff for the unique constraint.

Locally, use the Stripe CLI rather than a tunnel and a hand-made payload. stripe listen --forward-to gives you a real signed delivery against a real secret, and stripe trigger fires real fixtures. Automate two tests: the same fixture posted twice asserting identical state, and a wrong signature asserting a 400.

What this costs, and when not to build it

The bill for asynchronous processing is eventual consistency

The moment you return 200 before doing the work, the user who just paid can reach your dashboard before their entitlement exists. That needs a real answer, either a polling state on the success page or optimistic provisioning from the checkout session that the webhook later confirms. Either is extra code and an extra UI state. Anyone who tells you receive-fast is free hasn’t shipped it.

The second cost is operational. You now own an events table that grows forever, a queue, a worker, a dead-letter path and a reconciliation job with its own failure modes: four things to monitor where you had one. Plan the retention policy on that table on day one, or it’ll be your largest table inside a year.

Cases where you should not build this

If you’re pre-revenue with one plan and a handful of customers, a full webhook pipeline is premature. Stripe Checkout plus a scheduled script that reads subscriptions and syncs statuses gets you correct billing state with no endpoint to secure and no queue to run. It’s minutes behind reality, which at that stage is fine, and the webhook path bolts on later without changing the apply function.

If your entitlement model is a plain plan-to-feature mapping, look hard at Stripe’s own entitlements primitives before building a mapping table and the code to maintain it. And if you sell through a marketplace or app store, their billing system is the ledger rather than Stripe. Building against the wrong system of record costs more than any bug in this article.

The boundary on hiring anyone, stated plainly: a competent backend engineer with a week and this article doesn’t need us. Outside help pays on the other case, an existing broken integration with live revenue running through it that has to be migrated without a maintenance window. That’s most of the Stripe integration work we take on, and it’s closer to a rewrite-or-refactor judgement than to a greenfield build.

The checklist to run against your current endpoint

• Signature verified against the raw body, before any parsing, with the per-endpoint secret.
• A unique index on the Stripe event id in the schema, not a check-then-insert in code.
• Handler does verification, one durable write, and returns. Nothing else on the request path.
• Non-2xx returned only when you failed to persist. Unhandled event types return 200.
• Effects carry their own natural key, so several events describing one commercial fact cannot apply twice.
• Current state re-read from the Stripe API rather than trusted from the payload, with the endpoint API version pinned.
• A scheduled reconciliation job that calls the same apply function and emits a metric per repair.
• Tests for duplicate delivery and for a forged signature, in CI.

The constraints get stricter as the amounts do. All of this applies verbatim to fintech and payments platforms, where reconciliation is the primary control rather than a safety net added later. If you’re not sure which of the five steps you’re missing, describe the integration to us in a paragraph and we’ll tell you which one, whether or not you hire anyone to fix it.