The ambiguous-outcome problem unique to payments (a timeout that might mean success or failure), the reconciliation job that compares local records against the processor's ledger, webhook handling, and the dead-letter/alerting path for genuinely stuck payments.
Published September 23, 2026
Your service calls the payment processor -> the processor charges the card successfully
-> the SUCCESS RESPONSE is lost on the way back (network failure, timeout, your service crashes)
-> your service has NO WAY to distinguish "the charge failed" from "the charge succeeded but
the response was lost" purely from the timeout itself
This is the concrete version of Payment — Requirements' "no true undo" point: a timeout calling the payment processor is fundamentally ambiguous — it tells you the call didn't complete cleanly, not whether the charge itself happened. Treating every timeout as "definitely failed, safe to retry" risks a duplicate charge (mitigated by idempotency — Payment — Idempotency Implementation, but only if the retry uses the SAME idempotency key); treating every timeout as "definitely succeeded" risks never charging a customer who should have been charged.
public PaymentStatus resolveAmbiguousPayment(String idempotencyKey) {
// instead of guessing, ASK the processor directly using the idempotency key
// (most processors support looking up a transaction by the idempotency key you sent them)
ProcessorTransactionStatus actual = processorClient.lookupByIdempotencyKey(idempotencyKey);
return switch (actual) {
case SUCCEEDED -> PaymentStatus.CAPTURED; // update local record to match reality
case FAILED -> PaymentStatus.FAILED;
case NOT_FOUND -> PaymentStatus.FAILED; // processor never received it — safe to retry fresh
};
}
The correct response to an ambiguous timeout isn't guessing — it's querying the processor's own records directly (using the same idempotency key originally sent) to find out what actually happened, then updating the local record to match that ground truth. This is why the idempotency key needs to be something the processor itself can look up by, not just a local-only deduplication mechanism.
Scheduled job (e.g. every hour):
1. Pull the processor's transaction log for the period (most processors expose this via API)
2. For every processor transaction, find the matching local Payment record by processorReference
3. Flag any MISMATCH:
- processor shows SUCCEEDED, local record shows FAILED or missing -> investigate, likely
needs a local record correction (money moved, our records don't reflect it)
- processor shows FAILED/NOT_FOUND, local record shows CAPTURED -> serious, investigate
immediately (we believe we were paid but weren't)
Even with careful idempotency and ambiguity-resolution logic, a reconciliation job is still necessary as a safety net — it's the process that catches whatever the request-time logic missed (a bug, an edge case, a processor outage during the exact moment of ambiguity-resolution itself). This is a direct, concrete instance of Alerting Strategy's broader point: some correctness properties can't be fully guaranteed at request time alone and need an independent, periodic verification pass.
Most payment processors also push asynchronous notifications (webhooks) for events that happen on their side after the initial request (a delayed settlement, a chargeback, a dispute). Webhook handlers need their own idempotency handling (a webhook can be delivered more than once by the processor itself — this is standard, expected behavior, not a bug on the processor's end) and should verify the webhook's authenticity (a signature check) before trusting its payload, since an unauthenticated webhook endpoint is a real attack surface (Payment — Security covers this further).
A payment that remains ambiguous even after querying the processor (e.g. the processor itself is having an outage) shouldn't loop retrying silently forever — it should move to a dead-letter state after a bounded number of resolution attempts, triggering an alert (Alerting Strategy) for manual investigation. A payment stuck in an unresolved state with no human ever notified is a real, damaging failure mode — silent, unbounded retry loops and silent permanent-stuck states are both wrong; a bounded retry count followed by explicit escalation is the correct shape.
Q: How often should the reconciliation job run — does more frequent mean 'better'? A: More frequent reduces the window before a mismatch is caught, but most processors' transaction logs have their own reporting delay (transactions may not appear immediately), so running much faster than that delay just re-checks stale data — matching the job's frequency to the processor's actual reporting latency is more useful than an arbitrarily tight schedule.
Q: Why can't the ambiguity-resolution lookup itself also time out or fail? A: It can, and when it does, the correct response is the SAME bounded-retry-then-dead-letter pattern, not an infinite loop trying to resolve the ambiguity — reconciliation exists as the backstop specifically for cases where even the resolution attempt itself doesn't succeed within a reasonable number of tries.
Q: Should a webhook handler ever trust its payload without database cross-referencing? A: No — a webhook should be treated as a NOTIFICATION to go check the actual state (via the processor's authoritative API or against local records), not as the sole source of truth to act on blindly, since a delayed, duplicated, or (if verification is skipped) forged webhook could otherwise directly corrupt payment state.
Q: Is reconciliation only necessary because of engineering bugs, or is it fundamentally required regardless of code quality? A: Fundamentally required regardless of code quality — network partitions, processor-side outages, and the physical impossibility of guaranteeing a response is received exactly once over an unreliable network mean ambiguous outcomes are an inherent property of any system calling an external service over a network, not a symptom of bugs that could be coded away entirely.