Partner Integrations — OTA Sync, Retries, Webhooks, Reconciliation & Bulk Data — Interview Questions

How to use this lesson

External integrations fail in every possible way: they time out, rate-limit you, return partial success, send duplicates, deliver out of order, and change their APIs. Senior answers use a standard toolkit:

• adapters behind a canonical model;
• asynchronous queues;
• idempotency;
• retries with backoff, plus DLQs;
• circuit breakers;
• reconciliation, as the final safety net.

Q1. How does OTA rate and availability sync work? Design an OTA rate-sync system.

Short answer:

• Two directions:
  • ARI push (Availability, Rates, Inventory): from the CRS to the OTAs, usually through a channel manager or direct APIs (OTA-specific XML or JSON APIs, often based on OpenTravel (OTA) standards);
  • reservations pull or push: bookings made on OTAs flow into the CRS (a webhook or polling), and must decrement the inventory.
• The design:
  1. Inventory, rate and restriction changes emit events (outbox to Kafka), keyed by hotelId, which keeps the ordering per hotel.
  2. A sync service consumes them and builds delta updates per channel: coalesce rapid changes (many updates to the same hotel, room and date within seconds become one message), and batch the date ranges.
  3. Channel adapters (one per OTA) translate the canonical model to each partner's format, apply the channel's mapping (room and rate-plan codes) and markups, and respect the partner's rate limits.
  4. Send the requests with timeouts, retries (backoff and jitter), and circuit breakers; failed messages go to a per-channel DLQ for replay.
  5. Track the sync status per hotel, channel and date; run periodic full syncs (for example nightly) to heal any drift.
• Ordering and staleness: send the latest state (not increments like "minus 1 room"), with a version or timestamp, so an older update can't overwrite a newer one.

Q2. How do you scale OTA integrations?

Short answer:

• Partition the work by hotel (Kafka partitions keyed by hotel): parallelism across hotels, ordering within a hotel.
• A separate consumer group and worker pool per channel (bulkheads): a slow OTA doesn't block the others.
• Coalesce and batch updates, which reduces the call volume by orders of magnitude.
• Adaptive rate limiting per partner (a token bucket, and honouring 429 or Retry-After).
• Horizontal scaling of the stateless adapters, with autoscaling on consumer lag.
• Configuration-driven adapters (mapping tables, templates) make onboarding a new partner mostly configuration, not new code.
• Contract tests and partner sandboxes in CI.

Q3. How do you handle OTA failures? Design an OTA integration retry mechanism.

Short answer:

• Classify the failures:
  • transient (timeouts, 5xx, 429): retry;
  • permanent (4xx validation, a mapping error, authentication): don't retry. Send it to the DLQ and alert the right team;
  • unknown outcome (a timeout after sending): the request must be idempotent, or you must query the status before resending.
• The retry design:
  • exponential backoff with jitter and a maximum number of attempts or maximum time;
  • retry topics (Spring Kafka's @RetryableTopic: retry-1m, retry-5m, retry-30m, then a DLT), so the main flow isn't blocked;
  • a circuit breaker per partner (Resilience4j) that stops calls to an OTA that's down, and queues the updates;
  • latest-state semantics: when retrying, send the current state, not the stale payload;
  • DLQ tooling: inspect, fix and replay messages.
• Bookings coming in from an OTA: acknowledge them quickly, process them asynchronously and idempotently (dedupe on the OTA booking ID). If the room is no longer available (overbooking from sync lag), run the walk or relocation process and alert.
• Monitoring: sync lag per channel, error rates, DLQ depth, and reconciliation differences.

Q4. Design retry with exponential backoff.

Short answer:

• delay = min(maxDelay, base × 2^attempt), plus jitter (full jitter: random(0, delay)), which avoids thundering herds when many clients retry at the same moment.
• Retry only idempotent or safe operations, and retryable errors.
• Limit the attempts or the total time, and respect the caller's deadline (no retries past the overall timeout).
• Honour Retry-After headers.
• Use a retry budget (for example, retries at most 10% of the requests), so retries can't amplify an outage.
• Don't nest retries in every layer (the multiplication problem: 3 retries × 3 layers = 27 calls).

RetryConfig cfg = RetryConfig.custom()
    .maxAttempts(5)
    .intervalFunction(IntervalFunction.ofExponentialRandomBackoff(Duration.ofMillis(200), 2.0, 0.5))
    .retryOnException(e -> e instanceof TransientPartnerException)
    .build();
Retry retry = Retry.of("ota-expedia", cfg);
Supplier<Response> call = Retry.decorateSupplier(retry, () -> client.push(update));

Learn it in depth → Design a Retry Mechanism with Backoff

Q5. Design a webhook delivery system.

Short answer:

• Registration: partners register endpoint URLs, the event types they want, and a secret. Validate the URLs (block internal or private IP ranges, to prevent SSRF).
• Delivery pipeline:
  • events go into a durable queue (Kafka), then delivery workers;
  • each delivery is an HTTP POST with a signature (HMAC-SHA256 of the timestamp and body, in a header), an event ID (for the receiver to deduplicate), and a timestamp (the receiver rejects old ones, preventing replays);
  • short timeouts; any 2xx means success;
  • retries with exponential backoff over hours or days; after that, the endpoint is disabled and the owner is notified.
• Isolation: per-endpoint queues or rate limits, so one slow receiver doesn't delay others (bulkheads).
• Ordering: usually not guaranteed, so include a sequence number or version; receivers fetch the latest state if needed.
• Observability and self-service: a delivery log per endpoint, manual redelivery, and test events.
• Semantics: at-least-once. Document that receivers must be idempotent.

Q6. Design a payment reconciliation system.

Short answer:

• Goal: prove that our records (bookings and payment intents), the payment service provider's reports (settlements, captures, refunds, chargebacks) and the bank statements all agree, and investigate any differences.
• The pipeline:
  1. Ingest the PSP settlement files or APIs, and bank statements, daily (with idempotent file processing: a hash or file ID).
  2. Normalise them into a canonical transaction model (the amount in minor units, currency, reference IDs, dates, fees).
  3. Match on keys (the PSP transaction ID, our payment ID, merchant reference); fall back to fuzzy matching (amount, date window).
  4. Classify the result: matched; missing on our side; missing at the PSP; amount mismatch; duplicates; fees; FX differences.
  5. Create exceptions for finance ops, with a workflow (assign, resolve, adjust), and automated fixes for the known categories.
  6. Report: the match rate, the unreconciled amount, and aging.
• Design principles:
  • an immutable ledger (double-entry style) as the internal source of truth;
  • idempotent ingestion;
  • an audit trail;
  • handle time zones and cut-off times correctly.

Q7. Design a bulk data import pipeline (and a file upload service).

Short answer:

• Upload:
  • the client asks for a pre-signed URL and uploads directly to object storage (S3), in multipart or resumable form, so the application servers don't handle large streams;
  • validate the type, size and checksum; run a malware scan in a quarantine bucket before accepting;
  • store the metadata (owner, status, checksum) in a database.
• Processing:
  1. an upload event (S3 notification) triggers the import job;
  2. stream-parse the file (never load it all into memory) in chunks; Spring Batch gives chunk-oriented processing, restartability, skip and retry policies, and job metadata;
  3. validate each record, and write the errors to an error report (row, field, reason), instead of failing the entire file;
  4. batch writes (JDBC batch, COPY for PostgreSQL); upserts for idempotent reprocessing;
  5. parallelise by partitioning the file or the key ranges;
  6. track the progress and status (pending, processing, completed with errors), and notify the user.
• Correctness: idempotency (the file hash plus the row key), transactional chunks, and a staging table, then a swap, for all-or-nothing imports.

Learn it in depth → Design a Batch Processing System

Follow-up questions this topic invites — and their answers

Q: Why use a canonical model with adapters? A: Each partner has its own format and quirks. A canonical internal model plus one adapter per partner (the Adapter and Anti-Corruption Layer patterns) keeps the core domain independent of partners, and makes onboarding a new partner local work.

Q: Why send full state instead of deltas to partners? A: Full (latest) state per key is idempotent and self-healing: a retried or reordered message can't double-apply a change, and an older version can simply be ignored.

Q: How does the receiver verify a webhook? A: Recompute the HMAC over the timestamp and raw body with the shared secret, compare it in constant time, check the timestamp is recent, and deduplicate on the event ID.

Q: What's the difference between a retry topic and a DLQ? A: Retry topics delay and re-attempt messages automatically (transient failures); the DLQ holds messages that exhausted retries or failed permanently, for inspection, fixing and manual replay.