Extreme Scale, 99.99% Availability, DR & Project Deep-Dive Stories — Interview Questions

How to use this lesson

For scale questions, start with the numbers (back-of-envelope estimates), then remove the bottlenecks layer by layer: edge → stateless compute → data → asynchronous work. For availability, talk in nines, failure domains and recovery objectives (RTO/RPO).

For project deep-dives, prepare specific stories with metrics. Interviewers will dig 3–4 levels deep, so only claim what you really did.

Learn it in depth → Back-of-Envelope Estimation

Q1. How would you design a 1M TPS system?

Short answer:

• Clarify: reads or writes? What payload size? What latency target? What consistency is required? 1M TPS of cached reads is a very different problem from 1M TPS of durable writes.
• Edge:
  • a CDN and edge caching for anything cacheable;
  • GeoDNS or anycast routing to several regions;
  • L4/L7 load balancers;
  • rate limiting and WAF at the edge.
• Compute:
  • stateless services, horizontally scaled (at about 5–10k requests per second per instance, that's hundreds of instances across regions);
  • efficient I/O (non-blocking, or virtual threads);
  • connection reuse (HTTP/2, gRPC);
  • small payloads (Protobuf).
• Reads: a multi-level cache (in-process → Redis Cluster → database), so over 95–99% of reads never touch the database; read replicas.
• Writes:
  • partition everything (shard by a key with even distribution);
  • append-only, log-structured writes (Kafka as the write buffer; each partition handles tens of MB/s);
  • storage built for write throughput (Cassandra, ScyllaDB, DynamoDB);
  • batching and asynchronous acknowledgement where the business allows it.
• Avoid the global bottlenecks:
  • no single-row hot counters (use sharded counters);
  • no global locks;
  • no cross-shard transactions on the hot path (use sagas);
  • ID generation without coordination (Snowflake-style IDs, or UUIDv7).
• Operate:
  • load tests at scale;
  • autoscaling with pre-warming;
  • backpressure and load shedding (drop low-priority work first);
  • cell-based architecture (independent copies of the stack, each serving a subset of users) to limit the blast radius.

Q2. How do you optimise for high traffic, and handle peak traffic?

Short answer:

• Measure first: profile and load test to find the real bottleneck (database, connection pools, locks, GC, downstream services).
• Reduce the work: caching (HTTP, CDN, Redis, local), pagination, compression, avoiding N+1 queries, precomputation, and asynchronous processing of non-critical work (queues).
• Scale out: stateless instances with autoscaling; read replicas; sharding; a bigger connection pool only if the database can take it.
• Protect the system at peak:
  • rate limiting and queue-based load levelling;
  • a virtual waiting room for flash sales;
  • circuit breakers, timeouts and bulkheads;
  • graceful degradation (turn off recommendations or reviews, serve cached prices) using feature flags;
  • prioritise the revenue path (booking and payment over browsing extras).
• Prepare for known peaks (sales, holidays): capacity planning, pre-scaling (scheduled scaling), cache warming, a change freeze, war-room monitoring, and game days.

A story template: "Before a festival sale, load tests showed the database CPU saturating at 3× normal traffic. We added a Redis availability cache (95% hit ratio), moved confirmation emails to Kafka, and pre-scaled the pods. On the day we handled 8× normal traffic with p99 under 400 ms."

Q3. How do you design a system with 99.99% uptime?

Short answer:

• The budget: 99.99% allows about 52 minutes of downtime per year (about 4.3 minutes per month). That leaves little room for manual recovery, so failover must be automatic.
• Remove single points of failure:
  • at least 3 instances across 3 AZs; load balancers with health checks;
  • replicated databases with automatic failover (Aurora, Patroni for PostgreSQL, a managed service), Kafka RF=3;
  • redundant network paths; multi-region for the highest tiers (the region itself becomes the failure domain).
• Contain failures:
  • timeouts, retries with backoff, circuit breakers, bulkheads;
  • graceful degradation;
  • cell-based architecture;
  • dependencies' availability must be higher (availabilities multiply: 5 dependencies at 99.99% give about 99.95%), or be made non-critical with async and fallbacks.
• Safe change (most outages are caused by changes):
  • canary and blue-green deployments with automated rollback;
  • feature flags;
  • backward-compatible database migrations (expand and contract);
  • config changes rolled out gradually.
• Detect and recover fast: SLO-based alerting, automated remediation, runbooks, on-call, and practised incident response. MTTR matters as much as MTBF.
• Verify: chaos engineering and game days; DR drills.

Q4. What is a multi-region deployment strategy (for Spring applications)? Design a disaster recovery strategy.

Short answer:

• The DR objectives: RPO (how much data you can lose) and RTO (how long you can be down), per service. They drive the cost.
• Patterns, from cheapest to most expensive:
  • backup and restore (hours of RTO; RPO = the backup interval);
  • pilot light (data replicated; minimal compute in the DR region, scaled up on failover);
  • warm standby (a scaled-down copy running);
  • active-active multi-region (both regions serve traffic; RTO near zero; the most complex).
• Data is the hard part:
  • asynchronous cross-region replication (Aurora Global Database, PostgreSQL replicas, MirrorMaker 2 for Kafka) means RPO above zero (seconds);
  • active-active writes need conflict handling (a home region per tenant or hotel; CRDTs; last-writer-wins only for suitable data);
  • or globally distributed databases (Spanner, CockroachDB, DynamoDB global tables), with latency trade-offs.
• Traffic:
  • DNS or global load balancer failover (Route 53 health checks, Cloudflare), with low TTLs;
  • region affinity for users;
  • externalised configuration per region (Spring profiles or config per region; region-aware service discovery).
• Spring specifics: stateless services (sessions in Redis or tokens), idempotent consumers (duplicates after failover), and Flyway migrations compatible with both regions during a rollout.
• Practice: regular DR drills (actually fail over), runbooks, backup restore tests, and infrastructure as code, so a region can be rebuilt.

Q5. Design an auto-scaling strategy, and cold-start mitigation.

Short answer:

• What to scale on:
  • CPU works for CPU-bound services; for I/O-bound services, use requests per second per pod, latency, or concurrency;
  • Kafka consumer lag (with KEDA) for consumers;
  • queue depth for workers.
• Kubernetes:
  • the HPA (pods) with sensible targets (for example 60–70% CPU) and stabilisation windows (scale up fast, scale down slowly);
  • the Cluster Autoscaler or Karpenter (nodes);
  • VPA for right-sizing requests;
  • scheduled scaling for known peaks;
  • minimum replicas across AZs.
• Limits: a maximum replica count that the database and downstream services can support (autoscaling the application can overload the database).
• Cold-start mitigation (for the JVM):
  • readiness probes that pass only after warm-up (cache warming, connection pool initialisation, JIT warm-up through synthetic requests);
  • slow-start in the load balancer;
  • faster startup: Spring Boot lazy initialisation (carefully), Class Data Sharing / AppCDS, CRaC (checkpoint and restore), GraalVM native images (millisecond startup; for serverless or scale-to-zero, at some build complexity and peak throughput cost);
  • keep warm capacity (minimum replicas, provisioned concurrency for Lambda), plus predictive or scheduled scaling ahead of peaks;
  • smaller images, and pre-pulled images on nodes.

Learn it in depth → Probes & Autoscaling

Q6. How would you redesign your current architecture for 10× scale? What improvements would you suggest in the current system?

Short answer: Structure the answer:

1. The current baseline: traffic, data volume, p99 latency, cost, and the known bottlenecks (from load tests and production metrics).
2. Find what breaks first at 10×: usually the database (writes, connections, hot rows), then synchronous call chains, then shared caches or brokers, then operational processes.
3. Changes, by layer:
   • data: read replicas and caching → sharding by the natural key (hotel or tenant) → moving high-volume data (events, logs, search) to fit-for-purpose stores;
   • communication: synchronous chains → events (outbox, Kafka) for non-critical steps;
   • compute: statelessness, autoscaling, efficient I/O;
   • edge: CDN, gateway rate limiting;
   • cells or regions to limit the blast radius.
4. Non-technical: team ownership boundaries, platform tooling, observability, cost per transaction.
5. Migration plan: incremental (strangler), measurable milestones, and load tests at each step.

Improvements to suggest (a good "what would you change?" answer): the top 3 pain points with evidence (for example, "40% of incidents come from the OTA sync, so make it asynchronous with retries and reconciliation"), each with its expected impact and cost.

Q7. What performance issue have you solved? How do you debug a memory issue?

Short answer:

• Performance story (STAR with numbers):
  • "The availability API's p99 was 2.4 s. Tracing showed 60% of the time in the database: an N+1 query when loading rate plans, plus a missing composite index. We fetched with a join (entity graph), added (hotel_id, stay_date) indexing, and cached rate plans in Caffeine. The p99 dropped to 280 ms, and database CPU fell by 45%."
  • Mention the method: measure (APM, traces, profiler) → hypothesis → fix → verify with a load test → monitor.
• Debugging memory issues:
  1. Symptoms: OutOfMemoryError (which kind: heap, metaspace, direct memory, "unable to create native thread"), long GC pauses, a steadily growing heap after full GCs (a leak), or a pod OOMKilled by Kubernetes (the native memory exceeded the container limit, not the heap).
  2. Collect data:
     • GC logs (-Xlog:gc*);
     • -XX:+HeapDumpOnOutOfMemoryError;
     • jcmd <pid> GC.heap_info, and a heap dump with jcmd <pid> GC.heap_dump;
     • class histograms (jcmd GC.class_histogram);
     • JFR recordings (allocation profiling);
     • Native Memory Tracking (-XX:NativeMemoryTracking=summary) for off-heap issues.
  3. Analyse: Eclipse MAT: the dominator tree, leak suspects, and the path to GC roots. The common culprits are:
     • unbounded caches or maps;
     • ThreadLocals not removed in thread pools;
     • listeners never unregistered;
     • large result sets loaded fully;
     • classloader leaks on redeploys;
     • direct buffers.
  4. Fix and verify (bounded caches with Caffeine, streaming or pagination, cleanup in finally), and add memory metrics and alerts.

Common trap: in containers, set the heap as a percentage (-XX:MaxRAMPercentage=75) and leave headroom for metaspace, thread stacks, direct buffers and the code cache. Setting -Xmx equal to the container limit guarantees OOMKills.

Q8. What was the biggest production issue you faced?

Short answer: Use an incident story structure:

1. Impact: what users experienced, how long it lasted, and the business cost.
2. Detection: how it was noticed (an alert, or a customer?).
3. Response: your role (incident commander, or investigator), mitigation first (a rollback, feature flag, scaling, failover), then the diagnosis.
4. Root cause: technical and systemic (for example, "a config change reduced the database pool from 50 to 5; there was no validation, and no canary for config").
5. Follow-ups: the blameless postmortem, and the preventive actions (config validation, canary config rollout, a pool-saturation alert).
6. What you learned.

Show calm, structured thinking, and systemic fixes, not heroics.

Q9. What architectural decision did you take independently? Why microservices for your project?

Short answer:

• Independent decision: describe the context, the options considered, why you chose one (data, a spike), how you got buy-in (an RFC or ADR), the outcome with metrics, and what you'd reconsider. For example: introducing the outbox pattern, choosing Redis for the rate cache, or splitting a service.
• Why microservices: give real drivers:
  • independent scaling (search has 100× the load of booking);
  • independent deployment by several teams;
  • fault isolation (a partner-integration failure doesn't take down booking);
  • different technology needs.
  • Be honest about the costs: distributed transactions (sagas), operational complexity, observability, and network latency.
  • A senior answer can also be: "we deliberately kept a modular monolith for X, because the team was small and the domain boundaries weren't clear yet."

Q10. Why should we hire you as a Senior Java Developer?

Short answer: Give a concise, evidence-based pitch (60–90 seconds), matched to the job:

1. Technical depth: "N years of Java and Spring in production, building [domain] systems at [scale]".
2. Impact: 2 or 3 quantified achievements (latency, reliability, cost, delivery).
3. Senior behaviours: designing systems end to end, mentoring, improving engineering practices, owning production.
4. Fit: why this role and domain, and what you'd contribute in the first months.

Avoid generic claims ("hard-working, a team player") without evidence.

Follow-up questions this topic invites — and their answers

Q: What is cell-based architecture? A: Running several independent, identical copies ("cells") of the full stack, each serving a partition of users or tenants. A failure or bad deploy affects one cell only, limiting the blast radius; routing maps each tenant to a cell.

Q: How do RTO and RPO influence the design? A: RPO near zero needs synchronous replication (with latency costs) or multi-region consensus databases; RTO near zero needs active-active or hot standby with automatic failover. Each step down in RTO and RPO multiplies cost, so set them per service based on business impact.

Q: Why can autoscaling make an outage worse? A: Scaling application pods increases connections and load on a shared database or downstream service that can't scale, turning a slowdown into a collapse. Cap the replicas, pool connections (PgBouncer), and add load shedding.

Q: What does "expand and contract" mean for database migrations? A: Make schema changes backward compatible in steps: add new columns or tables (expand), deploy code that writes both and reads the new, backfill, switch reads, then remove the old structures (contract). No step breaks the running version.