Resilient Microservices: Timeouts, Retries and Circuit Breakers Done Right

In a microservice system, failures aren't exceptional — they're constant. A dependency gets slow, a pod restarts, the network drops packets. The question isn't whether a call will fail, but whether one failing call can cascade and take everything down. Four patterns prevent that, and they work best together.

Why cascades happen

Service A calls B. B slows down, from 50 ms to 10 seconds. Every request to A now holds a thread (and maybe a database connection) for 10 seconds. A's thread pool fills up, A stops responding, and the services calling A start failing too. One slow service took down three.

1. Timeouts: never wait forever

Every remote call needs a timeout. Many HTTP clients' defaults are infinite, or very long.

@Bean
RestClient paymentClient(RestClient.Builder builder) {
    var factory = new SimpleClientHttpRequestFactory();
    factory.setConnectTimeout(Duration.ofSeconds(1));
    factory.setReadTimeout(Duration.ofSeconds(2));
    return builder.baseUrl("http://payment-service").requestFactory(factory).build();
}

• Set timeouts from the dependency's real latency (for example, a little above its p99), not arbitrary round numbers.
• Budget them: if your API must answer in 1 second, its downstream calls can't have 5-second timeouts. Propagate deadlines where possible.

2. Retries: only for transient failures, with backoff and jitter

Retry transient errors (timeouts, 503s, connection resets) — never business errors (400s, validation failures).

• Use exponential backoff with jitter (for example 200 ms, 400 ms, 800 ms, each ±50% random), so thousands of clients don't retry in sync (a thundering herd).
• Cap the attempts (2–3 is usually enough).
• Only retry idempotent operations — or make them idempotent with an idempotency key. Retrying a non-idempotent payment can charge the customer twice.
• Don't retry in every layer: 3 retries × 3 layers = 27 calls.

resilience4j.retry.instances.payment:
  max-attempts: 3
  wait-duration: 200ms
  enable-exponential-backoff: true
  exponential-backoff-multiplier: 2
  enable-randomized-wait: true
  randomized-wait-factor: 0.5
  retry-exceptions:
    - java.io.IOException
    - org.springframework.web.client.ResourceAccessException

3. Circuit breaker: stop calling what's broken

A circuit breaker watches the failure rate of calls to a dependency:

• Closed: calls flow normally, and failures are counted over a sliding window.
• Open: once failures (or slow calls) exceed a threshold, calls fail immediately, without touching the dependency, for a wait period. This protects your threads, and gives the dependency room to recover.
• Half-open: after the wait, a few trial calls go through. If they succeed, the breaker closes; otherwise it opens again.

resilience4j.circuitbreaker.instances.payment:
  sliding-window-type: COUNT_BASED
  sliding-window-size: 20
  failure-rate-threshold: 50          # open when 50% of the last 20 calls failed
  slow-call-duration-threshold: 2s
  slow-call-rate-threshold: 60
  wait-duration-in-open-state: 10s
  permitted-number-of-calls-in-half-open-state: 3

@CircuitBreaker(name = "payment", fallbackMethod = "paymentFallback")
@Retry(name = "payment")
public PaymentResult charge(PaymentRequest req) {
    return paymentClient.post().uri("/charges").body(req).retrieve().body(PaymentResult.class);
}

private PaymentResult paymentFallback(PaymentRequest req, Throwable t) {
    return PaymentResult.pending(req.orderId());   // queue it for later, don't fail the checkout
}

Order matters. In Resilience4j's Spring annotations, the default aspect order applies Retry outside the CircuitBreaker, so every retry attempt is counted by the breaker, and an open breaker stops retries quickly.

4. Bulkheads: isolate the damage

Named after ship compartments: give each dependency its own limited pool (threads or concurrent calls), so one slow dependency can't consume all resources.

resilience4j.bulkhead.instances.recommendations:
  max-concurrent-calls: 20
  max-wait-duration: 0

If the recommendation service hangs, at most 20 requests wait on it, and checkout keeps working.

Fallbacks: decide what "degraded" looks like

A fallback should be a deliberate product decision:

• serve cached or default data (recommendations → "popular items");
• queue the work for later (payment → "pending", confirmed asynchronously);
• hide the non-essential feature (reviews widget);
• fail fast, with a clear error, when there's no safe alternative.

Observe it

Resilience4j publishes metrics through Micrometer: circuit breaker state, failure rates, retry counts and bulkhead saturation. Alert on breaker state changes and rising retry rates; they're early warnings of a sick dependency.

Follow-up questions this topic invites — and their answers

Q: Circuit breaker or retry, which comes first? A: Put Retry outside the CircuitBreaker (Resilience4j's default aspect order), so every attempt is counted by the breaker and an open breaker short-circuits the retries quickly.

Q: Is Hystrix still used? A: No. Netflix Hystrix is in maintenance mode, and Spring Cloud removed its support. Resilience4j (through Spring Cloud Circuit Breaker, or its own Spring Boot starter) is the standard choice.

Q: How do timeouts and circuit breakers relate? A: Timeouts turn a hanging call into a failure; the circuit breaker counts those failures (and slow calls) and stops calling. Without timeouts, a breaker may never see failures, just stuck threads.

Q: What's the difference between a bulkhead and a rate limiter? A: A bulkhead limits concurrent calls, protecting your resources from a slow dependency. A rate limiter limits calls per time window, protecting the dependency (or you) from too much traffic.

Dive deeper in the circuit breaker lesson, the retry and backoff lesson and our microservices interview questions.