Most distributed systems do not fail because of one dramatic outage. They fail because a small assumption held for two years and then quietly stopped being true. A network call that usually completes in 40ms starts taking three seconds. A retry policy that was supposed to be a safety net becomes the thing that finishes off a struggling dependency.

Resilience is less about preventing failure and more about making failure boring.

Start with the failure modes

Before choosing a pattern, write down what actually happens when each dependency misbehaves. There are only a few shapes worth naming:

  • Slow: the call succeeds, but it takes long enough to exhaust connection pools.
  • Down: the call fails fast and loudly.
  • Flaky: it works often enough that nobody builds a fallback.
  • Poisoned: the same bad input keeps producing the same expensive error.

Each one needs a different response. Timeouts help with slow. Circuit breakers help with down and flaky. Idempotency keys and dead-letter queues help with poisoned. Applying retries to all four is how you turn a small incident into a regional one.

Timeouts are a design decision

A timeout is not a number you copy from a template. It is a statement about how long your user is willing to wait and how much work you are willing to abandon. Budget it explicitly: if the edge has 800ms, the orchestration layer gets 500ms, and the leaf service gets 200ms. Every layer that invents its own timeout breaks the budget.

Make state recovery explicit

Event-driven systems tend to hide their recovery story. Ask three questions:

  1. If this consumer processes a message twice, what breaks?
  2. If it crashes mid-transaction, who repairs the state?
  3. If the queue backs up for an hour, what does the user see?

If the answer to any of those is “it probably won’t happen,” the design is not finished. Idempotent handlers, an outbox for writes, and a replay-safe event log cover most of the gap without adding much ceremony.

Observe the system you actually have

Dashboards are not observability. A useful signal answers a question someone asks at 2am: which dependency is slow, which tenant is affected, and is the system recovering? Track saturation, error budgets, and queue depth per service — not just CPU.

The goal is not zero incidents. It is incidents that stay small, recover without heroics, and teach you something after the fact.