A System That Lasts: Understandable × Observable × Recoverable

What makes a system last?

A system lasts when 3 conditions hold together: someone who did not build it can explain it, its failures are seen before users report them, and its recovery has been practised. I write this as a formula, a system that lasts = understandable × observable × recoverable, and the multiplication matters, because a zero in any one factor makes the whole result zero.

Most people judge a system by a single measure, which is whether it works today. The measure is easy to apply, and for a long time it appears to be enough. Pages load, reports run, automations fire, and the business concludes that the system is sound. The difficulty is that none of the 3 conditions above is visible on a normal day. The 3 conditions only show themselves when something changes, and by then it is late to build them.

Why does a system that works today still fail later?

A system that works today fails later because the conditions that made it work are quietly temporary. The person who understands it is present, the vendor behaves as expected, the data arrives in the expected format and the business is roughly the size it was when the system was designed. Under those conditions, almost any system looks reliable, so the test of reliability is delayed until the conditions move.

Then something does move. The one person who knows how it fits together leaves, a vendor changes its terms, volume grows tenfold, or an assumption made 2 years ago stops being true. The team finds out that its knowledge lived in one head, that nothing warned anyone as the problem developed, and that recovery depends on steps nobody has ever tried. What does a business lose in the days it takes to rediscover how its own system worked? And how much of what it called stability was in fact one person's memory?

If a system works only while its original conditions remain intact, the dependency has not been removed. The dependency has only been hidden.

What does understandable mean?

A system is understandable when someone who did not build it can explain what it does, why it does it, and what happens when something breaks. The test is not the quality of the documentation but whether a second person can answer those 3 questions without asking the author.

An understandable system has traceable decisions, visible processes and written knowledge. A new developer can read the documentation and find where the data comes from, which parts depend on which, and how a change is deployed. On a HIPAA-regulated platform that I led end to end, I treated production documentation and developer onboarding as part of the delivery and not as a task for later, because a platform that only its author can operate has a hidden single point of failure. Read the case study.

What does observable mean?

A system is observable when its failures are seen by the people responsible before users report them. Observability means that logs are collected in one place, that alerts carry enough context to be acted on, and that someone is named as the person who receives them.

Many teams have monitoring that produces alerts without producing understanding. An alert that says a threshold was crossed, with no indication of what it affects, trains people to ignore alerts. A useful alert says what failed, what depends on it and who owns it. When the first signal of a failure is a customer's complaint, the system is not observable, however many dashboards exist.

What does recoverable mean?

A system is recoverable when the restore has been practised, so the time to bring it back is known and not assumed. A backup that has never been restored is an unverified file, and a real incident is the wrong moment for the first restore.

Practising recovery means restoring from backup into a clean environment, recording how long it takes, and writing down what was missing. Each practice run finds something: a credential held by one person, a service that was never in the backup, an order of steps that nobody had written down. On the same platform I built Dockerised CI/CD and disaster recovery alongside the application, because deployment that can be repeated and a restore that has been tried are the same discipline applied to different moments.

Why multiply the 3 factors instead of adding them?

The 3 factors are multiplied because each one fails differently, and a strength in one does not compensate for a gap in another. Consider what happens when a single factor is missing:

• A system that is observable and recoverable but not understandable raises alerts that nobody can interpret and restores that nobody can verify.
• A system that is understandable and recoverable but not observable has well-documented steps for failures that nobody notices in time.
• A system that is understandable and observable but not recoverable shows the failure clearly, explains it perfectly, and cannot fix it.

In each case the high scores on 2 factors do not help. That is the reason for treating the formula as a product and not as a checklist where a good average is acceptable.

How can you score your own system?

You can score a system with 3 tests, one for each factor, and each test has a yes or no answer. Run them in this order:

1. Ask a colleague who did not build the system to explain what it does, why it does it and what happens when it fails.
2. Find out how the last incident was discovered, and whether the system or a user raised it first.
3. Restore the most recent backup into a clean environment, and record how long it takes and what is missing.

A no on any of the 3 tests means the system is not yet one that lasts, and that factor is the one to fix first, because the other 2 cannot make up for it. A useful first step is usually modest: write down who understands which part, route one alert to a named person with context, or run a single restore. Systems rarely become durable through a large project. Systems become durable through a few repeated checks that make the dependencies visible.