Almost no organization records all three.

The first is the number written into the plan. The second is the number measured during exercises, if exercises happen. The third is the number observed during real incidents.

The distance between them is the only metric that matters. It is also the metric that almost no one calculates.

The Three States of D.R. Capability

Disaster recovery capability exists in three forms simultaneously, and the three forms produce three different numbers.

1. Declared capability: the RPO and RTO values written into the D.R. plan. These are typically inherited from compliance requirements, business expectations, or vendor templates. They are aspirational by construction.

2. Tested capability: the actual recovery time and data loss observed during the most recent end-to-end exercise, if such an exercise has been performed. This is the measurement that most closely approximates real recovery, but only if the exercise conditions are realistic.

3. Observed capability: the actual recovery time and data loss measured during a real incident. This is the only number with no theoretical component. It is also the number that the organization discovers it has, rather than the number it had planned for.

The three numbers are rarely the same. The distance between them is the Validation Gap, and it is the most actionable measurement in disaster recovery.

A plan that has not been tested has only one number. A plan that has been tested has two. A plan that has survived an incident has three. Most organizations operate with one and assume it represents the others.

Calculating the Validation Gap

The Validation Gap is calculable, not estimable. Three inputs produce the number:

• Base gap: the difference, in hours, between Tested RTO and Declared RTO. A plan declaring 4 hours that tested at 9 hours has a base gap of 5 hours.

• Decay coefficient: a multiplier reflecting how stale the test is. Months since the last exercise multiplied by the platform’s change velocity. A stable platform might use 0.05 per month. A platform under active migration might use 0.15 per month. Twelve months on a stable platform produces a coefficient of 0.6. Twelve months on a fast-changing platform produces 1.8.

• Adjusted gap: base gap multiplied by (1 + decay coefficient). The same 5-hour base gap, on a stable platform tested 12 months ago, becomes 8 hours. On a fast-changing platform, it becomes 14 hours.

A D.R. plan with no recent test has a Validation Gap equal to the entire declared RTO, regardless of how confident the plan reads. The numbers are aspirational, not validated.

The Validation Gap is paid in currency. The product of the adjusted gap and the platform’s hourly business value is the unpriced exposure the organization is carrying. For a platform supporting US$ 200,000 per hour in transactions, an adjusted gap of 8 hours represents US$ 1.6 million in exposure that has been declared as covered but is not measurably so.

According to the Cockroach Labs State of Resilience 2025 report, only 20 percent of executives feel their organizations are fully prepared to prevent or respond to outages, and organizations average 86 hours of unplanned outage per year. Most of those hours are paid against a Validation Gap that was never calculated.

The Validation Gap is paid in full during the first incident. Until then, it accumulates without being charged.

Executive implication: Ask the platform team for three numbers: the declared RTO, the most recently tested RTO, and the date of that test. The adjusted Validation Gap, multiplied by the platform’s hourly business value, is the line item the organization is carrying without recording it.

Why the Number Is Not Recorded

The Validation Gap is rarely calculated, and the reason is structural rather than technical.

D.R. exercises, when they happen, are typically scoped narrowly. A cluster is recovered. A database is restored. A failover is demonstrated. None of these individually measure end-to-end recovery, because the dependencies that determine real recovery (identity infrastructure, certificate authorities, container registries, DNS, network paths) live outside the cluster boundary. The structural failure modes of these layers are documented in The Hidden Reliability Risks in Multi-Cluster Kubernetes and The SPOFs You Did Not Design. What matters here is that an exercise that does not include them measures something other than D.R. capability (FN-0003).

When exercises do happen, results are usually narrated rather than measured. “The exercise was successful” is not a number. The actual elapsed time, the deviations from the runbook, the dependencies that failed to activate, and the coordination overhead consumed before recovery began are all measurable. They are also rarely written down.

The optimism cascade (FN-0024) compounds this. The platform team reports the cluster is ready. The security team reports identity is ready. The network team reports DNS is ready. Each report is true within its scope. None of them validate the chain. The organization is preparing for an incident in pieces while incidents arrive whole.

The team that wrote the plan is rarely the team executing it eighteen months later. Knowledge transfer artifacts describe intent, not the operational details required to act on it (FN-0017). A runbook that worked when its author was on call may fail under any other rotation.

Tested recovery is recovery in ideal conditions. Real recovery is recovery in degraded ones. The Validation Gap is the distance between them.

Executive implication: D.R. governance requires authority across team boundaries. Without a designated owner with cross-functional mandate, every exercise will reflect the readiness of the strongest individual team and ignore the dependencies between teams.

From Internal Metric to Regulatory Exposure

Until recently, the Validation Gap was a useful internal measurement that almost no organization computed. Starting in 2025, it has begun to acquire regulatory weight.

The Digital Operational Resilience Act (DORA) entered into force across the European Union on January 17, 2025. Its requirements are explicit:

• Articles 24-25 require digital operational resilience testing, including scenario-based exercises with documented outcomes that demonstrate the capability of recovery, not just its plan.
• Articles 26-27 require threat-led penetration testing every three years for significant entities, conducted by accredited testers under conditions that approximate realistic adversary behavior.
• Articles 17-23 require ICT-related incident reporting, including a four-hour initial notification window for major incidents.
• Articles 28-30 require ICT third-party risk management, including contractual evidence that critical providers (cloud platforms among them) meet equivalent resilience standards.

For Kubernetes environments operating regulated workloads, these requirements translate the Validation Gap from an internal metric into a finding category. A plan that exists in a wiki article without measured exercise results does not satisfy DORA. A test that recovers a single cluster in isolation does not satisfy a scenario-based requirement. Incident detection and reporting must be instrumented to meet the four-hour notification window, which constrains the design of observability and incident response tooling.

DORA is the most explicit example. It is not the only one.

The NIS2 Directive entered into force in October 2024 with a broader scope than DORA, covering essential and important entities across energy, transport, banking, healthcare, digital infrastructure, and public administration. It mandates risk management measures explicitly including business continuity and incident handling. In the United States, the SEC’s cybersecurity disclosure rule (Item 1.05 of Form 8-K, effective late 2023) requires public companies to disclose material cybersecurity incidents within four business days. Banking sector guidance from the OCC, FRB, and FDIC continues to tighten heightened standards for operational resilience.

The pattern across all of these is structural:

Regulators no longer ask whether a plan exists. They ask whether the plan has been tested, by whom, under what conditions, and with what measured outcome.

The Validation Gap is the metric that answers that question. An organization that has not calculated it is now exposed not only to operational risk, but to regulatory finding risk, and increasingly to public disclosure obligations.

Executive implication: If the organization operates under DORA, NIS2, SEC cybersecurity disclosure, or any sectoral resilience framework, the Validation Gap has stopped being optional. The audit no longer ends when the plan is reviewed. It ends when the test results are reviewed.

How to Start Recording

The transition from declared D.R. to validated D.R. is structural, not procedural. It changes what an exercise is, who runs it, and how its results are recorded.

Exercises must be timed and end to end. A test that recovers a single cluster in isolation does not validate enterprise D.R. The exercise must include identity restoration, certificate validation, image availability, network reachability, and application-level recovery. The clock starts when the simulated incident is declared and stops when business operations are confirmed.

The team executing must not be the team that wrote the plan. The on-call rotation, not the original author, should drive the exercise. This surfaces the gap between documented intent and operationally usable instructions (FN-0015).

Conditions should be realistic, not ideal. Recovery exercises in pristine environments validate the procedure under conditions that will not exist during a real incident. Introducing controlled degradation (removed access to a documented system, simulated unavailability of a dependency, partial information about the failure mode) reveals failure modes that pristine tests hide (FN-0007).

Results must be measured, not narrated. The actual RTO, the actual RPO, the failures encountered, the recovery deviations from the runbook, and the time spent in coordination are the measurements that close the Validation Gap. “The exercise was successful” is not a measurement.

The Validation Gap must be recorded as a number, alongside the declared RTO. When leadership reviews the D.R. plan, both numbers should be visible. The declared value alone is no longer sufficient evidence of capability.

For an executive-focused treatment of these patterns specifically in Red Hat OpenShift environments, see Why Most OpenShift D.R. Strategies Fail at Executive Level.

Architectural Continuity

Disaster recovery is not the document that an auditor reviews. It is the number that the organization is willing to record alongside the declared one.

Declared capability is a hypothesis. Tested capability is a measurement. The Validation Gap is the distance the organization is carrying without recording it.

The tardigrade survives the vacuum of space, radiation a thousand times the human limit, temperatures from near absolute zero to 150 degrees Celsius. None of those capabilities are inferred. Each was measured under controlled conditions before the organism was claimed to possess them. Resilience that survives measurement is the only resilience that can be relied upon. Resilience that has only been described will be measured during the first incident, at the moment when the cost of measurement is highest and the time to act on it is shortest.

References

1. Cockroach Labs, “The State of Resilience 2025: Confronting Outages, Downtime, and Organizational Readiness”, 2024.

2. European Union, Regulation (EU) 2022/2554 (Digital Operational Resilience Act), in force January 17, 2025.

3. European Union, Directive (EU) 2022/2555 (NIS2 Directive), in force October 2024.

4. U.S. Securities and Exchange Commission, Cybersecurity Risk Management, Strategy, Governance, and Incident Disclosure, final rule, July 2023.

Modern platforms often contain internal infrastructure that is not visible in the primary operational model used by administrators.

These resources include internal networks, control-plane communication paths, service networks, operator-managed components, and reconciliation controllers.

They exist to support platform behavior rather than application workloads, and are frequently created automatically during cluster deployment.

Because they are not part of the infrastructure model operators typically reason about, they remain largely invisible until they interact with external resources or cause unexpected conflicts.

Implication:

When failures involve these internal mechanisms, troubleshooting becomes difficult because the affected infrastructure exists outside the mental model used to design and operate the environment.

Platform architectures increasingly depend on infrastructure that operators do not directly see.

Part of the Field Notes series documenting operational patterns observed in real-world platform architectures.

Every abstraction layer hides complexity from the user while introducing additional operational mechanics behind the scenes.

Controllers reconcile desired state.
Operators manage lifecycle logic.
Networking overlays create new routing paths.

These mechanisms remain mostly invisible during normal operation.

They become visible only when something fails.

Implication:

The operational overhead created by abstraction layers can be understood as an abstraction tax: a cost paid by the platform team in exchange for simplified interfaces offered to users.

This dynamic often contributes to Operational Gravity (FN-0014), where complexity gradually accumulates around platform teams.

Part of the Field Notes series documenting operational patterns observed in real-world platform architectures.

Modern systems are distributed.
But fragility didn’t disappear.
It just became harder to see.

They run across clusters, regions, providers . They are observable, containerized, orchestrated .

They look resilient.

And yet, they still fail in surprisingly simple ways.

Not because distribution failed.
But because our understanding didn’t evolve with it.

The Illusion of Resilience

Cloud-native architectures are often assumed to be resilient by default.

They are not.

What we actually built are systems that:

• scale well
• deploy fast
• look observable

But resilience is something else entirely.
And we rarely design for it.

A system is not resilient because it is distributed. It is resilient because it can survive the loss of what it depends on (FN-0013).

And most systems today cannot.

The Happy Path Trap

Most systems are designed around success.

Requests succeed.
Dependencies respond.
Flows complete.

Failure exists.
But as an afterthought.

• generic retries
• vague error handling
• logs that assume context

If your system only knows how to succeed, failure becomes undefined behavior (FN-0015).

This is where fragility begins.

Not in infrastructure.
In assumptions.

The Illusion of Testing

Modern delivery pipelines create confidence.

But often, it is misplaced.

We test components in isolation.
We mock dependencies.
We simulate behavior, not reality.
We validate expected outputs.

And then we assume the system will behave.

Mocks don’t fail like real systems do (FN-0004).

Integration is where reality lives.
And it is often the least tested part.

Passing tests prove consistency, not correctness under stress.

Hidden SPOFs in Plain Sight

Single points of failure did not disappear (FN-0002).

They became harder to see.

DNS

The most fundamental layer of the internet.

Still misconfigured.
Still under-tested.
Still capable of bringing entire systems down.

The most critical systems are often the least questioned.

Observability

Dashboards are everywhere.

But visibility is not understanding.

When the observability stack fails (or lacks context), diagnosis becomes guesswork.

A system is observable until it fails outside the path it was designed to show.

External Dependencies

Modern systems rely on external services:

• identity providers
• CI/CD platforms
• third-party APIs

Failures in these integrations are not just technical.

They are organizational.

Failures in integrated systems don’t just break flows, they break ownership.

No one knows who should fix the problem.
So no one does it fast enough.

Cognitive Fragility

As systems evolved, so did abstraction.

Platforms simplified complexity.
Interfaces reduced cognitive load .

This is necessary.

But it also distances decision-making from reality.

And it comes with a cost.

Abstractions reduce cognitive load, but they also hide the system (FN-0006).

Over time, this creates cognitive blind spots (FN-0010):

• dependencies no one maps
• behaviors no one understands
• failure modes no one anticipates

You cannot reason about what you cannot see.

And when the system fails:

The system breaks, and the organization struggles to respond.

Not Another Disaster Recovery Problem

This is not primarily a recovery problem.

It is an understanding problem.
And understanding does not scale by default.

Disaster recovery strategies often assume we know what failed.

In reality, we often don’t.

You can’t recover from failures you don’t understand.

For a deeper look into recovery strategies, see our previous notes on disaster recovery.

Closing

We built distributed systems.
But not distributed understanding.

And so, fragility remains.

Not where we used to look.
But exactly where we stopped looking.

Fragility Map

Platform abstractions reduce cognitive load for users.

A developer deploying an application rarely needs to understand how scheduling, networking, storage provisioning, or cluster lifecycle actually work.

The interface becomes simple: deploy, expose, scale.

However, the operational side of the platform moves in the opposite direction.

Each abstraction layer introduces additional controllers, reconciliation loops, networking paths, and state dependencies that must be understood when something fails.

Implication:

Abstractions successfully simplify usage, but they rarely simplify operation.

Instead, operational complexity becomes concentrated in the platform team responsible for maintaining the abstraction itself (FN-0010).

Part of the Field Notes series documenting operational patterns observed in real-world platform architectures.