<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>Validation-Gap on Elastocera</title>
    <link>https://elastocera.com/tags/validation-gap/</link>
    <description>Recent content in Validation-Gap on Elastocera</description>
    <image>
      <title>Elastocera</title>
      <url>https://elastocera.com/images/forest-og.jpg</url>
      <link>https://elastocera.com/images/forest-og.jpg</link>
    </image>
    <generator>Hugo -- 0.157.0</generator>
    <language>en</language>
    <lastBuildDate>Fri, 22 May 2026 10:00:00 -0300</lastBuildDate>
    <atom:link href="https://elastocera.com/tags/validation-gap/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>The DR Number Almost No One Records</title>
      <link>https://elastocera.com/posts/kubernetes-dr-strategies-fail-real-enterprises/</link>
      <pubDate>Fri, 22 May 2026 10:00:00 -0300</pubDate>
      <guid>https://elastocera.com/posts/kubernetes-dr-strategies-fail-real-enterprises/</guid>
      <description>Disaster recovery has three measurable states. Most organizations record only the first. The Validation Gap is the calculable distance between declared and tested capability, and starting in 2025, it is becoming a regulatory exposure.</description>
        <enclosure url="https://elastocera.com/images/tardigrade-og.jpg" length="0" type="image/jpeg"/>
      <content:encoded><![CDATA[<p>Disaster recovery has three numbers.</p>
<p>Almost no organization records all three.</p>
<p>The first is the number written into the plan. The second is the number measured during exercises, if exercises happen. The third is the number observed during real incidents.</p>
<p>The distance between them is the only metric that matters. It is also the metric that almost no one calculates.</p>
<hr>
<h3 id="the-three-states-of-dr-capability">The Three States of D.R. Capability</h3>
<p><span class="tooltip-term" data-tooltip="Disaster Recovery (D.R.): the set of policies, tools, and procedures designed to recover technology infrastructure and systems after a disruptive event. In Kubernetes environments, D.R. encompasses cluster recovery, data replication, identity and certificate restoration, and the network infrastructure required to reestablish operations."> Disaster recovery </span> capability exists in three forms simultaneously, and the three forms produce three different numbers.</p>
<ol>
<li>
<p><strong>Declared capability</strong>: the <span class="tooltip-term" data-tooltip="Recovery Point Objective (RPO): the maximum acceptable amount of data loss measured in time. An RPO of 1 hour means the organization accepts losing up to 1 hour of data. Recovery Time Objective (RTO): the maximum acceptable duration of downtime before business impact becomes critical."> RPO and RTO </span> values written into the D.R. plan. These are typically inherited from compliance requirements, business expectations, or vendor templates. They are aspirational by construction.</p>
</li>
<li>
<p><strong>Tested capability</strong>: the actual recovery time and data loss observed during the most recent end-to-end exercise, if such an exercise has been performed. This is the measurement that most closely approximates real recovery, but only if the exercise conditions are realistic.</p>
</li>
<li>
<p><strong>Observed capability</strong>: the actual recovery time and data loss measured during a real incident. This is the only number with no theoretical component. It is also the number that the organization discovers it has, rather than the number it had planned for.</p>
</li>
</ol>
<p>The three numbers are rarely the same. The distance between them is the <strong>Validation Gap</strong>, and it is the most actionable measurement in disaster recovery.</p>
<blockquote>
<p>A plan that has not been tested has only one number. A plan that has been tested has two. A plan that has survived an incident has three. Most organizations operate with one and assume it represents the others.</p>
</blockquote>
<hr>
<h3 id="calculating-the-validation-gap">Calculating the Validation Gap</h3>
<p>The Validation Gap is calculable, not estimable. Three inputs produce the number:</p>
<ul>
<li>
<p><strong>Base gap</strong>: the difference, in hours, between Tested RTO and Declared RTO. A plan declaring 4 hours that tested at 9 hours has a base gap of 5 hours.</p>
</li>
<li>
<p><strong>Decay coefficient</strong>: a multiplier reflecting how stale the test is. Months since the last exercise multiplied by the platform&rsquo;s change velocity. A stable platform might use 0.05 per month. A platform under active migration might use 0.15 per month. Twelve months on a stable platform produces a coefficient of 0.6. Twelve months on a fast-changing platform produces 1.8.</p>
</li>
<li>
<p><strong>Adjusted gap</strong>: base gap multiplied by (1 + decay coefficient). The same 5-hour base gap, on a stable platform tested 12 months ago, becomes 8 hours. On a fast-changing platform, it becomes 14 hours.</p>
</li>
</ul>
<p>A D.R. plan with no recent test has a Validation Gap equal to the entire declared RTO, regardless of how confident the plan reads. The numbers are aspirational, not validated.</p>
<p>The Validation Gap is paid in currency. The product of the adjusted gap and the platform&rsquo;s hourly business value is the <strong>unpriced exposure</strong> the organization is carrying. For a platform supporting US$ 200,000 per hour in transactions, an adjusted gap of 8 hours represents US$ 1.6 million in exposure that has been declared as covered but is not measurably so.</p>
<p>According to the Cockroach Labs State of Resilience 2025 report, only 20 percent of executives feel their organizations are fully prepared to prevent or respond to outages, and organizations average 86 hours of unplanned outage per year. Most of those hours are paid against a Validation Gap that was never calculated.</p>
<blockquote>
<p>The Validation Gap is paid in full during the first incident. Until then, it accumulates without being charged.</p>
</blockquote>
<p><strong>Executive implication:</strong> Ask the platform team for three numbers: the declared RTO, the most recently tested RTO, and the date of that test. The adjusted Validation Gap, multiplied by the platform&rsquo;s hourly business value, is the line item the organization is carrying without recording it.</p>
<hr>
<h3 id="why-the-number-is-not-recorded">Why the Number Is Not Recorded</h3>
<p>The Validation Gap is rarely calculated, and the reason is structural rather than technical.</p>
<p>D.R. exercises, when they happen, are typically scoped narrowly. A cluster is recovered. A database is restored. A failover is demonstrated. None of these individually measure end-to-end recovery, because the dependencies that determine real recovery (identity infrastructure, certificate authorities, container registries, DNS, network paths) live outside the cluster boundary. The structural failure modes of these layers are documented in <a href="/posts/hidden-reliability-risks-multi-cluster-kubernetes/">The Hidden Reliability Risks in Multi-Cluster Kubernetes</a> and <a href="/posts/spofs-modern-cloud-native-architectures/">The SPOFs You Did Not Design</a>. What matters here is that an exercise that does not include them measures something other than D.R. capability (<a href="https://elastocera.com/field-notes/operational-knowledge-vs-architectural-knowledge/" class="fn-ref" title="Operational Knowledge vs Architectural Knowledge">FN-0003</a>).</p>
<p>When exercises do happen, results are usually narrated rather than measured. &ldquo;The exercise was successful&rdquo; is not a number. The actual elapsed time, the deviations from the runbook, the dependencies that failed to activate, and the coordination overhead consumed before recovery began are all measurable. They are also rarely written down.</p>
<p>The optimism cascade (<a href="https://elastocera.com/field-notes/assumed-readiness/" class="fn-ref" title="Assumed Readiness">FN-0024</a>) compounds this. The platform team reports the cluster is ready. The security team reports identity is ready. The network team reports DNS is ready. Each report is true within its scope. None of them validate the chain. The organization is preparing for an incident in pieces while incidents arrive whole.</p>
<p>The team that wrote the plan is rarely the team executing it eighteen months later. Knowledge transfer artifacts describe intent, not the operational details required to act on it (<a href="https://elastocera.com/field-notes/available-knowledge-not-applied/" class="fn-ref" title="Available Knowledge Is Not Applied Knowledge">FN-0017</a>). A runbook that worked when its author was on call may fail under any other rotation.</p>
<blockquote>
<p>Tested recovery is recovery in ideal conditions. Real recovery is recovery in degraded ones. The Validation Gap is the distance between them.</p>
</blockquote>
<p><strong>Executive implication:</strong> D.R. governance requires authority across team boundaries. Without a designated owner with cross-functional mandate, every exercise will reflect the readiness of the strongest individual team and ignore the dependencies between teams.</p>
<hr>
<h3 id="from-internal-metric-to-regulatory-exposure">From Internal Metric to Regulatory Exposure</h3>
<p>Until recently, the Validation Gap was a useful internal measurement that almost no organization computed. Starting in 2025, it has begun to acquire regulatory weight.</p>
<p>The <span class="tooltip-term" data-tooltip="DORA (Digital Operational Resilience Act): EU regulation 2022/2554, in force across the European Union from January 17, 2025. Applies to financial entities and their critical ICT third-party service providers. Requires evidence of tested recovery capability, structured incident reporting, and threat-led penetration testing for significant entities."> Digital Operational Resilience Act </span> (DORA) entered into force across the European Union on January 17, 2025. Its requirements are explicit:</p>
<ul>
<li><strong>Articles 24-25</strong> require digital operational resilience testing, including scenario-based exercises with documented outcomes that demonstrate the capability of recovery, not just its plan.</li>
<li><strong>Articles 26-27</strong> require <span class="tooltip-term" data-tooltip="Threat-Led Penetration Testing (TLPT): adversary-simulation exercises required by DORA every three years for significant financial entities, conducted by accredited testers using current threat intelligence. The objective is to validate operational resilience under realistic attack conditions, not to confirm controls in isolation."> threat-led penetration testing </span> every three years for significant entities, conducted by accredited testers under conditions that approximate realistic adversary behavior.</li>
<li><strong>Articles 17-23</strong> require ICT-related incident reporting, including a four-hour initial notification window for major incidents.</li>
<li><strong>Articles 28-30</strong> require ICT third-party risk management, including contractual evidence that critical providers (cloud platforms among them) meet equivalent resilience standards.</li>
</ul>
<p>For Kubernetes environments operating regulated workloads, these requirements translate the Validation Gap from an internal metric into a finding category. A plan that exists in a wiki article without measured exercise results does not satisfy DORA. A test that recovers a single cluster in isolation does not satisfy a scenario-based requirement. Incident detection and reporting must be instrumented to meet the four-hour notification window, which constrains the design of observability and incident response tooling.</p>
<p>DORA is the most explicit example. It is not the only one.</p>
<p>The <span class="tooltip-term" data-tooltip="NIS2 Directive (EU 2022/2555): in force across the European Union from October 2024. Expands the scope of cybersecurity and operational resilience requirements to essential and important entities across multiple sectors. Mandates risk management measures including business continuity, incident handling, and supply chain security."> NIS2 Directive </span> entered into force in October 2024 with a broader scope than DORA, covering essential and important entities across energy, transport, banking, healthcare, digital infrastructure, and public administration. It mandates risk management measures explicitly including business continuity and incident handling. In the United States, the SEC&rsquo;s cybersecurity disclosure rule (Item 1.05 of Form 8-K, effective late 2023) requires public companies to disclose material cybersecurity incidents within four business days. Banking sector guidance from the OCC, FRB, and FDIC continues to tighten heightened standards for operational resilience.</p>
<p>The pattern across all of these is structural:</p>
<blockquote>
<p>Regulators no longer ask whether a plan exists. They ask whether the plan has been tested, by whom, under what conditions, and with what measured outcome.</p>
</blockquote>
<p>The Validation Gap is the metric that answers that question. An organization that has not calculated it is now exposed not only to operational risk, but to regulatory finding risk, and increasingly to public disclosure obligations.</p>
<p><strong>Executive implication:</strong> If the organization operates under DORA, NIS2, SEC cybersecurity disclosure, or any sectoral resilience framework, the Validation Gap has stopped being optional. The audit no longer ends when the plan is reviewed. It ends when the test results are reviewed.</p>
<hr>
<h3 id="how-to-start-recording">How to Start Recording</h3>
<p>The transition from declared D.R. to validated D.R. is structural, not procedural. It changes what an exercise is, who runs it, and how its results are recorded.</p>
<p><strong>Exercises must be timed and end to end.</strong> A test that recovers a single cluster in isolation does not validate enterprise D.R. The exercise must include identity restoration, certificate validation, image availability, network reachability, and application-level recovery. The clock starts when the simulated incident is declared and stops when business operations are confirmed.</p>
<p><strong>The team executing must not be the team that wrote the plan.</strong> The on-call rotation, not the original author, should drive the exercise. This surfaces the gap between documented intent and operationally usable instructions (<a href="https://elastocera.com/field-notes/the-first-incident-test/" class="fn-ref" title="The First Incident Test">FN-0015</a>).</p>
<p><strong>Conditions should be realistic, not ideal.</strong> Recovery exercises in pristine environments validate the procedure under conditions that will not exist during a real incident. Introducing controlled degradation (removed access to a documented system, simulated unavailability of a dependency, partial information about the failure mode) reveals failure modes that pristine tests hide (<a href="https://elastocera.com/field-notes/governance-drift/" class="fn-ref" title="Governance Drift">FN-0007</a>).</p>
<p><strong>Results must be measured, not narrated.</strong> The actual RTO, the actual RPO, the failures encountered, the recovery deviations from the runbook, and the time spent in coordination are the measurements that close the Validation Gap. &ldquo;The exercise was successful&rdquo; is not a measurement.</p>
<p><strong>The Validation Gap must be recorded as a number, alongside the declared RTO.</strong> When leadership reviews the D.R. plan, both numbers should be visible. The declared value alone is no longer sufficient evidence of capability.</p>
<p><em>For an executive-focused treatment of these patterns specifically in Red Hat OpenShift environments, see <a href="/posts/openshift-dr-strategies-fail-executive-level/">Why Most OpenShift D.R. Strategies Fail at Executive Level</a>.</em></p>
<hr>
<h3 id="architectural-continuity">Architectural Continuity</h3>
<p>Disaster recovery is not the document that an auditor reviews. It is the number that the organization is willing to record alongside the declared one.</p>
<blockquote>
<p>Declared capability is a hypothesis.
Tested capability is a measurement.
The Validation Gap is the distance the organization is carrying without recording it.</p>
</blockquote>
<p>The tardigrade survives the vacuum of space, radiation a thousand times the human limit, temperatures from near absolute zero to 150 degrees Celsius. None of those capabilities are inferred. Each was measured under controlled conditions before the organism was claimed to possess them. Resilience that survives measurement is the only resilience that can be relied upon. Resilience that has only been described will be measured during the first incident, at the moment when the cost of measurement is highest and the time to act on it is shortest.</p>
<hr>
<h3 id="references">References</h3>
<ol>
<li>
<p>Cockroach Labs, &ldquo;<a href="https://www.cockroachlabs.com/blog/the-state-of-resilience-2025-reveals-the-true-cost-of-downtime/">The State of Resilience 2025: Confronting Outages, Downtime, and Organizational Readiness</a>&rdquo;, 2024.</p>
</li>
<li>
<p>European Union, <a href="https://eur-lex.europa.eu/eli/reg/2022/2554/oj">Regulation (EU) 2022/2554 (Digital Operational Resilience Act)</a>, in force January 17, 2025.</p>
</li>
<li>
<p>European Union, <a href="https://eur-lex.europa.eu/eli/dir/2022/2555/oj">Directive (EU) 2022/2555 (NIS2 Directive)</a>, in force October 2024.</p>
</li>
<li>
<p>U.S. Securities and Exchange Commission, <a href="https://www.sec.gov/rules/final/2023/33-11216.pdf">Cybersecurity Risk Management, Strategy, Governance, and Incident Disclosure</a>, final rule, July 2023.</p>
</li>
</ol>
]]></content:encoded>
    </item>
  </channel>
</rss>
