Governance in multi-cluster Kubernetes is treated as a set of policies. Policies are not governance.
The control system that propagates, enforces, and reports them is. Policies that are defined but not propagated, propagated but not enforced, or enforced but not reported back are documentation. The system has many ways to fail silently, and each silence accumulates as risk that becomes visible only during audits or incidents.
Where Governance Fails Quietly
Organizations operating multi-cluster Kubernetes fleets face a structural risk that is rarely discussed in architectural reviews: governance gaps that remain invisible until an audit fails or an incident escalates.
The cost is measurable. Undetected configuration drift increases incident blast radius. Inconsistent RBAC baselines extend audit preparation from days to weeks, and let a single compromised credential reach every cluster sharing that baseline instead of one. Clusters onboarded without active policy enforcement create compliance blind spots that accumulate silently.
These are not tooling problems. They are symptoms of treating governance as configuration rather than as an architectural control system.
The Pattern of Silent Failure
In multi-cluster environments, governance failures rarely originate from missing policies.
They emerge from systemic misalignment across clusters:
- Configuration drift between environments
- Inconsistent RBAC baselines
- Selective policy enforcement
- Imported clusters without active governance agents
- Labeling schemes that do not scale
The recurring pattern is this:
Organizations believe they have centralized governance because policies exist on the hub.
In reality, enforcement is uneven, propagation is misunderstood, and compliance status is assumed rather than verified.
This creates silent governance gaps that only surface during audits or incidents.
For a production-level examination of how these gaps manifest as cascading deletions, infrastructure failures, and silent packet loss in multi-cluster environments, see The Hidden Reliability Risks in Multi-Cluster Kubernetes.
The Control System Lens
Governance in RHACM should be treated as a distributed control system, not as a configuration feature.
The system has five structural layers:
- Policy Definition: what must be enforced
- Targeting Logic (Placement): where enforcement applies
- Propagation Mechanism: how policies reach managed clusters
- Enforcement Agents: what evaluates compliance locally
- Feedback (Compliance State): what reports status back to the hub
Each layer is necessary. None is sufficient on its own.
Most operational failures occur at the boundaries between these layers:
- Policy defined, but Placement incorrect
- Placement correct, but governance addons not installed
- Enforcement active, but no alerting loop
- Compliance visible, but not operationalized
Governance therefore is not a YAML problem.
It is a Propagation Integrity problem: whether policies actually reach, get enforced by, and report back from every cluster they target.
Governing Principles
Principle 1: Governance Must Be Hub-Centric
Policy definitions belong to the hub cluster. No ad-hoc, cluster-level policy creation.
Cluster-by-cluster RBAC adjustments introduce entropy. Propagation eliminates variance.
Enforcement should be deterministic and uniform across the fleet.
This does not mean every cluster receives identical configuration. RHACM supports controlled customization through hub-side policy templates that reference managed cluster attributes via template functions. The distinction is architectural: variability is declared centrally and resolved at propagation time, not managed independently per cluster.
Hub centralization is the correct governance posture and the fleet’s largest single dependency at the same time. The recovery dimension of that trade-off, where the same hub becomes a tier-0 point of failure the whole fleet inherits, is examined in Why Most OpenShift DR Strategies Fail at Executive Level.
Principle 2: Targeting Must Scale Without Reconfiguration
ClusterSets and a strict label taxonomy are scaling primitives.
A sustainable targeting model requires:
- Functional classification (
environment) - Risk classification (
tier) - Geographic dimension (
region) - Architectural role (
cluster-type)
Adding a new cluster should require only correct labeling.
If policy rollout requires editing definitions for a new cluster, the architecture does not scale.
An operational detail that reinforces this: Placement only evaluates clusters within bound ClusterSets. ManagedClusterSetBindings must exist in the correct namespace for targeting to function. This is a common source of silent targeting failures where policies appear defined but never reach their intended clusters.
Principle 3: Enforcement Agents Are Part of Governance
Imported MCE clusters frequently lack governance addons when custom klusterlet-config is used.
This creates a blind state:
- Policies propagate via ManifestWork to the managed cluster
- The governance-policy-framework and config-policy-controller are absent
- No local evaluation occurs
- Compliance dashboards show the cluster but report no status
From an architectural standpoint, governance agents are enforcement endpoints in a distributed control plane.
If they are absent, the control system is partially blind. The hub has no way to distinguish between a compliant cluster and one that simply never evaluated.
Principle 4: Governance Is a Feedback Loop
Dashboards are passive artifacts.
Governance becomes operational only when compliance state transitions trigger action:
Compliant -> NonCompliant -> Alert -> Remediation
In practice, most organizations stop at NonCompliant. The compliance dashboard is checked periodically, but no automated alerting or remediation path exists. This turns governance into historical reporting rather than active control.
The gap between NonCompliant and Alert is where governance effectiveness is determined. Without integration into alerting systems, compliance state transitions are observed retroactively, not acted upon in real time.
Governance without feedback is documentation.
Principle 5: Policies Are Code, Not Configuration
Manual console-created policies break traceability.
A GitOps-managed policy lifecycle using PolicyGenerator with Kustomize and ArgoCD or OpenShift GitOps introduces:
- Change review
- Version history
- Auditability
- Rollback capability
In mature platform organizations, governance changes follow the same rigor as application deployments.
Measuring Propagation Integrity
The five principles describe how governance should be designed. The question that follows is operational: how does the team know whether its governance is actually working?
Propagation Integrity is the measurement that answers that question. It is calculable from three inputs. The denominator for all three is the same: the clusters the policy’s Placement targets.
-
Definition coverage: the percentage of intended targets that actually have the policy defined locally, where the intended set for each policy is the clusters its Placement targets. Read it as the count of selected clusters where the propagated policy is present (ManifestWork applied, replicated Policy in place), over that target count.
-
Enforcement coverage: the percentage of selected clusters that have the governance agents installed and operational. Read it as the count where the governance-policy-framework and config-policy-controller addons report Available, over the same set. A policy that propagated but cannot be evaluated locally is documentation.
-
Feedback coverage: the percentage of selected clusters that report compliance state back to the hub. Read it as the count whose replicated Policy carries a non-empty compliance status (Policy.status), over the same set. A policy that is enforced but whose state is never reported leaves the hub blind.
Propagation Integrity is the product of these three. A fleet with 100% definition coverage, 100% enforcement coverage, and 100% feedback coverage has 100% Propagation Integrity. Each gap reduces the score multiplicatively, not additively: a fleet at 90% across all three is operating at 73% Propagation Integrity, not 90%. The asymmetry matters more than the average: a fleet at 100% definition, 100% feedback, and 60% enforcement also scores 60%, which is precisely the partially blind state Principle 3 describes: 40% of clusters never evaluate at all.
The interpretation is structural. Propagation Integrity below 90% means a meaningful fraction of the fleet is not under active governance. The number quantifies the blind spot that compliance dashboards typically hide.
For illustration, take a 40-cluster fleet operating at 73% Propagation Integrity. The score does not map to a clean cluster count, because the gaps compound across layers, but the directional reading is concrete: roughly a quarter of the fleet’s governance capacity is unaccounted for, and that fraction carries unpriced audit and incident exposure. Multiply it by the hours of audit preparation the team spends per quarter, and the percentage becomes a line item leadership can fund against. Substituting a real fleet size and audit-hour figure is what turns the structure of the calculation into a number for a budget review.
Propagation Integrity is what compliance looks like when measured instead of assumed.
The same measured-not-assumed discipline, applied to disaster recovery, produces a different number: the Validation Gap, developed in The DR Number Almost No One Records. Propagation Integrity is its governance counterpart. One measures whether policy reaches the fleet, the other whether recovery capability was ever tested.
Executive implication: Ask the platform team for the three coverage numbers. If they cannot produce them, the governance program has not yet operationalized its own measurement, and producing them is the first deliverable to fund, not a free precondition. The work is concrete: querying policy compliance state across the hub, reconciling intended against actual placement, and instrumenting governance-agent presence on each cluster. The investment buys the instrument that makes the blind spot visible before it closes it. What is not measured cannot be improved.
Organizational Impact
When governance is treated as an architectural control system:
- Configuration drift decreases measurably across the fleet
- Security baselines stabilize across regions and environments
- Cluster onboarding becomes predictable, requiring only correct labeling
- Audit responses shift from reactive preparation to deterministic reporting
- Incident blast radius becomes bounded by consistent enforcement
When governance is treated as configuration:
- Compliance becomes assumed rather than verified
- Cluster variance increases with each manual exception
- Audit preparation consumes engineering time disproportionately
- Incidents surface latent misalignment that could have been detected earlier
- Risk becomes unmeasurable because the control system has gaps
The difference is structural discipline, not tooling.
Architectural Continuity
Governance in multi-cluster Kubernetes is not about RBAC objects or YAML definitions. It is about controlling entropy across distributed systems. It is a control system: one that senses deviation, applies corrective force, and continuously stabilizes the platform under changing conditions.
Without feedback loops, systems drift. Without enforcement, policies decay. Without structural intent, scale amplifies fragility instead of resilience.
The primitives exist. Whether they form a coherent control system or remain a collection of configuration artifacts depends on architectural discipline. Every cluster that is not actively governed by design is governed by assumption. And assumptions, in distributed systems, are where incidents begin.
Continue with: Translating OpenShift Health into Business Risk