Architecture documentation describes how a system was designed. It rarely captures how that system behaves under load, partial failure or prolonged operational pressure.

Implication:

The gap between designed and observed behavior grows as systems age. Teams that rely on documentation alone inherit risk that has no name in any diagram.

Part of the Field Notes series documenting operational patterns observed in real-world platform architectures.

Let’s explore five items and try to answer that question.

1. Multi Clusters

Organizations operating multi-cluster Kubernetes fleets face a structural risk that is rarely discussed in architectural reviews: governance gaps that remain invisible until an audit fails or an incident escalates.

The cost is measurable. Undetected configuration drift increases incident blast radius. Inconsistent RBAC baselines extend audit preparation from days to weeks. Clusters onboarded without active policy enforcement create compliance blind spots that accumulate silently.

These are not tooling problems. They are symptoms of treating governance as configuration rather than as an architectural control system.

This document frames governance in multi-cluster Kubernetes as a distributed control problem and proposes structural principles for solving it.

2. Problem Pattern

In multi-cluster environments, governance failures rarely originate from missing policies.

They emerge from systemic misalignment across clusters:

• Configuration drift between environments
• Inconsistent RBAC baselines
• Selective policy enforcement
• Imported clusters without active governance agents
• Labeling schemes that do not scale

The recurring pattern is this:

Organizations believe they have centralized governance because policies exist on the hub.

In reality, enforcement is uneven, propagation is misunderstood, and compliance status is assumed rather than verified.

This creates silent governance gaps that only surface during audits or incidents.

• For a production-level examination of how these gaps manifest as cascading deletions, infrastructure failures, and silent packet loss in multi-cluster environments, see The Hidden Reliability Risks in Multi-Cluster Kubernetes.

3. Architectural Lens

Governance in RHACM should be treated as a distributed control system, not as a configuration feature.

The system has five structural layers:

1. Policy Definition: what must be enforced
2. Targeting Logic (Placement): where enforcement applies
3. Propagation Mechanism: how policies reach managed clusters
4. Enforcement Agents: what evaluates compliance locally
5. Feedback (Compliance State): what reports status back to the hub

Each layer is independently necessary. None are sufficient alone.

Most operational failures occur at the boundaries between these layers:

• Policy defined, but Placement incorrect
• Placement correct, but governance addons not installed
• Enforcement active, but no alerting loop
• Compliance visible, but not operationalized

Governance therefore is not a YAML problem.

It is a propagation integrity problem.

4. Governing Principles

Principle 1: Governance Must Be Hub-Centric

Policy definitions belong to the hub cluster. No ad-hoc, cluster-level policy creation.

Cluster-by-cluster RBAC adjustments introduce entropy. Propagation eliminates variance.

Enforcement should be deterministic and uniform across the fleet.

This does not mean every cluster receives identical configuration. RHACM supports controlled customization through hub-side policy templates that reference managed cluster attributes via template functions. The distinction is architectural: variability is declared centrally and resolved at propagation time, not managed independently per cluster.

Principle 2: Targeting Must Scale Without Reconfiguration

ClusterSets and a strict label taxonomy are scaling primitives.

A sustainable targeting model requires:

• Functional classification (environment)
• Risk classification (tier)
• Geographic dimension (region)
• Architectural role (cluster-type)

Adding a new cluster should require only correct labeling.

If policy rollout requires editing definitions for a new cluster, the architecture does not scale.

An operational detail that reinforces this: Placement only evaluates clusters within bound ClusterSets. ManagedClusterSetBindings must exist in the correct namespace for targeting to function. This is a common source of silent targeting failures where policies appear defined but never reach their intended clusters.

Principle 3: Enforcement Agents Are Part of Governance

Imported MCE clusters frequently lack governance addons when custom klusterlet-config is used.

This creates a dangerous state:

• Policies propagate via ManifestWork to the managed cluster
• The policy-framework and config-policy-controller are absent
• No local evaluation occurs
• Compliance dashboards show the cluster but report no status

From an architectural standpoint, governance agents are enforcement endpoints in a distributed control plane.

If they are absent, the control system is partially blind. The hub has no way to distinguish between a compliant cluster and one that simply never evaluated.

Principle 4: Governance Is a Feedback Loop

Dashboards are passive artifacts.

Governance becomes operational only when compliance state transitions trigger action:

Compliant > NonCompliant > Alert > Remediation

In practice, most organizations stop at NonCompliant. The compliance dashboard is checked periodically, but no automated alerting or remediation path exists. This turns governance into historical reporting rather than active control.

The gap between NonCompliant and Alert is where governance effectiveness is determined. Without integration into alerting systems, compliance state transitions are observed retroactively, not acted upon in real time.

Governance without feedback is documentation.

Principle 5: Policies Are Code, Not Configuration

Manual console-created policies break traceability.

A GitOps-managed policy lifecycle using PolicyGenerator with Kustomize and ArgoCD or OpenShift GitOps introduces:

• Change review
• Version history
• Auditability
• Rollback capability

In mature platform organizations, governance changes follow the same rigor as application deployments.

5. Organizational Impact

When governance is treated as an architectural control system:

• Configuration drift decreases measurably across the fleet
• Security baselines stabilize across regions and environments
• Cluster onboarding becomes predictable, requiring only correct labeling
• Audit responses shift from reactive preparation to deterministic reporting
• Incident blast radius becomes bounded by consistent enforcement

When governance is treated as configuration:

• Compliance becomes assumed rather than verified
• Cluster variance increases with each manual exception
• Audit preparation consumes engineering time disproportionately
• Incidents surface latent misalignment that could have been detected earlier
• Risk becomes unmeasurable because the control system has gaps

The difference is structural discipline, not tooling.

Closing Insight

In multi-cluster Kubernetes environments, governance is not about RBAC objects or YAML definitions.

It is about controlling entropy across distributed systems.

The primitives for policy definition, targeting, propagation, and enforcement exist. Whether those primitives form a coherent control system or merely a collection of configuration artifacts depends on architectural discipline.

Every cluster that is not actively governed by design is governed by assumption. And assumptions, in distributed systems, are where incidents begin.

Architectural Continuity

Governance in multi-cluster environments is not a checklist, and it is not a collection of policies.

It is a control system. One that senses deviation, applies corrective force, and continuously stabilizes the platform under changing conditions.

Without feedback loops, systems drift.
Without enforcement, policies decay.
Without structural intent, scale amplifies fragility instead of resilience.

In distributed environments, governance is not overhead. It is the mechanism that determines whether complexity remains controlled, or becomes chaotic.

The next step is understanding how those control signals become executive risk indicators.

Continue with: Translating OpenShift Health into Business Risk