Connectivity
Workloads can find and reach one another regardless of placement.
Investigation 024 - Cluster Networking Principles
It can reach.
Should it?
The flat network solved communication. It never decided whether a public frontend, payment processor, customer database, and reporting job should all trust one another equally.
Begin the investigation downEvery path is openfrontendauthpaymentcustomer-dbreporting
Prologue
Movement and authorization are not the same decision.
An office without locked rooms makes movement effortless. It also treats payroll, customer records, and public reception as one trust domain.
Foundation
Every workload can find and reach every other workload.
Assumption
Anything inside the platform deserves equal access.
Pressure
Responsibilities and sensitivity diverge while reachability stays identical.
Mystery
How can selected paths be denied without dismantling the flat network?
First Principles
Reachability is infrastructure. Authorization is governance.
A network can correctly deliver a packet and still enable a communication the platform should prohibit. Connectivity and authorization are independent contracts.
Authorization
No platform declaration says which workload relationships are permitted.
Two outcomes
Routing capability can exist while policy prevents a prohibited communication outcome.
Unrestricted opportunity is not a network failure. It is a missing platform contract.
Naive Architecture
Trust every internal workload equally.
The simplest communication policy is no policy. Every path is available, no team waits for a rule, and new services communicate immediately.
Workloadany responsibility
Flat Networkevery path open
Workloadany sensitivity
No configuration
Communication works from the first deployment.
One trust domain
The same small team owns and validates every workload.
Valid when trust is uniform
Additional controls would add cost without solving a present problem.
The Architecture That Almost Worked
Let every application guard its own door.
Each service already authenticates callers. Add a local allowlist and let every application reject communication it considers inappropriate.
Connectionalready received
Application Checkteam owned
Business Logicaccept or reject
The application can reject a caller after the connection arrives. The platform cannot verify that every team performs the check.
Breaking Our Design
Three independent pressures earn one isolation contract.
Each experiment begins from its own state, earns one architectural step, and stops before the next episode's answer.
EPISODE 01
All Workloads Are Not Equal / Trust Was Assumed
Pressure. Five workloads with different responsibilities and sensitivity share identical reachability.
Prediction. If internal placement implies trust, equal reachability should match equal consequence.
Experiment
Reveal responsibilities and compare access.
Observation
Sensitivity differs; reachability does not.
Failure
Connectivity silently became trust.
Discovery
The platform needs an explicit authorization question.
Next pressure
What happens when one workload is compromised?
Boundary
No compromise or isolation solution appears here.
Independent simulation: unequal work behind equal network access.
EPISODE 02
One Compromise Becomes Many / Blast Radius Scales with Reachability
Pressure. One public frontend is compromised inside an otherwise functioning flat network.
Prediction. Cluster size should not change the consequence of compromising the same workload.
Experiment
Model 10, 100, and 500 workloads.
Observation
Potential targets are N-1: 9, 99, 499.
Failure
Existing relationships expand opportunity.
Discovery
Potential blast radius follows reachable relationships.
Next pressure
Who can enforce a uniform boundary?
Boundary
Detection remains delayed and unresolved.
Independent simulation: possible connection attempts after one compromise.
Cluster size10
Public frontendnot compromised
Potential targets-
Networkworking as designed
Detectionunknown and possibly delayed
EPISODE 03
The Platform Must Own Isolation / Declarative Isolation Requirement
Pressure. Four teams implement per-application allowlists, but every application receives the connection before its code can reject it.
Prediction. Independent application ownership should produce uniform protection the platform can verify.
Experiment
Omit one check across four teams.
Observation
The platform cannot verify uniform protection.
Failure
Perfect application behavior became an invariant.
Discovery
Declare authorization separately as platform-owned desired state.
Contract
Observe enforcement, then probe a denied path.
Boundary
Syntax, rule semantics, and mechanism remain deferred.
Independent simulation: application checks fail; declaration remains distinct from enforcement.
Application checks4 teams / 4 allowlists
Connection arrivalbefore application rejection
Platform auditnot attempted
Desired policynot declared
Observed enforcementnot observed
The Turning Point
Keep the reachability foundation.
Declare which communication is authorized.
Routing Capabilityflat network foundation
Declared Authorizationplatform owned
Communication Outcomepermitted or prevented
The policy constrains communication outcomes. It does not replace the network beneath them.
The Network Isolation Contract
Five responsibilities. One platform owner.
Contract not yet earned. Complete Episode 3: fail application ownership, declare desired policy separately, observe enforcement, and probe a denied path.
A mature platform preserves the flat routing foundation while governing which communication attempts are authorized to succeed.
Preserve the reachability foundation
Addresses, routes, and discovery retain their underlying capability. Policy enforcement prevents prohibited communication, so denied workload pairs do not have an authorized effective path.
Declare authorization separately
Desired communication relationships exist independently of application code as platform-owned state.
Enforce the contract
The platform prevents prohibited communication before application business logic must reject it.
Keep concerns independent
Connectivity establishes routing capability. Authorization determines the permitted communication outcome.
Limit blast radius
A compromise retains only the communication relationships intentionally authorized for that workload group.
Isolation does not prevent compromise. It limits the reachable opportunities available afterward.
Only Now: Kubernetes
NetworkPolicy is a bounded realization of the contract.
Kubernetes NetworkPolicy controls layer 3 and layer 4 traffic involving Pods. Its selectors target Pods and namespaces, and its IP blocks target CIDR ranges. It does not target Services by name.
A supporting network plugin must implement enforcement. Creating a NetworkPolicy resource alone may have no effect. Applicable policies combine additively, and evaluation order does not change the result.
Policy handling is asynchronous. Existing or newly created Pods can temporarily observe lag or inconsistent application while a distributed plugin converges. The Kubernetes API exposes no exact signal for when a policy has been applied everywhere.
This investigation does not teach syntax, default-deny patterns, or CNI-specific mechanisms. NetworkPolicy does not provide TLS, Service-name targeting, guarantees for every protocol, compromise detection, or enforcement observability.
The architecture is platform-owned declarative isolation. NetworkPolicy realizes a bounded L3/L4 portion of it.
Engineering Reflection
Timeless Engineering Principle
Universal reachability enables communication. Declarative isolation governs it. A resilient platform requires both.
Architectural Honesty
One small trust domain
A cohesive team with a few uniformly trusted workloads may gain nothing from another policy layer.
Trust becomes heterogeneous
Public entry points, sensitive workloads, multiple teams, tenants, or regulatory boundaries make unrestricted access unnecessary risk.
Costs Accepted
Communication declarations become another platform artifact.
Engineers must describe legitimate workload relationships.
An incorrect declaration can interrupt valid communication.
Troubleshooting separates routing capability from authorization outcome.
Was the flat network wrong?
No. It remains the correct reachability foundation. The missing responsibility was authorization.
Does isolation detect compromise?
No. Detection can be delayed and partial. Isolation bounds opportunity without claiming to identify the attacker.
Investigation Exercise
Compare an open network with declared authorization.
Predict the possible targets available to one compromised frontend in each design.
Draw the same four workloads first with every path open, then with only necessary paths authorized.
Trace possible connection attempts without assuming every attempt compromises its target.
The vulnerability is unchanged. Only the opportunities after compromise differ.
Run the synthesis trace after writing your prediction.
Bridge to Movement IV
Communication is governed.
But every workload still forgets.
Every workload can discover and reach the others it needs.
Communication authorization is declared separately and enforced by the platform.
A compromise no longer inherits every possible path by default.
But every process loses what it held in memory when it stops.
State must outlive the process that produced it.
Movement III closes with governed communication. Movement IV opens with information that must survive execution.
Intellectual Lineage
This investigation inherits the book's separation of desired state from observed state and its decomposition of independently owned control responsibilities. Control theory supplies the feedback-loop model; Borg and Omega provide lineage for decomposition around shared state. Applying that inheritance to communication boundaries does not claim Borg or Omega invented Kubernetes NetworkPolicy.
Deliberate Simplifications Ledger
Declarative policy syntax, selectors, and rule definitionsFuture: Network Policy Semantics (unassigned)
Policy defaults, ordering, and conflict semanticsFuture: Network Policy Semantics (unassigned)
Namespace, egress, IP-block, and cluster-wide strategiesFuture: Isolation Scope and Strategy (unassigned)
Service identity, mutual authentication, and service-mesh controlsFuture: Workload Identity and Mutual Authentication (unassigned)
CNI enforcement, compromise detection, and observabilityFuture: Policy Enforcement and Observation (unassigned)
Sources
- Kubernetes Documentation - Network Policies
- Saltzer, Jerome H., and Michael D. Schroeder. The Protection of Information in Computer Systems. Proceedings of the IEEE, 1975.
- Rose, Scott, Oliver Borchert, Stu Mitchell, and Sean Connelly. Zero Trust Architecture. NIST SP 800-207, 2020.
- Burns, Brendan, Brian Grant, David Oppenheimer, Eric Brewer, and John Wilkes. Borg, Omega, and Kubernetes. ACM Queue, 2016.