Routes target machines
Routers are built to forward toward host and subnet destinations.
Investigation 020 - Cluster Networking Principles
Machines can reach machines.
Can workloads reach workloads?
We inherit two contracts without rediscovery: INV-018 established that a running Pod instance owns its own network identity, and INV-019 established that a node must own finite resource boundaries. The mystery now is cross-node communication: why does a packet addressed to a remote Pod vanish when physical links are healthy?
Begin the investigation downPhysical RoutesMACHINES KNOWN, WORKLOADS UNKNOWN
Prologue
The packet leaves Node A and disappears.
Orders runs on one node. Inventory runs on another. Orders sends to Inventory's Pod address. The physical network carries packets between machines, but it has never been taught where Pod addresses live.
Node A
Orders Pod sends traffic.
Physical fabric
Routers evaluate destination against machine routes.
Missing map
No route exists for the destination Pod address.
Result
The machines are linked; workload reachability is still unresolved.
First Principles
Physical routing and workload identity are distinct realities.
Physical networks route machine addresses. Workload identity answers which running instance a packet is for. These responsibilities overlap on one node, but diverge immediately across nodes.
Pods are separate identities
A Pod IP identifies one running instance, not the node itself.
Cross-node meaning is missing
A workload address can be correct locally but meaningless to the physical fabric.
Do not reveal the answer early: first isolate each failure caused by forcing workload communication through machine identity.
Naive Architecture
Pods borrow node IP and use node ports.
To avoid creating a new network model, every Pod is addressed through its node IP plus a selected port.
Node A192.168.1.10
Orders Podnode :8080
Inventory Podnode :9090
The Architecture That Almost Worked
Under constrained conditions, shared node identity looks sufficient.
If one Pod runs per node, or if operators coordinate unique node ports manually, traffic appears stable and the design seems successful.
One Pod per node
No local port contention exists.
Coordinated ports
Teams avoid collisions by agreement and sequencing.
Identity remains machine-owned
The design has not proven independent workload addressability.
The design works while reality is cooperative. Breaking episodes test when cooperation ends.
Breaking Our Design
Five causal episodes derive the cluster network requirement.
EPISODE 01
Port Ownership Collision
Pressure. Two independent Pods on one node both need tcp/8080.
Prediction. If borrowed node identity is sufficient, both should bind independently.
Experiment
Both Pods bind node IP port 8080.
Observation
First bind succeeds; second bind is rejected.
Failure
Port ownership becomes a shared node bottleneck.
Discovery
Shared node identity breaks independent Pod execution.
Next Pressure
Try translation to keep one node identity while avoiding collisions.
Constraint
This is an endpoint ownership conflict, not an app bug.
Boundary Delta
Node identity cannot safely represent many independent listeners.
Local experiment: two independent Pod listeners on one node endpoint.
Orders Podpending
Inventory Podpending
Node endpoint192.168.1.10:8080
Run the bind attempt to test shared node endpoint ownership.
EPISODE 02
NAT or Port Translation Workaround
Pressure. Keep one node IP while allowing both Pods to keep internal port 8080.
Prediction. Distinct node ports translated to each Pod should restore delivery.
Experiment
Map node :30001 and :30002 to separate Pod :8080 listeners.
Observation
Both requests arrive at their targets.
Success (temporary)
Connectivity appears repaired.
Discovery
Translation can hide port collisions.
Next Pressure
Now inspect what receiver believes the sender identity is.
Constraint
Allow this repair to appear successful before deeper tests.
Boundary Delta
Delivery restored does not yet prove identity correctness.
Local experiment: external node ports translated to internal Pod ports.
Translation mapnot configured
Request Apending
Request Bpending
Configure translation and send both requests.
EPISODE 03
Source Masquerading Corrupts Identity
Pressure. Destination-port translation restored delivery. Now add source masquerading on the forwarded path while the receiver relies on source identity.
Prediction. If this additional translation preserves identity, the receiver should still see the caller Pod IP.
Experiment
Enable source masquerading, then inspect the receiver log.
Observation
Receiver records node IP as sender.
Failure
End-to-end source identity is obscured.
Discovery
Connectivity and identity are different guarantees.
Next Pressure
Try direct Pod-addressed routing without translation.
Constraint
This is a separate identity test, independent from Episode 2 setup.
Boundary Delta
A node-forwarded identity is not workload identity.
Local experiment: receiver examines observed source identity.
Actual senderPod 10.42.1.5
Observed senderunknown
Policy impactundetermined
Enable source masquerading and inspect receiver-side identity.
EPISODE 04
Direct Pod Address Has No Physical Route
Pressure. Preserve Pod identity and address destination Pod directly across nodes.
Prediction. If physical routing already knows Pod addresses, packet should arrive.
Experiment
Send direct packet to remote Pod address 10.42.7.3.
Observation
Physical route lookup returns no destination path.
Failure
Route knowledge for Pod space is absent.
Discovery
Identity correctness alone does not create reachability.
Next Pressure
Derive minimum network guarantees workloads require.
Constraint
This test isolates routing absence without adding a mechanism.
Boundary Delta
A second network meaning must be maintained by the platform.
Local experiment: remote Pod address lookup on physical route table.
Destination10.42.7.3
Route lookupnot queried
Deliveryunknown
Perform direct route lookup for the destination Pod address.
EPISODE 05
Derive the Flat Reachability Requirement
Pressure. Previous failures must be resolved by explicit workload-facing guarantees.
Prediction. A contract set that guarantees identity and direct reachability should close all prior pressures.
Experiment
Assemble required guarantees one by one.
Observation
Missing guarantees reproduce earlier failures.
Success
Complete set resolves causal chain at contract level.
Discovery
Cluster-wide direct Pod reachability is required.
Next Pressure
Move to stable address identity in INV-021.
Constraint
Do not implement CNI, BGP, overlays, underlays, or plugins here.
Boundary Delta
Contract earned before mechanism naming.
Local experiment: classify and assemble guarantees only.
Unique Pod identitymissing
Direct Pod reachabilitymissing
Source identity preservedmissing
Assemble all guarantees to complete the causal chain.
Optional Episode Review
The Turning Point
If workload identity is independent,
reachability must be cluster-wide.
The requirement is now earned as architecture: any Pod must be directly reachable by Pod identity across node boundaries without translation in the path.
Generic Cluster Network Contract
Guarantees before implementation.
Every workload may assume that Pod identity is unique cluster-wide, directly reachable without address translation, source-preserving end to end, and independent of node placement.
Unique Pod-instance identity
One Pod IP belongs to one running temporary Pod instance at a time.
Direct Pod-to-Pod reachability
Any Pod can address any other Pod directly with no translation hop in the path.
End-to-end source identity
Receivers observe true sender workload identity, not a forwarding node identity.
Placement-independent addressing
Applications address workloads without learning which node currently hosts them.
Physical network is implementation
Workloads depend on the cluster reachability guarantees, not on the mechanism used beneath them.
Precision boundary: a Pod IP does not survive replacement. Replacement may receive a different Pod IP.
Only Now: Kubernetes
Kubernetes names this as the Pod IP network model.
Kubernetes requires Pod-level addressing that remains meaningful across nodes. Workloads communicate by Pod identity, not by coordinating node ports.
This chapter intentionally does not explain IP allocation internals, plugin internals, overlay versus underlay design, BGP, eBPF, or route convergence implementation details.
Name after discovery, not before discovery.
Engineering Reflection
Timeless principle: execution identity and communication identity must align.
When workload instances are placed and replaced across machines, network assumptions must continue to name workloads directly, not machines that host them.
Architectural Honesty
Coordinated edge patterns
Shared node identity can remain acceptable for tightly controlled ingress or single-workload-per-node situations.
Independent cross-node workloads
Independent teams, same-port workloads, and identity-sensitive policies expose the hidden coupling quickly.
Costs Accepted
Every node must carry enough route knowledge to reach remote Pods.
Route knowledge can be delayed as Pod instances appear, disappear, and are replaced.
The platform must maintain a shared logical map above physical links.
Applications gain simplicity because platform networking absorbs complexity.
Investigation Exercise (Optional)
Run the synthesis trace, then stop at evidence boundaries.
Predict which fixes only delivery and which preserves identity.
Run the five-step trace from collision to contract assembly.
Separate temporary success, identity corruption, and routing absence.
State the minimum guarantees without naming implementation mechanism.
Run the trace after writing your prediction.
Bridge to INV-021
Reachability is solved.
Address stability is not.
A Pod address is now reachable across nodes, but the address belongs to one temporary Pod instance. If that Pod is replaced, the replacement may receive a new address and old callers still target the vanished instance address.
The network can deliver to a Pod address. It cannot make one Pod address survive Pod replacement.
Deliberate Simplifications Ledger
Pod IP allocation and pool managementOpen backlog
Network plugin, overlay, and underlay realizationOpen backlog
Route convergence and stale route recovery behaviorOpen backlog
Network policy and isolation semanticsINV-024
Stable service-level addressing above Pod instancesINV-021