Investigation 039 - Distributed Systems Economics

The work is still owned.
Capacity is still finite.

An unfinished correction remains legitimate after a failed attempt. Trying again consumes finite shared execution capacity. Ownership does not grant unlimited access to that capacity. What economic authority may govern another attempt?

Unfinished correction

Still owned
Dependency downAttempt failedCapacity finite

Legitimate to retry. Free to retry now — not yet.

Author's Note

Retry feels like a virtue. It is not free.

A failed attempt does not necessarily mean the responsibility is gone. The dependency may recover. So another attempt may be exactly what the system needs. But an attempt spends a worker, a network request, dependency processing, and coordination capacity. The cost is tiny once. It is not tiny repeated without bound.

This investigation is not about a retry algorithm, a backoff curve, or a workqueue implementation. It is about the architectural boundary any such policy must respect: unfinished responsibility and access to scarce execution capacity are not the same fact.

The purpose of this investigation is not to discover a universal retry algorithm. It is to discover the boundary any such policy must respect.

Prologue

An attempt is not free, and ownership is not a rate limit.

Something has failed. The responsibility has not necessarily disappeared, so another attempt may still be necessary. But that attempt consumes a finite, shared pool that other legitimate work also needs. Restricting effort raises its own question: does withholding an attempt change the underlying responsibility, or only the present opportunity to spend capacity on it?

Inherited

Unfinished work remains owned after failure; observation is delayed and partial; a preference needs an explicit objective.

Temptation

Give every item the same bounded allowance and call the system protected.

Mystery

How can ownership survive without granting unlimited access to finite corrective effort?

First Principles

Work, attempts, and decisions are not the same fact.

Work vs. attempt?

Unfinished work can outlive many attempts. A retry is another attempt, not another piece of ownership.

Attempts cost capacity?

Worker time, network, dependency processing, and coordination are all finite and shared.

Decision vs. subject?

An economic decision about effort is not automatically a decision about the work's status.

Evidence vs. decision?

What the system observed, concluded, and decided are three different things that can drift apart.

Unfinished work, eligibility for another attempt, selection for execution, an attempt in progress, and success are five different states.

Naive Architecture

Retry until it works.

When an attempt fails, execute another immediately. No allowance, no waiting period, no economic calculation. The responsibility never silently disappears, and for rare, cheap, quickly-recovering failures this is often sufficient.

Attempt fails
→
Execute again immediately
→
Repeat until success

The Architecture That Almost Worked

Give each item a bounded allowance per accounting interval.

An unfinished item may attempt repeatedly, but only until its allowance for the current interval is exhausted. The work is not deleted when the allowance runs out — it simply waits until the next interval. One item's effort is now bounded.

Retry until it works
→
Bounded allowance per interval
→
Effort constrained, ownership intact
One unfinished item cannot consume unlimited corrective effort within the accounting interval. That is all this candidate establishes.

Breaking Our Design

Four pressures separate the allowance from what it actually bounds.

Episode 01 starts from the per-item allowance candidate. Each later episode unlocks only after the preceding discovery creates its boundary.

EPISODE 01

The Allowance Spent at Once

One item stays within its allowance but consumes it in a concentrated burst rather than spread across the interval.

Allowance10 attempts / interval
Spread patternnot run
Burst patternnot run
Contention claimunknown

Run the same allowance spread across the interval first.

EPISODE 02

A Thousand Bounded Failures

Every item independently respects its allowance, but the population keeps growing.

Population1 item
Per-item allowance10 attempts
Aggregate demand10 permitted
Accounting domainundefined

Grow the population while every item stays individually compliant.

EPISODE 03

The Work the Budget Did Not Select

Shared capacity is finite and demand exceeds it. Some legitimate work is not selected for the current attempt.

Shared capacityfinite
Selected worknot evaluated
Unselected worknot evaluated
Ownershipunknown

Evaluate demand against finite shared capacity.

EPISODE 04

Yesterday's Allocation

An economic decision was reasonable given its evidence. The evidence has since changed.

Original evidencedependency unavailable
Economic decisionnot made
New evidencenot observed
Decision authorityunknown

Make an economic decision from today's evidence.

Review the four experiments

    The Turning Point

    An allowance is not a guarantee.
    Ownership is not an entitlement to capacity.

    Explicit subject and domain
    →
    Nonselection ≠ lifecycle change
    →
    Claim bounded by its evidence

    The Retry Economy Contract

    Govern access to effort without redefining ownership.

    Economic authority may constrain present access to finite corrective execution capacity. It must remain explicitly scoped, and it must never silently become a lifecycle decision.

    1. Amount ≠ concentration

    A cumulative allowance over an interval does not, by itself, bound how concentrated that effort is.

    2. Per-subject ≠ system-wide

    Independent per-item bounds need an explicit accounting domain before they support a system-wide claim.

    3. Nonselection ≠ lifecycle

    Not receiving a present execution opportunity does not complete, cancel, or release durable ownership.

    4. Claim bounded by evidence

    A future-effort decision must not silently become permanent exclusion once its supporting evidence changes.

    Unfinished work remains owned (INV-006)
    Economic decision with explicit subject and accounting domain
    Present execution opportunity granted or withheld
    Lifecycle state preserved regardless of selection
    Claim remains bounded by the evidence that supports it
    No universal interval, curve, quota, or fairness rule prescribed
    Ownership is not unlimited access to effort. Economic authority is not lifecycle authority. A past decision is not permanent authority.

    Only Now: Kubernetes

    client-go's workqueue realizes part of this boundary.

    A delaying queue can make a remembered item eligible again after a chosen duration instead of immediately. A rate-limiting queue asks a configured limiter when an item may be added again and tracks how many times it has been requeued. The default typed limiter combines a per-item policy with an overall token-bucket policy — making the per-item-versus-aggregate distinction concrete.

    Calling Forget clears an item's retry history. It does not, by itself, prove that desired and observed state agree or that the underlying responsibility is resolved — the controller still owns that lifecycle judgment.

    Client-go illustrates the contract through one family of mechanisms. It does not define the contract for every corrective system.

    Engineering Reflection

    Control over scarce capacity must stay separate from ownership.

    Keep immediate retry

    Failures are rare, attempts are cheap, dependencies recover quickly, and little unrelated work competes for the same capacity.

    Adopt the economic contract

    Persistent failures, large subject populations, or shared dependencies make retry effort a material cost.

    Costs Accepted

    Some work waits

    Protecting finite capacity means not every unfinished item gets an immediate attempt.

    Explicit scope required

    Aggregate claims need a defined subject and accounting domain, not an assumed one.

    No eventual-progress guarantee

    Preserved ownership does not prove that work will eventually execute or succeed.

    Decisions stay revisable

    An economic decision cannot be treated as timeless just because it still exists.

    Investigation Exercise

    Trace amount, population, selection, and evidence separately.

    Prediction

    State whether one item's allowance bounds contention, and whether nonselection changes ownership.

    Experiment

    Vary concentration, population size, available capacity, and supporting evidence while holding the allowance fixed.

    Observation

    Record whether an old economic decision keeps claiming authority after its evidence has changed.

    Reflection

    Separate the architectural boundary from the interval, curve, or quota a concrete system chooses.

    retry economy trace
    Complete all four experiments before running the synthesis trace.

    Bridge from Movement VI

    Six boundaries.
    One recurring lesson.

    Movement VI's six economic contracts converge on one lesson: a system may govern access to scarce resources without letting that decision redefine responsibility, authority, or evidence
    A system may govern access to scarce resources without allowing that economic decision to redefine the responsibility, authority, or evidence on which correctness depends.

    Intellectual Lineage

    Borg made finite shared resources, isolation, and large workload populations explicit operational concerns. Omega showed that individually legitimate decisions can interfere over shared state. Control theory separates the persistence of an error signal from the policy and actuation used to influence it. Kubernetes client-go realizes these ideas through queues that remember work and limiters that govern when another attempt becomes eligible.

    Deliberate Simplifications Ledger

    Retry intervals, counts, backoff curves, jitter, and delay capsOpen backlog
    Worker sizing, concurrency, fairness, starvation prevention, and accounting-domain selectionOpen backlog
    Permanent-failure classification, cancellation, and abandonment authorityOpen backlog
    Evidence refresh, reconsideration triggers, and decision-revision mechanismsOpen backlog
    Concrete client-go queue and limiter configuration, metrics, and API evolutionOpen backlog

    Sources