Systems often advertise fault tolerance while leaving the precise consequences of a repeated command, a stale lease or a partially committed operation ambiguous. Recovering availability is not identical to recovering truth.
1. Specify what cannot happen
Before designing a retry loop, define invariants: only one authority may accept a transition, a generation cannot move backwards, an effect cannot be counted twice, and a recovery event must not silently rewrite the accepted history. Different systems need different invariant sets; their scope is part of the claim.
An invariant is useful only when its input and enforcement boundary are defined. Declaring 'exactly once' without identifying the sender, receiver, storage semantics and failure model is insufficient.
2. Replay is a state problem
Duplicate delivery is normal in real systems. The dangerous case is not receiving a repeated message, but allowing it to generate a second effect. Generation identifiers, receipts, authority epochs and idempotent decisions help define the boundary. N.E.M.E.S.I.S. illustrates replay and stale-generation decisions; T.H.O.R.A.N. illustrates state admissibility.
A valid rejection should be evidenced as clearly as a valid acceptance. Test expired identifiers, sequence reversals, conflicting versions, broken receipts and crash/restart boundaries.
3. Authority and coordination
Concurrent workers do not require concurrent writers to a single authority record. Clear ownership, lease boundaries and deterministic integration reduce conflict. These principles extend from state engines to orchestration: independent execution can coexist with centralised authority over accepted transitions.
The research catalog includes StateSeal / InvariantStore, ConcordGrid / Agent Factory, CHRONOS and other assets whose scopes touch this problem from different angles. Their presence in the portfolio is not a declaration of a shared industrial certification.
4. Evidence after a crash
Preserve the pre-crash baseline, failure injection, restarted state, duplicate accounting and archive verification. A recovery trace lacking the rejected alternatives is too easy to overinterpret. Hashes establish byte identity, not semantic correctness; the two checks must be combined.
For multi-node claims, local-process simulations are insufficient. Partitions, time disagreement, real external effects and off-host observations should be tested inside a separately documented qualification.
5. The commercial question
A buyer should ask which invariant is actually enforced on its integration path, how failure is detected, whether manual repair is required and which tests can be rerun without access to proprietary source. Controlled technical diligence turns those questions into an acceptance contract.
Explore the supporting public context
These pages provide related demonstrators and evidence boundaries, not independent certification or a direct proof of every claim in this essay.
Independent research note. All reproduction-sensitive material and buyer-specific experimental protocols are withheld from public disclosure.