hraness

idempotent command records: the audit log an agent runtime owes you

record the mutation before you dispatch it

Drafted by an AI agent at Ben Guo's direct request from the Hraness source repositories, and checked against those sources before publication.

Send a command to a running agent from another device and three things can happen: the command runs, the command fails, or you never find out which. The third case is the one that breaks naive implementations, because the natural recovery (send it again) can apply the effect twice. The fix is a rule about ordering: record the mutation before you dispatch it, and bind that record to an idempotency key and the exact authority that will execute it.

Oompa implements this rule literally. Every effect-bearing command in its sync path is a durable row with a state, an idempotency key, and an authority tuple. The state machine and the admission policy are pure functions in src/cloud/commands.ts, which makes them easy to show and easy to test.

the ordering rule

“Record before dispatch” means the durable write comes first. When you submit a command, the system persists an intent row in pending, and only then does the custodian’s sync cycle claim and execute it. If the process dies between the record and the effect, the record survives and recovery can ask the only question that matters: did the effect run? If the response is lost after the effect ran, replaying the same idempotency key returns the stored outcome instead of running the effect again.

This inverts the usual client intuition. The client does not need to know whether its request succeeded; it needs to know that repeating the exact request is safe. Safety comes from the record, not from the response.

the state machine

Commands move through a closed set of states:

pending ──> prepared ──> effect_started ──> applied
    │            │              │
    │            │              ├──> failed
    │            │              │
    │            │              └──> ambiguous
    ├──> cancelled               (applied, failed, ambiguous are terminal)
    └──> expired

pending can become prepared, cancelled, or expired. prepared can become effect_started, cancelled, or expired. effect_started can resolve to applied, failed, or ambiguous. Everything except ambiguous from effect_started is terminal, and ambiguous itself is terminal: the transition table lists it with no outgoing edges.

Two details carry the weight. First, effect_started is recorded as its own committed state before the provider effect is attempted to resolve. Once a command is there, the honest answers are applied, failed, or ambiguous, and the code cannot quietly step back to prepared to pretend nothing happened. Second, ambiguous exists as a first-class terminal state. A timeout after dispatch is not a failure; it is genuinely unknown, and the record says so. That record blocks speculative retry, because retrying a possibly-applied effect is how one “steer” becomes two.

replay and conflict

The idempotency check is three lines of policy:

export function idempotencyDisposition(
  existing: StoredIdempotency | null,
  requestDigest: string,
): IdempotencyDisposition {
  if (existing === null) return "new";
  return existing.requestDigest === requestDigest ? "replay" : "conflict";
}

new admits the request. replay returns the stored result for a byte-identical request under the same key. conflict refuses: the key was already spent on a different request. That last branch is what makes idempotency keys real keys rather than dedup hints. Reusing a key for a different command is a bug the store rejects instead of a subtle corruption it permits.

Combined with the state machine, this gives the guarantee an agent runtime owes you: every mutation is recorded, replayable by its key, bound to one authority generation, and honest about the difference between “failed” and “unknown.”

inside the admission policy

The part worth studying is commandAuthorityTransitionDisposition, the pure admission function that decides whether a claimed command may move forward. Its signature takes the command’s current state and bound authority, the lease’s leaseUntil, the live authority tuple, the requested authority, the target state, and now. Its rejections are a closed union: lease_not_live, live_authority, bound_authority, invalid_transition.

The order of checks encodes the policy:

  1. Time first. A non-finite now, a deadline already in the past, or a malformed window is lease_not_live. No authority question is even asked.
  2. Then the live fence. authorityMatches compares bootGeneration, bootId, and fence between the request and the lease. A command submitted under the previous custodian (or an expired lease the same daemon used to hold) is live_authority.
  3. Then binding. Moving pending to prepared attaches the requesting authority as boundAuthority and requires the slot to be empty. Re-preparing under the same authority is a replay; under a different live authority it is a rebound, which is how a command that was prepared but never executed can be safely re-admitted after custody moves.
  4. Then the transition itself. Anything past prepared requires the bound authority to match the requested authority exactly, then delegates to the transition table.

Two properties fall out of the design and are worth stealing. Rejections return the unchanged projection, so the test suite can assert that a rejected call moved nothing (the lease lesson’s fencing depends on this). And rebound is explicit: authority changes are recorded transitions, not silent rewrites, which keeps the audit log truthful about which generation actually dispatched the effect.

reconciling the ambiguous

effect_started resolving to ambiguous is where the design earns its name. The runbook rule across the codebase is that a dispatched-but-unconfirmed effect is indeterminate: reconcile by idempotency key before any retry, and never replay speculatively. For provider turns, that means reading the provider’s own state (did the turn start? did it complete?) and settling the record to applied or failed from evidence, not from hope.

The same posture appears in the operational layer. The hosted-deployment helpers write protected intents before mutations and dispatch receipts after them; a result like created_dispatched_reconciliation_required must be resumed with the same command and identifiers, never a fresh attempt. Expiry is handled by a separate pure function, schedulerExpiryDisposition: only pending commands can expire, and only at their recorded deadline. A command already in prepared or effect_started does not silently lapse; it is fenced by authority instead.

Durability has a retention story too. The sync layer’s idempotency receipts are registered in costs.json as convex:idempotencyReceipts with retention ttl:P30D, a 512-row budget per account, and an owner module (convex/idempotency.ts). The registry matters less as bookkeeping than as a forcing function: in this codebase a new table, bucket, or stream fails check:cost-surfaces until it registers a kind, retention class, owner, and budget, so every durable record carries its lifecycle in the same review as its schema.

The hosted path shows the same discipline at the transport layer. Remote commands live in the convex:deviceCommands table (ephemeral retention, 256 rows per account), are claimed under the custodian’s execution lease, and settle under their idempotency key. A websocket subscription wakes the daemon when its device’s pending set changes, but the subscription carries no authority: a wake that races the poll timer costs one extra cycle, never a second execution, because the command still has to be claimed and bound to the exact authority generation the ordinary way. oompa remote command <uuidv7> reads the durable record back, which is the operator-facing half of the audit log: the record is inspectable, not merely enforced.

what the record cannot prove

The record is an audit log, not an oracle. A durable applied proves the custodian recorded that outcome under a specific authority; it does not prove what a provider did with its own hidden state. A conflict proves key reuse, not which request was correct. And the whole mechanism assumes the record itself is durable: Oompa’s hosted layer stores commands in Convex tables with bounded budgets, while purely local execution writes through its own SQLite store. Neither tier claims that a local custody receipt proves a remote effect.

If you implement this pattern yourself, the load-bearing decisions are the uncomfortable ones: keep ambiguous as a terminal state you reconcile rather than a retry queue you drain, treat conflict as a refusal rather than an overwrite, bind commands to an authority generation that can expire, and write the record before the dispatch. The dispatch is the easy part.

sources

  • oompa: command states, idempotency, and authority admission in src/cloud/commands.ts; lease authority in src/cloud/leases.ts.
  • Oompa hosted sync runbook: durable intents, dispatch receipts, and reconciliation posture for hosted operations.
  • The costs.json registry in the same repository: how every durable surface declares kind, retention, owner, and budget.

keep reading: free for subscribers

the rest of this lesson is free. add your email once and every subscriber lesson on this site stays unlocked.

already subscribed? enter the same email to unlock.