hraness

journaling for p2p apps: wal ideas outside the database

the local record is the audit trail

Drafted by an AI agent at Ben Guo's direct request from the Hraness source repositories, and checked against those sources before publication.

Databases have write-ahead logs for a reason: before you say a change happened, you write down that it happened, in an order you can replay, somewhere that survives the process dying mid-sentence. Peer-to-peer applications need the same thing and rarely get it. The “state” is scattered across a UI, a socket, a filesystem, and a consensus engine, and the crash is always in the gap between two of them.

Valhalla’s answer is vhalla-journal: a durable commit log that is not a database, does not know what it is storing, and exists for exactly one reason: to be the authority on the order in which things committed. The module doc is precise about the scope: “This crate is the order authority, not consensus… it does not verify certificates, admit values, or decide anything.” The journal is where “decided” becomes “happened,” and that distinction is the whole design.

the local record is the audit trail

The problem the journal solves is narrower than storage. A consensus engine decides a batch. The application state (in Valhalla’s case, a social archive and a rooms registry) has to reflect it. Between “the engine decided” and “the stores show it” are several filesystem writes, and a crash can land anywhere in the sequence. Without an ordering authority, recovery has to guess: did this store update commit? Was that batch applied?

The journal removes the guess by making one thing durable first: the bundle. A Bundle is an immutable, content-addressed record: SHA-256 over nine length-prefixed fields including the certificate bytes, the predecessor and next frontiers, the batch, and the height. Its identity is its content; Bundle::decode re-derives the hash from the parsed fields so “a corrupted or substituted file cannot impersonate the expected id.” The journal does not interpret the fields. It orders them.

the commit protocol

Journal::commit is a fixed sequence, each step a named filesystem operation the Store trait exposes:

  1. Lock: take the exclusive writer lock (released on process death).
  2. ReadPin: re-read the current HEAD pin under the lock.
  3. CreateBundle: write bundles/<id> exclusively (fails Ok(false) if it exists).
  4. SyncBundle: fsync the bundle file.
  5. WriteHeightMarker: write heights/<n> binding the height to the bundle id.
  6. SyncHeightMarker: fsync the marker.
  7. WritePinTmp: write pin.tmp carrying predecessor, next, bundle id, height.
  8. SyncPinTmp: fsync it.
  9. RenamePin: atomically rename pin.tmp over HEAD. This is the publication point.
  10. SyncDir: fsync the directory so the rename and bundle entry are durable.

Then, and only then, the caller is acknowledged. Every step is ordered by the rule that a crash before the rename leaves a recoverable mess and a crash after it leaves a committed pin. The pin file itself is VHP1 magic plus 104 bytes (two 32-byte frontiers, a 32-byte bundle id, a u64 height), small enough that its write is as close to atomic as a filesystem gives you.

The retry path is where the design earns its keep. commit re-reads the pin under the lock, so a crash that already published the exact bundle reconciles to Outcome::AlreadyCommitted instead of applying twice: the uncertain outcome is resolved by looking at what is durable, not by guessing. A different bundle claiming the same predecessor or height is JournalError::Conflict. Nothing is silently retried, and nothing is silently overwritten.

recovery is a read

Journal::recover trusts only what is on disk, and the rules are a direct consequence of the commit order:

  • A renamed pin is committed. Its bundle is read back and re-verified: the pin’s claimed bundle id, predecessor, and next are checked against the bundle’s own content-derived values.
  • A leftover pin.tmp is discarded. It was written but never renamed, so it never happened.
  • Height markers above the committed pin are “unpublished residue” (a crash between marker write and pin rename) and dropped so a different bundle can claim the height later.
  • Orphan bundles (files under bundles/ the pin does not reference) “carry no authority.” They are reported, not trusted.
  • A corrupt pin fails closed. Recovery does not repair; it refuses.

The Recovered struct reports all of it: the pin, the orphan list, whether a tmp was dropped, which height markers were discarded. Recovery is not “make it consistent”; it is “tell the caller exactly what is durable and let the layer above decide.” That is what makes it an audit trail rather than a repair log.

The tests get at this through FaultingStore, a Store wrapper that injects a programmed Fault at any Step (CrashBefore, CrashAfter, FailIo) while logging every attempted step. crash_after_rename_recovers_committed_and_retry_is_idempotent dies between RenamePin and SyncDir, reopens, and asserts the pin committed and the retry reconciles. crash_before_rename_leaves_orphan_and_retry_completes dies earlier and asserts the bundle is orphaned and the retry finishes the pin. The protocol’s promises are tested at exactly the points where a process can die.

the ledger below consensus

Under the journal sits vhalla-ledger, which solves the other half of the problem: if the journal is the order authority, the ledger is the content authority. It keeps a bounded, linear event history (MAX_EVENTS = 4096 events, MAX_PAYLOAD = 1024 bytes each) where every event names its parent by digest and append rejects an event whose parent is not the current tip (UnknownOrForkedParent). Actor sequences must be strictly monotonic. The StateRoot is not stored. It is re-derived from the retained ancestor chain every time, so “caller-supplied roots are rejected”: a checkpoint is only accepted when its claimed head is the current tip and its claimed root is the one the history actually produces.

This is the WAL idea split in two. The journal answers “in what order did things commit” with durable pins. The ledger answers “what is the state at this point” with a re-derivable root. Neither trusts a claim (the journal re-reads its pin, the ledger re-computes its root) because in a system where a peer’s evidence can conflict with itself, the local record is the only thing you can audit.

The boundedness is what makes the ledger portable. MAX_SNAPSHOT_BYTES caps a snapshot at 16 MiB, and the whole crate is no_std, so the same append-log can run inside a WASM adapter and produce bit-identical StateRoots. The honest caveat is that snapshots are not authenticated: a Snapshot is Copy content-addressed data, and the crate treats it as untrusted until a signed storage layer wraps it. The ledger fixes the order and the derivation; who vouched for a snapshot is a different layer’s problem.

the pattern repeats

The same commit discipline shows up in miniature wherever the codebase persists anything. vhalla-native’s SpentFile (the bounded set of consumed invitation nonces) republishes the whole VSN1 file on every consume: write the temporary sibling, sync it, rename over the old, sync the directory. vhalla-social-store and vhalla-rooms-store publish snapshots the same way: durable intent reconciled before publication, checksum as integrity evidence and “never an external freshness proof.” Even vhalla-identity’s key creation follows it: the identity.tmp file is fully synced, hard-linked to identity, then the directory is synced.

The pattern is always: write the complete new thing to a temporary name, fsync it, rename it over the old, fsync the directory, then acknowledge. The rename is the commit point; everything before it is preparation and everything after it is cleanup. It is the journal’s pin.tmp over HEAD at a smaller scale: the same reason, the same crash semantics, the same refusal to acknowledge before the durable record exists.

The lesson the workspace keeps demonstrating is that this is not a database luxury. Anywhere a process can die between “decided” and “recorded” (and in a peer-to-peer app that is everywhere), the local durable record is the audit trail, the recovery, and the acknowledgement boundary all at once. The journal is small because it only does that one thing. It is also the reason anything above it can be trusted.

sources

  • vhalla
  • crates/vhalla-journal/src/lib.rs: the commit protocol, Store trait, FaultingStore, and recovery rules
  • crates/vhalla-journal/src/tests.rs: crash-injection tests at each Step
  • crates/vhalla-ledger/src/lib.rs: the derived-root event chain below consensus
  • crates/vhalla-rooms-consensus/src/lib.rs: journal-first durability order above the adapter
  • crates/vhalla-native/src/spent.rs: VSN1 republish in miniature
  • crates/vhalla-social-store/src/lib.rs and crates/vhalla-rooms-store/src/lib.rs: the snapshot stores using the same discipline