hraness

consensus for a group chat: why raft assumes things your peers can't promise

agreement among a handful of flaky writers

Drafted by an AI agent at Ben Guo's direct request from the Hraness source repositories, and checked against those sources before publication.

Consensus is the word peer-to-peer systems reach for when they need a group to agree on something. For a room (a chat space, a shared directory, a multiplayer game), the temptation is to reach for Raft, because Raft is the consensus algorithm people actually understand. Then the deployment reality arrives: your “servers” are laptops that sleep, browsers that close, peers behind NATs that drop in and out. Raft’s assumptions do not survive contact with that.

Valhalla’s room layer shows both the reasoning and the result. It uses real Byzantine consensus (Malachite, a Tendermint-family engine), but only where consensus is actually needed, with a private validator set, and with a durable journal between the engine’s decision and any acknowledgement. The interesting part is not which algorithm was picked. It is what the system decided consensus was for, and what it refused to let a certificate mean.

what consensus is for here

The first thing the workspace does is shrink the problem. The blockchain architecture plan (kb/plans/valhalla-blockchain-architecture.md) has a table that asks, for each claim, what primitive it needs and whether a global chain is required. Signatures, receipts, “these peers saw the same result,” membership changes: none of those need a worldwide order. Only two do: “this voucher has not been spent twice” (and only within its spending domain) and “every participant agrees on one worldwide order.” Everything else stays as signed evidence.

So what the rooms nodes run consensus on is narrow: a shared room directory (the registry of rooms, their records, their admitted state) where the validators are a known, private set. Chat messages do not go through consensus. Neither do social records, discovery hits, or attention notifications. The thing being agreed is a sequence of batches that mutate the registry, and the thing being agreed about is small enough that a handful of validators can decide it.

why raft can’t run on a room

Raft assumes three things a room of peers cannot promise.

Stable membership. Raft’s cluster is a fixed set; adding a server is a joint-consensus operation that itself requires the cluster to be alive. A room’s members are whoever is online. Valhalla’s rooms do not even try: the validator set is private and configured, not derived from who showed up.

A leader with a durable log. Raft’s leader owns the log for its term; followers trust its ordering and replicate it. A peer that sleeps mid-term, a partition that isolates the leader, a follower whose disk is a browser’s IndexedDB. Raft’s model has no room for “the leader is a laptop that closed.” Tendermint-family consensus replaces the leader’s authority with rounds of voting: proposers rotate, validators prevote and precommit, and a value commits when more than two-thirds of voting power signs it. No single peer’s durability is load-bearing.

Crash-fault, not Byzantine. Raft assumes failed nodes stop: they do not lie. A peer-to-peer room cannot assume that. A validator that signs two conflicting precommits is not crashed; it is adversarial. BFT consensus exists precisely for this: the quorum threshold (>2/3 voting power) tolerates up to a third of the set being wrong in any way: crashed, partitioned, or actively malicious.

what was chosen instead

The implementation is vhalla-rooms-node: Malachite engines (pinned to a specific commit of circlefin/malachite, the arc-malachitebft-* crates) on libp2p networking, deciding vhalla-rooms-consensus batches. The engine runs the full Tendermint stack: proposals stream in parts, validators prevote and precommit by round, and a CommitCertificate collects the signatures that crossed the quorum.

The part that is not the engine is the boundary around it. vhalla-rooms-consensus is the adapter between the engine’s Decided/Finalized events and the durable application state, and its rule is stated in the crate doc: the acknowledgement is emitted only after the journal commit and the store publishes succeed. The engine decides; the adapter decides whether the decision is durable. A certificate is a claim about a quorum. It is not, by itself, a commit.

the certificate is not the commit

The certificate verification in cert.rs is a good example of what “verify” means when you take it seriously. verify_commit_certificate does not trust the engine’s verdict. It re-checks the evidence:

  • The round must be non-Nil (a Nil round decided nothing).
  • At most MAX_CERT_SIGNATURES = 64 signatures; empty and oversized both fail.
  • Every signer must be in the trusted validator set and sign at most once (UnknownSigner, DuplicateSigner).
  • Each signature is verified over a rebuilt precommit (RoomVote::new(Precommit, height, round, Val(value_id), address)), the canonical RV1 sign bytes, not the bytes the certificate happened to carry.
  • Distinct voting power must exceed two-thirds: signed_power * 3 > total * 2, else BelowQuorum.

Then the certificate is canonicalized to VC2 bytes (magic, height, round, the full 32-byte value_id, count, and (address || signature) pairs), and those bytes are what the journal bundle stores. A replica that never ran the engine can verify the canonical form against the validator set with verify_canonical_certificate: the same checks, on the bytes, independently.

And still, the certificate is not the commit. Adapter::decide in vhalla-rooms-consensus does the full sequence: check whether the journal already committed this height to this value (reconcile if so, reject if it committed a different value, which is equivocation at a committed height), look up the held batch by value commitment, replay the batch against the pinned frontier and compare the claimed result digests, then, only then, write the bundle through the journal’s commit protocol, publish the social and rooms stores in that order, apply in memory, and send CommitAck. DecidedOutcome has three values: Acked, Rejected, Withheld. The middle one exists because “the engine decided but the evidence doesn’t check out” is a real outcome, and the third because “the durable write is uncertain” is a third.

The promotion-gates plan records the regression test that pins the distinction: two conflicting certificates that both verify are constructed deliberately, because “certificate validity is not consensus, ancestry, finality, or freshness.” A certificate proves a quorum signed something. It does not prove it is the thing, that nothing conflicting was signed, or that it is durable.

the parts that actually break

The implementation detail is where the room-scale constraints show. Proposals are big (a batch is up to MAX_BATCH_BYTES = 48 KiB) and gossipsub coalesces bursts into single writes that the relayed transport truncates above roughly 1.1–1.2 KiB. So PROPOSAL_CHUNK_BYTES is 768 and PART_PUBLISH_SPACING is 20 milliseconds: the parts are deliberately small and deliberately paced, because the transport’s real behavior, not its specification, is the constraint.

Liveness has its own machinery. GetValue requests carry the deadline the engine assigned; past it, the connector resolves the reply with a tombstone: a value derived from b"VHTOMB" plus the validator’s address, height, and round, that can never validate and is never published. A held reply would park the sequential connector forever; a dropped one kills the actor. The tombstone is the third option: answer with something that cannot commit, so the engine moves on.

The WAL is versioned (VRW2) because the WAL is what lets a restarted node resume its votes, and an old format’s votes would be worse than none. The validator’s durable state is split the same way the trust is: the engine’s WAL restores votes and locks, the node’s own store/ restores the values those votes referred to, and the journal is the authority on what committed. Three kinds of state, three owners, each restored independently.

what the evidence shows

The honest status is in the plan: this is a private validator set, not open membership, not a public room, not qualified for the public internet. The qualification harness (NetGate, WalPlan/WalFault) exists to drive partition and fault-injection evidence in tests, and vhalla-rooms-consensus’s own tests replay committed batches against stores and check divergence fails closed. The four-validator partition evidence in the blockchain plan is explicit about its limits: it verifies quorum boundaries, “not BFT finality or partition safety.”

That is the answer to “why not Raft” that the code actually gives. Not “Raft is bad”; Raft is fine for what it assumes. The answer is that a room of flaky peers fails every assumption Raft makes, and a BFT engine with a private validator set, a verified certificate that is still not a commit, and a journal that gates the acknowledgement is what the failure modes actually require. Consensus for a group chat is not smaller than consensus for a blockchain. It is the same problem, scoped to who is allowed to vote and what they are voting on.

sources

  • vhalla
  • crates/vhalla-rooms-node/src/cert.rs: verify_commit_certificate and the VC2 canonical form
  • crates/vhalla-rooms-consensus/src/lib.rs: the journal-gated decide path and DecidedOutcome
  • crates/vhalla-rooms-node/src/unix.rs: PROPOSAL_CHUNK_BYTES, PART_PUBLISH_SPACING, tombstones, VRW2
  • crates/vhalla-rooms-node/Cargo.toml: the Malachite pin
  • valhalla-blockchain-architecture.md: which claims need consensus at all
  • valhalla-promotion-gates.md: “certificate validity is not consensus, ancestry, finality, or freshness”

keep reading: free for subscribers

the rest of this lesson is free. add your email once and every subscriber lesson on this site stays unlocked.

already subscribed? enter the same email to unlock.