Skip to content

Delta Log (WAL)

The write buffer is the write-ahead log — one mechanism, not two.

Framing

u32 payload length | u32 CRC32-C(payload) | payload

Payloads are varint-encoded RowDelta records: table, flags, entity, ts plus the channels that arrived (id, kind, value). A tombstone is a flag bit with no channels.

Group commit

Appenders enqueue frames; the first one in a batch kicks the flusher, which lets the -group-commit window elapse so followers pile in, then performs one write+fsync and releases everyone. Append returns only after its batch is durable. 0 disables batching (fsync per append).

Generations & pruning

Log files are numbered (wal/<seq>.log). Each file pairs with the memtable holding exactly its rows — a generation. A merge:

  1. freezes the active generation, opens seq+1,
  2. waits for in-flight appends to land,
  3. folds the frozen memtable into segments (tmp→rename),
  4. deletes every log file ≤ seq — only after all covered segments are durable.

Recovery

On open, all surviving logs replay in order into a fresh memtable. Scanning stops at the first torn or CRC-corrupt frame and truncates the file there — the classic crash artifact of a half-written tail. If segments already contain some replayed rows (crash between rename and log delete), the next merge drops exact duplicates: re-merging is idempotent.