Skip to content

Merge Worker

The HTAP bridge: folds frozen delta generations into columnar segments in the background.

Cycle

  1. Freeze & rotate — the active generation (log + memtable) is frozen; a fresh one takes over. Frozen memtables keep serving reads until their rows are published in segments.
  2. Group — frozen rows are grouped per (table, channel, day-partition); a table's tombstones force a rewrite of all its channel segments.
  3. Fold — per partition, the existing segment (if any) merges with the sorted new rows into *.tmp:
    • immutable: 2-way ordered merge; exact duplicates dropped (crash-replay idempotency)
    • mutable: newest version per entity wins; entities whose tombstone is newer vanish
  4. Publish — per channel, all partition renames + registry update + dropping the frozen rows happen in one critical section, so a concurrent reader never sees rows twice or not at all.
  5. Prune — with every covered segment durable, the old WAL files are deleted.

Triggers & throttle

Size (-merge-bytes, default 8 MiB of delta) or time (-merge-every, default 30s), whichever first. Segment writing is paced by -merge-throttle (bytes/s) so a merge never starves foreground ingest on small hardware.

Retention

The same worker drops expired day-partitions (whole directories) for tables declared with RETENTION.