On July 29, 2026 we chased a panic — "BUG: idx too big" — from a crash-looping desktop daemon all the way down to a single 1.3 MB change blob, and found out it had also been quietly crashing the production gateway for days. This is the story of the bug, the three ways it hurt us, and the three fixes.

The symptom

After merging the directory-listing performance work (PR #900), one team machine started crash-looping at boot: the backfill reindex would run for about five minutes, then the daemon died with a panic — BUG: idx too big — deep inside the document CRDT. Every restart replayed the same crash, because the reindex retries until it completes. The panic unwound the entire reindex transaction, so the work was rolled back every time. We reverted #900 and #904 from main while investigating.

The root cause: quadratic op IDs

Every operation inside a document change gets an op ID of (timestamp, index, actor). The index is supposed to count ops within one change. But the apply loop advances it like this for multi-block ops:

Adding i on every iteration instead of 1 makes the index grow quadratically: a move op carrying n blocks consumes about n²/2 index values. The index was capped at 2^24−1 (16.7M) — a leftover from a packed 15-byte op ID encoding (48-bit timestamp, 24-bit index, 48-bit actor) that was removed in 2024, but the cap and its panic stayed. With quadratic growth, a single move op of only ~5,800 blocks reaches the cap.

The document that found the bug

The trigger is real content: a large legal-code import under the "demo publish" account. One of its subdocuments — the Spanish Código Civil of 1889 — was published as a single change containing 6,642 block replacements and one MoveBlocks op with 6,411 blocks. Applying that op crosses the 2^24−1 ceiling at block #5,791 (the index peaks around 20.5M), and newOpID panics. We verified this by pulling the exact blob out of the production database and simulating the apply loop.

Notably, today's writer cannot author such a change — creating a change re-applies its own ops, so it would have panicked at authoring time. The blob came from an older or different writer. But content is content-addressed and immutable: once a poisonous blob exists, it syncs everywhere and stays forever.

Three ways it hurt

1. Boot crash-loop. PR #900 added an indexer step that rebuilds each document in memory (to derive a fallback cover image) for every document Ref — including during the one-time backfill reindex at boot. On any machine holding the poisonous blob, boot → reindex → panic → restart → repeat. The machine is bricked until the code changes.

2. Production crashes — before #900 ever shipped there. The production gateway runs release images that never contained #900, yet its daemon panicked 7 times in 72 hours with the same "idx too big". The stack traces show plain GetDocument calls: the daemon has no gRPC panic-recovery interceptor, so serving that one document kills the whole process. Anyone opening the page — or any crawler — takes the gateway down. A public URL that works as a denial-of-service button.

3. The revert trap. Reverting #900/#904 removed their storage migrations from the code. But any machine that had already booted the merged code had recorded the newer migration version, and the migration runner refuses to run when the data dir version is newer than the code ("OLD VERSION: cannot be downgraded"). So affected machines were stuck both ways: the new build crash-looped, and the reverted build refused to boot. Reverting code that shipped migrations is not actually a revert for anyone who ran it.

The fixes

PR #908 — reland #900/#904 with a panic-safe deriver. The cover-image derivation now recovers panics into an error; the indexer already treated deriver errors as best-effort (log, leave the field unset, clients fall back to fetching content). One bad blob now costs a log line and a missing thumbnail instead of the daemon. Relanding also restored the migration versions, un-bricking the machines caught by the revert trap.

PR #909 — fail to apply instead of panicking. The apply loop now validates the op index (and timestamp) and returns an error, so GetDocument on an oversized document fails that one request while the daemon keeps running. The write path keeps its panic: locally-authored ops count linearly and cannot legitimately reach the limit, so a panic there still means a real bug.

PR #910 — widen the index so the document actually loads. The 24-bit cap is policy, not representation: op IDs live in a plain in-memory struct and serialize as full uint64s on the wire; they are recomputed on every load and never persisted. So we widened the field to int64 and raised the cap to 2^31−1, which allows ~65,000 blocks in a single move op — 10× the largest seen. No migration, no reindex, and documents that load today produce identical op IDs. We verified by replaying the real Código Civil blobs from production through the new code: the document applies and hydrates completely.

Still open

The quadratic numbering itself stays, deliberately: op IDs are derived deterministically at apply time, and later changes reference earlier ops by these IDs, so renumbering would change identity for every multi-block op in existing documents. Fixing it for real is a versioned protocol change. There is also a related write/apply asymmetry (the writer numbers ops linearly while apply numbers them quadratically) worth resolving in the same effort.

Rollout order matters: nodes on old builds still panic (pre-#909) or error (post-#909) on documents that exceed their cap. The graceful-error release should be widely deployed before anyone edits the previously-frozen documents, because new changes built on high-index ops would be unappliable on old builds. Production needs a release cut to pick any of this up. A gRPC panic-recovery interceptor would also be cheap defense-in-depth.

Lessons

Invariant panics are only safe on paths where the process owns the data. The same check that is a bug-catcher on the write path is a denial-of-service on the network-input path. Data arriving from the network must always fail soft.

When you delete a representation, delete its limits — or turn them into validated errors. The 24-bit cap outlived the encoding it protected by two years and kept its panic the whole time.

Content-addressed networks never forget. A single malformed-but-signed blob, authored once by an old client, will keep resurfacing on every node that syncs it, forever. Robustness has to live on the read path.

And a revert is not a revert once migrations have shipped. If a migration went out, roll forward.