[jak-pan]

Fixes #808.

When a segment seals its inverted index, four defects can make the seal lose an error or keep an incomplete range index. After that, range queries (<, <=, >, >=) silently return too few documents. One of them also crashes the process after a single failed MANIFEST sync. This PR fixes the four defects, with one commit and one regression test per defect.

Problem on main

• 1. A column seal error is lost. InvertedIndexer::seal() logs a failed column seal and goes on. flush() and compact() then overwrite the status, so the seal returns OK and the segment is sealed with that column unsealed.
• 2. A seal marker can reach disk before the range index it describes. The indexes write without a WAL, so data becomes durable only when its column family is flushed. The seal writes the marker into the default family, next to the new range and histogram data, and nothing orders their flushes. A writable open() lists the default family first, so RocksdbContext::flush() writes the markers first. That happens, for example, to the writing segment's index after a restart. A crash in between keeps a sealed column with only part of its range index, and a later seal skips the column because it is already sealed.
• 3. reset_cf() keeps a deleted handle. It deletes the handle once DropColumnFamily() succeeds. It replaces the handle in cf_handles_ only if CreateColumnFamily() succeeds. After a failed re-create, the next flush() or close() uses the deleted handle. Together with defect 1, one failed MANIFEST sync during a seal crashes the process with SIGSEGV in DBImpl::Flush.
• 4. A read error ends the term scan as if it were the last term. generate_statistical_indexes() loops while iter_term->Valid() and never checks status(). A read error in the middle of the term table builds the range index from the terms before it, and the column is sealed. If the error was transient, the seal's compaction succeeds and nothing reports it.

Changes

• Return the column error (inverted_indexer.cc, inverted_column_indexer.h, inverted_column_indexer_write.cc): InvertedColumnIndexer::seal() is split into prepare_seal() and mark_sealed(). prepare_seal() writes the special values, the range index and the histogram, and mark_sealed() writes the marker. seal() still runs both.
  • InvertedIndexer::seal() prepares every unsealed column and returns the first error, as flush() already does. It marks the columns sealed only after all of them were prepared.
  • So a failed seal leaves no column read-only. The caller keeps writing to that index, and returning at the first failed column would leave the columns before it sealed and rejecting inserts.
• Write the markers after their data (same files): InvertedIndexer::seal() flushes between preparing and marking. InvertedColumnIndexer::seal() flushes between prepare_seal() and mark_sealed().
  • Each Flush() waits until its tables are recorded in the MANIFEST, so a marker can no longer reach disk before its data, whatever RocksDB flushes later.
  • InvertedIndexer::seal() adds one flush per seal. Most of that data was flushed anyway by the flush that follows.
  • I first flushed the default family last in flush_unlocked(). Review showed that this orders only that explicit flush: an automatic flush of a full default memtable during the seal could still go first. The flush order in flush_unlocked() is therefore unchanged.
• Remove the deleted handle (rocksdb_context.cc, +2 lines): after a failed re-create, reset_cf() erases the handle from cf_handles_, as drop_cf() does.
  • The family stays dropped, as it was before this change. get_cf() returns null, and the seal fails with an error.
  • On the next open, the missing family is reported, as after a crash between the drop and the create ([Bug]: A crash while a segment seals can leave a collection that cannot be opened ("Column family[...$RANGES] does not exist") #798).
• Check the term iterator (inverted_column_indexer_write.cc, +14 lines): the seal fails if the iterator's status is not OK after the scan.
  • It also fails if SeekToLast() does not find a term. There is at least one term at that point, so an invalid iterator means an error, and the check guards the key() read that follows.

Tests

tests/db/index/column/inverted_column/inverted_indexer_seal_error_test.cc, new:

Test	98aa126	this PR
SealReturnsColumnSealError	FAIL	PASS
SealFlushesItsDataBeforeTheMarker	FAIL	PASS
FailedResetKeepsNoDeletedHandle	FAIL	PASS
ColumnSealFailsOnTermReadError	FAIL	PASS

• SealReturnsColumnSealError stores a term value that cannot be decoded in one of four columns. On 98aa126, seal() returns OK, and the other three columns are sealed and reject inserts.
• SealFlushesItsDataBeforeTheMarker seals a column and then reads the memtable entry counts. On 98aa126, the range family still holds 1,000 entries in memory next to the marker. With this PR, only the marker is left.
• FailedResetKeepsNoDeletedHandle fails the re-create's MANIFEST sync. On 98aa126, the deleted handle stays in cf_handles_.
• ColumnSealFailsOnTermReadError fails the reads of the middle of a 100,000-term table during the column seal. It then seals again without the fault and checks three range queries. On 98aa126, the column seal returns OK.
• Two rocksdb::FileSystemWrapper classes in the test file inject the faults, through the public create_opts_.env of RocksdbContext. Nothing in src/ changes for the tests.
• Not covered by a test: the check after SeekToLast(). After a successful scan, the last block is in the block cache, so a test cannot make that read fail.
• Each commit builds, and its own tests pass.

Longer reproductions, not part of this PR, pass with these changes. On 98aa126 they fail:

• Crash during a seal at a default-family flush. An index of 4,500,000 documents is reopened with default options, sealed, and killed.
  • On 98aa126, the crash comes right after the flush that carries the marker. 768 of 1,000 range entries survive, and 82 of 164 range queries are wrong.
  • With this PR, a crash at the first default-family flush leaves the column unsealed with exact answers. A crash at the second one, which carries the marker, leaves it sealed with all 1,000 entries. In both cases 0 queries are wrong.
  • The same holds with 200,000 documents and a 64 KB range memtable.
• One failed MANIFEST sync during InvertedIndexer::seal(). On 98aa126 the process crashes with SIGSEGV. With this PR, seal() returns the error.
• Read error during the column seal, followed by the segment seal. On 98aa126, 40 of 164 range queries are wrong after a reopen. With this PR, the column seal fails, the segment seal then succeeds, and 0 queries are wrong.

Existing suites with this PR (macOS arm64, CMake Release), all passing:

• the six inverted_column_indexer_* tests and inverted_indexer_util_test;
• fts_column_indexer_test (93) and fts_rocksdb_reducer_test (18);
• segment_test (108), segment_helper_test (15), segment_row_id_test (10) and add_column_batch_boundaries_test (1);
• fts_multi_segment_test (3), write_recovery_test (7), optimize_recovery_test (1) and collection_test (104).

Merging with #802 and #803

• fix(index): empty the range column family in place at seal #803 ([Bug]: A crash while a segment seals can leave a collection that cannot be opened ("Column family[...$RANGES] does not exist") #798) also changes inverted_column_indexer_write.cc: it replaces the two reset_cf() calls of the seal with an in-place clear. This PR touches other parts of that file: the end of the term loop, SeekToLast(), and seal(). fix(index): empty the range column family in place at seal #803 keeps reset_cf() as a fallback for an unreadable range family, and fix 3 covers that fallback.
• perf(db): create RocksDB column families in one call #802 ([Enhance]: Creating a RocksDB instance rewrites the OPTIONS file once per column family #799) also changes rocksdb_context.cc, but only create(). This PR changes only reset_cf() there.
• git merge-tree reports no conflict with either PR, so whichever merges second needs only a rebase.
• With all three merged, the tests of this PR (4), perf(db): create RocksDB column families in one call #802 (3) and fix(index): empty the range column family in place at seal #803 (5) pass, together with the sequential (8) and cyclic (7) number suites.

Out of scope

• SegmentImpl ignores the results of column_indexer->seal() and invert_indexers_->flush() when it adds or alters a column (segment.cc, lines 3254 and 3406). There, the segment meta is already updated before the seal, so returning the error needs a decision on how to roll the meta back. Not changed here.
• A seal that fails after some markers were written still leaves those columns read-only while the segment keeps accepting writes. A later insert then fails in such a column after the id map and the forward store have taken the document.
  • On main this already happens when the final flush or compaction of the seal fails. This PR adds one rare way into it: a failed marker write into the memtable.
  • The fix belongs to segment write recovery. Not changed here.
• The crash window between the drop and the create in reset_cf(): [Bug]: A crash while a segment seals can leave a collection that cannot be opened ("Column family[...$RANGES] does not exist") #798 / fix(index): empty the range column family in place at seal #803.

Not covered

• Linux and Windows: not built or run here. The tests use no platform-specific code.
• The tests ran on a 98aa126 tree with this PR's files. Every file this PR touches is byte-identical between 98aa126 and main (7f54d94).
• clang-format 21.1.8 was used; the repository pins 18.1.8, which is not installed here.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-12T09:23:38.235009Z	a8e804d	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.