Commit b249992
* fix(mongodb): partition the catch-up probe by _id range, breaking the mongot->mongod wire-limit ceiling (#313)
The exhaustive catch-up probe added in #309 asks mongot to score the whole
corpus in one $vectorSearch+$count call. $count only shrinks the CLIENT
reply — mongot still streams one entry per matched document to mongod first,
and that internal hop hits mongod's own message-size limits above roughly
888k documents (BSONObjectTooLarge / ProtocolError), which is every dataset
this suite ships. Live against Atlas 8.0.30 on glove-100-angular (1,183,514
docs), every config failed identically after ~3h, even though the index
itself had finished building.
Partition 0..expected_count into bounded _id ranges (dense and sequential
for every dataset reader), probe each range independently with a new _id
filter field on the index, and sum — with adaptive bisection if a partition
still overflows the wire. `want` stays the full uncapped expected_count
throughout, so this doesn't loosen #305's guarantee, only how the
completeness count is transported.
Validated live against a real Atlas M10 + 2x S10_HIGHCPU cluster: a
1,000,000-document corpus (comfortably past the ~888k ceiling) reached
100% confirmed coverage in 276s, with index_coverage_fraction: 1.0 recorded
in the published result.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* style(mongodb): run cargo fmt
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
1 parent 246d005 commit b249992
2 files changed
Lines changed: 598 additions & 48 deletions
0 commit comments