Skip to content

Commit b249992

Browse files
fix(mongodb): partition the catch-up probe by _id range, breaking the mongot->mongod wire-limit ceiling (#313) (#314)
* fix(mongodb): partition the catch-up probe by _id range, breaking the mongot->mongod wire-limit ceiling (#313) The exhaustive catch-up probe added in #309 asks mongot to score the whole corpus in one $vectorSearch+$count call. $count only shrinks the CLIENT reply — mongot still streams one entry per matched document to mongod first, and that internal hop hits mongod's own message-size limits above roughly 888k documents (BSONObjectTooLarge / ProtocolError), which is every dataset this suite ships. Live against Atlas 8.0.30 on glove-100-angular (1,183,514 docs), every config failed identically after ~3h, even though the index itself had finished building. Partition 0..expected_count into bounded _id ranges (dense and sequential for every dataset reader), probe each range independently with a new _id filter field on the index, and sum — with adaptive bisection if a partition still overflows the wire. `want` stays the full uncapped expected_count throughout, so this doesn't loosen #305's guarantee, only how the completeness count is transported. Validated live against a real Atlas M10 + 2x S10_HIGHCPU cluster: a 1,000,000-document corpus (comfortably past the ~888k ceiling) reached 100% confirmed coverage in 276s, with index_coverage_fraction: 1.0 recorded in the published result. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * style(mongodb): run cargo fmt Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
1 parent 246d005 commit b249992

2 files changed

Lines changed: 598 additions & 48 deletions

File tree

0 commit comments

Comments
 (0)