You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e2a7a09
Browse filesBrowse the repository at this point in the historyBrowse files
One final run per size is published; run-to-run variability was not measured.
223
+
Memory and disk sizes use GiB; input labels are nominal targets
224
+
(1 GB = 1,000 MiB of extracted text). A page is approximately 2,000 bytes.
225
+
Chunks are Qdrant's approximate count at completion; the per-size curves
226
+
retain the maximum observed count used by the completion detector.
227
+
228
+
Documents flow through the pipeline rather than accumulating in it.
229
+
The file-watch index retains metadata per watched file to detect edits and
230
+
deletions, so its cost depends on file count. The right panel estimates this
231
+
cost from the connector worker's extra private memory: the fitted slope is
232
+
**~361 bytes per file**. This proxy also includes unequal worker
233
+
buffers; it does not isolate individual allocations.
234
+
235
+
The embedding runtime, engine and working buffers account for the worker
236
+
baseline. The curves show how total memory evolves during indexing,
237
+
including any steps as file-watch metadata and buffers grow.
238
+
Parsed-text cache, persistence snapshots and stored embeddings live on disk.
239
+
Across these runs a **500× larger input target costs 2.1× the peak PSS**.
240
+
Both file count and text volume change between sizes; this comparison does
241
+
not establish constant memory usage.
240
242
241
243
<palign="center">
242
-
<imgsrc="https://raw.githubusercontent.com/pathwaycom/serviette/main/docs/assets/bench-memory.png"alt="Left: indexer PSS over time for corpora from 100 MB to 50 GB, from the run before the allocator tuning; every curve reaches its plateau in the first minutes and holds it for hours. Right: connector-worker extra memory across six corpus sizes follows ~318 bytes per watched file"width="100%">
244
+
<imgsrc="docs/assets/bench-memory.png"alt="Measured indexer PSS over time for all six corpora; legends use the same peaks as the table. Right: estimated file-watch memory versus file count, fitted at approximately 361 bytes per file."width="100%">
243
245
</p>
244
-
<palign="center"><em>Curves from the run <strong>before</strong> the allocator tuning (peaks 6.6–16.4 GB); the table above has the final numbers from the tuned rerun (6.6–13.1 GB). The shape is the point: each curve reaches its plateau in the first minutes and holds it for the rest of the run — 5.5 hours for 50 GB.</em></p>
245
-
246
-
The peak itself is dominated by the embedding stack, not the engine — a
247
-
Pathway worker process is ~200 MB; the rest is the price of running
248
-
embeddings locally (8 × PyTorch runtime + model), i.e. of paying no
249
-
per-token API fees. Fewer workers or an API embedder shrink it accordingly.
246
+
<palign="center"><em>All curves and table values come from the same runs with allocator background purging enabled. File-watch memory is estimated from rank-0 private-memory excess over the median of the other workers.</em></p>
250
247
251
248
<palign="center">
252
-
<imgsrc="https://raw.githubusercontent.com/pathwaycom/serviette/main/docs/assets/bench-memory-breakdown.png"alt="Breakdown of the 10 GB corpus peak as measured in the run before the allocator tuning (8.8 GB): three quarters is the local PyTorch embedding stack across 8 workers; file-watch metadata is about 1.1 GB; supervisors and shared code make up the rest"width="85%">
249
+
<imgsrc="docs/assets/bench-memory-breakdown.png"alt="Breakdown at the 10 GB run's 8.16 GiB PSS peak: estimated local-embedder private and shared memory, estimated file-watch memory, remaining private memory and other shared pages."width="85%">
253
250
</p>
254
-
<palign="center"><em>Same earlier run (8.8 GB peak on the 10 GB corpus; 7.7 GB after tuning). The proportions are what matters here: the embedding stack dominates, the engine and the file-watch index are small.</em></p>
255
-
256
-
Memory is measured as PSS (proportional set size) summed over the container:
257
-
shared pages — e.g. the PyTorch libraries mapped by every worker — are
258
-
counted once, not once per process. Setup: 96-core CPU host, streaming mode,
259
-
8 worker processes, local `static-retrieval-mrl-en-v1` embeddings (no API
260
-
calls; Matryoshka-truncated to 256 dims), 512-token chunks, Qdrant, and
261
-
jemalloc's `background_thread` purging enabled in the indexer containers
262
-
(measured free; it keeps idle workers from retaining freed pages). Numbers
263
-
were measured on a nightly Pathway build whose engine matches the released
264
-
wheel (pathway ≥ 0.32.1 — what the benchmark's docker image and the
265
-
Development section install), so they are reproducible as-is.
251
+
<palign="center"><em>Attribution at the sampled 10 GB peak. Local-embedder memory is estimated from a separate eight-process initialization diagnostic and library mappings; libraries already needed without the embedder remain in the residual. The 3.27 GiB estimate without the local embedder is calculated by subtraction, not measured in an external-embedder indexing run. It excludes the embedding service and does not measure client overhead or changes in working buffers. Shared pages are accounted for proportionally through PSS.</em></p>
252
+
253
+
Setup: 96-logical-CPU host (see the saved CPU model),
254
+
streaming mode, 8 worker processes with 12 OpenMP threads each,
255
+
local `static-retrieval-mrl-en-v1` embeddings, Matryoshka-truncated to 256 dims,
256
+
512-token chunks, Qdrant, and jemalloc `background_thread:true`.
257
+
No embedding API calls. Installed versions: serviette 0.1.7,
258
+
Pathway 0.33.0, PyTorch 2.14.1,
259
+
sentence-transformers 6.1.0.
260
+
Peak memory is the maximum sampled PSS summed over the indexer container.
261
+
Qdrant and API-server memory, unmapped filesystem cache and kernel memory
262
+
are outside that measurement; it is not a RAM requirement for the whole stack.
263
+
Brief peaks between samples may be missed.
264
+
265
+
Indexing time is an estimate ending at the last increase in the running
266
+
maximum of Qdrant's approximate chunk count. That moment can precede the
267
+
final write; output-completeness checks do not revise the timestamp or
268
+
measure when the backend's search index is ready. The recorded environment,
269
+
cache conditions and limits on reproducing it are described in
0 commit comments