Skip to content

Commit 5df557b

Browse files
committed
docs(comparison): add continent-scale Europe section
First continent-scale run lands on the comparison page. Hetzner bare-metal host (48 cores, 124 GB RAM, NVMe LVM at /mnt/fast), Geofabrik europe-latest.osm.pbf at 32 GB. Cairn OSM ingest finishes in 4 h 25 m: pass 0 (max-id scan) → pass 1 (flatnode write) → pass 2a (node-place emit) → pass 2b0 (relation way-ref scan) → pass 2b1 (parallel way-place emit, 22.16 M ways emitted) → pass 2b2 (admin relations, 363 131 polygons after dedup, 25.45 M places after dedup). The bundle-write tail (admin tiles, text index, place tiles, point tiles, manifest, SBOM) is in flight at commit time; final disk / hot RSS / p95 land in the next update. The Phase 6e v2 build pipeline is what makes this run feasible on the contended cluster — single shared AdminIndex across both enrichment passes, admin-tile write moved ahead of the tantivy + place-tile peaks (so the polygon ring set is freed before allocation pressure rises), and interleaved place-tile + point-tile emission off one sorted slice (no Vec<PlacePoint> materialization intermediate). Plus rkyv size_64 to lift the 2 GB rel-pointer ceiling on Russia / Norway / Greece-scale admin tile archives. Pelias / Photon / Nominatim Europe measurements: not yet run. Projected ranges noted in the table (12-18 h, 6-10 h, 24-48 h respectively) based on incumbent docs and our country-scale observations. Full parity benches ship in a follow-up. Caveat block notes the operational knobs needed on this specific cluster (pause airflow Python workers during the build window; bump vm.max_map_count to 1 048 576 so Cairn's mmap pressure doesn't run into the 65 530 default). They're host-config, not algorithmic, and don't show up on country-scale runs.
1 parent 7bb41d1 commit 5df557b

1 file changed

Lines changed: 79 additions & 0 deletions

File tree

comparison/index.html

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -454,6 +454,85 @@ <h3>Recall on noisy queries</h3>
454454
gotchas is at
455455
<a href="https://github.com/cairn-geocoder/cairn/blob/main/benchmarks/results/REPORT.md"><code>benchmarks/results/REPORT.md</code></a>.
456456
</p>
457+
458+
<h3>Continent scale — Europe</h3>
459+
<p>
460+
First continent-scale run, on a different host: a Hetzner
461+
bare-metal box with 48 cores, 124 GB RAM, 32 GB swap (rotating)
462+
+ 64 GB NVMe swap, NVMe LVM volume mounted at
463+
<code>/mnt/fast</code>. Input is Geofabrik
464+
<code>europe-latest.osm.pbf</code> (32 GB), 7× larger than
465+
Germany. Cairn runs against this PBF through the same
466+
<code>cairn-build</code> binary as the country-scale runs,
467+
with rkyv <code>size_64</code> rel-pointers (lifts the per-tile
468+
2 GB ceiling that 32-bit relative pointers impose on Russia /
469+
Norway-scale admin polygons) and the Phase 6e streaming
470+
pipeline (single shared AdminIndex, admin-tile write before
471+
tantivy + place-tile write, interleaved place-tile + point-tile
472+
emission off the same sorted slice).
473+
</p>
474+
<table class="api">
475+
<thead>
476+
<tr>
477+
<th>Engine</th>
478+
<th>OSM ingest</th>
479+
<th>Places after dedup</th>
480+
<th>Admin polygons</th>
481+
<th>Bundle disk</th>
482+
<th>Hot RSS</th>
483+
<th>p95</th>
484+
</tr>
485+
</thead>
486+
<tbody>
487+
<tr>
488+
<td><strong>Cairn</strong></td>
489+
<td><strong>4 h 25 m</strong></td>
490+
<td><strong>25.45 M</strong></td>
491+
<td><strong>363 131</strong></td>
492+
<td><em>pending — build in flight</em></td>
493+
<td><em>pending</em></td>
494+
<td><em>pending</em></td>
495+
</tr>
496+
<tr>
497+
<td>Pelias</td>
498+
<td colspan="6"><em>not run yet — projected ~12–18 h on the same hardware (Elasticsearch import + WhosOnFirst + OpenAddresses + OSM importers run sequentially via docker-compose; 32 GB PBF dominates the OSM importer pass)</em></td>
499+
</tr>
500+
<tr>
501+
<td>Photon</td>
502+
<td colspan="6"><em>not run yet — projected ~6–10 h (Nominatim DB import is the long pole; Photon's Elasticsearch index build is fast once the Postgres source exists)</em></td>
503+
</tr>
504+
<tr>
505+
<td>Nominatim</td>
506+
<td colspan="6"><em>not run yet — projected ~24–48 h on a single Postgres instance (osm2pgsql import + indexes; cluster has the I/O headroom but the runtime is single-process bound)</em></td>
507+
</tr>
508+
</tbody>
509+
</table>
510+
<p>
511+
Cairn's OSM ingest on Europe — pass 0 (max-id scan) → pass 1
512+
(flatnode write, 110 GB virtual / ~88 GB on NVMe sparse) →
513+
pass 2a (node-place emit) → pass 2b0 (relation way-ref scan)
514+
→ pass 2b1 (parallel way-place emit, 458 M ways scanned,
515+
22.16 M emitted) → pass 2b2 (admin relations, 363 K polygons
516+
out of 8.31 M relations) — completes in 4 h 25 m on this box,
517+
sustaining ~2300 % CPU (≈ 23 cores) through the
518+
rayon-parallel passes. Final bundle write (admin tiles, text
519+
index, place tiles, point tiles, manifest, SBOM) runs after
520+
OSM ingest finishes; full numbers + the corresponding
521+
Pelias / Photon / Nominatim measurements ship in the next
522+
update.
523+
</p>
524+
<p class="caveat">
525+
The cluster also runs an Airflow-driven planetary-imagery
526+
pipeline on the same host. We pause those workers
527+
(<code>kill -KILL</code> the Python pool, airflow respawns
528+
them after the build window) and bump
529+
<code>vm.max_map_count</code> to 1 048 576 so Cairn's mmap
530+
pressure (110 GB flatnode + per-tile output mappings + tantivy
531+
segment files) doesn't run into the 65 530 default limit.
532+
Both knobs are operational, not algorithmic — they don't show
533+
up in country-scale runs because the cluster has more
534+
headroom there.
535+
</p>
457536
</div>
458537
</section>
459538

0 commit comments

Comments
 (0)