Merge pull request #51 from osgeonepal/feat/hybrid-history

feat(hisotry): add osm hisotry for the stats

Author: kshitijrajsharma

.github/workflows/windows-exe.yml

@@ -0,0 +1,43 @@
+name: windows-exe
+
+on:
+  push:
+    tags: ["v*", "*.*.*"]
+  workflow_dispatch:
+
+permissions:
+  contents: write
+
+jobs:
+  build:
+    runs-on: windows-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install osmsg + PyInstaller
+        run: |
+          uv venv
+          uv pip install . pyinstaller
+
+      - name: Build osmsg.exe
+        run: uv run pyinstaller --clean --noconfirm packaging/pyinstaller/osmsg.spec
+
+      - name: Smoke-test the exe
+        run: dist\osmsg.exe --version
+
+      - name: Upload build artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: osmsg-windows-x64
+          path: dist/osmsg.exe
+
+      - name: Attach to release
+        if: startsWith(github.ref, 'refs/tags/')
+        uses: softprops/action-gh-release@v2
+        with:
+          files: dist/osmsg.exe

.gitignore

@@ -9,3 +9,19 @@ dist
 node_modules/
 stats.*
 *.parquet
+
+.claude/
+
+# experiment artifacts (large)
+experiments/parquet-history/cache/
+experiments/parquet-history/data/
+experiments/parquet-history/out/
+experiments/parquet-history/out_demo/
+experiments/parquet-history/planet_work/
+
+# QA / scratch (generated)
+experiments/parquet-history/qa_alltime/
+experiments/parquet-history/*_work/
+/qa/
+
+CLAUDE.md

.pre-commit-config.yaml

@@ -11,6 +11,7 @@ repos:
       - id: check-merge-conflict
       - id: check-toml
       - id: check-yaml
+        exclude: ^packaging/conda/meta\.yaml$  # conda recipe uses Jinja templating, not plain YAML
       - id: check-json
       - id: fix-byte-order-marker
 

README.md

@@ -14,13 +14,15 @@ of nodes, ways, and relations created, modified, or deleted, written to parquet,
 
 A Project of [OSGeo Nepal](https://osgeonepal.org).
 
-## Features
+## What does it do?
 
 - Per-user create/modify/delete counts over any time window.
 - Tag and hashtag breakdowns (e.g. `building`, `#hotosm`).
 - Country and custom-boundary filters via Geofabrik.
 - Cron-friendly resume with `--update`.
+- One-command setup: `osmsg --insert` loads all history into your store, `osmsg --update` keeps it current.
 - Outputs you can query: parquet, csv, json, markdown, DuckDB, Postgres.
+- Cloud-native history: months covered by a published parquet dataset are read remotely.
 
 ## Install
 
@@ -36,6 +38,16 @@ docker run --rm -v "$PWD:/work" -w /work ghcr.io/osgeonepal/osmsg:latest --last
 `uvx` can run osmsg in a throwaway environment , no install, no virtualenv to manage. Works
 with any flag combination, e.g. `uvx --from osmsg osmsg --last hour --tags building --summary -f parquet -f markdown`.
 
+More ways to install:
+
+```bash
+conda install -c conda-forge osmsg                 # conda / mamba
+brew install osgeonepal/tap/osmsg          # macOS / Linux (Homebrew tap)
+```
+
+On Windows, download `osmsg.exe` from the [latest release](https://github.com/osgeonepal/osmsg/releases)
+and run it directly, no Python required.
+
 ## Quick start
 
 ```bash
@@ -46,6 +58,38 @@ osmsg --hashtags hotosm --last day       # only changesets tagged #hotosm
 
 That's it. A `stats.duckdb` and a `stats.parquet` show up in your current folder.
 
+## Set up a full history store
+
+Two commands give you a complete, self-updating store. The first loads all of OSM history from the
+published dataset and records where to resume; the second catches up to now and runs on a schedule.
+
+```bash
+osmsg --insert            # load all history into stats.duckdb, then exit
+osmsg --update            # catch up to now (repeat on cron)
+```
+
+`osmsg` clears the multi-week backlog on day diffs, then refines to finer diffs as the store stays
+current. For near-real-time, run `osmsg --update --url minute`.
+
+Pick your store with one flag. DuckDB is the default (`stats.duckdb`); add a DSN for Postgres:
+
+```bash
+osmsg --insert --psql-dsn "postgresql://user:pass@localhost/osmsg"
+osmsg --update --psql-dsn "postgresql://user:pass@localhost/osmsg"
+```
+
+Load only a slice with `--start/--end`; `--update` then continues from the end of that slice:
+
+```bash
+osmsg --insert --start 2020-01-01 --end 2023-01-01
+```
+
+Already have the planet files? Insert from them directly:
+
+```bash
+osmsg --insert --osh-file history-latest.osh.pbf --changeset-file changesets-latest.osm.bz2
+```
+
 ## Tutorials
 
 ### 1. Stats for a country
@@ -153,13 +197,31 @@ docker-compose `environment:` block all reach the same setting. CLI flag wins ov
 | `--output-dir` | `OSMSG_OUTPUT_DIR` | `.` | Where `<name>.duckdb` and exports are written. |
 | `--format` / `-f` | `OSMSG_FORMAT` | `parquet` | Repeat for multiple. Comma-separated when set via env. |
 | `--psql-dsn` | `OSMSG_PSQL_DSN` | unset | libpq DSN for `-f psql`. |
+| `--psql-bulk` | `OSMSG_PSQL_BULK` | off | Faster first full load to Postgres. |
+| `--history` / `--no-history` | `OSMSG_HISTORY` | on | Read covered months from the published dataset. |
+| `--history-url` | `OSMSG_HISTORY_URL` | `osmsg-history` | Published dataset location. |
+| `--insert` | (none) | off | Load history into the store and seed resume, then exit. No window loads all of it. |
+| `--osh-file` / `--changeset-file` | (none) | unset | Insert from local planet history + changeset files instead of the dataset. |
 | `--changeset-pad-hours` | `OSMSG_CHANGESET_PAD_HOURS` | `1` | See below. |
 | (auto-bootstrap on `--update`) | `OSMSG_BOOTSTRAP` | `hour` | `hour`, `day`, or `week`. Used when `--update` runs against an empty DB. |
 | (auto-bootstrap on `--update`) | `OSMSG_BOOTSTRAP_DAYS` | unset | Integer N; overrides `OSMSG_BOOTSTRAP`. |
 | OSM credentials (Geofabrik) | `OSM_USERNAME`, `OSM_PASSWORD` | unset | Required only when a Geofabrik URL is in use. |
 
 A `.env` file at the working directory is loaded automatically.
 
+## Maintainers
+
+Generating and publishing the history dataset is the `osmsg maintain` group:
+
+```bash
+osmsg maintain month 2026-06 --repo osgeonepal/osmsg-history   # append one finished month
+osmsg maintain month 2026-06 --no-upload                       # generate locally, review, upload later
+osmsg maintain convert history.osh.pbf changesets.osm.bz2 2005-01-01 2026-06-01 work --parts 24
+osmsg maintain publish work/out --repo osgeonepal/osmsg-history
+```
+
+See [experiments/parquet-history](./experiments/parquet-history/README.md) for the full-history batch.
+
 ## Documentation
 
 - [Installation](./docs/Installation.md)

api/routers/v1.py

@@ -44,7 +44,7 @@ async def get_user_stats(
             list[str] | None, Parameter(description="Filter to changesets carrying any of these hashtags. Repeatable.")
         ] = None,
         tags: Annotated[bool, Parameter(description="Include per-user tag_stats breakdown in the response.")] = True,
-        limit: Annotated[int, Parameter(ge=1, le=1000, description="Page size (1–1000).")] = 100,
+        limit: Annotated[int, Parameter(ge=1, le=1000, description="Page size (1 to 1000).")] = 100,
         offset: Annotated[int, Parameter(ge=0, description="Page offset.")] = 0,
     ) -> UserStatsResponse:
         start = start or (datetime.min.replace(tzinfo=UTC) if end else None)

docs/Manual.md

@@ -25,7 +25,7 @@ osmsg --country nepal --country india --country africa   # Geofabrik regions, re
 ```
 
 > When `--url` is omitted, osmsg picks a planet replication granularity that fits the requested
-> span: minute for spans under 6h, hour for 6h–7d, day for ≥7d. A warning prints when the
+> span: minute for spans under 6h, hour for 6h to 7d, day for 7d or more. A warning prints when the
 > auto-switch happens; pass `--url` explicitly to override (also suppressed by `--country`,
 > `--update`, or multiple `--url` values).
 
@@ -41,7 +41,7 @@ osmsg --boundary '{"type":"Polygon",...}'     # inline GeoJSON string
 ```
 
 > `--boundary` filters changesets whose bounding box intersects the given geometry.
-> A Geofabrik region name resolves from the same index as `--country` — no separate file needed.
+> A Geofabrik region name resolves from the same index as `--country`, no separate file needed.
 > `--boundary` only filters; it does not change the replication source.
 > To scope the replication source to a country's diffs, use `--country` instead.
 >
@@ -107,6 +107,72 @@ Downloaded `.osc.gz` files cache to a per-user dir (`~/Library/Caches/osmsg` on
 `~/.cache/osmsg` on Linux). Re-running the same range reuses them, so no network is needed.
 `--cache-dir` to relocate, `--delete-temp` to clean up after a run.
 
+## Setting up a store
+
+`--insert` loads history into the store and seeds the resume position, then exits. Follow it with
+`--update` to catch up to now and keep current. DuckDB is the default store; pass `--psql-dsn` to use
+Postgres (no separate `-f psql` needed).
+
+```bash
+osmsg --insert                                   # load all published history into stats.duckdb
+osmsg --update                                   # catch up to now, then run on cron
+
+osmsg --insert --psql-dsn "host=localhost dbname=osm user=osm"   # into Postgres (bulk first load)
+osmsg --insert --start 2020-01-01 --end 2023-01-01               # a slice; --update continues from its end
+osmsg --insert --osh-file history.osh.pbf --changeset-file changesets.osm.bz2  # from local files
+```
+
+- No window loads the whole dataset; `--start/--end` loads a slice and resumes from the slice end.
+- `--osh-file` with `--changeset-file` converts local planet files into the store (offline, or a custom
+  extract). Give both together.
+- The Postgres load uses the bulk path (drops indexes and keys, rebuilds after).
+
+`--insert` and `--update` pick the replication granularity from how far behind the store is. A fresh
+store clears the multi-week backlog on day diffs (tens of files), then refines to hour and minute as it
+stays current. For near-real-time, run `osmsg --update --url minute`. A store tracks one granularity at
+a time; changing it hands off at the day boundary, so the windows stay disjoint. Pass `--url` to either
+command to set the granularity yourself.
+
+## Cloud-native history
+
+Months covered by a published parquet dataset (default `kshitijrajsharma/osmsg-history` on
+HuggingFace) are read remotely. The recent uncovered tail uses the live replication path. This is on
+by default.
+
+```bash
+osmsg --start 2015-01-01 --end 2020-01-01     # read from the dataset
+osmsg --start 2024-01-01                       # covered months remote, current month live
+osmsg --last week --no-history                  # live path only
+```
+
+- `--no-history` (env `OSMSG_HISTORY=0`) uses the live path.
+- `--history-url` (env `OSMSG_HISTORY_URL`) sets the dataset location.
+- The live path is used when the dataset is unreachable, with `--update`, and with `--length`.
+
+### Postgres as a source of truth
+
+`osmsg --insert --psql-dsn ...` loads the dataset into osmsg's schema and seeds the resume position;
+`osmsg --update --psql-dsn ...` then keeps Postgres current. `--psql-bulk` (env `OSMSG_PSQL_BULK`)
+forces the bulk path on a plain run; `--insert` already uses it.
+
+## Maintaining the dataset
+
+`osmsg maintain` builds and publishes the history parquet.
+
+```bash
+osmsg maintain month 2026-06 --repo osgeonepal/osmsg-history   # build one finished month and upload
+osmsg maintain month 2026-06 --no-upload                       # build locally, review, upload later
+osmsg maintain publish out --repo osgeonepal/osmsg-history     # write + upload manifest.json
+osmsg maintain convert history.osh.pbf changesets.osm.bz2 2005-01-01 2026-06-01 work --parts 24
+```
+
+`month` builds from the live day diffs, exports the two partitions, uploads, and advances the
+manifest. It refuses to publish a month whose data stops short of the month boundary (pass
+`--allow-incomplete` to override), so published months are complete by construction. Re-running
+`osmsg maintain month <YYYY-MM>` rebuilds a month and overwrites its published partition, which repairs
+a month that was first generated from a mid-day planet snapshot. `convert` turns local planet files
+into the datasets out of core. Uploads use the `hf` CLI (`uvx`), so be logged in to HuggingFace.
+
 ## Credentials
 
 `--country` and any `geofabrik` URL need OSM credentials. Resolution order:

docs/infra.md

@@ -43,18 +43,18 @@ $EDITOR infra/.env
 
 | Variable | Default | Notes |
 | --- | --- | --- |
-| `OSMSG_DOMAIN` | `localhost` | Your domain — enables automatic HTTPS via Caddy |
+| `OSMSG_DOMAIN` | `localhost` | Your domain, enables automatic HTTPS via Caddy |
 | `OSMSG_SCHEDULE` | `*/2 * * * *` | supercronic cron expression |
 | `OSMSG_BOOTSTRAP` | `hour` | First-run window: `hour`/`day`/`week`/`month`/`year` |
 | `OSMSG_BOOTSTRAP_DAYS` | _unset_ | Exact day count for first run (alternative to `OSMSG_BOOTSTRAP`) |
 | `OSM_USERNAME` | _unset_ | OSM account username (required for Geofabrik country replication) |
 | `OSM_PASSWORD` | _unset_ | OSM account password (required for Geofabrik country replication) |
-| `OSMSG_EXTRA_ARGS` | _see example_ | osmsg args applied on every tick — country, format, tags, boundary, etc. |
+| `OSMSG_EXTRA_ARGS` | _see example_ | osmsg args applied on every tick (country, format, tags, boundary, etc.) |
 
-`OSMSG_EXTRA_ARGS` runs on every tick. Do not put `--last`, `--days`, or `--update` here —
+`OSMSG_EXTRA_ARGS` runs on every tick. Do not put `--last`, `--days`, or `--update` here;
 tick adds those automatically based on whether state exists.
 
-Geofabrik sub-daily replication uses your OSM credentials directly — no browser opt-in required.
+Geofabrik sub-daily replication uses your OSM credentials directly, no browser opt-in required.
 
 ### Start
 
@@ -75,7 +75,7 @@ docker compose pull && docker compose up -d
 
 ## Run as a systemd service
 
-Only the `infra/` directory needs to be on the server — no source code or build tools required.
+Only the `infra/` directory needs to be on the server; no source code or build tools required.
 
 **1. Place files:**
 
@@ -85,7 +85,7 @@ cp infra/docker-compose.yml infra/Caddyfile infra/osmsg.service /opt/osmsg/infra
 cp infra/.env.example /opt/osmsg/infra/.env
 $EDITOR /opt/osmsg/infra/.env
 
-# The pgdata Docker volume binds to /mnt — create the directory first
+# The pgdata Docker volume binds to /mnt, create the directory first
 mkdir -p /mnt/osmsg/pgdata
 ```
 

experiments/parquet-history/README.md

@@ -0,0 +1,43 @@
+# Historical parquet datasets
+
+osmsg precomputes OSM history into time-partitioned parquet published on HuggingFace, so past-window
+queries read remotely instead of re-downloading replication diffs. The read side is built into osmsg
+(`osmsg --start ... --end ...` serves covered months from the dataset, `osmsg --insert` loads it into
+a store). The build and publish side is the `osmsg maintain` subcommand group; this directory holds
+only the server batch script that wraps it.
+
+## Datasets
+
+Two datasets, both partitioned `year=*/month=*` and Morton(centroid)-sorted for time and bbox pruning:
+
+- **changefiles**, per-changeset counts + poi + `tag_stats` JSON + `created_at` + bbox.
+- **changesets**, per-changeset metadata: uid, username, `created_at`, editor, hashtags, bbox.
+
+## Build the full history (server batch)
+
+```bash
+./planet_batch.sh <start YYYY-MM-DD> <end YYYY-MM-DD>
+```
+
+Downloads `history-latest.osm.pbf` + `changesets-latest.osm.bz2`, time-filters with `osmium`, then
+runs `osmsg maintain convert` to stream and aggregate out of core. Needs the `osmium` CLI and ~150 GB
+free. The conversion alone is `osmsg maintain convert <osh> <changesets> <start> <end> <work_dir>
+--parts N`.
+
+## Publish and maintain
+
+```bash
+osmsg maintain publish <out_dir> --drop-last --repo <repo>   # write + upload manifest.json
+osmsg maintain month <YYYY-MM> --repo <repo>                 # append one finished month
+osmsg maintain month <YYYY-MM> --no-upload                   # generate locally, review, upload later
+```
+
+`osmsg maintain month` builds the month from the live day diffs, exports the two partitions, uploads
+them, and advances the manifest. It refuses to publish a month that stops short of its boundary
+(`--allow-incomplete` overrides), and re-running it rebuilds and overwrites a month, which repairs one
+first generated from a mid-day planet snapshot.
+
+## Load into a store
+
+`osmsg --insert` loads the published parquet into an osmsg DuckDB or Postgres store and seeds the
+resume position; see the [Manual](../../docs/Manual.md).

experiments/parquet-history/planet_batch.sh

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+# Planet batch: download once, time-filter (C++), convert out-of-core to changefiles/changesets parquet.
+# Server job, not in-session: the history file is ~148 GB and the streaming pass is hours.
+#
+# Usage: ./planet_batch.sh <start YYYY-MM-DD> <end YYYY-MM-DD> [history_url]
+#
+# history_url defaults to the planet full-history directory's latest file. Confirm the current name at
+# https://planet.openstreetmap.org/pbf/full-history/ (the dated history-YYMMDD.osh.pbf). Needs the
+# osmium CLI (osmium-tool) for time-filter and ~150 GB free for the download plus the windowed extract.
+set -euo pipefail
+
+start="${1:?start date YYYY-MM-DD}"
+end="${2:?end date YYYY-MM-DD}"
+history_url="${3:-https://planet.openstreetmap.org/pbf/full-history/history-latest.osm.pbf}"
+changeset_url="https://planet.openstreetmap.org/planet/changesets-latest.osm.bz2"
+
+here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+work="$here/planet_work"
+mkdir -p "$work"
+
+command -v osmium >/dev/null || { echo "osmium-tool required (time-filter). Install osmium-tool."; exit 1; }
+
+echo ">>> [1/4] download history + changeset dump (resumable)"
+curl -fSL -C - -o "$work/history.osh.pbf" "$history_url"
+curl -fSL -C - -o "$work/changesets.osm.bz2" "$changeset_url"
+
+echo ">>> [2/4] time-filter history to [$start, $end] (C++, avoids Python touching all versions)"
+osmium time-filter -O -o "$work/history-window.osh.pbf" \
+    "$work/history.osh.pbf" "${start}T00:00:00Z" "${end}T00:00:00Z"
+
+echo ">>> [3/4] stream + aggregate out-of-core to parquet"
+uv run --project "$here/../.." osmsg maintain convert \
+    "$work/history-window.osh.pbf" "$work/changesets.osm.bz2" "$start" "$end" "$work" --parts 24
+
+echo ">>> [4/4] done. datasets in $work/out/{changefiles,changesets}."
+echo "    publish: uv run --project $here/../.. osmsg maintain publish $work/out --repo <repo>"