|
1 | 1 | # osmsg |
2 | 2 |
|
3 | | -OpenStreetMap stats generator. UTC-only, parquet-first, OAuth 2.0. |
| 3 | +Generate OpenStreetMap user stats from the command line. Point it at a time window, get back per-user counts of nodes/ways/relations created, modified, and deleted, in parquet, csv, json, markdown, or straight into Postgres. |
4 | 4 |
|
5 | 5 | ## Install |
6 | 6 |
|
7 | 7 | ```bash |
8 | 8 | pip install osmsg |
9 | | -# or |
| 9 | +# or, as a standalone CLI |
10 | 10 | uv tool install osmsg |
11 | | -# or |
| 11 | +# or, no install |
12 | 12 | docker run --rm -v "$PWD:/work" -w /work ghcr.io/osgeonepal/osmsg:latest --last hour |
13 | 13 | ``` |
14 | 14 |
|
15 | | -## Quick start (CLI) |
| 15 | +## Examples |
16 | 16 |
|
17 | 17 | ```bash |
18 | | -# Last hour, planet replication |
| 18 | +# What happened in the last hour, planet-wide |
19 | 19 | osmsg --last hour |
20 | 20 |
|
21 | | -# Country-level (live Geofabrik index lookup, OAuth 2.0) |
22 | | -export OSM_USERNAME=... OSM_PASSWORD=... # or use a .env |
| 21 | +# Yesterday's stats for a country (needs OSM credentials, see below) |
23 | 22 | osmsg --country nepal --last day |
24 | 23 |
|
25 | | -# Custom range + per-key tag totals + daily summary |
26 | | -osmsg --start "2026-04-01 00:00:00" --end "2026-04-08 00:00:00" \ |
| 24 | +# Custom range, with per-key tag breakdowns and a daily summary |
| 25 | +osmsg --start "2026-04-01" --end "2026-04-08" \ |
27 | 26 | --tags building --tags highway --summary |
28 | 27 |
|
29 | | -# Long flag lists are easier as YAML; CLI args still override |
30 | | -osmsg --config nepal.yaml --rows 50 |
| 28 | +# Only changesets tagged #hotosm (substring by default; --exact-lookup for whole-word) |
| 29 | +osmsg --hashtags hotosm --last day |
31 | 30 |
|
32 | | -# Cron-friendly: resume from where the last run left off |
| 31 | +# Cron-friendly: pick up where the last run left off |
33 | 32 | osmsg --country nepal --update |
34 | 33 | ``` |
35 | 34 |
|
36 | | -## Library usage |
| 35 | +YAML configs work too if your flag list gets long: `osmsg --config nepal.yaml`. |
37 | 36 |
|
38 | | -```python |
39 | | -from datetime import datetime, UTC |
| 37 | +## Output |
40 | 38 |
|
41 | | -from osmsg import RunConfig, run, OsmsgError |
| 39 | +Every run writes `stats.duckdb` (or `<--name>.duckdb`) plus whatever formats you ask for via `-f parquet|csv|json|markdown|psql`. Parquet is the default. Open it with duckdb, polars, pandas, whatever. |
42 | 40 |
|
43 | | -cfg = RunConfig( |
44 | | - name="nepal", |
45 | | - countries=["nepal"], |
46 | | - start_date=datetime(2026, 4, 25, tzinfo=UTC), |
47 | | - end_date=datetime(2026, 4, 26, tzinfo=UTC), |
48 | | - formats=["parquet", "psql"], |
49 | | - psql_dsn="host=localhost dbname=osm user=osm", |
50 | | -) |
51 | | -try: |
52 | | - result = run(cfg) # OSM credentials picked up from OSM_USERNAME / OSM_PASSWORD |
53 | | -except OsmsgError as exc: |
54 | | - ... |
55 | | - |
56 | | -print(result["files"]["parquet"]) # → 'nepal.parquet' |
57 | | -print(result["rows"]) # → user count |
58 | | -``` |
59 | | - |
60 | | -Query a stored database (DuckDB file or Postgres) without re-running: |
61 | | - |
62 | | -```python |
63 | | -from osmsg import connect, user_stats, daily_summary |
64 | | - |
65 | | -conn = connect("nepal.duckdb") |
66 | | -top_10 = user_stats(conn, top_n=10) |
67 | | -days = daily_summary(conn) |
| 41 | +```bash |
| 42 | +duckdb stats.duckdb -c "SELECT username, SUM(nodes_created) AS n |
| 43 | + FROM users JOIN changeset_stats USING (uid) |
| 44 | + GROUP BY username ORDER BY n DESC LIMIT 10" |
68 | 45 | ``` |
69 | 46 |
|
70 | | -Typed exceptions (`OsmsgError` base, plus `UnknownRegionError` / `CredentialsRequiredError` / `GeofabrikAuthError` / `NoDataFoundError`) are catchable by callers; the CLI maps them to exit codes (2 = config/auth, 1 = no data). The package ships a `py.typed` marker for `mypy` / `ty`. |
71 | | - |
72 | | -## Output formats |
73 | | - |
74 | | -`-f parquet` (default) `-f csv` `-f json` `-f markdown` `-f psql` |
75 | | - |
76 | | -Every run writes a portable `<name>.duckdb` (queryable with the duckdb CLI) plus the formats you asked for. Parquet is the canonical exchange format — open it directly with DuckDB / polars / pandas. |
77 | | - |
78 | | -```sql |
79 | | --- Query the duckdb file from anywhere |
80 | | -duckdb stats.duckdb -c "SELECT name, map_changes FROM ( |
81 | | - SELECT u.username AS name, SUM(s.nodes_created+s.ways_created) AS map_changes |
82 | | - FROM users u JOIN changeset_stats s USING (uid) GROUP BY 1 ORDER BY 2 DESC LIMIT 10 |
83 | | -)" |
84 | | -``` |
| 47 | +The schema is the same in DuckDB and Postgres. Four tables: `users`, `changesets`, `changeset_stats`, and `state` (the resume marker for `--update`). |
85 | 48 |
|
86 | 49 | ## Credentials |
87 | 50 |
|
88 | | -`--country` and Geofabrik internal URLs need OSM credentials (OAuth 2.0; OAuth 1.0a was retired June 2024). Public planet replication (`--url minute|hour|day`) needs none. Resolution order: |
| 51 | +`--country` (and Geofabrik URLs) need an OSM account; public planet replication (`--url minute|hour|day`) doesn't. |
89 | 52 |
|
90 | | -1. `--username` (CLI) + `--password-stdin` (one line on stdin), or `osm_username` / `osm_password` (`RunConfig`) |
91 | | -2. `OSM_USERNAME` / `OSM_PASSWORD` env vars (auto-loaded from `.env`) |
92 | | -3. Interactive `getpass` prompt — TTY only. Headless library callers raise `CredentialsRequiredError`. |
| 53 | +Set `OSM_USERNAME` and `OSM_PASSWORD` in your environment or a `.env` file. Or pass `--username` and pipe the password to `--password-stdin`. OAuth 2.0 happens behind the scenes. |
93 | 54 |
|
94 | | -Passwords are not accepted as a plain CLI flag — they would leak into shell history and `ps` output. |
| 55 | +## Library |
95 | 56 |
|
96 | | -## Schema |
97 | | - |
98 | | -Four tables, identical in DuckDB and PostgreSQL — write once, query anywhere. |
| 57 | +```python |
| 58 | +from datetime import datetime, UTC |
| 59 | +from osmsg import RunConfig, run |
99 | 60 |
|
100 | | -| Table | Purpose | |
101 | | -|---|---| |
102 | | -| `users` | uid → username | |
103 | | -| `changesets` | metadata: hashtags, editor, bbox | |
104 | | -| `changeset_stats` | flat counts + nested `tag_stats` JSON | |
105 | | -| `state` | resume marker — exactly one row per source_url (UPSERTed each run) | |
| 61 | +result = run(RunConfig( |
| 62 | + name="nepal", |
| 63 | + countries=["nepal"], |
| 64 | + start_date=datetime(2026, 4, 25, tzinfo=UTC), |
| 65 | + end_date=datetime(2026, 4, 26, tzinfo=UTC), |
| 66 | +)) |
| 67 | +print(result["files"]["parquet"]) |
| 68 | +``` |
106 | 69 |
|
107 | | -`map_changes` is computed at query time as the sum of the nine element columns (`{nodes,ways,rels}_{created,modified,deleted}`); POI counters are tracked separately and never folded in. |
| 70 | +That's the same pipeline the CLI runs. See [docs/Manual.md](./docs/Manual.md) for everything else. |
108 | 71 |
|
109 | | -## Developer setup |
| 72 | +## Develop |
110 | 73 |
|
111 | 74 | ```bash |
112 | 75 | git clone https://github.com/osgeonepal/osmsg && cd osmsg |
113 | 76 | uv sync |
114 | | -uv run pytest -m "not network" |
| 77 | +uv run pytest |
115 | 78 | uv run osmsg --help |
116 | 79 | ``` |
117 | | - |
118 | | -See [docs/Manual.md](./docs/Manual.md) for the flag reference and [docs/Installation.md](./docs/Installation.md) for environment notes. |
0 commit comments