Skip to content

Commit fc31d00

Browse files
fix(ci): fix lib creds on ci
1 parent c2e8bb9 commit fc31d00

3 files changed

Lines changed: 48 additions & 75 deletions

File tree

README.md

Lines changed: 36 additions & 75 deletions
Original file line numberDiff line numberDiff line change
@@ -1,118 +1,79 @@
11
# osmsg
22

3-
OpenStreetMap stats generator. UTC-only, parquet-first, OAuth 2.0.
3+
Generate OpenStreetMap user stats from the command line. Point it at a time window, get back per-user counts of nodes/ways/relations created, modified, and deleted, in parquet, csv, json, markdown, or straight into Postgres.
44

55
## Install
66

77
```bash
88
pip install osmsg
9-
# or
9+
# or, as a standalone CLI
1010
uv tool install osmsg
11-
# or
11+
# or, no install
1212
docker run --rm -v "$PWD:/work" -w /work ghcr.io/osgeonepal/osmsg:latest --last hour
1313
```
1414

15-
## Quick start (CLI)
15+
## Examples
1616

1717
```bash
18-
# Last hour, planet replication
18+
# What happened in the last hour, planet-wide
1919
osmsg --last hour
2020

21-
# Country-level (live Geofabrik index lookup, OAuth 2.0)
22-
export OSM_USERNAME=... OSM_PASSWORD=... # or use a .env
21+
# Yesterday's stats for a country (needs OSM credentials, see below)
2322
osmsg --country nepal --last day
2423

25-
# Custom range + per-key tag totals + daily summary
26-
osmsg --start "2026-04-01 00:00:00" --end "2026-04-08 00:00:00" \
24+
# Custom range, with per-key tag breakdowns and a daily summary
25+
osmsg --start "2026-04-01" --end "2026-04-08" \
2726
--tags building --tags highway --summary
2827

29-
# Long flag lists are easier as YAML; CLI args still override
30-
osmsg --config nepal.yaml --rows 50
28+
# Only changesets tagged #hotosm (substring by default; --exact-lookup for whole-word)
29+
osmsg --hashtags hotosm --last day
3130

32-
# Cron-friendly: resume from where the last run left off
31+
# Cron-friendly: pick up where the last run left off
3332
osmsg --country nepal --update
3433
```
3534

36-
## Library usage
35+
YAML configs work too if your flag list gets long: `osmsg --config nepal.yaml`.
3736

38-
```python
39-
from datetime import datetime, UTC
37+
## Output
4038

41-
from osmsg import RunConfig, run, OsmsgError
39+
Every run writes `stats.duckdb` (or `<--name>.duckdb`) plus whatever formats you ask for via `-f parquet|csv|json|markdown|psql`. Parquet is the default. Open it with duckdb, polars, pandas, whatever.
4240

43-
cfg = RunConfig(
44-
name="nepal",
45-
countries=["nepal"],
46-
start_date=datetime(2026, 4, 25, tzinfo=UTC),
47-
end_date=datetime(2026, 4, 26, tzinfo=UTC),
48-
formats=["parquet", "psql"],
49-
psql_dsn="host=localhost dbname=osm user=osm",
50-
)
51-
try:
52-
result = run(cfg) # OSM credentials picked up from OSM_USERNAME / OSM_PASSWORD
53-
except OsmsgError as exc:
54-
...
55-
56-
print(result["files"]["parquet"]) # → 'nepal.parquet'
57-
print(result["rows"]) # → user count
58-
```
59-
60-
Query a stored database (DuckDB file or Postgres) without re-running:
61-
62-
```python
63-
from osmsg import connect, user_stats, daily_summary
64-
65-
conn = connect("nepal.duckdb")
66-
top_10 = user_stats(conn, top_n=10)
67-
days = daily_summary(conn)
41+
```bash
42+
duckdb stats.duckdb -c "SELECT username, SUM(nodes_created) AS n
43+
FROM users JOIN changeset_stats USING (uid)
44+
GROUP BY username ORDER BY n DESC LIMIT 10"
6845
```
6946

70-
Typed exceptions (`OsmsgError` base, plus `UnknownRegionError` / `CredentialsRequiredError` / `GeofabrikAuthError` / `NoDataFoundError`) are catchable by callers; the CLI maps them to exit codes (2 = config/auth, 1 = no data). The package ships a `py.typed` marker for `mypy` / `ty`.
71-
72-
## Output formats
73-
74-
`-f parquet` (default) `-f csv` `-f json` `-f markdown` `-f psql`
75-
76-
Every run writes a portable `<name>.duckdb` (queryable with the duckdb CLI) plus the formats you asked for. Parquet is the canonical exchange format — open it directly with DuckDB / polars / pandas.
77-
78-
```sql
79-
-- Query the duckdb file from anywhere
80-
duckdb stats.duckdb -c "SELECT name, map_changes FROM (
81-
SELECT u.username AS name, SUM(s.nodes_created+s.ways_created) AS map_changes
82-
FROM users u JOIN changeset_stats s USING (uid) GROUP BY 1 ORDER BY 2 DESC LIMIT 10
83-
)"
84-
```
47+
The schema is the same in DuckDB and Postgres. Four tables: `users`, `changesets`, `changeset_stats`, and `state` (the resume marker for `--update`).
8548

8649
## Credentials
8750

88-
`--country` and Geofabrik internal URLs need OSM credentials (OAuth 2.0; OAuth 1.0a was retired June 2024). Public planet replication (`--url minute|hour|day`) needs none. Resolution order:
51+
`--country` (and Geofabrik URLs) need an OSM account; public planet replication (`--url minute|hour|day`) doesn't.
8952

90-
1. `--username` (CLI) + `--password-stdin` (one line on stdin), or `osm_username` / `osm_password` (`RunConfig`)
91-
2. `OSM_USERNAME` / `OSM_PASSWORD` env vars (auto-loaded from `.env`)
92-
3. Interactive `getpass` prompt — TTY only. Headless library callers raise `CredentialsRequiredError`.
53+
Set `OSM_USERNAME` and `OSM_PASSWORD` in your environment or a `.env` file. Or pass `--username` and pipe the password to `--password-stdin`. OAuth 2.0 happens behind the scenes.
9354

94-
Passwords are not accepted as a plain CLI flag — they would leak into shell history and `ps` output.
55+
## Library
9556

96-
## Schema
97-
98-
Four tables, identical in DuckDB and PostgreSQL — write once, query anywhere.
57+
```python
58+
from datetime import datetime, UTC
59+
from osmsg import RunConfig, run
9960

100-
| Table | Purpose |
101-
|---|---|
102-
| `users` | uid → username |
103-
| `changesets` | metadata: hashtags, editor, bbox |
104-
| `changeset_stats` | flat counts + nested `tag_stats` JSON |
105-
| `state` | resume marker — exactly one row per source_url (UPSERTed each run) |
61+
result = run(RunConfig(
62+
name="nepal",
63+
countries=["nepal"],
64+
start_date=datetime(2026, 4, 25, tzinfo=UTC),
65+
end_date=datetime(2026, 4, 26, tzinfo=UTC),
66+
))
67+
print(result["files"]["parquet"])
68+
```
10669

107-
`map_changes` is computed at query time as the sum of the nine element columns (`{nodes,ways,rels}_{created,modified,deleted}`); POI counters are tracked separately and never folded in.
70+
That's the same pipeline the CLI runs. See [docs/Manual.md](./docs/Manual.md) for everything else.
10871

109-
## Developer setup
72+
## Develop
11073

11174
```bash
11275
git clone https://github.com/osgeonepal/osmsg && cd osmsg
11376
uv sync
114-
uv run pytest -m "not network"
77+
uv run pytest
11578
uv run osmsg --help
11679
```
117-
118-
See [docs/Manual.md](./docs/Manual.md) for the flag reference and [docs/Installation.md](./docs/Installation.md) for environment notes.

pyproject.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ dependencies = [
1515
"pyarrow>=24.0.0",
1616
"pydantic>=2.13.3",
1717
"python-dotenv>=1.2.2",
18+
"pytz>=2024.1",
1819
"requests>=2.32.5",
1920
"rich>=13.0",
2021
"shapely>=2.1.2",

uv.lock

Lines changed: 11 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)