Skip to content

Commit e1a8f40

Browse files
feat(api): serve export data on data.devartslab.com via Worker + R2
Pull model: the Worker cron (*/15m) syncs both export files from GitHub into R2; GET /v1/* serves filtered JSON from the bucket. A Bearer token (PRIVATE_API_TOKEN secret) upgrades responses to the private dataset. /v1/admin/sync forces a resync. No CF credentials needed on the GitHub side. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
1 parent f3d3eaa commit e1a8f40

7 files changed

Lines changed: 2117 additions & 1 deletion

File tree

‎.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,8 @@ htmlcov/
2323

2424
# Never commit the private export to this public repo
2525
data/repos.private.json
26+
27+
# Node / worker
28+
node_modules/
29+
.wrangler/
30+
worker/.dev.vars

‎README.md‎

Lines changed: 42 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,9 @@ export.config.yaml --> github-export (GraphQL v4) --> data/repos.public.json
1111
| --> data/repos.private.json (pushed to a private repo)
1212
v
1313
categories | overrides | filters
14+
15+
GitHub Action (every 6h) writes the files; the data.devartslab.com Worker
16+
(cron */15m) pulls them into R2 and serves a filtered JSON API on top.
1417
```
1518

1619
A scheduled GitHub Action runs the exporter every 6 hours (or on demand via `workflow_dispatch`), validates the output against the JSON schema, and commits `data/repos.public.json` back to this repo. `repos.private.json` is pushed to a separate private repo (see [Private output](#private-output)).
@@ -26,6 +29,7 @@ A scheduled GitHub Action runs the exporter every 6 hours (or on demand via `wor
2629

2730
| Endpoint | URL | Notes |
2831
| ------------ | ------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------- |
32+
| **API** | `https://data.devartslab.com` | Cloudflare Worker + R2, filtered endpoints, ~15 min freshness |
2933
| raw | `https://raw.githubusercontent.com/DevArtsLab/tool-github-export/main/data/repos.public.json` | always fresh, rate-limited lightly |
3034
| jsDelivr CDN | `https://cdn.jsdelivr.net/gh/DevArtsLab/tool-github-export@main/data/repos.public.json` | cached, fast, CORS-friendly; pin `@<commit>` for deterministic reads |
3135
| schema | `https://raw.githubusercontent.com/DevArtsLab/tool-github-export/main/schemas/repos.schema.json` | JSON Schema 2020-12 |
@@ -56,6 +60,43 @@ data = httpx.get(
5660

5761
See `examples/consumer.html` for a working browser page (serve it with `uv run python -m http.server`).
5862

63+
## API: data.devartslab.com
64+
65+
The `worker/` directory is a Cloudflare Worker serving both exports from R2 at `https://data.devartslab.com`. It syncs from GitHub every 15 minutes (cron) and also accepts `POST /v1/admin/sync` to force a refresh.
66+
67+
| Endpoint | Description |
68+
| ------------------------------ | ----------------------------- |
69+
| `GET /` | service index + endpoint docs |
70+
| `GET /health` | liveness + R2 dataset status |
71+
| `GET /v1/repos` | filtered list (params below) |
72+
| `GET /v1/repos/{owner}/{name}` | single repository |
73+
| `GET /v1/stats` | export stats block |
74+
| `GET /v1/categories` | repo count per category |
75+
| `GET /v1/tags` | repo count per tag |
76+
| `GET /v1/languages` | repo count per language |
77+
| `GET /v1/export.json` | the whole export document |
78+
79+
`/v1/repos` query params: `category`, `owner`, `tag`, `lang`, `featured`, `q` (free text), `sort` (pushed_at/stars/forks/name/created_at/updated_at/size_kb/open_issues), `order` (asc/desc), `limit`, `offset`.
80+
81+
```bash
82+
curl -s "https://data.devartslab.com/v1/repos?category=tool&sort=stars&order=desc"
83+
curl -s "https://data.devartslab.com/v1/repos/DevArtsLab/tool-github-export"
84+
```
85+
86+
**Private dataset:** any `GET /v1/*` endpoint served with `Authorization: Bearer $PRIVATE_API_TOKEN` returns the private export instead (superset of the public one). On the private dataset, `visibility=private` is honored as an extra filter. Authed responses are never cached.
87+
88+
### Worker ops
89+
90+
```bash
91+
cd worker
92+
npm install
93+
npx wrangler deploy # deploys + attaches data.devartslab.com
94+
npx wrangler secret put GH_EXPORT_TOKEN # cron fetch of the private export
95+
npx wrangler secret put PRIVATE_API_TOKEN # consumer bearer for private dataset
96+
```
97+
98+
CI deploys are intentionally not wired yet (needs a scoped Cloudflare API token; the wrangler OAuth token expires and should not be stored in repo secrets).
99+
59100
## The contract
60101

61102
Top-level fields: `schema_version`, `generated_at`, `generator`, `sources`, `stats` (incl. `by_category`), `repositories[]`.
@@ -131,7 +172,7 @@ Private repos and repos marked `exclude: true` never reach `repos.public.json`.
131172
## Roadmap
132173

133174
- Optional AI enrichment block (`ai` field per repo: summary, tech stack, highlights) via a model API in the Action
134-
- Phase 2 serving: Cloudflare Worker + R2 for `data.devartslab.com`-style endpoints with query filters
175+
- CI deploy for `worker/` on merge (needs a scoped Cloudflare API token)
135176
- `repos.featured.json` convenience view
136177

137178
## License

0 commit comments

Comments
 (0)