description	Add a dataset (CSV, TSV, JSON, or GeoJSON) to an existing PortalJS portal. Appends an entry to datasets.json so the catalog and showcase render it automatically; routes the data by source (local file vs remote URL) — R2 via Git LFS by default, remote URLs by passthrough.
allowed-tools	Read, Write, Edit, Bash, WebFetch

/portaljs-add-dataset

Add a dataset to an existing PortalJS (portaljs-catalog) portal. Appends one entry to datasets.json — the single source of truth for the catalog — and routes the data to the right place by source first, then size. No per-dataset page is created: the catalog at /search lists it and the dynamic showcase route pages/[owner]/[slug].tsx renders it automatically at /@<namespace>/<slug>.

The routing matrix — branch on SOURCE, then size

Where the bytes end up depends first on where they come from, then on size/intent. The manifest's resource.path (or single-file file) is the unifying abstraction: resourceUrl() in lib/datasets.ts serves a repo-relative path from /public/data and passes an absolute URL straight through. So every route below just decides what string to write into path.

Source	Default	What happens	Manifest `path`
Local file	R2 via Git LFS	`mv` into the repo, `git lfs track <path>`, commit a ~134 B pointer, `git lfs push` → Giftless → R2 (no GitHub remote needed)	absolute R2 URL (browser fetches R2 directly)
Local file (fenced)	inline	Only bundled SAMPLE data or an OSS self-host with no R2: `cp` into `public/data/` (stays inline per `.gitattributes` fence)	bare filename → `/data/<file>`
Remote URL	passthrough	record the URL as-is — no download, no upload, zero duplication	the absolute URL, unchanged
Remote URL	adopt (opt-in)	user wants it hosted/versioned under the portal: fetch → route as a local file (R2/LFS)	absolute R2 URL

Storage decision (epic po-g9y): the default for added data is R2, regardless of size or format. Inline is not a size threshold — it is a fenced exception for bundled sample data and the OSS-no-R2 fallback. Remote URLs default to passthrough (copy nothing); adopting one into R2 is an explicit opt-in.

Remote-URL passthrough caveat: serving and linking always work. But in-browser range / DuckDB queries against a 3rd-party URL need CORS + range support on that host, which we don't control. If the user needs querying (not just preview/download) and the remote lacks CORS/range, recommend the adopt-into-R2 option.

Supported formats for the showcase preview/registration: CSV, TSV, JSON (array), GeoJSON. (LFS itself is format-agnostic — any binary can be tracked — but the showcase Table/map only previews these.)

Required input — ask, don't error

Source — a local file path (./data/file.csv) or a public URL (https://…/data.csv)
Portal directory — path to the portal project (defaults to current directory)
Namespace — the dataset's namespace value (the portal's NAMESPACE_TYPE group: a subject for 'theme' portals, a publisher for 'owner' portals)

If the source is missing, ask for it — never dead-end. The user can say "use defaults" to accept the defaults below.

Steps

1. Gather input from `$ARGUMENTS` (interview if thin)

Extract what's present:

SOURCE — file path or URL
PORTAL_DIR — portal directory (default: .)
DATASET_NAME — human-readable name (default: derived from filename)
DATASET_SLUG — URL slug (default: lowercase hyphenated filename without extension)
DESCRIPTION — optional one-line description
NAMESPACE — namespace value (default: read the existing first entry's namespace from datasets.json, else reference)
ADOPT — for a remote URL only: whether to adopt the file into R2 (default: no — passthrough). Only ask if the URL route is taken and querying is plausibly needed.

If SOURCE is missing, ask (one focused prompt) and wait:

To add a dataset I need:
1. Source: local file path or public URL (required)
2. Portal directory (Enter for current directory)
3. Dataset name (Enter to use the filename)
4. Namespace value — the group this dataset belongs to
   (subject if the portal is "theme" mode, publisher if "owner" mode; Enter to reuse the catalog's existing namespace)

Check the portal's namespace mode if helpful: read NAMESPACE_TYPE from PORTAL_DIR/lib/datasets.ts so you can phrase the namespace question correctly ("subject" vs "publisher").

2. Validate the portal directory

The target must be a portaljs-catalog portal. Confirm PORTAL_DIR/datasets.json, PORTAL_DIR/package.json, and PORTAL_DIR/pages/[owner]/[slug].tsx exist. If datasets.json is missing, tell the user this portal isn't the catalog template (it may be an older single-page template) and ask how to proceed rather than failing silently.

3. Detect the format

If SOURCE is a URL: fetch headers (or a HEAD) and check the status. If not 200, tell the user the fetch failed (with the HTTP status) and ask them to confirm the URL is publicly accessible, then retry. Detect format from Content-Type or the URL extension. (For the default passthrough route you do not download the body — only enough to detect format.)

If SOURCE is a local file path: check the file exists. If not, tell the user the path wasn't found and ask for a correct path. Detect format from the file extension.

Format detection rules:

.csv or text/csv → CSV
.tsv or text/tab-separated-values → TSV
.geojson or application/geo+json or (JSON-parseable and parsed.type === "FeatureCollection") → GeoJSON
.json or application/json → JSON array
Vector geo formats — .zip (zipped Shapefile), .gpkg, .kml/.kmz, .fgb, or a .csv with a geometry/lat-lon column → hand off to /portaljs-add-geo. That skill auto-ingests geo sources into a dual-tier dataset (PMTiles render + GeoParquet query) on the user's machine. This tabular skill only previews flat files; it does not tile or build GeoParquet. (A small GeoJSON you just want listed with a download link can stay here as the geojson format; for a map/query view, prefer /portaljs-add-geo or /portaljs-add-map.)
Anything else: tell the user the showcase can't preview this format and ask them to convert to CSV, TSV, JSON array, or GeoJSON first.

4. Route the data (the core of this skill)

Pick the branch from the matrix above. Decide on SOURCE first.

4a. Remote URL — passthrough (DEFAULT for URLs)

Do nothing to the bytes. The manifest path is the absolute URL itself; resourceUrl() passes it through and the browser fetches it directly. Skip to Step 5 with MANIFEST_PATH=<the URL>.

Mention the CORS/range caveat if the user may need querying, and offer the adopt route.

4b. Remote URL — adopt into R2 (OPT-IN only)

Only when the user explicitly wants the file hosted/versioned under the portal. Download it to a temp path, then fall through to 4c treating that temp file as the local source:

curl -fL "$SOURCE" -o "/tmp/$DATASET_SLUG.$EXT"

4c. Local file — R2 via Git LFS (DEFAULT)

This versions a tiny pointer in git and streams the bytes to Cloudflare R2 through Giftless. The browser then fetches the bytes straight from R2 (manifest path = absolute R2 URL).

cd PORTAL_DIR
mkdir -p data
# mv (NOT cp) so an external local file isn't tripled on disk:
mv "$SOURCE" "data/$DATASET_SLUG.$EXT"     # for 4b, $SOURCE is the temp download

# Ensure the LFS clean/smudge filters are installed for THIS repo before tracking.
# On a fresh machine `git lfs install` may never have run (brew install git-lfs does
# NOT auto-run it), so without this `git add` commits the RAW bytes instead of a
# ~134 B pointer and the later `git lfs push` has nothing to stream. Idempotent.
if ! git config --get filter.lfs.clean >/dev/null 2>&1; then
  git lfs install --local
fi

# Per-file, format-agnostic LFS tracking (appends a path entry to .gitattributes,
# NOT an extension glob):
git lfs track "data/$DATASET_SLUG.$EXT"
git add .gitattributes "data/$DATASET_SLUG.$EXT"
git commit -m "data: add $DATASET_SLUG via LFS"

mv, not cp. When bringing an external local file into the repo, move it in place — a cp leaves a 3rd on-disk copy (original + working tree + .git/lfs cache).

Authenticate git-lfs (phase-2 gotcha, po-g9y.1). The committed .lfsconfig carries the Giftless URL but no credentials. Mint a repo-scoped JWT and set the credentialed URL in local git config only (never commit it). Use the _jwt HTTP Basic piggyback — do not set a broad http.extraHeader, which git-lfs would replay onto the presigned R2 PUT (R2 → 400) and the verify callback.

Mint the token from the Arc API (default — no private key on your machine). The Arc API is the token issuer (po-g9y.13): it holds the RS256 signer as a Worker secret and mints a scoped, short-TTL LFS token for any authenticated Arc user. Reuse the same Arc token /portaljs-deploy resolves (PORTALJS_TOKEN, else ~/.portaljs/credentials); if you have neither, run /portaljs-deploy's sign-in step once. The endpoint returns a ready-to-use credentialed lfs_url:

API="${PORTALJS_ARC_API:-https://api.arc.portaljs.com}"
ARC_TOKEN="${PORTALJS_TOKEN:-$(node -e "try{process.stdout.write(JSON.parse(require('fs').readFileSync(process.env.HOME+'/.portaljs/credentials','utf8')).token||'')}catch{}")}"
# Claim the slug under your Arc account (idempotent; safe to re-run). Minting never
# creates a slug (po-g9y.13), so claim once before the FIRST push on a portal that
# hasn't been deployed yet — this is what makes create → add-data → deploy seamless.
# 403 = the slug belongs to another account.
curl -fsS -X POST "$API/v1/repos/<project-slug>/claim" -H "Authorization: Bearer $ARC_TOKEN" >/dev/null
# Pushing data needs write: request ?actions=read,write,verify (default is read-only,
# pull). ?ttl=<secs> to tune.
LFS_URL=$(curl -fsS -X POST "$API/v1/repos/<project-slug>/lfs-token?actions=read,write,verify" \
  -H "Authorization: Bearer $ARC_TOKEN" \
  | node -e "process.stdin.on('data',d=>process.stdout.write(JSON.parse(d).lfs_url))")
git config lfs.url "$LFS_URL"     # local only — carries the token, never committed

OSS self-host fallback. Running your own Giftless without an Arc account? Sign the token locally with the issuer's private key instead:
TOKEN=$(python3 ../../giftless/mint-token.py --org datopian --repo <project-slug> \
  --ttl 3600 --algorithm RS256 --key-file ../../giftless/jwt_private_key)
git config lfs.url "https://_jwt:$TOKEN@lfs.portaljs.com/datopian/<project-slug>"

(See giftless/README.md → "Authenticating a Git LFS client".)

Push the bytes to R2 (no GitHub remote required), then reclaim local disk. The bytes route to wherever lfs.url points (set above) — a git remote is not a prerequisite. git lfs push needs only a remote name to enumerate objects from the branch; with lfs.url set, that remote's own URL is never contacted for the transfer. Scaffolded portals are local-only (create-portaljs does git init + an initial commit, no remote; a bare git push would fail with "No configured push destination"), so branch on whether a real remote exists:

BRANCH=$(git rev-parse --abbrev-ref HEAD)
if git remote get-url --push origin >/dev/null 2>&1; then
  # A GitHub remote exists (collaboration setup): a normal push carries commits +
  # LFS pointers to GitHub AND streams the bytes to R2 via the pre-push hook.
  git push -u origin "$BRANCH"
else
  # Local-only scaffold (the DEFAULT): no GitHub remote. Add a throwaway local
  # remote just so git-lfs has an object source, then push the LFS objects — the
  # bytes go to Giftless/R2 via lfs.url; GitHub is never contacted. (locksverify
  # is off because Giftless serves no locking API — the check is cosmetic noise.)
  git remote add r2-lfs . 2>/dev/null || true
  git -c lfs.locksverify=false lfs push r2-lfs "$BRANCH"
fi
git lfs prune    # working tree + .git/lfs cache ≈ 2× locally until pruned

A GitHub remote is optional — it's for collaboration, not storage. Data-to-R2 is git-lfs talking to Giftless over HTTP (the LFS batch API at lfs.url); that URL alone determines the upload destination, independent of any git remote. A GitHub remote only shares commits + LFS pointers with collaborators — orthogonal to where the bytes live. This is what makes create → add-data → deploy work for a non-dev user who has never configured a remote.

Build the absolute R2 URL for the manifest. The browser fetches R2 directly, so path must be the public R2 URL — resourceUrl() passes absolute URLs through unchanged. The Giftless object key is lfs/datopian/<project-slug>/<oid>, where <oid> is the SHA-256 in the committed LFS pointer:

OID=$(git cat-file -p :data/$DATASET_SLUG.$EXT | sed -n 's/^oid sha256://p')
# MANIFEST_PATH = $R2_PUBLIC_BASE/lfs/datopian/<project-slug>/$OID

R2_PUBLIC_BASE defaults to https://data.portaljs.com — the public read domain on the portaljs-giftless R2 bucket (GET/HEAD + range + CORS, edge-cached). So the browser fetches https://data.portaljs.com/lfs/datopian/<project-slug>/<oid> directly. Override it only for an OSS self-host with its own bucket/domain (export R2_PUBLIC_BASE=…).

4d. Local file — inline (FENCED exception only)

Only for bundled SAMPLE data shipped with a template, or an OSS self-host running without R2 credentials. Copy into public/data/, which the .gitattributes fence keeps inline:

mkdir -p PORTAL_DIR/public/data
cp "$SOURCE" "PORTAL_DIR/public/data/$DATASET_SLUG.$EXT"

MANIFEST_PATH is the bare filename ($DATASET_SLUG.$EXT) → served from /data/<file>.

5. Append the entry to `datasets.json`

Open PORTAL_DIR/datasets.json (a JSON array) and append one entry, keeping all existing entries. Match the Dataset shape from lib/providers/types.ts:

{
  "slug": "DATASET_SLUG",
  "namespace": "NAMESPACE",
  "name": "DATASET_NAME",
  "description": "DESCRIPTION",
  "file": "MANIFEST_PATH",
  "format": "csv"
}

file is MANIFEST_PATH from Step 4: a bare filename (inline), or an absolute URL (passthrough or R2). resourceUrl() resolves both.
format is one of csv | tsv | json | geojson (lowercase), matching the detected format.
namespace is the value gathered in Step 1. (namespace, slug) must be unique across the manifest — if a clash exists, ask the user for a different slug or namespace.
Drop description only if there genuinely is none (it's optional in the type).

That is the entire registration. getStaticPaths in pages/[owner]/[slug].tsx picks up the new (namespace, slug) pair at build time, and the catalog at /search filters over it. Do not create any page file — there is no pages/datasets/[slug].tsx in this template.

6. Verify the build

cd PORTAL_DIR
npx next build > /tmp/portaljs-add-dataset-build.log 2>&1
BUILD_EXIT=$?
tail -20 /tmp/portaljs-add-dataset-build.log

If BUILD_EXIT is non-zero, print the log and fix the error (commonly malformed JSON in datasets.json) before reporting success.

7. Report success

✓ Dataset added: DATASET_NAME
  - Route:     <local→R2 LFS | remote URL passthrough | adopted→R2 | inline sample>
  - Data:      <data/DATASET_SLUG.EXT (LFS → R2) | the source URL | public/data/DATASET_SLUG.EXT>
  - Manifest:  datasets.json (entry appended; path = MANIFEST_PATH)
  - Showcase:  /@NAMESPACE/DATASET_SLUG  (rendered by pages/[owner]/[slug].tsx)
  - Catalog:   appears in /search automatically

For an LFS route, remind the user the bytes are in R2 and git lfs prune has reclaimed local disk. Then:

Next: run `npm run dev` and visit http://localhost:3000/@NAMESPACE/DATASET_SLUG to verify,
or run /portaljs-add-chart or /portaljs-add-map to add a view to its showcase.

Notes

No per-dataset page. Registration is one JSON entry. The dynamic route pages/[owner]/[slug].tsx renders metadata + a Table data preview + a Download & API section + a Views placeholder for every manifest entry.
path is the seam. Repo-relative → /data/<file> (inline); absolute URL → passed through (remote passthrough or R2). One abstraction (resourceUrl()) serves all routes — never branch on the source at the call site.
CSV/TSV preview via the template's <Table />, which fetches the resolved URL in the browser and auto-detects the delimiter (papaparse). JSON / GeoJSON entries show a download link in the showcase; use /portaljs-add-map for a Leaflet view of GeoJSON.
LFS is format-agnostic. .gitattributes ships minimal (only the inline fence); this skill appends a per-file tracking entry for whatever it routes to R2 — including images and arbitrary binaries — never an extension glob.
Column names with spaces are preserved by papaparse and wrap fine in the table.

Navigation Menu

Uh oh!

/portaljs-add-dataset

The routing matrix — branch on SOURCE, then size

Required input — ask, don't error

Steps

1. Gather input from `$ARGUMENTS` (interview if thin)

2. Validate the portal directory

3. Detect the format

4. Route the data (the core of this skill)

4a. Remote URL — passthrough (DEFAULT for URLs)

4b. Remote URL — adopt into R2 (OPT-IN only)

4c. Local file — R2 via Git LFS (DEFAULT)

4d. Local file — inline (FENCED exception only)

5. Append the entry to `datasets.json`

6. Verify the build

7. Report success

Notes

Uh oh!

FilesExpand file tree

portaljs-add-dataset.md

Latest commit

History

portaljs-add-dataset.md

File metadata and controls

/portaljs-add-dataset

The routing matrix — branch on SOURCE, then size

Required input — ask, don't error

Steps

1. Gather input from $ARGUMENTS (interview if thin)

2. Validate the portal directory

3. Detect the format

4. Route the data (the core of this skill)

4a. Remote URL — passthrough (DEFAULT for URLs)

4b. Remote URL — adopt into R2 (OPT-IN only)

4c. Local file — R2 via Git LFS (DEFAULT)

4d. Local file — inline (FENCED exception only)

5. Append the entry to datasets.json

6. Verify the build

7. Report success

Notes

1. Gather input from `$ARGUMENTS` (interview if thin)

5. Append the entry to `datasets.json`