|
| 1 | +--- |
| 2 | +title: "Corpus adds don't fetch metadata — and the row gives no cue either way, and there's no inspector to see or fix it" |
| 3 | +lede: "Adding a corpus item to New America raised the question 'did it fetch the metadata?' — and the honest answer is no, and nothing on the row would tell you either way. The ➕ path writes url/kind/domain and stops; the Jina metadata fetch we already run for domain-corpus sources never fires for entity corpus." |
| 4 | +date_created: 2026-07-24 |
| 5 | +date_modified: 2026-07-24 |
| 6 | +authors: |
| 7 | + - Michael Staton |
| 8 | +augmented_with: |
| 9 | + - Claude Code on Claude Fable 5 |
| 10 | +semantic_version: 0.0.0.1 |
| 11 | +tags: |
| 12 | + - Issue |
| 13 | + - Usability |
| 14 | + - Augment-It |
| 15 | + - Corpus |
| 16 | + - Org-Workbench |
| 17 | + - Content-Items |
| 18 | + - Metadata |
| 19 | +status: Open · Jotted |
| 20 | +--- |
| 21 | + |
| 22 | +# Corpus adds don't fetch metadata; no cue; no inspector |
| 23 | + |
| 24 | +## The answer to the operator's question |
| 25 | + |
| 26 | +**It didn't.** `organization.corpus.add` / `person.corpus.add` → |
| 27 | +`findOrCreateContent` writes `url`, `kind`, `url_domain`, and counters to the |
| 28 | +`content_items` ledger — full stop. No title, no fetch, no enrichment. This |
| 29 | +is the same hole [[Corpus-Items-Not-Visible-On-Person-Cards-Coverage-Hard-To-Assess]] |
| 30 | +noted from the read side ("content_items has no title field"); this doc is |
| 31 | +the write side of it. |
| 32 | + |
| 33 | +## Three wants (one flow) |
| 34 | + |
| 35 | +1. **Fetch metadata on add — the way we already do it.** The precedent is |
| 36 | + content-ingest's `source.add`: a Jina metadata fetch (title, excerpt, |
| 37 | + authors, publisher, bibliographic fields — `services/content-ingest/src/corpus.ts`, |
| 38 | + the "METADATA-ONLY file" path). Entity corpus adds should ride the same |
| 39 | + machinery: the ➕ (and search-and-add's per-row add) triggers the fetch, |
| 40 | + `content_items` grows the metadata columns (title, authors?, |
| 41 | + published_at?), and every corpus render hydrates from the ledger. |
| 42 | +2. **A visual cue on the row.** Fetched → the row shows the title (host+path |
| 43 | + stays the un-fetched fallback per |
| 44 | + [[List-Rows-Show-Hostname-Only-Same-Domain-Entries-Indistinguishable]]); |
| 45 | + un-fetched or failed → a visible mark (badge/dot) that says "bare URL, |
| 46 | + metadata pending/failed". The operator should never have to ask the |
| 47 | + question this issue opens with. |
| 48 | +3. **An inspector.** Click a corpus row → see what the ledger holds and |
| 49 | + adjust it: title, kind, date, maybe authors. The in-place patch shipped |
| 50 | + for streams ([[Pulse-Streams-Need-Editable-Kind-And-User-Facing-Names]]) |
| 51 | + is the edit precedent; a corpus row wants the same, plus the fetched |
| 52 | + fields. Human adjusts, per [[human-in-drivers-seat]] — Jina's guess is a |
| 53 | + candidate, not truth. |
| 54 | + |
| 55 | +## Machinery notes (jotted) |
| 56 | + |
| 57 | +- `content_items` is SCHEMALESS — adding `title` etc. is free; the work is |
| 58 | + the fetch trigger + hydration reads + the row UI. |
| 59 | +- Where does the fetch run? The resolver could call content-ingest over NATS |
| 60 | + at add time (sync — the cue is immediate), or a fire-and-forget that |
| 61 | + patches the ledger after (async — needs the pending/failed cue anyway). |
| 62 | + The failed/pending badge is wanted regardless, so async is viable. |
| 63 | +- `organization.detail` / `affiliation.detail` / `organization.affiliations` |
| 64 | + would hydrate title via `content_id` → `content_items` (the join noted in |
| 65 | + the #20 issue). |
| 66 | +- Backfill: existing bare corpus entries (all of them, today) want a |
| 67 | + re-fetch pass — same additive discipline as the person-name backfill |
| 68 | + (fill missing metadata, never overwrite operator-adjusted fields). |
| 69 | + |
| 70 | +## Open questions |
| 71 | + |
| 72 | +- [ ] Sync fetch on add (immediate cue, slower ➕) vs async patch (fast ➕, |
| 73 | + needs pending state) — or Jina-on-add with a short timeout and async |
| 74 | + retry? |
| 75 | +- [ ] Does the inspector edit `content_items` (shared ledger — edits visible |
| 76 | + to every referencing entity) or the per-entity corpus entry? Probably the |
| 77 | + ledger for metadata, the entry for kind. |
| 78 | +- [ ] Does search-and-add's one-click add fire the same fetch (it should — |
| 79 | + same verb underneath)? |
0 commit comments