|
| 1 | +--- |
| 2 | +title: "Jina metadata parser is blog-only — needs two profiles + fuzzy routing (academic sources lose their date/publisher)" |
| 3 | +lede: "Jina returns rich metadata for every source, but the parser only reads OpenGraph/blog keys (article:published_time, og:site_name). Scholarly articles carry their metadata under Highwire/Dublin-Core/PRISM keys (citation_*, dc.*, prism.*), so their date and publisher fall through empty and the operator hand-fills them. Fix: two parser profiles — structured and opengraph — with fuzzy routing by source kind, and a manual override so a human can re-route when detection is wrong." |
| 4 | +date_created: 2026-08-02 |
| 5 | +date_modified: 2026-08-02 |
| 6 | +authors: |
| 7 | + - Michael Staton |
| 8 | +augmented_with: |
| 9 | + - Claude Code on Claude Opus 4.8 |
| 10 | +semantic_version: 0.0.0.1 |
| 11 | +tags: |
| 12 | + - Issue |
| 13 | + - Augment-It |
| 14 | + - Content-Ingest |
| 15 | + - Jina |
| 16 | + - Metadata |
| 17 | + - Bug |
| 18 | +status: Resolved |
| 19 | +--- |
| 20 | + |
| 21 | +# Jina metadata parser is blog-only |
| 22 | + |
| 23 | +## Why Care? |
| 24 | + |
| 25 | +The operator has been hand-entering metadata that Jina already returned — |
| 26 | +"supremely inefficient." A live fetch of a Springer article proved it: Jina |
| 27 | +returned authors (`dc.creator`, `citation_author`), publisher |
| 28 | +(`citation_publisher = Springer US`), and date (`dc.date = 2023-05-27`, |
| 29 | +`prism.publicationDate`, `citation_publication_date`) — but the parser only |
| 30 | +looked for blog/OpenGraph keys (`article:published_time`, `meta.date`, |
| 31 | +`og:site_name`), which academic pages don't set. So date + publisher fell |
| 32 | +through empty. A Jina fetch of a **company landing page** is different again |
| 33 | +(og/site metadata, no author or date at all). |
| 34 | + |
| 35 | +## The design (operator-directed) |
| 36 | + |
| 37 | +Not one parser reaching for more keys — **two parser profiles plus fuzzy |
| 38 | +routing**, so when auto-detection is wrong the human can press a button to |
| 39 | +re-parse under the other profile: |
| 40 | + |
| 41 | +- **`structured`** — Highwire `citation_*`, Dublin Core `dc.*`, PRISM |
| 42 | + `prism.*`. For scholarly articles, journals, anything with a DOI. |
| 43 | +- **`opengraph`** — `og:*`, `article:*`, Jina top-level, generic. For blogs, |
| 44 | + news, company landing pages, plain web pages. |
| 45 | + |
| 46 | +Routing detects the **source kind** (`academic-paper` / `article` / |
| 47 | +`company-landing` / `web-page`) from which keys are present, records it on the |
| 48 | +source, and picks the profile. A `forceProfile` override lets the operator |
| 49 | +re-route (button wiring tracked as a fast-follow). |
| 50 | + |
| 51 | +Each profile resolves every field across an ordered alias list (first hit |
| 52 | +wins), so a field missing under one convention still resolves under another. |
| 53 | +Different kinds expect different fields — a company landing page is not nagged |
| 54 | +for an author/date it doesn't have. |
| 55 | + |
| 56 | +## Authors |
| 57 | + |
| 58 | +Authors stay an **array of strings** (Jina's `citation_author` / `dc.creator` |
| 59 | +already arrive as arrays; one array element = one author, never comma-split, |
| 60 | +so "Pal, Soumen" survives). The UI author field already coerces a |
| 61 | +comma-separated entry into an array. Future: array of strings that resolve to |
| 62 | +unique author profiles (the persons canonical layer). |
| 63 | + |
| 64 | +## Fix |
| 65 | + |
| 66 | +`jina.ts` gains `detectProfile()` (routing → profile + kind), `extractBib()` |
| 67 | +(pure, profile-aware, `forceProfile`-overridable), and per-profile field alias |
| 68 | +lists. `fetchViaJina()` accepts `forceProfile`. Vitest fixtures cover the |
| 69 | +academic (real Springer metadata), article, and company-landing kinds. |
| 70 | + |
| 71 | +## Resolution |
| 72 | + |
| 73 | +Fixed 2026-08-02. Verified: content-ingest `vitest run` + `tsc --noEmit`. |
| 74 | +Reaches augment.didi.sh on the next redeploy. The manual re-route **button** |
| 75 | +(UI + a re-extract capability that reuses the cached Jina result) is a |
| 76 | +fast-follow, tracked separately. |
| 77 | + |
| 78 | +## See also |
| 79 | + |
| 80 | +- [[Fetch-Full-Content-Clobbers-Operator-Metadata]] — the sibling fetch fix. |
| 81 | +- [[feedback_additive_enrichment_never_overrides_accepted]] · [[feedback_human_in_drivers_seat]] |
0 commit comments