|
| 1 | +--- |
| 2 | +title: "Jina learns two metadata profiles — academic sources stop losing their dates and publishers" |
| 3 | +lede: "The metadata parser was reading only blog/OpenGraph keys, so scholarly articles — whose data hides under Highwire/Dublin-Core/PRISM tags — arrived with no date and a weak publisher, and the operator hand-filled them. Now it detects the source KIND (academic-paper / article / company-landing / web-page), routes to the right parser profile, and resolves every field across all conventions. A Springer article now fills its authors, publisher, and date automatically." |
| 4 | +date_created: 2026-08-02 |
| 5 | +date_modified: 2026-08-02 |
| 6 | +publish: true |
| 7 | +authors: |
| 8 | + - Michael Staton |
| 9 | +augmented_with: |
| 10 | + - Claude Code on Claude Opus 4.8 |
| 11 | +files_changed: |
| 12 | + - services/content-ingest/src/jina.ts |
| 13 | + - services/content-ingest/test/jina-extract.test.ts |
| 14 | +tags: |
| 15 | + - Augment-It |
| 16 | + - Content-Ingest |
| 17 | + - Jina |
| 18 | + - Metadata |
| 19 | + - Bug-Fix |
| 20 | +--- |
| 21 | + |
| 22 | +# Jina learns two metadata profiles |
| 23 | + |
| 24 | +Jina always returned rich metadata for every source — we just weren't reading |
| 25 | +it. The parser looked for blog/OpenGraph keys (`article:published_time`, |
| 26 | +`og:site_name`), which scholarly pages don't set: they carry their metadata |
| 27 | +under Highwire, Dublin-Core, and PRISM tags (`citation_*`, `dc.*`, `prism.*`). |
| 28 | +So academic sources lost their publish date and got a weak publisher, and you |
| 29 | +filled them in by hand. |
| 30 | + |
| 31 | +Now `jina.ts` **detects the source kind** and **routes to one of two parser |
| 32 | +profiles**: |
| 33 | + |
| 34 | +- **`structured`** — `citation_*` / `dc.*` / `prism.*`, for scholarly articles |
| 35 | + and anything with a DOI. |
| 36 | +- **`opengraph`** — `og:*` / `article:*` / generic, for blogs, news, company |
| 37 | + landing pages, and plain web pages. |
| 38 | + |
| 39 | +Each field resolves across an ordered list of aliases spanning every |
| 40 | +convention, so a value missing under one still resolves under another. Every |
| 41 | +source now records its detected `source_kind` and `parser_profile`, and the |
| 42 | +extractor accepts a `forceProfile` override — the foundation for a "re-parse |
| 43 | +as…" button when detection is wrong (fast-follow, #79). |
| 44 | + |
| 45 | +A live Springer article that used to arrive dateless now fills its four |
| 46 | +authors, `Springer US`, and `2023-05-27` automatically. |
| 47 | + |
| 48 | +Verified with a vitest suite whose academic fixture is the real metadata |
| 49 | +captured live from that article. Reaches augment.didi.sh on the next redeploy. |
0 commit comments