Skip to content

Commit fdc9646

Browse files
mpstatonclaude
andcommitted
issue(content-ingest): Jina metadata parser is blog-only — needs two profiles + routing
Jina returns rich metadata for every source, but the parser reads only OpenGraph/blog keys, so academic sources (citation_*/dc.*/prism.*) lose their date + publisher and the operator hand-fills them. Design: two parser profiles (structured + opengraph) with fuzzy routing by source kind and a forceProfile override for manual re-routing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018UYTYu4MAFZ7iyr2VTo2kq
1 parent ad8a289 commit fdc9646

1 file changed

Lines changed: 81 additions & 0 deletions

File tree

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
---
2+
title: "Jina metadata parser is blog-only — needs two profiles + fuzzy routing (academic sources lose their date/publisher)"
3+
lede: "Jina returns rich metadata for every source, but the parser only reads OpenGraph/blog keys (article:published_time, og:site_name). Scholarly articles carry their metadata under Highwire/Dublin-Core/PRISM keys (citation_*, dc.*, prism.*), so their date and publisher fall through empty and the operator hand-fills them. Fix: two parser profiles — structured and opengraph — with fuzzy routing by source kind, and a manual override so a human can re-route when detection is wrong."
4+
date_created: 2026-08-02
5+
date_modified: 2026-08-02
6+
authors:
7+
- Michael Staton
8+
augmented_with:
9+
- Claude Code on Claude Opus 4.8
10+
semantic_version: 0.0.0.1
11+
tags:
12+
- Issue
13+
- Augment-It
14+
- Content-Ingest
15+
- Jina
16+
- Metadata
17+
- Bug
18+
status: Resolved
19+
---
20+
21+
# Jina metadata parser is blog-only
22+
23+
## Why Care?
24+
25+
The operator has been hand-entering metadata that Jina already returned —
26+
"supremely inefficient." A live fetch of a Springer article proved it: Jina
27+
returned authors (`dc.creator`, `citation_author`), publisher
28+
(`citation_publisher = Springer US`), and date (`dc.date = 2023-05-27`,
29+
`prism.publicationDate`, `citation_publication_date`) — but the parser only
30+
looked for blog/OpenGraph keys (`article:published_time`, `meta.date`,
31+
`og:site_name`), which academic pages don't set. So date + publisher fell
32+
through empty. A Jina fetch of a **company landing page** is different again
33+
(og/site metadata, no author or date at all).
34+
35+
## The design (operator-directed)
36+
37+
Not one parser reaching for more keys — **two parser profiles plus fuzzy
38+
routing**, so when auto-detection is wrong the human can press a button to
39+
re-parse under the other profile:
40+
41+
- **`structured`** — Highwire `citation_*`, Dublin Core `dc.*`, PRISM
42+
`prism.*`. For scholarly articles, journals, anything with a DOI.
43+
- **`opengraph`**`og:*`, `article:*`, Jina top-level, generic. For blogs,
44+
news, company landing pages, plain web pages.
45+
46+
Routing detects the **source kind** (`academic-paper` / `article` /
47+
`company-landing` / `web-page`) from which keys are present, records it on the
48+
source, and picks the profile. A `forceProfile` override lets the operator
49+
re-route (button wiring tracked as a fast-follow).
50+
51+
Each profile resolves every field across an ordered alias list (first hit
52+
wins), so a field missing under one convention still resolves under another.
53+
Different kinds expect different fields — a company landing page is not nagged
54+
for an author/date it doesn't have.
55+
56+
## Authors
57+
58+
Authors stay an **array of strings** (Jina's `citation_author` / `dc.creator`
59+
already arrive as arrays; one array element = one author, never comma-split,
60+
so "Pal, Soumen" survives). The UI author field already coerces a
61+
comma-separated entry into an array. Future: array of strings that resolve to
62+
unique author profiles (the persons canonical layer).
63+
64+
## Fix
65+
66+
`jina.ts` gains `detectProfile()` (routing → profile + kind), `extractBib()`
67+
(pure, profile-aware, `forceProfile`-overridable), and per-profile field alias
68+
lists. `fetchViaJina()` accepts `forceProfile`. Vitest fixtures cover the
69+
academic (real Springer metadata), article, and company-landing kinds.
70+
71+
## Resolution
72+
73+
Fixed 2026-08-02. Verified: content-ingest `vitest run` + `tsc --noEmit`.
74+
Reaches augment.didi.sh on the next redeploy. The manual re-route **button**
75+
(UI + a re-extract capability that reuses the cached Jina result) is a
76+
fast-follow, tracked separately.
77+
78+
## See also
79+
80+
- [[Fetch-Full-Content-Clobbers-Operator-Metadata]] — the sibling fetch fix.
81+
- [[feedback_additive_enrichment_never_overrides_accepted]] · [[feedback_human_in_drivers_seat]]

0 commit comments

Comments
 (0)