Skip to content

Commit 48577bf

Browse files
mpstatonclaude
andcommitted
feat(content-ingest, resolver, strategy-curator): sources auto-capture authors, publisher, date from Jina
Fetching a source now fills in its own bibliographic fields — author(s), publisher, and publication date — pulled from Jina's structured (JSON) metadata and written to both the markdown frontmatter and the SurrealDB sources registry. Each field is editable in the source form, so curation starts from filled-in rather than blank. authors is an array (one author -> a one-element list). The data was always in Jina's response; we were requesting the wrong format. A stale `Accept: 'text/markdown'` header routed every fetch through the markdown fallback (title + date only) while the JSON-extraction code sat dead behind a failing JSON.parse. Switching to `Accept: 'application/json'` lit up author, og:site_name (publisher), and article:published_time (date). content-ingest (jina.ts, corpus.ts): - request application/json; extract authors (string|array -> string[]), publisher, published_date; write to frontmatter; preserve hand-corrected values across a re-fetch. resolver (domains.ts): - sources registry gains an authors[] column; source.add / fetch persist the bib via applyBibToRegistry and now ECHO source_slug back so an in-session source connects to its on-disk file immediately (Filename field + rename were blank until a reload). strategy-curator UI: - Author(s) / Publisher / Published date editable fields, authors as a comma-separated input backed by a string[]. - commitOnEdit action: commit on Enter/blur, pulse the field border green on success (--color-confidence-high), fade back; no pulse on error. - withSlug fallback derives source_slug from corpus_path so the Filename field never goes blank for a file that exists. Docs: - Spec -> 0.0.0.6 with an Increment 4 section. - changelog 2026-06-29_02. Files changed: - services/content-ingest/src/{jina.ts,corpus.ts} - services/record-surrealdb-resolver/src/domains.ts - apps/strategy-curator/src/{curation.svelte.ts,SourceDetail.svelte,types.ts,app.css} - context-v/specs/Strategy-Curator-Entry-Point-for-Augment-It.md - changelog/2026-06-29_02_Sources-Capture-Their-Own-Bibliography-Authors-Publisher-Date.md Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 1077500 commit 48577bf

9 files changed

Lines changed: 408 additions & 65 deletions

File tree

apps/strategy-curator/src/SourceDetail.svelte

Lines changed: 42 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,40 @@
1111
extractText = '';
1212
void curation.addExtract(extractKind, t);
1313
}
14+
15+
// Pulse the border green to confirm a save, then fade back (see .sc-saved).
16+
function flash(node: HTMLElement): void {
17+
node.classList.remove('sc-saved');
18+
void node.offsetWidth; // reflow → restart the animation on rapid repeats
19+
node.classList.add('sc-saved');
20+
window.setTimeout(() => node.classList.remove('sc-saved'), 1600);
21+
}
22+
23+
// Action: commit on Enter (blur) or change, then flash on success.
24+
function commitOnEdit(node: HTMLInputElement, run: (value: string) => unknown) {
25+
let current = run;
26+
const onChange = async (): Promise<void> => {
27+
await current(node.value);
28+
if (!curation.lastError) flash(node);
29+
};
30+
const onKey = (e: KeyboardEvent): void => {
31+
if (e.key === 'Enter') {
32+
e.preventDefault();
33+
node.blur(); // triggers change → save + flash
34+
}
35+
};
36+
node.addEventListener('change', onChange);
37+
node.addEventListener('keydown', onKey);
38+
return {
39+
update(next: (value: string) => unknown) {
40+
current = next;
41+
},
42+
destroy() {
43+
node.removeEventListener('change', onChange);
44+
node.removeEventListener('keydown', onKey);
45+
},
46+
};
47+
}
1448
</script>
1549

1650
{#if curation.focused}
@@ -23,7 +57,7 @@
2357
<input
2458
value={s.title ?? ''}
2559
placeholder="(no title — fetch, retry, or just type one)"
26-
onchange={(e) => curation.updateSource('title', e.currentTarget.value)}
60+
use:commitOnEdit={(v) => curation.updateSource('title', v)}
2761
/>
2862
</div>
2963
<div class="sc-field">
@@ -33,17 +67,21 @@
3367
value={s.source_slug ?? ''}
3468
placeholder="(filename appears after first save/fetch)"
3569
disabled={!s.source_slug}
36-
onchange={(e) => curation.renameSource(e.currentTarget.value)}
70+
use:commitOnEdit={(v) => curation.renameSource(v)}
3771
/>
3872
</div>
73+
<div class="sc-field">
74+
<span class="sc-label">Author(s) <span class="sc-muted sc-mini">— comma-separated</span></span>
75+
<input value={(s.authors ?? []).join(', ')} placeholder="(auto-filled on fetch)" use:commitOnEdit={(v) => curation.updateAuthors(v)} />
76+
</div>
3977
<div class="grid2">
4078
<div class="sc-field">
4179
<span class="sc-label">Publisher</span>
42-
<input value={s.publisher ?? ''} onchange={(e) => curation.updateSource('publisher', e.currentTarget.value)} />
80+
<input value={s.publisher ?? ''} placeholder="(auto-filled on fetch)" use:commitOnEdit={(v) => curation.updateSource('publisher', v)} />
4381
</div>
4482
<div class="sc-field">
4583
<span class="sc-label">Published date</span>
46-
<input value={s.published_date ?? ''} placeholder="YYYY-MM-DD" onchange={(e) => curation.updateSource('published_date', e.currentTarget.value)} />
84+
<input value={s.published_date ?? ''} placeholder="YYYY-MM-DD" use:commitOnEdit={(v) => curation.updateSource('published_date', v)} />
4785
</div>
4886
</div>
4987
<div class="sc-field">

apps/strategy-curator/src/app.css

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,22 @@
102102
.sc-app input:focus,
103103
.sc-app textarea:focus,
104104
.sc-app select:focus { outline: none; border-color: var(--color-field-focus); box-shadow: var(--fx-accent-glow); }
105+
106+
/* Saved-confirmation pulse: border glows green on commit, then fades back. */
107+
@keyframes sc-save-flash {
108+
0% {
109+
border-color: var(--color-confidence-high);
110+
box-shadow: 0 0 0 3px color-mix(in srgb, var(--color-confidence-high) 45%, transparent);
111+
}
112+
100% {
113+
border-color: var(--color-border);
114+
box-shadow: none;
115+
}
116+
}
117+
.sc-app input.sc-saved { animation: sc-save-flash 1.6s ease-out; }
118+
@media (prefers-reduced-motion: reduce) {
119+
.sc-app input.sc-saved { animation-duration: 1ms; }
120+
}
105121
.sc-app textarea { resize: vertical; min-height: 3.5em; }
106122

107123
.sc-app button {

apps/strategy-curator/src/curation.svelte.ts

Lines changed: 32 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -178,7 +178,7 @@ class CurationState {
178178
this.focusIdx = 0;
179179
if (typeof localStorage !== 'undefined') localStorage.setItem(ACTIVE_STRATEGY_KEY, slug);
180180
const r = await this.call<{ sources: Source[] }>('domain.assemble', { type: DOMAIN_TYPE, slug, client_slug: this.clientSlug });
181-
this.sources = r?.sources ?? [];
181+
this.sources = (r?.sources ?? []).map(withSlug);
182182
const tv = await this.call<{ tags: string[] }>('tag.suggest', { prefix: '', client_slug: this.clientSlug });
183183
this.tagVocab = tv?.tags ?? [];
184184
}
@@ -193,7 +193,7 @@ class CurationState {
193193
this.saveStatus = this.lastError ?? 'add failed';
194194
return;
195195
}
196-
this.sources = [...this.sources, r.source];
196+
this.sources = [...this.sources, withSlug(r.source)];
197197
this.focusIdx = this.sources.length - 1;
198198
this.saveStatus = `added — ${this.sources.length} sources`;
199199
}
@@ -234,6 +234,25 @@ class CurationState {
234234
}
235235

236236
// edit a bibliographic field (registry + file frontmatter). Filename is unchanged.
237+
// authors is an array (one author → one-element array); the form edits it as a
238+
// comma-separated string. Splits, trims, writes to registry + file frontmatter.
239+
async updateAuthors(value: string): Promise<void> {
240+
const f = this.focused;
241+
if (!f || !this.activeSlug) return;
242+
const authors = value.split(',').map((s) => s.trim()).filter(Boolean);
243+
if ((f.authors ?? []).join('|') === authors.join('|')) return; // no-op
244+
this.replaceSource({ ...f, authors });
245+
await this.call('source.update', {
246+
source_uuid: f.source_uuid,
247+
domain_type: DOMAIN_TYPE,
248+
domain_slug: this.activeSlug,
249+
client_slug: this.clientSlug,
250+
fields: {},
251+
authors,
252+
});
253+
this.saveStatus = this.lastError ? 'authors update failed' : '✓ authors saved';
254+
}
255+
237256
async updateSource(field: 'title' | 'publisher' | 'published_date', value: string): Promise<void> {
238257
const f = this.focused;
239258
if (!f || !this.activeSlug) return;
@@ -377,8 +396,18 @@ class CurationState {
377396

378397
private replaceSource(s: Source): void {
379398
const i = this.sources.findIndex((x) => x.source_uuid === s.source_uuid);
380-
if (i >= 0) this.sources[i] = s;
399+
if (i >= 0) this.sources[i] = withSlug(s);
381400
}
382401
}
383402

403+
// Belt-and-suspenders: if a source arrives without source_slug but with a
404+
// corpus_path, derive the on-disk filename from the path so the Filename field
405+
// (and rename) always connect to a file that exists.
406+
function withSlug(s: Source): Source {
407+
if (s.source_slug || !s.corpus_path) return s;
408+
const base = s.corpus_path.split('/').pop() ?? '';
409+
const slug = base.replace(/\.md$/, '');
410+
return slug ? { ...s, source_slug: slug } : s;
411+
}
412+
384413
export const curation = new CurationState();

apps/strategy-curator/src/types.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ export type Source = {
2323
normalized_url?: string;
2424
url: string;
2525
title?: string;
26+
authors?: string[];
2627
publisher?: string;
2728
published_date?: string;
2829
strategy_slugs?: string[];
Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
---
2+
date_created: 2026-06-29
3+
date_modified: 2026-06-29
4+
title: "Sources capture their own bibliography — authors, publisher, date"
5+
lede: "Fetch a source and it now fills in its own author(s), publisher, and publication date, straight from Jina's structured metadata — authors as an array, all editable, all written to both the file and the registry. Plus a green save-pulse on every field, and a fix so renamed files actually stay connected."
6+
publish: true
7+
authors:
8+
- Michael Staton
9+
augmented_with:
10+
- Claude Code on Claude Opus 4.8 (1M context)
11+
files_changed:
12+
- services/content-ingest/src/jina.ts
13+
- services/content-ingest/src/corpus.ts
14+
- services/record-surrealdb-resolver/src/domains.ts
15+
- apps/strategy-curator/src/curation.svelte.ts
16+
- apps/strategy-curator/src/SourceDetail.svelte
17+
- apps/strategy-curator/src/types.ts
18+
- apps/strategy-curator/src/app.css
19+
- context-v/specs/Strategy-Curator-Entry-Point-for-Augment-It.md
20+
tags:
21+
- Progress-Update
22+
- Strategy-Curator
23+
- Source-Curation
24+
- Jina
25+
- Bibliography
26+
- SurrealDB
27+
- reach-edu
28+
---
29+
30+
## Why Care?
31+
32+
A source isn't just a URL — it's a citation. To ground anything in it you need the
33+
author, the publisher, and the date. Until now the curator caught the title and
34+
nothing else, so every source needed those typed in by hand. That's exactly the
35+
tedious, error-prone work the tool exists to remove.
36+
37+
Now: fetch a source and its **author(s), publisher, and publication date appear on
38+
their own**, pulled from the page's structured metadata. You still own them — every
39+
field is editable — but you start from filled-in, not blank.
40+
41+
## What's New?
42+
43+
- **Auto-filled bibliography.** `source.add` and `source.fetch` extract author,
44+
publisher (`og:site_name`), and published date from Jina and write them to both the
45+
markdown frontmatter and the SurrealDB registry.
46+
- **Authors is an array.** One author → a one-element list; multiple authors → multiple
47+
entries. (Jina hands back a string for one and an array for several; we normalize.)
48+
- **Editable everything.** Title, Filename, Author(s), Publisher, Published date — all
49+
in the source form, all hand-correctable, each writing through to file + registry.
50+
- **Green save-pulse.** Hit Enter (or click away) in a field and its border pulses green,
51+
then fades — a "saved ✓" right where you typed. No pulse if the save errored.
52+
- **Renamed files stay connected.** Fixed a gap where an in-session source could lose its
53+
link to its file on disk.
54+
55+
## Under the Hood — the one-line bug that hid everything
56+
57+
The bibliographic data was always in Jina's response — we were just asking for the
58+
wrong format. `r.jina.ai` returns a markdown preamble (title + date only) *or* a JSON
59+
body whose `data.metadata` carries author and `og:site_name`. We switched to JSON, wrote
60+
all the extraction code... and it did nothing. The author and publisher kept coming back
61+
empty.
62+
63+
The cause: a stale `Accept: 'text/markdown'` header. An earlier edit replaced the
64+
response-*handling* code but the window started one line below the header, so the request
65+
still asked for markdown. Every fetch ran the JSON parser against a markdown body,
66+
`JSON.parse` threw, and the code fell to the partial fallback path. **All the right code,
67+
dead behind a failing parse.** One character — `application/json` — and authors,
68+
publisher, and date all lit up:
69+
70+
```yaml
71+
authors:
72+
- "Alana Semuels"
73+
publisher: "The Atlantic"
74+
published_date: "2016-06-02"
75+
```
76+
77+
The lesson logged for next time: when a feature is "implemented but inert," check the
78+
*request*, not just the handler — and verify the running container actually has the line
79+
you think it does.
80+
81+
## The "renamed files won't connect" fix
82+
83+
A source added during a session came back from the resolver without its `source_slug`, so
84+
its Filename field sat empty and rename had nothing to point at — even though the database
85+
knew the file perfectly well. Two fixes: the `source.add` / `fetch` / `retry` responses now
86+
echo `source_slug`, and the UI derives the slug from `corpus_path` as a fallback. A file
87+
that exists never shows a blank filename again.
88+
89+
## What's Next
90+
91+
- A **paid Jina key** (`JINA_API_KEY` in `augment-it/.env`) for more consistent extraction
92+
and higher rate limits — the free tier honors the JSON request less reliably.
93+
- The still-pending **interstitial detector** and **auto text-extraction from attached PDFs**
94+
from the prior entry, [[2026-06-29_01_Strategy-Curator-Source-Surface-Curate-Sources-By-Domain-Type]].
95+
96+
## Files Touched
97+
98+
content-ingest's `jina.ts` (JSON extraction) and `corpus.ts` (bib frontmatter + authors
99+
list), the resolver's `domains.ts` (registry columns + response echo), and the
100+
strategy-curator UI (`curation.svelte.ts`, `SourceDetail.svelte`, `types.ts`, `app.css`).
101+
Spec bumped to 0.0.0.6 with an Increment 4 section.

context-v/specs/Strategy-Curator-Entry-Point-for-Augment-It.md

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ authors:
77
- Michael Staton
88
augmented_with:
99
- Claude Code on Claude Opus 4.8 (1M context)
10-
semantic_version: 0.0.0.5
10+
semantic_version: 0.0.0.6
1111
status: Implementing
1212
tags:
1313
- Spec
@@ -527,6 +527,31 @@ base64 over NATS, so `nats.conf` `max_payload` gates it: bumped **8MB → 48MB**
527527
If the backend ever moves off-box (no shared filesystem), the right evolution is a direct HTTP
528528
upload endpoint that bypasses NATS entirely — noted, not built.
529529

530+
### Increment 4 — bibliographic capture (Jina JSON) + save UX
531+
- **Jina JSON, not markdown.** content-ingest now requests `Accept: application/json` from
532+
`r.jina.ai`. The markdown preamble only carries title + published time; the JSON `data.metadata`
533+
block carries `author`, `og:site_name` (→ publisher), and `article:published_time` (→ date). We
534+
extract all three on `source.add` and `source.fetch` and write them to the file frontmatter
535+
**and** the SurrealDB `sources` registry. A re-fetch preserves analyst-corrected values when
536+
Jina omits a field. (Gotcha: a stale `Accept: 'text/markdown'` header silently routed every
537+
fetch through the markdown fallback for a while — all the JSON-metadata code was dead behind a
538+
failing `JSON.parse`. The header is the load-bearing line.)
539+
- **`authors` is an array.** Jina returns author as a *string* for one author and an *array* for
540+
several; we normalize to `string[]` (one author → a one-element array). Stored as a YAML list
541+
in frontmatter, an array column in the registry, edited in the form as a comma-separated field.
542+
- **Editable bib fields.** `SourceDetail` exposes Title / Filename / **Author(s)** / Publisher /
543+
Published date — all auto-filled on fetch, all hand-editable, each writing through to registry
544+
+ file.
545+
- **Paid Jina key** is read from `JINA_API_KEY` (compose env ← `augment-it/.env`); absent → free
546+
tier, which honors `Accept: application/json` less consistently.
547+
- **Save-confirmation pulse.** A `commitOnEdit` Svelte action commits each field on Enter (blur)
548+
or change and pulses the border green (`--color-confidence-high`) on success, fading back over
549+
~1.6s — "saved ✓" on the field itself. No green on error.
550+
- **source_slug echo + fallback.** `source.add` / `fetch` / `retry` now echo `source_slug` so an
551+
in-session source connects to its on-disk file immediately (the Filename field + rename had been
552+
blank until a reload). The UI also derives the slug from `corpus_path` as a fallback, so the
553+
field never goes blank for a file that exists.
554+
530555
### Gotchas the live stack taught us (checklist for the next SurrealDB-over-NATS service)
531556
1. **Strict mode → `DEFINE TABLE/INDEX IF NOT EXISTS` before any use** (run once via an
532557
`ensureDomainSchema` guard).

0 commit comments

Comments
 (0)