Skip to content

Commit af3b1d9

Browse files
authored
feat(feed-directory): last_result demotion and fail-closed catalog v2 (#1250)
* feat(feed-directory): last_result demotion and chrome Support catalog_version 2 only with required last_result; demote empty/error in sort, show ambient indicators, and warn before subscribe on failing rows. * docs: describe catalog last_result Clarify that Feed Directory scrape hints are instance last-known signals; cold instances show no badge and this is not /health. * fix(feed-directory): warn on failing OPML and honest count copy Align bulk export with subscribe warnings and drop "ready-to-use" when the catalog can include empty/error last scrapes.
1 parent 6290117 commit af3b1d9

16 files changed

Lines changed: 332 additions & 24 deletions

File tree

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ If a cross-repo behavior changed but upstream is not updated yet, document the g
4343

4444
- The browse UI is a **thin client**: fetch catalog JSON from the active instance, render rows client-side, build RSS links from each entry's `path`.
4545
- Do not reintroduce `bin/data-update`, `src/data/configs.json`, or a `html2rss-configs` gem dependency in this repo.
46-
- Wire shape v1 is defined in `html2rss-web` request specs and OpenAPI (`catalog_version`, `parameters.schema`, `parameters.defaults`).
46+
- Wire shape v2 is defined in `html2rss-web` request specs and OpenAPI (`catalog_version: 2`, required `last_result`, `meta.starters`). The browse client supports **`[2]` only** and fails closed on v1.
4747
- When the instance is unreachable or returns `404` with `catalog_disabled`, show an error state — no static fallback list.
4848
- **Wire parsing only in** `src/components/feed-directory/adapters/catalog-api.ts`. Domain modules must not parse API envelopes or wire rows.
4949
- See `CONTEXT.md` for glossary (`FeedDirectoryEntry`, catalog seam, instance persistence contract).

‎CONTEXT.md‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,16 +18,30 @@ Normalized domain type for one catalog row after wire parsing. Required fields o
1818
| `title`, `summary`, `topics` | Directory metadata from YAML |
1919
| `channelUrl`, `language` | Channel metadata |
2020
| `parameterSchema`, `parameterDefaults` | Dynamic feed parameters |
21+
| `lastResult` | Instance last-known scrape outcome (see below) |
22+
23+
## LastResult
24+
25+
Required ambient signal from catalog_version **2**. Closed set of `state` values only — do not invent green/yellow/red domain enums; map state to UI chrome in `ui/`.
26+
27+
| `state` | Meaning | Browse UX |
28+
| --------- | ------------------------------------------------------------ | -------------------------------------------- |
29+
| `ok` | Last directory-defaults scrape succeeded | Ambient “Last scrape ok” indicator |
30+
| `empty` | Last scrape returned no items | Demote in sort; warn before subscribe; badge |
31+
| `error` | Last scrape failed | Demote in sort; warn before subscribe; badge |
32+
| `unknown` | Never scraped with directory defaults on this process (cold) | Neutral — no badge; sorts with non-failing |
33+
34+
Wire fields: `code` (string \| null), `at` (ISO timestamp \| null). Missing or invalid `last_result` on a row fails closed (row dropped).
2135

2236
## Catalog seam
2337

2438
The boundary between the instance API and domain logic:
2539

26-
- **Wire:** `GET /api/v1/configs` envelope (`success`, `data.configs`, `meta.catalog_version`)
27-
- **Adapter:** `adapters/catalog-api.ts` — fetch, envelope validation, row validation, version gate (supported: `[1]`)
40+
- **Wire:** `GET /api/v1/configs` envelope (`success`, `data.configs`, `meta.catalog_version`, `meta.starters`)
41+
- **Adapter:** `adapters/catalog-api.ts` — fetch, envelope validation, row validation, version gate (supported: **`[2]` only**; v1 fail closed)
2842
- **Domain:** `FeedDirectoryEntry[]` consumed by filters, OPML build, and render
2943

30-
Wire parsing must stay in `adapters/catalog-api.ts` only.
44+
Wire parsing must stay in `adapters/catalog-api.ts` only. `meta.starters` is parsed for forward compatibility; browse does not render a featured strip today.
3145

3246
## Instance persistence contract
3347

‎src/components/feed-directory/adapters/catalog-api.test.ts‎

Lines changed: 72 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -17,26 +17,63 @@ const validEnvelope = {
1717
channel: { url: 'https://www.anthropic.com/news', language: 'en' },
1818
directory: { title: 'Anthropic — News', summary: 'Announcements.', topics: ['news'] },
1919
parameters: { schema: {}, defaults: {} },
20+
last_result: { state: 'ok', code: null, at: '2026-08-29T08:00:00Z' },
2021
},
2122
{
2223
id: 'bbc.co.uk/available_episodes',
2324
path: '/bbc.co.uk/available_episodes.rss',
2425
channel: { url: 'https://www.bbc.co.uk/programmes/%<id>s/episodes/player', language: 'en-GB' },
2526
directory: { title: 'BBC Sounds — Programme episodes', summary: 'Episodes.', topics: ['media'] },
2627
parameters: { schema: { id: { type: 'string' } }, defaults: { id: 'b006wkfp' } },
28+
last_result: { state: 'unknown', code: null, at: null },
29+
},
30+
{
31+
id: 'example.com/broken-scrape',
32+
path: '/example.com/broken-scrape.rss',
33+
channel: { url: 'https://example.com/broken', language: 'en' },
34+
directory: { title: 'Broken', summary: '', topics: [] },
35+
parameters: { schema: {}, defaults: {} },
36+
last_result: { state: 'error', code: 'EXTRACTION_EMPTY', at: '2026-08-29T09:00:00Z' },
37+
},
38+
{
39+
id: 'example.com/empty-scrape',
40+
path: '/example.com/empty-scrape.rss',
41+
channel: { url: 'https://example.com/empty', language: 'en' },
42+
directory: { title: 'Empty', summary: '', topics: [] },
43+
parameters: { schema: {}, defaults: {} },
44+
last_result: { state: 'empty', code: 'EXTRACTION_EMPTY', at: '2026-08-29T09:30:00Z' },
2745
},
2846
{ id: 'broken' },
47+
{
48+
id: 'missing.last/result',
49+
path: '/missing.last/result.rss',
50+
channel: { url: 'https://missing.example/', language: 'en' },
51+
directory: { title: 'Missing last_result', summary: '', topics: [] },
52+
parameters: { schema: {}, defaults: {} },
53+
},
54+
{
55+
id: 'invalid.last/result',
56+
path: '/invalid.last/result.rss',
57+
channel: { url: 'https://invalid.example/', language: 'en' },
58+
directory: { title: 'Invalid last_result', summary: '', topics: [] },
59+
parameters: { schema: {}, defaults: {} },
60+
last_result: { state: 'green', code: null, at: null },
61+
},
2962
],
3063
},
31-
meta: { total: 2, catalog_version: 1 },
64+
meta: {
65+
total: 4,
66+
catalog_version: 2,
67+
starters: ['anthropic.com/news', 'bbc.co.uk/available_episodes'],
68+
},
3269
};
3370

3471
function mockFetch(response: Partial<Response> & Pick<Response, 'status'>): typeof fetch {
3572
return (async () => response) as typeof fetch;
3673
}
3774

3875
describe('fetchCatalogResponse', () => {
39-
it('maps valid envelope rows and drops invalid ones', async () => {
76+
it('maps valid v2 envelope rows and drops invalid ones', async () => {
4077
const fetchImpl = mockFetch({
4178
ok: true,
4279
status: 200,
@@ -45,16 +82,32 @@ describe('fetchCatalogResponse', () => {
4582

4683
const { entries, meta } = await fetchCatalogResponse('https://example.test/', fetchImpl);
4784

48-
expect(entries).toHaveLength(2);
85+
expect(entries).toHaveLength(4);
4986
expect(entries[0]).toMatchObject({
5087
id: 'anthropic.com/news',
5188
siteKey: 'anthropic.com',
5289
title: 'Anthropic — News',
5390
topics: ['news'],
5491
language: 'en',
92+
lastResult: { state: 'ok', code: null, at: '2026-08-29T08:00:00Z' },
5593
});
5694
expect(entries[1]?.parameterDefaults).toEqual({ id: 'b006wkfp' });
57-
expect(meta).toEqual({ total: 2, catalogVersion: 1 });
95+
expect(entries[1]?.lastResult).toEqual({ state: 'unknown', code: null, at: null });
96+
expect(entries[2]?.lastResult).toEqual({
97+
state: 'error',
98+
code: 'EXTRACTION_EMPTY',
99+
at: '2026-08-29T09:00:00Z',
100+
});
101+
expect(entries[3]?.lastResult).toEqual({
102+
state: 'empty',
103+
code: 'EXTRACTION_EMPTY',
104+
at: '2026-08-29T09:30:00Z',
105+
});
106+
expect(meta).toEqual({
107+
total: 4,
108+
catalogVersion: 2,
109+
starters: ['anthropic.com/news', 'bbc.co.uk/available_episodes'],
110+
});
58111
});
59112

60113
it('throws disabled on 404', async () => {
@@ -75,13 +128,27 @@ describe('fetchCatalogResponse', () => {
75128
);
76129
});
77130

131+
it('throws unsupported version for catalog_version 1 (fail closed)', async () => {
132+
const fetchImpl = mockFetch({
133+
ok: true,
134+
status: 200,
135+
json: async () => ({
136+
...validEnvelope,
137+
meta: { total: 3, catalog_version: 1, starters: [] },
138+
}),
139+
} as Response);
140+
await expect(fetchCatalogResponse('https://example.test/', fetchImpl)).rejects.toBeInstanceOf(
141+
CatalogUnsupportedVersionError
142+
);
143+
});
144+
78145
it('throws unsupported version when catalog_version is not supported', async () => {
79146
const fetchImpl = mockFetch({
80147
ok: true,
81148
status: 200,
82149
json: async () => ({
83150
...validEnvelope,
84-
meta: { total: 2, catalog_version: 99 },
151+
meta: { total: 3, catalog_version: 99, starters: [] },
85152
}),
86153
} as Response);
87154
await expect(fetchCatalogResponse('https://example.test/', fetchImpl)).rejects.toBeInstanceOf(

‎src/components/feed-directory/adapters/catalog-api.ts‎

Lines changed: 36 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,22 @@
11
import { siteKeyFromId } from '../domain/entry';
2-
import type { CatalogLoadError, FeedDirectoryEntry } from '../domain/types';
2+
import { isLastResultState } from '../domain/last-result';
3+
import type { CatalogLoadError, FeedDirectoryEntry, LastResult } from '../domain/types';
34

4-
const SUPPORTED_CATALOG_VERSIONS = [1] as const;
5+
const SUPPORTED_CATALOG_VERSIONS = [2] as const;
56

67
interface CatalogWireEntry {
78
id?: unknown;
89
path?: unknown;
910
channel?: { url?: unknown; language?: unknown };
1011
directory?: { title?: unknown; summary?: unknown; topics?: unknown };
1112
parameters?: { schema?: unknown; defaults?: unknown };
13+
last_result?: unknown;
1214
}
1315

1416
interface CatalogEnvelope {
1517
success?: unknown;
1618
data?: { configs?: unknown };
17-
meta?: { total?: unknown; catalog_version?: unknown };
19+
meta?: { total?: unknown; catalog_version?: unknown; starters?: unknown };
1820
}
1921

2022
export class CatalogDisabledError extends Error {
@@ -80,6 +82,24 @@ function parseParameterDefaults(value: unknown): Readonly<Record<string, string>
8082
return defaults;
8183
}
8284

85+
/** Fail closed: missing or invalid last_result rejects the row. */
86+
function parseLastResult(value: unknown): LastResult | null {
87+
if (!isRecord(value)) return null;
88+
if (!isLastResultState(value.state)) return null;
89+
90+
const code = value.code;
91+
if (!(code === null || typeof code === 'string')) return null;
92+
93+
const at = value.at;
94+
if (!(at === null || typeof at === 'string')) return null;
95+
96+
return {
97+
state: value.state,
98+
code: code === null || code.trim() === '' ? null : code,
99+
at: at === null || at.trim() === '' ? null : at,
100+
};
101+
}
102+
83103
function parseCatalogEntries(configs: unknown): FeedDirectoryEntry[] {
84104
if (!Array.isArray(configs)) return [];
85105

@@ -90,7 +110,8 @@ function parseCatalogEntries(configs: unknown): FeedDirectoryEntry[] {
90110
const id = asString(wire.id);
91111
const path = asString(wire.path);
92112
const channelUrl = asString(wire.channel?.url);
93-
if (!id || !path || !channelUrl) continue;
113+
const lastResult = parseLastResult(wire.last_result);
114+
if (!id || !path || !channelUrl || !lastResult) continue;
94115

95116
entries.push({
96117
id,
@@ -103,6 +124,7 @@ function parseCatalogEntries(configs: unknown): FeedDirectoryEntry[] {
103124
language: asString(wire.channel?.language) ?? '',
104125
parameterSchema: parseParameterSchema(wire.parameters?.schema),
105126
parameterDefaults: parseParameterDefaults(wire.parameters?.defaults),
127+
lastResult,
106128
});
107129
}
108130

@@ -120,9 +142,15 @@ function parseCatalogVersion(meta: CatalogEnvelope['meta']): number {
120142
return version;
121143
}
122144

145+
export interface CatalogMeta {
146+
total: number;
147+
catalogVersion: number;
148+
starters: readonly string[];
149+
}
150+
123151
function parseCatalogEnvelope(payload: unknown): {
124152
entries: FeedDirectoryEntry[];
125-
meta: { total: number; catalogVersion: number };
153+
meta: CatalogMeta;
126154
} {
127155
if (!isRecord(payload)) {
128156
throw new CatalogInvalidEnvelopeError();
@@ -137,17 +165,18 @@ function parseCatalogEnvelope(payload: unknown): {
137165
const catalogVersion = parseCatalogVersion(envelope.meta);
138166
const totalRaw = envelope.meta?.total;
139167
const total = typeof totalRaw === 'number' && Number.isFinite(totalRaw) ? totalRaw : entries.length;
168+
const starters = parseStringArray(envelope.meta?.starters);
140169

141170
return {
142171
entries,
143-
meta: { total, catalogVersion },
172+
meta: { total, catalogVersion, starters },
144173
};
145174
}
146175

147176
export async function fetchCatalogResponse(
148177
instanceUrl: string,
149178
fetchImpl: typeof fetch = fetch
150-
): Promise<{ entries: FeedDirectoryEntry[]; meta: { total: number; catalogVersion: number } }> {
179+
): Promise<{ entries: FeedDirectoryEntry[]; meta: CatalogMeta }> {
151180
const catalogUrl = new URL('/api/v1/configs', instanceUrl).toString();
152181

153182
let response: Response;

‎src/components/feed-directory/app/FeedDirectoryApp.ts‎

Lines changed: 39 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,11 @@ import {
88
} from '../adapters/browser-storage';
99
import { downloadOpml } from '../adapters/browser-download';
1010
import { buildFeedUrl } from '../domain/feed-url';
11+
import { isFailingLastResult } from '../domain/last-result';
1112
import { buildOpmlDocument } from '../domain/opml';
1213
import { normalizeFilterLanguage } from '../domain/language';
1314
import { debounce } from '../lib/debounce';
15+
import type { FeedDirectoryEntry } from '../domain/types';
1416
import { renderFeedDirectory } from '../ui/render';
1517
import {
1618
applyFilterPatch,
@@ -182,6 +184,15 @@ export class FeedDirectoryApp {
182184
this.render();
183185
break;
184186
}
187+
case 'open-feed': {
188+
const entryId = actionEl.dataset.entryId;
189+
const entry = this.findEntry(entryId);
190+
if (!entry) return;
191+
if (isFailingLastResult(entry.lastResult) && !this.confirmFailingSubscribe(entry)) {
192+
event.preventDefault();
193+
}
194+
break;
195+
}
185196
case 'copy-feed':
186197
void this.copyFeed(actionEl.dataset.entryId);
187198
break;
@@ -193,6 +204,19 @@ export class FeedDirectoryApp {
193204
}
194205
}
195206

207+
private findEntry(entryId: string | undefined): FeedDirectoryEntry | undefined {
208+
if (!entryId) return undefined;
209+
return this.state.entries.find((item) => item.id === entryId);
210+
}
211+
212+
private confirmFailingSubscribe(entry: FeedDirectoryEntry): boolean {
213+
const detail =
214+
entry.lastResult.state === 'empty'
215+
? 'The last known scrape on this instance returned no items.'
216+
: 'The last known scrape on this instance failed.';
217+
return window.confirm(`${detail} Feeds that recently failed often fail again. Subscribe anyway?`);
218+
}
219+
196220
private async applyInstance(): Promise<void> {
197221
const normalized = normalizeInstanceUrl(this.state.instanceDraft);
198222
if (!normalized) {
@@ -217,9 +241,9 @@ export class FeedDirectoryApp {
217241
}
218242

219243
private async copyFeed(entryId: string | undefined): Promise<void> {
220-
if (!entryId) return;
221-
const entry = this.state.entries.find((item) => item.id === entryId);
222-
if (!entry) return;
244+
const entry = this.findEntry(entryId);
245+
if (!entry || !entryId) return;
246+
if (isFailingLastResult(entry.lastResult) && !this.confirmFailingSubscribe(entry)) return;
223247

224248
const url = buildFeedUrl(this.state.instanceUrl, entry, this.state.parametersById[entryId] ?? {});
225249
try {
@@ -242,6 +266,18 @@ export class FeedDirectoryApp {
242266
private exportOpml(): void {
243267
const { filteredEntries } = selectPagedEntries(this.state);
244268
if (filteredEntries.length === 0) return;
269+
270+
const failingCount = filteredEntries.filter((entry) => isFailingLastResult(entry.lastResult)).length;
271+
if (failingCount > 0) {
272+
const detail =
273+
failingCount === 1
274+
? '1 feed in this export had an empty or failed last scrape on this instance.'
275+
: `${failingCount} feeds in this export had an empty or failed last scrape on this instance.`;
276+
if (!window.confirm(`${detail} Feeds that recently failed often fail again. Export anyway?`)) {
277+
return;
278+
}
279+
}
280+
245281
const opml = buildOpmlDocument(this.state.instanceUrl, filteredEntries, this.state.parametersById);
246282
downloadOpml(opml);
247283
}

‎src/components/feed-directory/app/directory-state.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ const entry = (id: string, title: string): FeedDirectoryEntry => ({
1414
language: '',
1515
parameterSchema: {},
1616
parameterDefaults: {},
17+
lastResult: { state: 'unknown', code: null, at: null },
1718
});
1819

1920
describe('applyFilterPatch', () => {

0 commit comments

Comments
 (0)