milestone(augment-from-db): Augment from DB working is v1, will need usability iterations

Marks the five-step run landing the flow end to end (steps 1–5,
commits 8d141ca..251dda4): service capabilities + Exa, the
org-workbench remote (:3014), the search-and-add remote (:3016),
the people reveal with automatic affiliations, and stream-scan
with corpus dedup. Spec context-v/specs/Augment-From-DB-Flow.md
is Shipped at v0.1.0.0; running pinned to reach-edu.

Known follow-ups live in the spec's post_ship_note: operator
usability passes on all five surfaces, fire-log persistence,
pack-template seed terms, and pinned-deploy remote URLs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014zrFkWSVgTkoQyiobdBjrd

Author: mpstaton

README.md

@@ -29,7 +29,7 @@ The codebase is a federated set of small Svelte 5 microfrontends mounted into a
 - **SurrealDB Cloud** as the canonical entity layer — persons, organizations, and domains/sources (the corpus catalog: strategies, theses, topics) live here, shared across every client workspace and resolved via `record-surrealdb-resolver`. This is the "start from data already in the canonical layer" side of the app, alongside the original CSV-first pipeline — see [Core concepts](#core-concepts)
 - **didi.sh** as the shared identity plane — magic-link sign-in, org/membership-gated access, actor attribution (`created_by`/`updated_by`) on every canonical write. One account works across this app and its sibling Lossless VC-tooling apps (dididecks, memopop)
 - **Anthropic** for free-form prompt enrichment (via `prompt-runner`) and the in-app chat agent ("didi") — same service, two jobs
-- **Search providers are pluggable** (via `social-search`): a `connectors/` seam with a common `Connector` interface, dispatched per-fire with an optional `provider_override`. **SearXNG** (self-hosted, no API key) is the default for the social packs; **Tavily** stays wired in as a peer for content-RAG packs. Response Reviewer's by-record view exposes both — each record has a SearXNG row and a Tavily row of per-pack run icons, so any source can be re-fired on any record through either provider (additive; never overrides accepted data)
+- **Search providers are pluggable** (via `social-search`): a `connectors/` seam with a common `Connector` interface, dispatched per-fire with an optional `provider_override`, plus a capability-style connector registry (`registry/`) that resolves providers by intent, free-tier first. **SearXNG** (self-hosted, no API key) is the default for the social packs; **Tavily**, **SerpApi**, **GDELT**, **Google News RSS**, and **Exa** (neural search, `EXA_AI_API_KEY`) are registered peers. Response Reviewer's by-record view exposes per-provider re-fires, and the `search-and-add` remote exposes the same registry as a chip palette behind an always-editable search term (additive; never overrides accepted data)
 
 ## App structure
 
@@ -48,7 +48,9 @@ augment-it/
 │   ├── records-surface/         :3011 # Live view of promoted record sets over the workspace WebSocket
 │   ├── affiliation-rating-resolver/ :3012 # Rate + enrich person↔org affiliations in place — links, corpus, relevance — one screen per edge
 │   ├── sort-filter-lens/        :3013 # First Lens — sort/filter/inline-edit over the active record set
+│   ├── org-workbench/           :3014 # "Augment from DB" — org-first workbench: search to a canonical org, work its links/streams/corpus/people card in place
 │   ├── person-enrichment/       :3015 # Sparse-person triage — name/socials/org/web presence, one attendee at a time
+│   ├── search-and-add/          :3016 # Editable-term web search + stream scan, paired with org-workbench — one-click add to the launching entity
 │   ├── strategy-curator/        :3017 # "Corpora Curator" — build a strategy/thesis corpus from the SurrealDB canonical layer, live multi-operator sync
 │   ├── highlight-collector/           # planned — collect highlights from AI responses (scaffold)
 │   └── insight-manager/               # planned — manage insights across responses (scaffold)
@@ -100,11 +102,12 @@ augment-it/
 - **Packs and bundles.** A **pack** is the atomic enrichment unit — one source, one prompt-snippet template, one extraction schema, one render config. A **bundle** is a workflow composition of packs with orchestration, carry-forward between passes, and a single chat verb that fires the whole thing. The contracts are locked in `context-v/blueprints/Packs-and-Bundles-Pattern.md`.
 - **Sibling-payload responses.** Every response record carries both prose AND an optional structured `Candidate` (url, display_name, confidence 0–100, snippet, source_metadata), plus an `outcome` enum (`found | not_found | error | skipped | pending`). The renderer in Response Reviewer branches on outcome.
 - **Inline correction + human-supply on one surface.** The by-record view in Response Reviewer lets the user edit URLs, edit display_names, edit entity-names (the row's identity column), or supply a URL the pack didn't find — all riding the same `response.set_structured` subject.
-- **Federation-host shell, multiple Flows.** The shell used to walk one hardcoded `ROTATION`; it now owns a `FLOWS` registry (`shell/src/flows.svelte.ts`) — each Flow (`Improve a CSV`, `Build Corpora`, `Augment a CSV of Event Attendees`, `Augment a CSV of People`, `Rate Affiliations`, …) is a named use-case with its own rotation, switched via the header's "Flows" popdown. Within a Flow the shell still handles peek-deck tile rotation, co-existence (50/50 splits), and runtime cross-remote communication via `window` events + localStorage. A single-tenant (pinned) deploy defaults to its one relevant Flow rather than the first-registered one.
+- **Federation-host shell, multiple Flows.** The shell used to walk one hardcoded `ROTATION`; it now owns a `FLOWS` registry (`shell/src/flows.svelte.ts`) — each Flow (`Improve a CSV`, `Build Corpora`, `Augment a CSV of Event Attendees`, `Augment a CSV of People`, `Rate Affiliations`, `Augment from DB`, …) is a named use-case with its own rotation, switched via the header's "Flows" popdown. Within a Flow the shell still handles peek-deck tile rotation, co-existence (50/50 splits), and runtime cross-remote communication via `window` events + localStorage. A single-tenant (pinned) deploy defaults to its one relevant Flow rather than the first-registered one.
 - **Lenses.** A *lens* is a federated remote that re-presents the active record set under a different affordance shape — sort/filter, inline-edit, per-row corpus add — without leaving the record. `sort-filter-lens` is the first; registered as a third member of `AUGMENT_COMPOSITE` alongside PTM + Pack Runner. Lenses auto-fall-back to the newest non-archived record set when localStorage points at an archived one, so they survive `/promote-snapshot` cleanly.
 - **Funder-content corpus.** Per-client, per-funder directory of source materials backing each row. Two entry vectors land into the same shape: the chat `/inbox <url>` verb (with active-client context) and the per-row inline "+ URL" affordance in the lens. Both run fire-and-forget through `services/content-ingest/`, Jina-extract markdown, preserve original PDFs as LFS binaries, and stamp `record_uuid` + `published_at` into frontmatter. Manual-paste URLs land regardless of domain (operator curation trumps the same-host rule, which only binds pack outputs).
 - **Corpus chips tell the truth.** `corpus.list_for_record` joins by `corpus_funder_slug` as primary (one dir scan) with `record_uuid` lineage as fallback — chips stay accurate across `/promote-snapshot` cuts. `/promote-snapshot` itself derives `corpus_*` columns from filesystem state when cutting a new record set, and stitches `predecessor_record_set_id` for lineage walks.
 - **The canonical layer — a second, DB-first way to build a corpus.** Every flow above starts from an uploaded CSV. `strategy-curator` ("Corpora Curator" in the UI) starts the other direction: pick or create a **domain** — a `type`-discriminated grouping (`strategy`, `thesis`, `topic`, `market-segment`, …) stored in SurrealDB, not the filesystem — and gather **sources** into it (Jina-fetched metadata, full-content fetch, tags, pasted extracts). `record-surrealdb-resolver` owns the DB side (the `domains`/`sources`/`source_usages` tables plus `persons`/`organizations`/`affiliations`); `content-ingest` mirrors every write to an on-disk corpus file. `person-db-resolver`, `record-db-resolver`, and `affiliation-rating-resolver` are the sibling canonical-layer flows — matching/creating people and organizations, then rating the relevance of the affiliation between them — all writing into the same shared entity graph rather than a per-upload CSV schema.
+- **Augment from DB — the org-first workbench.** The newest canonical-layer flow inverts the CSV pipeline completely: start from an **organization already in SurrealDB**. `org-workbench` autocompletes to an org (names, aliases, or domains), then shows one card that views AND edits in place — identity/social links, pulse streams (`media_streams`), corpus items, and a reveal of every affiliated person with nested links of their own. Adding a person generates the `affiliations` edge + observation automatically (`person.affiliate`, org pre-bound — no affiliation UI needed, N orgs per person by construction). Every list carries a 🔍 that opens `search-and-add` in a paired tile: the search term stays always-editable, providers swap via the registry palette (SearXNG default, Exa a chip away), and each result row one-click-adds to exactly the list that launched the search. Streams go further — a per-stream **scan** fires the entity-pulse blog machinery at the stream URL and badges anything already in the `content_items` ledger, so "what's new on their blog" is one click and only genuinely-new items get added. Spec: `context-v/specs/Augment-From-DB-Flow.md` (Shipped).
 - **Live multi-operator sync.** Domain and source mutations on the canonical layer broadcast over NATS (`domain.created`, `source.added`, …); every connected session in the same workspace refetches automatically. Two people signed into the same client see each other's edits without a refresh.
 - **didi.sh identity + actor attribution.** Sign-in is magic-link only (no passwords, invite-only), via the shared `id.didi.sh` service — one account works across this app and its sibling Lossless VC-tooling apps. A signed-in session's `didi_id` rides every capability call and gets stamped as `created_by`/`updated_by` on canonical writes (DB rows and corpus frontmatter alike), so every mutation on the canonical layer carries who did it. The in-app chat agent ("didi") writes through the same envelope, tagged `via: didi-agent`, so an agent-driven edit is distinguishable from a manual one.
 

context-v/issues/Search-Providers-as-First-Class-SearXNG-Default.md

@@ -10,10 +10,10 @@ augmented_with:
   - Claude Code on Claude Fable 5
 semantic_version: 0.0.0.4
 revisions:
-  - 2026-07-21 — Status sweep, resolved. The core ask fully landed: SearXNG shipped as peer provider + social-pack default with provider_override on 2026-06-01 (changelog 2026-06-01_01), and the capability registry (registry/, register-connectors.ts booting searxng/tavily/serpapi/gdelt/google-news-rss) landed 2026-06-02 — going beyond this issue's Layer 1. Never done from the original plan: step 7's foundation-dataset smoke (the ≥60% acceptance number) and step 8's blueprint write-up. Those are noted, not blocking — the architecture the issue argued for is in production. (semver 0.0.0.4)
-  - 2026-05-26 — Initial draft as "Switch Search Substrate from Tavily to SearXNG." Framed the move as a substrate swap with Tavily preserved as a side-effect. (semver 0.0.0.1)
-  - 2026-05-27 — Reframed. The decision is **provider plurality as a first-class architectural concern**, not a substrate swap. SearXNG becomes the new default for social packs; Tavily stays as a peer for content-RAG packs; future providers (Brave, Google CSE, ProPublica NPO, Candid, LinkedIn-direct, MCP-server-as-provider) plug in via the same connector interface. Added §The iteration loop we're building toward — per-row, per-pack provider selection as a future product affordance. File renamed from `Switch-Search-Substrate-from-Tavily-to-Searxng.md` to `Search-Providers-as-First-Class-SearXNG-Default.md` to match. (semver 0.0.0.2)
-  - 2026-05-28 — Layer 1 (the connector plumbing, proposed-work steps 1–6) landed in code. `connectors/{types,index,tavily,searxng}.ts` exist, `PackConfig` is provider-aware, all common-seven social packs default to `connector: 'searxng'`, `runOnePackSearch` dispatches on provider and accepts `provider_override`, and the SearXNG container + `settings.yml` are in `docker-compose.yml`. Steps 7 (foundation-dataset smoke / the ≥60% acceptance number) and 8 (blueprint write-up) remain, as does the per-record iteration UI in response-reviewer :3005. (semver 0.0.0.3)
+  - "2026-07-21 — Status sweep, resolved. The core ask fully landed: SearXNG shipped as peer provider + social-pack default with provider_override on 2026-06-01 (changelog 2026-06-01_01), and the capability registry (registry/, register-connectors.ts booting searxng/tavily/serpapi/gdelt/google-news-rss) landed 2026-06-02 — going beyond this issue's Layer 1. Never done from the original plan: step 7's foundation-dataset smoke (the ≥60% acceptance number) and step 8's blueprint write-up. Those are noted, not blocking — the architecture the issue argued for is in production. (semver 0.0.0.4)"
+  - "2026-05-26 — Initial draft as \"Switch Search Substrate from Tavily to SearXNG.\" Framed the move as a substrate swap with Tavily preserved as a side-effect. (semver 0.0.0.1)"
+  - "2026-05-27 — Reframed. The decision is **provider plurality as a first-class architectural concern**, not a substrate swap. SearXNG becomes the new default for social packs; Tavily stays as a peer for content-RAG packs; future providers (Brave, Google CSE, ProPublica NPO, Candid, LinkedIn-direct, MCP-server-as-provider) plug in via the same connector interface. Added §The iteration loop we're building toward — per-row, per-pack provider selection as a future product affordance. File renamed from `Switch-Search-Substrate-from-Tavily-to-Searxng.md` to `Search-Providers-as-First-Class-SearXNG-Default.md` to match. (semver 0.0.0.2)"
+  - "2026-05-28 — Layer 1 (the connector plumbing, proposed-work steps 1–6) landed in code. `connectors/{types,index,tavily,searxng}.ts` exist, `PackConfig` is provider-aware, all common-seven social packs default to `connector: 'searxng'`, `runOnePackSearch` dispatches on provider and accepts `provider_override`, and the SearXNG container + `settings.yml` are in `docker-compose.yml`. Steps 7 (foundation-dataset smoke / the ≥60% acceptance number) and 8 (blueprint write-up) remain, as does the per-record iteration UI in response-reviewer :3005. (semver 0.0.0.3)"
 tags:
   - Issue
   - Augment-It

context-v/specs/Augment-From-DB-Flow.md

@@ -12,10 +12,10 @@ date_first_published: 2026-07-22
 exploration_of_record: "[[../explorations/Augment-From-DB-Flow-Two-New-Microfrontends]]"
 post_ship_note: "All five phases implemented, proven, and pushed on 2026-07-22 (attempt(augment-from-db, …, step1–5)). Pending: the operator browser walk-throughs each phase plan names, and the parked open questions (fire-log persistence, pack-template seed terms, pinned-deploy env-configurable remote URLs). Deviations from the as-specced snippets are recorded in each phase plan's post_ship_note — notably organization.streams.add (a verb the spec missed), the localStorage-hardened D2 launch contract, and relevance as string|null."
 revisions:
-  - 2026-07-22 — v0.1.0.0: status → Shipped. Phases 2–5 executed same day ([[../plans/Augment-From-DB-Phase-2-Org-Workbench-Remote]], [[../plans/Augment-From-DB-Phase-3-Search-And-Add-Remote]], [[../plans/Augment-From-DB-Phase-4-People-Reveal-And-Add-Person]], [[../plans/Augment-From-DB-Phase-5-Stream-Scan-Mode]]; changelog 2026-07-22_02 through _05). Both remotes live (:3014, :3016); stream-scan flip-test proven against Aspen's blog.
-  - 2026-07-22 — v0.0.1.1: status → Implementing. Phase 1 executed and shipped same day ([[../plans/Augment-From-DB-Phase-1-Service-Capabilities]], changelog 2026-07-22_01) — all four capabilities live and proven over NATS; both flagged SurrealQL constructs worked without fallbacks. Phases 2–5 remain.
-  - 2026-07-22 — v0.0.1.0: user sign-off. Narrative pass ran clean — the doc was written in one pass after the exploration settled, so structure and prose stand as drafted; no TBD markers existed. Next step per the spec→plan cascade: cut Phase 1 as the first plan in context-v/plans/.
-  - 2026-07-22 — v0.0.0.1: initial draft from the exploration of record, grounded in a same-day code read (flows registry, capability dispatcher, resolver handlers, connector registry). Six exploration open questions locked as decisions D1–D6.
+  - "2026-07-22 — v0.1.0.0: status → Shipped. Phases 2–5 executed same day ([[../plans/Augment-From-DB-Phase-2-Org-Workbench-Remote]], [[../plans/Augment-From-DB-Phase-3-Search-And-Add-Remote]], [[../plans/Augment-From-DB-Phase-4-People-Reveal-And-Add-Person]], [[../plans/Augment-From-DB-Phase-5-Stream-Scan-Mode]]; changelog 2026-07-22_02 through _05). Both remotes live (:3014, :3016); stream-scan flip-test proven against Aspen's blog."
+  - "2026-07-22 — v0.0.1.1: status → Implementing. Phase 1 executed and shipped same day ([[../plans/Augment-From-DB-Phase-1-Service-Capabilities]], changelog 2026-07-22_01) — all four capabilities live and proven over NATS; both flagged SurrealQL constructs worked without fallbacks. Phases 2–5 remain."
+  - "2026-07-22 — v0.0.1.0: user sign-off. Narrative pass ran clean — the doc was written in one pass after the exploration settled, so structure and prose stand as drafted; no TBD markers existed. Next step per the spec→plan cascade: cut Phase 1 as the first plan in context-v/plans/."
+  - "2026-07-22 — v0.0.0.1: initial draft from the exploration of record, grounded in a same-day code read (flows registry, capability dispatcher, resolver handlers, connector registry). Six exploration open questions locked as decisions D1–D6."
 tags:
   - Spec
   - Augment-It