Skip to content

Commit 79a10ea

Browse files
authored
Merge pull request #98 from stack127labs/research/traffic-growth-causes
Add a repeatable way to publish chosen self-host apps
2 parents 9ed1292 + 59cc352 commit 79a10ea

17 files changed

Lines changed: 724 additions & 171 deletions

‎content_docs/self-host-apps/data-workflow.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ recurring discovery
1212
-> provider refresh
1313
```
1414

15-
Production is canonical. Local is a disposable mirror for development and QA.
15+
Production is canonical. Local is a disposable mirror for development and QA. Claim, review, release, estimate and single-app upsert examples target development explicitly. For authorized production work on homelab, replace `--environment development` with `--environment production --confirm stack127-production`. These commands ignore `NODE_ENV` and inherited `DATABASE_URL` and reject `--database-url`. Discovery and scheduled refresh commands keep their existing environment behavior.
1616

1717
## Discovery
1818

@@ -31,12 +31,14 @@ npm run self-host-apps:discover-app-catalogs -- --catalog all --limit 5000 --wri
3131
```bash
3232
npm run self-host-apps:dedupe-candidates -- --dry-run
3333
npm run self-host-apps:dedupe-candidates
34-
npm run self-host-apps:select-candidate-batch -- --limit 100 --out <batch.json>
34+
npm run self-host-apps:select-candidate-batch -- --environment development --limit 100 --out <batch.json>
3535
npm run self-host-apps:collect-github-evidence -- --file <batch.json>
3636
```
3737

3838
Selection uses `FOR UPDATE SKIP LOCKED` and marks claimed rows `enriching`, so two workers do not take the same work. Reviewers use `references/review-format.md`. Put supporting links in `sources` and `data_source_urls`; use `technical_fact_status` for checked-but-unpublished technical facts. There is no field-source table.
3939

40+
Before a review commits, claiming the same IDs again is safe. Release an abandoned claim with `npm run self-host-apps:release-candidate-batch -- --environment development --file <claimed.json> --to needs_review`; production requires the production environment and confirmation above. After the review commits, resume at estimates without reclaiming or reapplying the review. Estimates and repository refreshes can be re-run.
41+
4042
The legacy `self-host-apps:reset-categories` command is retired and exits without database access. It previously replaced all category links using imported labels and keyword matches. Existing categories need source review too. Correct them per app with `apply-qa-overrides.mjs`, `review_scope: "partial"`, expected categories, and official evidence URLs. Do not infer purpose from incidental words such as image, model, or server.
4143

4244
The review applier validates and moves data. Publication requires an official identity, a managed icon in `stack127-media`, one to three controlled categories, a verified self-hosting classification, install documentation, and at least one source URL. Open and source-available apps need a verified repository and license. Closed-source apps need an official website and publisher identity, but no repository.
@@ -59,7 +61,7 @@ Estimate RAM and storage for every published app for a typical small self-hosted
5961

6062
```bash
6163
npm run self-host-apps:propose-resource-estimates -- --limit 5000 --out <resource-estimates.json>
62-
npm run self-host-apps:apply-resource-estimates -- --file <reviewed-estimates.json> --apply-reviewed
64+
npm run self-host-apps:apply-resource-estimates -- --environment development --file <reviewed-estimates.json> --apply-reviewed
6365
```
6466

6567
An estimate record needs a range, kind, confidence, a short internal basis, and a link to the official project or publisher page. Use `review.method: "app_estimate"` for a practical estimate. Keep published requirements as source evidence for the estimate, not as a separate public RAM or storage label. Do not use a package memory cap as an app requirement.

‎scripts/self-host-apps/README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,10 @@ Discovery stores raw evidence. It cannot publish or assign final categories, sum
2121

2222
## Review movement
2323

24-
- `select-candidate-batch.mjs` atomically claims queued and name-review rows.
24+
- `select-candidate-batch.mjs` atomically claims queued and name-review rows by rank, or exact IDs with `--ids` / `--ids-from <reviewed.json>`. `--match <name-or-repository>` is read-only lookup.
2525
- `collect-github-evidence.mjs` collects official API evidence without editing apps.
2626
- `apply-candidate-review.mjs` validates reviewed decisions and writes them transactionally.
27-
- `release-candidate-batch.mjs` releases an abandoned claim.
27+
- `release-candidate-batch.mjs` releases only abandoned enriching claims, through the shared environment and production confirmation gate.
2828

2929
`apply-candidate-review.mjs` requires source links for published records. Unknown values stay `null` or `unknown`; checked-but-unpublished facts use `technical_fact_status`.
3030

@@ -39,4 +39,6 @@ Discovery stores raw evidence. It cannot publish or assign final categories, sum
3939

4040
Production is canonical. `sync-prod-to-local.mjs` rebuilds the disposable local mirror. `sync-local-to-prod.mjs` is an exceptional guarded upsert, requires `--confirm-reviewed-push`, and does not draft missing production rows unless `--draft-missing` is also explicit.
4141

42+
For targeted additions, follow “Add one app on request” in `stack127_docs/self-host-apps-data.md`. Do not use `sync-local-to-prod` for single apps. Editorial commands (including claim release) use `--environment development` (default) or `--environment production --confirm stack127-production`, including production previews. GitLab/Codeberg and install-option refreshes accept `--slug`; scheduled refresh callers keep their existing environment behavior.
43+
4244
See `content_docs/self-host-apps/data-workflow.md` and `scripts/self-host-apps/scripts.json`.

‎scripts/self-host-apps/apply-candidate-review.mjs‎

Lines changed: 45 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ import { buildInstallOptions } from "./lib/install-options.mjs";
1616
import { installMethods, installOptionsToRow } from "./lib/streamlined-storage.mjs";
1717
import { isManagedMediaUrl } from "../stack127/media/storage.mjs";
1818

19+
import { databaseOptions } from "./lib/environment.mjs";
20+
1921
const decisions = new Set(["publish", "duplicate", "reject", "needs_review"]);
2022

2123
function reviewRecords(input) {
@@ -100,7 +102,7 @@ async function ensureCategories(client, categories) {
100102
}
101103
}
102104

103-
export async function upsertApp(client, record, expectedAppId) {
105+
export async function upsertApp(client, record, expectedAppId, dryRun = false) {
104106
const existing = await client.query("select id from public.self_host_apps where slug = $1", [record.slug]);
105107
if (existing.rows[0] && expectedAppId && existing.rows[0].id !== expectedAppId) {
106108
throw new Error(`${record.slug} belongs to a different app than app_id`);
@@ -109,6 +111,8 @@ export async function upsertApp(client, record, expectedAppId) {
109111
throw new Error(`${record.slug} already exists; provide app_id after confirming the merge`);
110112
}
111113

114+
if (dryRun) return existing.rows[0]?.id ?? null;
115+
112116
const values = appColumns.map((column) => record[column]);
113117
const updates = appColumns
114118
.filter((column) => column !== "slug")
@@ -281,23 +285,26 @@ export async function upsertInstallOptions(client, appId, options, catalogSource
281285
);
282286
}
283287

284-
async function candidateById(client, id) {
285-
const result = await client.query("select * from public.self_host_app_candidates where id = $1 for update", [id]);
288+
async function candidateById(client, id, dryRun) {
289+
const result = await client.query("select * from public.self_host_app_candidates where id = $1" + (dryRun ? "" : " for no key update"), [id]);
286290
if (!result.rows[0]) throw new Error(`Missing candidate: ${id}`);
287291
return result.rows[0];
288292
}
289293

290-
async function applyReview(client, review) {
294+
async function applyReview(client, review, dryRun = false) {
291295
const decision = review.decision ?? "publish";
292296
if (!decisions.has(decision)) throw new Error(`Unsupported decision: ${decision}`);
293297
if (!review.candidate_id) throw new Error("candidate_id is required");
294-
const candidate = await candidateById(client, review.candidate_id);
298+
const candidate = await candidateById(client, review.candidate_id, dryRun);
295299
if (candidate.status !== "enriching") {
296300
throw new Error(`Candidate ${candidate.id} is ${candidate.status}; claim it before applying a review`);
297301
}
298302

299303
if (decision === "duplicate") {
300304
if (!review.app_id) throw new Error("duplicate decisions require app_id");
305+
const target = await client.query("select id from public.self_host_apps where id = $1", [review.app_id]);
306+
if (!target.rows[0]) throw new Error(`Missing duplicate app: ${review.app_id}`);
307+
if (dryRun) return { candidate_id: candidate.id, decision, app_id: review.app_id };
301308
await preserveCandidateSources(client, candidate, review.app_id);
302309
await client.query(
303310
`
@@ -313,6 +320,7 @@ async function applyReview(client, review) {
313320

314321
if (decision === "reject" || decision === "needs_review") {
315322
if (!review.reason) throw new Error(`${decision} decisions require reason`);
323+
if (dryRun) return { candidate_id: candidate.id, decision };
316324
const candidateStatus = decision === "reject" ? "rejected" : decision;
317325
await client.query(
318326
`
@@ -331,6 +339,16 @@ async function applyReview(client, review) {
331339
throw new Error(`${validation.record.slug}: ${validation.errors.join("; ")}`);
332340
}
333341

342+
for (const slug of validation.record.categories) {
343+
if (!getSelfHostAppCategory(slug)) throw new Error(`Unknown category: ${slug}`);
344+
}
345+
if (dryRun) {
346+
const sourceRows = catalogSourceRows(candidate, review);
347+
const options = Array.isArray(review.install_options) ? review.install_options : buildInstallOptions(validation.record, sourceRows);
348+
if (options.length) installOptionsToRow(options, { catalogSources: sourceRows, checkedAt: new Date().toISOString() });
349+
const appId = await upsertApp(client, validation.record, review.app_id ?? null, true);
350+
return { candidate_id: candidate.id, decision, app_id: appId, slug: validation.record.slug };
351+
}
334352
await ensureCategories(client, validation.record.categories);
335353
const appId = await upsertApp(client, validation.record, review.app_id ?? null);
336354
await replaceCategoryLinks(client, appId, validation.record.categories);
@@ -361,24 +379,31 @@ async function applyReview(client, review) {
361379
return { candidate_id: candidate.id, decision, app_id: appId, slug: validation.record.slug };
362380
}
363381

382+
export async function applyReviews(client, reviews, dryRun = false) {
383+
const results = [];
384+
const ids = reviews.map((review) => review.candidate_id);
385+
if (new Set(ids).size !== ids.length) throw new Error("Repeated candidate_id in review batch.");
386+
await client.query(dryRun ? "begin read only" : "begin");
387+
try {
388+
// Lock in ID order, the same order claims and dedupe use, so batches in any order can't deadlock with them.
389+
if (!dryRun) {
390+
await client.query("select id from public.self_host_app_candidates where id = any($1::uuid[]) order by id for no key update", [ids.filter(Boolean)]);
391+
}
392+
for (const review of reviews) results.push(await applyReview(client, review, dryRun));
393+
await client.query(dryRun ? "rollback" : "commit");
394+
return results;
395+
} catch (error) {
396+
await client.query("rollback");
397+
throw error;
398+
}
399+
}
400+
364401
async function main() {
365402
const args = parseArgs();
403+
const databaseUrl = databaseOptions(args);
366404
if (!args.file) throw new Error("Pass --file with reviewed candidate decisions.");
367-
const input = await readJson(args.file);
368-
const reviews = reviewRecords(input);
369-
const results = [];
370-
371-
await withClient(async (client) => {
372-
await client.query("begin");
373-
try {
374-
for (const review of reviews) results.push(await applyReview(client, review));
375-
if (args["dry-run"]) await client.query("rollback");
376-
else await client.query("commit");
377-
} catch (error) {
378-
await client.query("rollback");
379-
throw error;
380-
}
381-
}, args["database-url"]);
405+
const reviews = reviewRecords(await readJson(args.file));
406+
const results = await withClient((client) => applyReviews(client, reviews, Boolean(args["dry-run"])), databaseUrl);
382407

383408
console.log(JSON.stringify({ dry_run: Boolean(args["dry-run"]), results }, null, 2));
384409
}

0 commit comments

Comments
 (0)