Skip to content

Commit b155680

Browse files
shhdwiShrish Dwivedi
andauthored
feat(skills): skills store, compile, save and validate (#656)
* feat(trail): add shell parsing, command risk checks and learning fields * feat(trail): read what ran in a session from its transcript * feat(skills): add the skills store, compile, save and validate * fix(trail): find programs, scratch folders and Trail's files on Windows - anchors and the timeout guard split PATH on ';' and look for git.exe, npm.cmd and the like (PATHEXT); Windows has no exec bit to check - a scratchpad folder is found with either slash, and Windows paths are compared without case (scratch folders, tamper's protected places) - tamper's cd from an unknown folder stays unknown on Windows, where resolve() puts a drive before the '/\0' it starts from * test(trail): make part 2's tests hold on Windows - POSIX file modes are asserted only off Windows; the FIFO and /dev/zero cases run only where those exist - expected paths are built with resolve()/join(), and the fold fixture compares checkouts resolved - capped children import the source by file URL, and say why they failed - os.homedir() follows USERPROFILE on Windows, so the cd test sets it too * fix(trail): log only the file's name for a hook error on Windows too A Windows stack frame joins its folders with '\', so the hook-errors log kept the whole path, user name included, with the slashes stripped. * test(trail): give the secret-check speed bar a 250 ms floor, which a backtracking pattern still misses by seconds * fix(skills): a ready probe finds git.exe and npm.cmd on a Windows PATH command -v split PATH on the platform's delimiter but looked only for the bare name, so on Windows every program read as not installed. It now uses pathDirs and programFiles from guard-fast, as anchors and the timeout guard do. * test(skills): make part 3's tests hold on Windows - POSIX file modes are asserted only off Windows - the FIFO cases and the socket, /tmp and ':' PATH probe test run only off Windows - a non-executable timeout on PATH counts on Windows, which has no exec bit - children import the source by file URL * fix(skills): see Windows temp folders as scratch paths when compiling A helper run from C:\Users\<me>\AppData\Local\Temp\...\scratchpad\ was not seen as scratch: the patterns knew only /tmp, /var/folders, $TMPDIR and a /scratchpad/ segment. So it was not copied into the skill, and the skill kept the absolute temp path. Now a scratchpad segment with either slash, the person's AppData\Local\Temp (C:\ or Git Bash's /c/), Windows\Temp, %TEMP% and $env:TEMP count, in any case, and a log into one is dropped. * test(skills): hold part 3's tests on Linux and Windows CI - the helper tests make their folder in /tmp off Windows: the run's random temp folder name read as a secret about one run in ten on Linux - a plain file on PATH counts as a program on Windows, which has no exec bit - the warm wall-hit bar is scaled by machine speed with a 50 ms floor, since Windows counts CPU in ~15.6 ms steps; a lost index still fails * test(skills): let the git probes test finish probes past the 300 ms budget On Windows five git spawns don't fit in one runProbes call, so the last probe is left out, as designed. The test calls again until every probe is in; the done ones come from the cache. --------- Co-authored-by: Shrish Dwivedi <shrish@nanonets.com>
1 parent 89248a9 commit b155680

95 files changed

Lines changed: 15339 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ dist/
66
# `graft build` locally (it re-adds this line itself), so the folder is a
77
# personal cache like node_modules, never a shared artifact.
88
graft/
9+
# Except the graft skill's SKILL.md fixture, which CI checks against the spec.
10+
!test/fixtures/skills/graft/
911

1012
# Env
1113
.env

‎src/init/install-record.ts‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
/**
2+
* D120 (2): the install record, `<trail home>/state/installed/<checkout hash>.json`:
3+
* the repo paths `trail init` wrote in one checkout, each with the sha256 of what
4+
* it wrote (null for a file it deleted). The clean-tree ready probe ignores a
5+
* dirty path only while it is in this record and still holds those bytes, so any
6+
* later edit blocks again.
7+
*
8+
* A path counts as init's when init changed it and it was clean in git just
9+
* before (or the last record already had it, unchanged since). A path the person
10+
* had dirty before init stays theirs, whatever init then wrote into it. Links,
11+
* FIFOs and files over 1 MB are never recorded or vouched for.
12+
*/
13+
import { spawnSync } from 'node:child_process';
14+
import { createHash } from 'node:crypto';
15+
import { closeSync, fstatSync, lstatSync, readSync, realpathSync } from 'node:fs';
16+
import { join } from 'node:path';
17+
import { trailPath, writePrivate } from '../util/private-fs.js';
18+
import { openPlain, readPlain } from './plain-file.js';
19+
20+
const MAX_FILE = 1024 * 1024;
21+
const MAX_PATHS = 200;
22+
const REL = /^(?!\/)(?!.*(?:^|\/)\.\.?(?:\/|$))[\w.@+-]+(?:\/[\w.@+-]+)*$/;
23+
const SHA = /^[0-9a-f]{64}$/;
24+
25+
/** What a recorded path holds: a sha256, or null for "init deleted it". */
26+
export type InstallFiles = Record<string, string | null>;
27+
28+
export interface InstallSnap {
29+
checkout: string;
30+
rels: string[];
31+
before: Map<string, string | null | false>;
32+
dirty: Set<string>;
33+
prior: InstallFiles | null;
34+
}
35+
36+
function real(checkout: string): string {
37+
try {
38+
return realpathSync.native(checkout);
39+
} catch {
40+
return checkout;
41+
}
42+
}
43+
44+
export function installRecordPath(checkout: string, env?: NodeJS.ProcessEnv): string {
45+
const key = createHash('sha256').update(real(checkout)).digest('hex').slice(0, 16);
46+
return trailPath(env, 'state', 'installed', `${key}.json`);
47+
}
48+
49+
/** The sha256 of a plain file's bytes; null when nothing is there; false for a link, a FIFO, a folder or a big file. */
50+
export function fileSha(path: string): string | null | false {
51+
try {
52+
if (!lstatSync(path).isFile()) return false;
53+
} catch {
54+
return null;
55+
}
56+
let fd = -1;
57+
try {
58+
fd = openPlain(path);
59+
const size = fstatSync(fd).size;
60+
if (size > MAX_FILE) return false;
61+
const buf = Buffer.alloc(size);
62+
let off = 0;
63+
while (off < size) {
64+
const n = readSync(fd, buf, off, size - off, off);
65+
if (n <= 0) break;
66+
off += n;
67+
}
68+
return createHash('sha256').update(buf.subarray(0, off)).digest('hex');
69+
} catch {
70+
return false;
71+
} finally {
72+
if (fd >= 0) closeSync(fd);
73+
}
74+
}
75+
76+
/** The record for this checkout, or null when there is none or it names another checkout. */
77+
export function readInstallRecord(checkout: string, env?: NodeJS.ProcessEnv): InstallFiles | null {
78+
let raw: unknown;
79+
try {
80+
const text = readPlain(installRecordPath(checkout, env), MAX_FILE);
81+
if (text === null) return null;
82+
raw = JSON.parse(text);
83+
} catch {
84+
return null;
85+
}
86+
const r = raw as { v?: unknown; checkout?: unknown; files?: unknown };
87+
if (!r || r.v !== 1 || r.checkout !== real(checkout) || !r.files || typeof r.files !== 'object' || Array.isArray(r.files)) return null;
88+
const out: InstallFiles = {};
89+
for (const [rel, sha] of Object.entries(r.files as Record<string, unknown>).slice(0, MAX_PATHS)) {
90+
if (REL.test(rel) && (sha === null || (typeof sha === 'string' && SHA.test(sha)))) out[rel] = sha;
91+
}
92+
return out;
93+
}
94+
95+
/** Whether `rel` is a path init recorded here that still holds what init left (absent, for one it deleted). */
96+
export function initLeftIt(files: InstallFiles | null, checkout: string, rel: string): boolean {
97+
if (!files || !Object.hasOwn(files, rel) || !REL.test(rel)) return false;
98+
return fileSha(join(checkout, rel)) === files[rel];
99+
}
100+
101+
/** The paths of `git status --porcelain -z` (a rename's both sides), for the paths asked about. */
102+
function dirtyPaths(checkout: string, rels: string[], env: NodeJS.ProcessEnv): Set<string> | null {
103+
if (!rels.length) return new Set();
104+
const r = spawnSync('git', ['status', '--porcelain', '-z', '--untracked-files=all', '--', ...rels], {
105+
cwd: checkout, encoding: 'utf8', timeout: 10_000, env: { ...env, GIT_OPTIONAL_LOCKS: '0', GIT_TERMINAL_PROMPT: '0' },
106+
});
107+
if (r.status !== 0 || typeof r.stdout !== 'string') return null;
108+
const out = new Set<string>();
109+
const parts = r.stdout.split('\0');
110+
for (let i = 0; i < parts.length; i++) {
111+
const p = parts[i]!;
112+
if (p.length < 4) continue;
113+
out.add(p.slice(3));
114+
if (p[0] === 'R' || p[0] === 'C') out.add(parts[++i] ?? '');
115+
}
116+
return out;
117+
}
118+
119+
/** Before init writes: what each path holds, which were dirty already, and the last record. */
120+
export function snapInstall(checkout: string, rels: string[], env: NodeJS.ProcessEnv = process.env): InstallSnap {
121+
const list = [...new Set(rels.filter((r) => REL.test(r)))].slice(0, MAX_PATHS);
122+
const dirty = dirtyPaths(checkout, list, env);
123+
const before = new Map(list.map((rel) => [rel, fileSha(join(checkout, rel))] as const));
124+
// When git can't say, every path counts as dirty: nothing is vouched for that the probe couldn't check either.
125+
return { checkout, rels: list, before, dirty: dirty ?? new Set(list), prior: readInstallRecord(checkout, env) };
126+
}
127+
128+
/** After init wrote: record the paths it made its own, and return them. */
129+
export function recordInstall(snap: InstallSnap, env: NodeJS.ProcessEnv = process.env): string[] {
130+
const files: InstallFiles = {};
131+
for (const rel of snap.rels) {
132+
const before = snap.before.get(rel);
133+
const after = fileSha(join(snap.checkout, rel));
134+
if (before === false || after === false) continue;
135+
const vouched = !!snap.prior && Object.hasOwn(snap.prior, rel) && snap.prior[rel] === before;
136+
if (vouched || (before !== after && !snap.dirty.has(rel))) files[rel] = after ?? null;
137+
}
138+
// Nothing init wrote, and nothing there before: no record at all.
139+
const kept = Object.keys(files);
140+
if (!kept.length && !snap.prior) return kept;
141+
writePrivate(installRecordPath(snap.checkout, env), `${JSON.stringify({ v: 1, checkout: real(snap.checkout), files }, null, 2)}\n`);
142+
return kept;
143+
}

‎src/notes/every-repo.ts‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
/**
2+
* Every-repo personal learnings (design 1.0, P21, R6): learnings that hold
3+
* on this machine in every repo, in `<trail home>/learnings/`. A kept
4+
* guardrail, a preference the person typed in many repos, a machine wall's
5+
* lesson. Beside `notes.ts`, which keeps a personal learning per repo.
6+
*
7+
* A guardrail template has a fixed file name (`no-timeout.md`,
8+
* `push-over-ssh.md`) so the PreToolUse fast path can read that one file
9+
* without listing the folder (R4, R56). Every path goes through
10+
* `trailPath(env)`, never a home path (0 rule 11).
11+
*/
12+
import { lstatSync, readFileSync, readdirSync } from "node:fs";
13+
import { join } from "node:path";
14+
import { flatLine } from "./flatten.js";
15+
import { EVERY_REPO, liveLearnings, newLearningId, readLearningFields, type LearningFields } from "./learning-fields.js";
16+
import { parseNote, renderNote, type Note, type WriteNoteInput } from "./notes.js";
17+
import { ensurePrivateDir, trailPath, writePrivate } from "../util/private-fs.js";
18+
import { refuseLinkedIntoCheckout } from "./linked-learnings.js";
19+
20+
/** What a new every-repo learning is written from: phase 0's input plus 1b's fields. Its id, scope and status come from where it is written. */
21+
export type LearningInput = WriteNoteInput & Pick<LearningFields, "skill" | "check" | "enforce" | "wall">;
22+
23+
/** A plain file name: no folder, no leading dot, `.md`. */
24+
const FILE_NAME = /^[a-z0-9][a-z0-9._-]{0,80}\.md$/;
25+
/** A learning is a few lines; a bigger file is not one Trail wrote, and is skipped rather than read at every start. */
26+
const MAX_FILE = 64 * 1024;
27+
28+
/** `<trail home>/learnings` (P21). */
29+
export function everyRepoDir(env: NodeJS.ProcessEnv = process.env): string {
30+
return trailPath(env, "learnings");
31+
}
32+
33+
/**
34+
* Writes one every-repo personal learning (0600 in a 0700 folder), id from its file name, `applies: every repo`
35+
* whatever the input says. A file of that name is replaced: a guardrail template keeps its one file; callers
36+
* pick other names so they don't meet. Returns the path.
37+
*/
38+
export function writeEveryRepo(input: LearningInput, fileName: string, env: NodeJS.ProcessEnv = process.env): string {
39+
if (typeof fileName !== "string" || !FILE_NAME.test(fileName) || fileName.includes("..")) {
40+
throw new Error("writeEveryRepo: a plain .md file name is needed");
41+
}
42+
const dir = everyRepoDir(env);
43+
// D114 (1): a linked folder is fine (dotfiles), unless it leads into a Trail checkout.
44+
refuseLinkedIntoCheckout(dir, env);
45+
ensurePrivateDir(dir);
46+
const note: Omit<Note, "path"> = {
47+
title: flatLine(input.title),
48+
author: flatLine(input.author, 80),
49+
date: input.date,
50+
branch: input.branch,
51+
cost: input.cost,
52+
touches: input.touches ?? [],
53+
body: input.body,
54+
...(input.standing ? { standing: true } : {}),
55+
...readLearningFields({ ...input, id: newLearningId(fileName), scope: "personal", applies: EVERY_REPO, status: "active" }, fileName),
56+
};
57+
const path = join(dir, fileName);
58+
writePrivate(path, renderNote(note));
59+
return path;
60+
}
61+
62+
/**
63+
* Active, not replaced, kind not lesson: what session start loads in every repo (R47). A file that says it holds
64+
* in one repo, a link, or anything that doesn't parse is left out.
65+
*/
66+
export function everyRepoLearnings(env: NodeJS.ProcessEnv = process.env): Note[] {
67+
const dir = everyRepoDir(env);
68+
let files: string[];
69+
try {
70+
files = readdirSync(dir).filter((f) => f.endsWith(".md")).sort();
71+
} catch {
72+
return [];
73+
}
74+
const out: Note[] = [];
75+
for (const f of files) {
76+
const path = join(dir, f);
77+
try {
78+
const st = lstatSync(path);
79+
if (!st.isFile() || st.size > MAX_FILE) continue;
80+
const n = parseNote(readFileSync(path, "utf8"), path);
81+
if (!n || (n.applies && n.applies !== EVERY_REPO)) continue;
82+
out.push({ ...n, scope: "personal", applies: EVERY_REPO });
83+
} catch {
84+
/* unreadable: skip */
85+
}
86+
}
87+
return liveLearnings(out).filter((n) => n.kind !== "lesson");
88+
}

‎src/notes/guardrails.ts‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
/**
2+
* The learning a kept guardrail writes (design 2.2): the catalog's template
3+
* text, never model-written. `no-timeout` is a machine fact, so it holds in
4+
* every repo whatever `applies` says (R49), and carries the check the
5+
* PreToolUse fast path enforces; `push-over-ssh` is text only and holds
6+
* where it was kept.
7+
*/
8+
import { flatLine } from "./flatten.js";
9+
import { EVERY_REPO } from "./learning-fields.js";
10+
import type { LearningInput } from "./every-repo.js";
11+
import { WALLS } from "../trace/walls.js";
12+
import type { WallClass } from "../trace/types.js";
13+
14+
/** YYYY-MM-DD of `d` in local time (days are local dates). */
15+
function localDay(d: Date): string {
16+
return `${d.getFullYear()}-${String(d.getMonth() + 1).padStart(2, "0")}-${String(d.getDate()).padStart(2, "0")}`;
17+
}
18+
19+
/**
20+
* The learning a kept guardrail writes (2.2). Template text only; never model-written. Throws for a class with no
21+
* template. `author` is '' (the person kept it at init; the screen named it).
22+
*/
23+
export function guardrailLearning(cls: WallClass, applies: string, evidence: string, now: Date = new Date()): LearningInput {
24+
const w = WALLS.find((x) => x.cls === cls);
25+
const g = w?.guardrail;
26+
if (!w || !g) throw new Error(`guardrailLearning: ${String(cls)} has no guardrail`);
27+
const where = g.applies === "every repo" ? EVERY_REPO : applies;
28+
return {
29+
title: g.title,
30+
body: g.text,
31+
author: "",
32+
date: localDay(now),
33+
kind: "guardrail",
34+
applies: where,
35+
src: "error-class",
36+
...(g.check ? { check: g.check, enforce: "sure-fail" as const } : {}),
37+
wall: cls,
38+
evidence: flatLine(evidence, 160),
39+
};
40+
}

0 commit comments

Comments
 (0)