Skip to content

Commit 30b09fb

Browse files
NiveditJainclaude
andauthored
[failproofai-legion-950] Restore the framework floors #935 moved, and stop Dependabot proposing the move (#950)
Dependabot's grouped security update #935 opened three minutes after #934 put the ai-4, langchain-0.3 and llamaindex-0.11 pins back, and moved them off their releases again, mastra-0 with them: ai-4 ai 4.3.19 -> 5.0.52, zod 3.25.76 -> 4.6.7 langchain-0.3 @langchain/core 0.3.80 -> 1.3.0, @langchain/langgraph 0.4.10 -> 1.5.2, zod -> 4.6.7 llamaindex-0.11 llamaindex 0.11.4 -> 0.12.1, zod -> 4.6.7 mastra-0 @mastra/core 0.24.9 -> 1.79.0, @mastra/memory 0.15.13 -> 1.39.0, zod 4.6.6 -> 4.6.7 Since then __tests__/ci/ts-sdk-fixture-pins.test.ts has failed in all three `test` configurations, and the frameworks (node 20, 24) and runtimes shards of failproofai-ts-sdk-integrations have failed in global setup (`code.charCodeAt is not a function`, vite masking the fixture install failure), on main and so on every open PR, including all thirteen open Dependabot PRs. All four fixtures are back on their #934 package.json and lockfile, byte-for-byte (`git diff cdedbe1^ -- <fixture>` is empty). langchain-dup-core keeps #935's zod 4.6.6 -> 4.6.7: it names no zod line, and the fix is in range. Restoring alone would last minutes: the advisories SECURITY.md accepts for these floors (ai GHSA-rwvc-j5jr-mgvh, @ai-sdk/provider-utils GHSA-866g-f22w-33x8, zod GHSA-mrhx-c7xc-m3w5) reopen with the old lockfiles, and the only fix Dependabot can reach for each is the major the fixture exists to stay below. So .github/dependabot.yml gets a security-updates-only npm entry per floor (open-pull-requests-limit: 0 turns version updates off) whose `ignore` ranges hold the framework on its line, and zod on 3 where the framework requires it. `ignore` is one of the options security updates read; the ranges are `versions`, because `update-types` is documented as version-updates only. Fixes on the line still arrive. The file validates against SchemaStore's dependabot-2.0 schema. ts-sdk-fixture-pins.test.ts now also fails when a floor has no such entry, has more than one, leaves version updates on, ignores its framework only by update-types, ignores a range that would also block its own pin, or lets zod leave 3 where the fixture pins it there; and when an entry names a fixture that no longer exists. Each was checked to fail by mutating the config. SECURITY.md says why the entries exist and that a Dependabot PR moving a pin anyway should be closed, not merged. Verified locally: - bun run test:run: 426 files, 8835 passed / 10 skipped - bun run test:e2e: 358 passed / 7 skipped - lint 0 errors (5 existing warnings, untouched files); tsc clean - failproofai-ts-sdk-integrations in node:20 / node:24 containers mirroring the CI job: frameworks 1036 passed / 4 skipped on each Node, runtimes 1132 passed Claude-Session: https://claude.ai/code/session_01KYzGfVxwGZMvdyyi97XyZY Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 705d34b commit 30b09fb

13 files changed

Lines changed: 5668 additions & 2553 deletions

File tree

‎.github/dependabot.yml‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,3 +71,73 @@ updates:
7171
python-dependencies:
7272
patterns:
7373
- "*"
74+
75+
# The TypeScript SDK's integration fixtures that hold each framework's supported
76+
# FLOOR. These four entries are for security updates alone —
77+
# `open-pull-requests-limit: 0` turns version updates off — and exist to stop one
78+
# move. Each floor carries an advisory whose only fix is a major its framework
79+
# release cannot take (SECURITY.md, "Accepted, with no fix available"), and
80+
# Dependabot's grouped security updates reach that fix by moving the framework off
81+
# the line the fixture exists to test: #838, #931, and #935, which opened three
82+
# minutes after #934 put the pins back. `ignore` applies to security updates too,
83+
# so each range below keeps the framework on its line, and zod on 3 where the
84+
# framework requires it; every fix that stays on the line still arrives. Ranges
85+
# are `versions`, never `update-types`, which security updates do not read. A new
86+
# floor fixture needs an entry here: __tests__/ci/ts-sdk-fixture-pins.test.ts
87+
# fails without one.
88+
- package-ecosystem: npm
89+
directory: /sdk/typescript/integration/fixtures/ai-4
90+
schedule:
91+
interval: weekly
92+
day: monday
93+
open-pull-requests-limit: 0
94+
ignore:
95+
- dependency-name: ai
96+
versions: [">=5.0.0"]
97+
# ai@4.3.19 peers on zod ^3.23.8.
98+
- dependency-name: zod
99+
versions: [">=4.0.0"]
100+
101+
- package-ecosystem: npm
102+
directory: /sdk/typescript/integration/fixtures/langchain-0.3
103+
schedule:
104+
interval: weekly
105+
day: monday
106+
open-pull-requests-limit: 0
107+
ignore:
108+
- dependency-name: "@langchain/core"
109+
versions: [">=0.4.0"]
110+
# @langchain/langgraph@0.4.10 peers on @langchain/core <0.4.0.
111+
- dependency-name: "@langchain/langgraph"
112+
versions: [">=0.5.0"]
113+
# Both depend on zod ^3.25.32. A zod 4 here would still install, nested
114+
# under them, so nothing but this range keeps the fixture's own zod on 3.
115+
- dependency-name: zod
116+
versions: [">=4.0.0"]
117+
118+
- package-ecosystem: npm
119+
directory: /sdk/typescript/integration/fixtures/llamaindex-0.11
120+
schedule:
121+
interval: weekly
122+
day: monday
123+
open-pull-requests-limit: 0
124+
ignore:
125+
- dependency-name: llamaindex
126+
versions: [">=0.12.0"]
127+
# @llamaindex/core@0.6.8 and @llamaindex/workflow@1.1.5 need zod ^3.23.8.
128+
- dependency-name: zod
129+
versions: [">=4.0.0"]
130+
131+
- package-ecosystem: npm
132+
directory: /sdk/typescript/integration/fixtures/mastra-0
133+
schedule:
134+
interval: weekly
135+
day: monday
136+
open-pull-requests-limit: 0
137+
ignore:
138+
# @mastra/core, and the @mastra/memory and @mastra/mcp releases paired with it.
139+
- dependency-name: "@mastra/*"
140+
versions: [">=1.0.0"]
141+
# @mastra/core@0.24.9 requires ai@^4. This fixture's own zod is already 4.
142+
- dependency-name: ai
143+
versions: [">=5.0.0"]

‎CHANGELOG.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,14 @@
11
# Changelog
22

3-
## 1.0.13-beta.0 — 2026-10-25
3+
## 1.0.13-beta.0 — 2026-10-27
44

55
### Features
66

77
- **`fp-cloud-cli` cuts `0.0.2` stable.** The `0.0.2b0` line opened after `0.0.1` was never published; it is cut stable rather than published, so `pipx install fp-cloud-cli` resolves the `fp` that drives what FailproofAI Cloud now serves (entries in `fp-cloud-cli/CHANGELOG.md`). This is the PyPI package's own line and does not move the npm version, which stays at `1.0.13-beta.0` (#936)
88

99
### Dependencies
1010

11+
- Put the `ai-4`, `langchain-0.3`, `llamaindex-0.11` and `mastra-0` SDK integration fixtures back on the releases they exist to test, and stop Dependabot moving them. Its grouped security update #935, opened three minutes after #934 restored the first three, moved them again: `ai@4.3.19` → `5.0.52`, `@langchain/core@0.3.80` → `1.3.0`, `llamaindex@0.11.4` → `0.12.1` (each with `zod@3.25.76` → `4.6.7`), and `@mastra/core@0.24.9` → `1.79.0`. `__tests__/ci/ts-sdk-fixture-pins.test.ts` and the `frameworks` and `runtimes` integration shards have failed on `main` since, and so on every open PR. All four are back on their #934 manifests and lockfiles, and `.github/dependabot.yml` gives each a security-updates-only entry whose `ignore` ranges hold its framework on its line, and zod on 3 where the framework requires it, so the move is no longer proposed; the pin test now fails if a floor fixture loses that entry (#950)
1112
- Restore the `ai-4`, `langchain-0.3` and `llamaindex-0.11` SDK integration fixtures to the releases they exist to test: `ai@4.3.19`, `@langchain/core@0.3.80` with `@langchain/langgraph@0.4.10`, and `llamaindex@0.11.4`, each with `zod@3.25.76`. Dependabot's grouped security update (#931) reached the zod fix by moving them to `ai@7.0.152`, `@langchain/core@1.3.0` and `llamaindex@0.12.1`, the pin move `SECURITY.md` rules out, so the matrix silently stopped testing three supported floors. `llamaindex-0.11` then stopped installing (`@llamaindex/workflow@1.1.5` peers on `zod@^3.23.8`, so `npm ci` failed `ERESOLVE`), and because global setup installs every fixture first, the `frameworks` and `runtimes` integration shards failed on `main`. #931 also moved the `runtimes` fixture to `zod@4.6.7` while its Deno program still imported `npm:zod@4.6.5`, which Deno could no longer find; it now imports the installed version. `__tests__/ci/ts-sdk-fixture-pins.test.ts` fails on either move before anything installs (#934)
1213
- Patch the fixable SDK integration fixture advisories with fixture-local `overrides`, moving no pin: `undici` 7.29.1 → 7.30.1 in `ai-5` (GHSA-xjj6-f33q-929c, GHSA-4qv8-2p9j-2685, GHSA-f72v-6w45-q472); `fast-uri` 3.1.8 → 3.1.9 in `llamaindex-0.11` (GHSA-w74x-v6f9-j3xg, GHSA-wjr9-j795-jxcr, GHSA-xhq7-xmqj-769h, GHSA-333x-w4r6-6crp); and in `mastra-1`, `@modelcontextprotocol/server` 2.0.0 → 2.3.0 and `@modelcontextprotocol/client` 2.2.0 → 2.3.0 (GHSA-jm63-cq47-j5jp, GHSA-46jr-4wx8-v453, GHSA-rvq5-wwqv-78pq, GHSA-6prh-2h8m-c8cw), both pinned exactly by `@mastra/mcp@2.0.0`, each now with its own `@modelcontextprotocol/core@2.3.0` nested under it (#934)
1314
- `SECURITY.md` accepts zod GHSA-mrhx-c7xc-m3w5 in the five fixtures held to zod 3: `ai-4`, `langchain-0.3` and `llamaindex-0.11`, and the nested copies in `langchain-dup-core` and `mastra-0`. It is fixed only in zod 4.6.6, no zod 3 release carries the fix, and every copy is required at `^3` by the framework release its fixture pins (#934)

‎SECURITY.md‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -56,8 +56,9 @@ The gate above scans the lockfiles it is *given* (the `--lockfile` list in the
5656
workflow). GitHub's dependency graph reads **every** lockfile in the repository, so
5757
Dependabot alerts cover trees the gate does not — in particular the TypeScript SDK's
5858
integration fixtures, `sdk/typescript/integration/fixtures/*/package-lock.json`.
59-
Those are not in the scanner's list and are not a Dependabot *update* ecosystem in
60-
[`.github/dependabot.yml`](.github/dependabot.yml) either, so an alert is the only
59+
Those are not in the scanner's list and get no Dependabot *version* updates either
60+
(the four floor fixtures have entries in [`.github/dependabot.yml`](.github/dependabot.yml),
61+
but only to restrain its security updates; see below), so an alert is the only
6162
thing that will ever mention them. Treat the alert list as authoritative for those
6263
paths, and see *Triaging a Dependabot alert in an integration fixture* below for how
6364
to resolve one.
@@ -118,6 +119,17 @@ That splits an alert in one of these lockfiles into two cases:
118119
the scanner does not read these lockfiles, so an `[[IgnoredVulns]]` entry for one
119120
would filter nothing and be reported as an unused ignore.
120121

122+
Dependabot's own security updates know neither case. Given a case-2 advisory they
123+
reach the fix by moving the framework, which is how #838, #931 and #935 happened
124+
(#935 opened three minutes after #934 put the pins back). So each floor fixture —
125+
`ai-4`, `langchain-0.3`, `llamaindex-0.11`, `mastra-0` — has a security-updates-only
126+
entry in [`.github/dependabot.yml`](.github/dependabot.yml)
127+
(`open-pull-requests-limit: 0`) whose `ignore` ranges hold its framework on its
128+
line, and zod on 3 where that framework requires it; a fix that stays on the line
129+
still arrives. A new floor fixture needs its own entry, and
130+
`__tests__/ci/ts-sdk-fixture-pins.test.ts` fails without one. A Dependabot PR that
131+
moves a pin anyway fails that same test: close it rather than merge it.
132+
121133
### Accepted, with no fix available
122134

123135
| Advisory | Package | Fixtures | Why it stays |

‎__tests__/ci/ts-sdk-fixture-pins.test.ts‎

Lines changed: 91 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,23 +2,30 @@ import { readFileSync, readdirSync } from "node:fs";
22
import { join } from "node:path";
33

44
import { describe, expect, it } from "vitest";
5+
import { parse } from "yaml";
56

67
/**
78
* A TypeScript SDK integration fixture named for a release line — `ai-4`,
89
* `langchain-0.3`, `mastra-0` — exists to hold its framework on that line: the
910
* pinned release is the subject of the test (SECURITY.md, "Triaging a Dependabot
1011
* alert in an integration fixture"). A move off it still installs and still
1112
* passes, against a different major, so the matrix loses a supported floor with
12-
* every check green. Dependabot security updates have made that move twice
13-
* (#838, #931), both times to reach a fix the fixture could not take. This fails
14-
* on the manifest, before anything installs.
13+
* every check green. Dependabot security updates have made that move three times
14+
* (#838, #931, #935), each to reach a fix the fixture could not take; #935 opened
15+
* three minutes after #934 put the pins back. This fails on the manifest, before
16+
* anything installs, and on a `.github/dependabot.yml` that no longer stops the
17+
* move from being proposed for a supported floor.
1518
*
1619
* #931 also moved the `runtimes` fixture's zod without the `npm:zod@…` specifier
1720
* its Deno program imports, which Deno resolves against the fixture's
1821
* `node_modules` and so could no longer find. That half is checked here too.
1922
*/
2023

21-
const fixturesDir = join(__dirname, "..", "..", "sdk", "typescript", "integration", "fixtures");
24+
const root = join(__dirname, "..", "..");
25+
const fixturesDir = join(root, "sdk", "typescript", "integration", "fixtures");
26+
27+
/** Every framework's supported floor: where the SDK's peer range starts. */
28+
const FLOORS = ["ai-4", "langchain-0.3", "llamaindex-0.11", "mastra-0"];
2229

2330
/** The package each fixture-name prefix pins. */
2431
const FRAMEWORK: Record<string, string> = {
@@ -58,11 +65,60 @@ const specifiers = fixtures.flatMap((fixture) =>
5865
}),
5966
);
6067

68+
type DependabotUpdate = {
69+
"package-ecosystem": string;
70+
directory?: string;
71+
directories?: string[];
72+
"open-pull-requests-limit"?: number;
73+
ignore?: { "dependency-name": string; versions?: string[] }[];
74+
};
75+
76+
/** Each fixture directory `.github/dependabot.yml` names, with the entry naming it. */
77+
const FIXTURE_DIR = "/sdk/typescript/integration/fixtures/";
78+
const dependabot = parse(readFileSync(join(root, ".github", "dependabot.yml"), "utf8")) as {
79+
updates: DependabotUpdate[];
80+
};
81+
const fixtureEntries = dependabot.updates
82+
.filter((update) => update["package-ecosystem"] === "npm")
83+
.flatMap((update) =>
84+
[update.directory, ...(update.directories ?? [])]
85+
.filter((dir): dir is string => dir?.startsWith(FIXTURE_DIR) === true)
86+
.map((dir) => ({ fixture: dir.slice(FIXTURE_DIR.length), update })),
87+
);
88+
89+
type Version = [number, number, number];
90+
const versionOf = (text: string): Version => {
91+
const [major = 0, minor = 0, patch = 0] = text.split(".").map(Number);
92+
return [major, minor, patch];
93+
};
94+
const compare = (a: Version, b: Version) => a[0] - b[0] || a[1] - b[1] || a[2] - b[2];
95+
96+
/** The first release past a line: `4` → 5.0.0, `0.3` → 0.4.0. */
97+
const nextLine = (line: string): Version => {
98+
const [major, minor] = line.split(".").map(Number);
99+
return minor === undefined ? [major! + 1, 0, 0] : [major!, minor + 1, 0];
100+
};
101+
102+
/** Whether a `dependency-name`, which may use `*`, matches `name`. */
103+
const namesMatch = (pattern: string, name: string) =>
104+
new RegExp(`^${pattern.replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*")}$`).test(name);
105+
106+
/**
107+
* Whether `update` ignores `name` from the next line on, while still letting a
108+
* fix on the line replace `pinned`. Only `>=` ranges in `versions` count: security
109+
* updates read `versions`, and do not read `update-types`.
110+
*/
111+
const holdsOnLine = (update: DependabotUpdate, name: string, pinned: string, line: string) =>
112+
(update.ignore ?? [])
113+
.filter((rule) => namesMatch(rule["dependency-name"], name))
114+
.flatMap((rule) => rule.versions ?? [])
115+
.map((range) => /^>=\s*(\d+(?:\.\d+){0,2})$/.exec(range.trim())?.[1])
116+
.filter((bound): bound is string => bound !== undefined)
117+
.some((bound) => compare(versionOf(bound), nextLine(line)) <= 0 && compare(versionOf(bound), versionOf(pinned)) > 0);
118+
61119
describe("TS SDK integration fixtures named for a release line", () => {
62120
it("finds them, every framework's supported floor among them", () => {
63-
expect(named.map((f) => f.fixture)).toEqual(
64-
expect.arrayContaining(["ai-4", "langchain-0.3", "llamaindex-0.11", "mastra-0"]),
65-
);
121+
expect(named.map((f) => f.fixture)).toEqual(expect.arrayContaining(FLOORS));
66122
});
67123

68124
it("know which framework each name pins", () => {
@@ -78,6 +134,34 @@ describe("TS SDK integration fixtures named for a release line", () => {
78134
});
79135
});
80136

137+
describe("Dependabot security updates on the supported floors", () => {
138+
const floors = named.filter((f) => FLOORS.includes(f.fixture));
139+
const zod3 = floors.filter((f) => readJson(f.fixture, "package.json").dependencies?.zod?.startsWith("3."));
140+
141+
it("finds every floor, and the ones held to zod 3", () => {
142+
expect(floors.map((f) => f.fixture).sort()).toEqual([...FLOORS].sort());
143+
expect(zod3.map((f) => f.fixture)).toEqual(expect.arrayContaining(["ai-4", "langchain-0.3", "llamaindex-0.11"]));
144+
});
145+
146+
it.each(floors)("$fixture has one security-only entry holding $framework on $line", ({ fixture, framework, line }) => {
147+
const entries = fixtureEntries.filter((e) => e.fixture === fixture).map((e) => e.update);
148+
expect(entries, `${fixture}: npm entries in .github/dependabot.yml`).toHaveLength(1);
149+
expect(entries[0]!["open-pull-requests-limit"], `${fixture}: version updates`).toBe(0);
150+
const pinned = readJson(fixture, "package.json").dependencies![framework!]!;
151+
expect(holdsOnLine(entries[0]!, framework!, pinned, line), `${fixture}: ignore for ${framework}`).toBe(true);
152+
});
153+
154+
it.each(zod3)("$fixture holds zod on 3, which its framework requires", ({ fixture }) => {
155+
const entry = fixtureEntries.find((e) => e.fixture === fixture)!.update;
156+
const pinned = readJson(fixture, "package.json").dependencies!.zod!;
157+
expect(holdsOnLine(entry, "zod", pinned, "3"), `${fixture}: ignore for zod`).toBe(true);
158+
});
159+
160+
it("names only fixtures that exist", () => {
161+
expect(fixtureEntries.map((e) => e.fixture).filter((f) => !fixtures.includes(f))).toEqual([]);
162+
});
163+
});
164+
81165
describe("TS SDK integration fixtures' npm: specifiers", () => {
82166
it("finds the Deno program's", () => {
83167
expect(specifiers.map((s) => s.name)).toEqual(expect.arrayContaining(["ai", "zod"]));

‎sdk/typescript/CHANGELOG.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,21 @@ section is missing or empty is refused before anything is built.
3939
for a release line pinning its framework off that line, or an `npm:` specifier
4040
naming a version the fixture does not install. (#934)
4141

42+
- **The four framework floors stay put.** Dependabot's grouped security update
43+
#935, opened three minutes after #934 merged, moved `ai-4`, `langchain-0.3` and
44+
`llamaindex-0.11` off their releases again (`ai@4.3.19` → `5.0.52`,
45+
`@langchain/core@0.3.80` → `1.3.0` with `@langchain/langgraph@0.4.10` →
46+
`1.5.2`, `llamaindex@0.11.4` → `0.12.1`, each with `zod@3.25.76` → `4.6.7`), and
47+
`mastra-0` too (`@mastra/core@0.24.9` → `1.79.0`, `@mastra/memory@0.15.13` →
48+
`1.39.0`). All four are back on their #934 manifests and lockfiles. So that the
49+
move stops being proposed rather than being undone a fourth time,
50+
`.github/dependabot.yml` gives each floor a security-updates-only entry
51+
(`open-pull-requests-limit: 0`) whose `ignore` ranges hold its framework on its
52+
line, and zod on 3 where that framework requires it; a fix that stays on the
53+
line still arrives. `__tests__/ci/ts-sdk-fixture-pins.test.ts` now also fails
54+
when a floor has no such entry, or one that would let the framework leave its
55+
line. (#950)
56+
4257
- **The remaining fixable fixture advisories clear, each with an override.** `ai-5`
4358
moves `undici` 7.29.1 → 7.30.1 (GHSA-xjj6-f33q-929c, GHSA-4qv8-2p9j-2685,
4459
GHSA-f72v-6w45-q472) and `llamaindex-0.11` moves `fast-uri` 3.1.8 → 3.1.9

0 commit comments

Comments
 (0)