Skip to content

Commit 64c5668

Browse files
feat(connector-openapi): normalize-openapi --trim writes only the operations --include-pattern matches (#28331)
Fixes #28309 Clause-②: no Metadata-protocol: none ## 维护者速读 - 改了什么:离线脚本 `normalize-openapi` 新增 `--trim` 开关,配合 `--include-pattern` 只写出被匹配的操作;两处文档同步。 - 为什么改:大型厂商文档(Stripe 612 个操作、63 MB)原样打包不现实;Stripe 连接器应用要求打包文件就是脚本输出并记录其 sha256。 - 风险与回滚:只改仓库脚本(不随 npm 发布)与文档;不加 `--trim` 时输出逐字节不变;回滚即 revert 本 PR。 - 先看:`packages/connectors/connector-openapi/scripts/normalize-openapi.mts` 的 `trimPaths` 与第 4 步前的裁剪调用;其测试文件的 `--trim` 一节;`content/docs/automation/connectors.mdx` 新段落。 - 两分钟验证:`pnpm --filter @objectstack/connector-openapi exec vitest run scripts/normalize-openapi.test.ts`,预期 20 个测试全部通过。 - 需要你做的:无,席位落地。 ## What changes - `scripts/normalize-openapi.mts` gains `--trim` (option `trim` on `normalizeOpenApi`). With it, every path operation no `--include-pattern` matches, every path item left with no operation, and `webhooks` are dropped **before** dereferencing, so only the kept operations are walked and nothing reachable only from a dropped operation reaches the output. A `$ref` path item is resolved (not walked) before its operations are selected. The `include` list is computed exactly as before (same patterns, same imported `operationActionKey`), and the printed `sha256` is still the hash of the bytes written, which are now the trimmed bytes. The summary gains `trimmedFrom` (operations in the source) when trimming. `--trim` with no `--include-pattern` is refused (exit 1); a pattern matching nothing is refused as before. - `securitySchemes` stay, as in the untrimmed path (they are referenced by name, never by `$ref`, and the provider does not read them); every other component was already dropped after inlining, and under `--trim` it is never reached from a dropped operation. - Docs: `content/docs/automation/connectors.mdx` (the script's paragraph: the example command now carries `--trim`, and a new paragraph says what trimming writes and when to leave it off); `content/docs/deployment/marketplace-handbook.mdx`, the one README-section sentence that still described the normalizer as forthcoming now says how a declarative OpenAPI app produces its packaged document and which sha256 to record. - The runtime (`src/**`) does not change; the script stays a repository script (not in `files[]`, no `bin`), so this carries `skip-changeset`. ### Why a separate `--trim` flag, not trimming whenever `--include-pattern` is given - An implied trim changes what the one documented command writes (its bytes and its hash) with no change to the command line. With a flag, every existing invocation writes exactly what it wrote before: the untrimmed Stripe run below reproduces the #28276 reading byte for byte. - The two outputs serve two different authors. Untrimmed plus `include` lets an app widen `include` later by editing config alone; trimmed means widening requires re-running the script. Which one an app wants is the author's call, and the command line that produced the packaged bytes should say which one it was. - Today's callers: the only documented call is the example in `connectors.mdx` (the shelf pipeline is not built yet), so the flag costs nobody a migration; the example now uses `--trim`, which is what recipe 1 teaches (trim to the operations the app calls). ## Tests `scripts/normalize-openapi.test.ts`, on the existing fixtures, a new `--trim` block and one new CLI case: - the trimmed output holds exactly the matched operations (`GetCustomers`, `GetCustomersCustomer`, `get_v1_balance`), drops the unmatched POST from a path item whose GET is kept, has the same `include` list as the untrimmed run, loads through `createOpenApiProviderFactory` with exactly those actions and no `$ref` in the document or the materialized def; - trimming to `balance` alone drops the customer path items and leaves no text reachable only from them (`A customer.`, `Error response.`, the shared `limit` parameter, `subscription`), with zero cuts counted (only the kept operation was walked); - a Swagger 2.0 document is trimmed after conversion, keyed by `operationId`; - a `$ref` path item is resolved before selection (its sibling `summary` kept) and `webhooks` are dropped; - refusals: `--trim` with no pattern; a pattern matching nothing; - control: without `--trim` the whole document is written, as before; - CLI: with `--trim`, `bytes` and `sha256` equal the written file's length and `sha256` of its bytes, `operations` is 2 and `trimmedFrom` is 4; `--trim` alone exits 1 and writes nothing. <details><summary>Package test and typecheck (head 08bf282)</summary> ```text pnpm --filter @objectstack/connector-openapi test && pnpm --filter @objectstack/connector-openapi typecheck Test Files 6 passed (6) Tests 82 passed (82) check:test-typecheck: OK — @objectstack/connector-openapi's test layer compiles under packages/connectors/connector-openapi/tsconfig.test.json os-verify-lock: VERDICT command-exit 0 scripts/normalize-openapi.test.ts alone: Tests 20 passed (20) (13 before this PR, 7 new) ``` </details> <details><summary>Ablation (one-off, committed fix first, restored)</summary> `node scripts/ablation-replace.mjs` on `normalize-openapi.mts`, anchor `if (matches(patterns, { key: operationActionKey(...` (hits 1 to 0, blob a2190cd to 70610c5bb588) replaced by a condition that keeps every operation. The test file then reads `Tests 5 failed | 15 passed (20)`: the four trim selection cases and the CLI `--trim` case went red; the refusal and control cases stayed green, as expected. Restored: blob equals HEAD (a2190cd) and `git diff HEAD` is empty. No build or `dist/` leg: the test imports the script from source. </details> ## Acceptance notes Measured once against the real Stripe document, `stripe/openapi` `openapi/spec3.json` at `51610f5f` (MIT; 8,317,513 bytes, sha256 `7cff4cc46d0654101a36a3302f7544640f5f73e4f3cad6f264f0e23c19fa1776`), with the pattern `^(GetCustomers|PostCustomers|GetCustomersCustomer|GetCharges|PostPaymentIntents|GetPaymentIntentsIntent)$` at the default `--max-depth 2`: | run | bytes | operations | sha256 | |:---|---:|---:|:---| | without `--trim` (control) | 63,317,655 | 612 | `0ea3c37a0854ad2646a156404b1ef86cca57403855e3803cd4630601eb6fc7b2` | | with `--trim` | 1,934,435 | 6 (`trimmedFrom` 612) | `08d6e40bccec707a0bfc355b1cbf936df29018c22e7be658d616683e501b13d4` | - The control equals the #28276 dev's reading byte for byte (same size, same hash), so the untrimmed path is unchanged on the real document. - `sha256sum` of the trimmed file equals the printed hash; `refsLeft` 0; cuts recursive 38, max-depth 1064, extension 240; the two runs print the same `include` list. - The trimmed file loads through `createOpenApiProviderFactory` with exactly the six actions and zero `$ref` keys in the text and in the materialized def. - The trimmed size is about 1.9 MB, not the about 1.1 MB the card quoted: the #28276 dev's six operations are not named on that card, so this is a different six (two of them creates carrying request bodies). The number is a reading of this pattern, not a target. ## Verification <details><summary>Derived gates (dispatch-gates --commands --repo objectstack-ai/objectstack, head 08bf282)</summary> 78 commands derived from the 4 changed paths; each run with its exit code captured before any pipe. `--ran` reconciliation: 78 derived, 78 run, 0 NOT-MEASURED (a derived zero). On the first run (head e6e7833) `check:skill-examples` and `check:dual-build-cjs-loads` answered exit 3 PREREQUISITE NOT MET (unbuilt sibling packages); after building the `@objectstack/client-react` closure both measured exit 0. Re-run in full on the final head 08bf282 (after merging origin/main): all 78 exit 0. </details> Not run locally, by rule: `test:repo`, `pnpm lint` and the other repo-wide scans (CI). --- _Generated by [Claude Code](https://claude.ai/code/session_017xRBcVBKAFk1ntXSjvPt7U)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 3fd8033 commit 64c5668

4 files changed

Lines changed: 283 additions & 29 deletions

File tree

‎content/docs/automation/connectors.mdx‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -184,9 +184,20 @@ match.
184184
```bash
185185
pnpm --filter @objectstack/connector-openapi normalize-openapi \
186186
--in https://example.com/openapi.yaml --out ./specs/crm-openapi.json \
187-
--include-pattern '^GET /v1/customers'
187+
--include-pattern '^GET /v1/customers' --trim
188188
```
189189

190+
With `--trim` the document it writes holds **only** the operations the
191+
patterns match: every other operation, every path item left with none, and
192+
`webhooks` are dropped before inlining, so nothing reachable only from a
193+
dropped operation is written, and the printed sha256 is the hash of those
194+
trimmed bytes. Package that output — a large vendor document shrinks to the
195+
operations the app calls (Stripe's 612 operations, 63 MB at the default
196+
depth, become about 1.9 MB for six). Without `--trim` the whole document is
197+
written and the patterns only compute `include`, so `include` can be widened
198+
later by editing config alone; with it, widening means re-running the script.
199+
`--trim` without an `--include-pattern` is refused.
200+
190201
Inlining is bounded: a recursive schema, or one more than `--max-depth`
191202
(default 2) named schemas deep, becomes a typed stub whose description names
192203
the reference it replaced. A Swagger 2.0 operation whose empty `consumes` list

‎content/docs/deployment/marketplace-handbook.mdx‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -113,8 +113,10 @@ administrator, not for a contributor:
113113
every one of them ([Setting a credential](/docs/automation/connectors#setting-a-credential)).
114114
The README is still where the administrator learns what each value is and
115115
where to get it. An app that ships declarative
116-
connectors gains an offline spec-normalization tool with
117-
[#26709](https://github.com/objectstack-ai/objectstack/issues/26709).
116+
OpenAPI connectors produces its packaged document with the offline
117+
`normalize-openapi` script: `--trim` keeps only the operations the app calls,
118+
and the sha256 it prints is the one to record here
119+
([`provider: 'openapi'`](/docs/automation/connectors#provider-openapi--one-action-per-operation)).
118120

119121
## Build and verify before publishing
120122

‎packages/connectors/connector-openapi/scripts/normalize-openapi.mts‎

Lines changed: 138 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,8 @@
77
*
88
* pnpm --filter @objectstack/connector-openapi normalize-openapi \
99
* --in <file | http(s) URL> --out <file.json> \
10-
* [--include-pattern <regex>]... [--max-depth <n>] [--server-url <url>] \
11-
* [--empty-consumes <media type>]
10+
* [--include-pattern <regex>]... [--trim] [--max-depth <n>] \
11+
* [--server-url <url>] [--empty-consumes <media type>]
1212
*
1313
* The runtime never converts: `openapi-provider.ts` refuses a document that is
1414
* not 3.x, and the generator skips a `$ref` parameter, request body or response
@@ -25,9 +25,25 @@
2525
* `securitySchemes` stay: they are referenced by name, not by `$ref`);
2626
* 5. make `servers[0].url` absolute (server variables take their defaults; a
2727
* relative URL resolves against the source URL, or `--server-url`);
28-
* 6. optionally compute the `providerConfig.include` list from allow-patterns;
28+
* 6. optionally compute the `providerConfig.include` list from allow-patterns,
29+
* and with `--trim` keep ONLY the operations they match (below);
2930
* 7. write the JSON and print its sha256 -- the hash of the bytes written.
3031
*
32+
* ## `--trim`: package the operations the app calls, not the vendor's catalogue
33+
*
34+
* Without `--trim` the whole document is written and the allow-patterns only
35+
* compute `include`: the app can widen `include` later by editing config. A
36+
* large vendor document is then shipped whole -- measured on Stripe's
37+
* `spec3.json`, 612 operations and 63,317,655 bytes at the default depth for an
38+
* app that calls six. `--trim` (which requires at least one
39+
* `--include-pattern`) drops every path operation no pattern matches, every
40+
* path item left with none, and `webhooks`, BEFORE step 4 -- so only the kept
41+
* operations are dereferenced, and nothing reachable only from a dropped one
42+
* survives into the output. The `include` list is the same one an untrimmed
43+
* run computes (the same patterns, the same `operationActionKey`), and the
44+
* printed sha256 is still the hash of the bytes written: the trimmed bytes.
45+
* The `securitySchemes` stay, as in step 4: they are referenced by name.
46+
*
3147
* ## Where a `$ref` is cut, and why it has to be
3248
*
3349
* "No `$ref` left" and "every schema fully inlined" cannot both hold for a real
@@ -89,6 +105,8 @@ export interface NormalizeOptions {
89105
input: string;
90106
/** Allow-patterns (JavaScript regular expressions) for the generated `include` list. */
91107
includePatterns?: readonly string[];
108+
/** Keep only the operations {@link includePatterns} match (requires at least one pattern). */
109+
trim?: boolean;
92110
/** Named-schema hops inlined before a stub (default {@link DEFAULT_MAX_DEPTH}). */
93111
maxDepth?: number;
94112
/** Absolute http(s) URL that replaces `servers[0].url`. */
@@ -113,8 +131,10 @@ export interface NormalizeResult {
113131
sourceVersion: string;
114132
/** The output document's `openapi` version. */
115133
openapi: string;
116-
/** Every operation's action key, in document order, de-duplicated. */
134+
/** Every written operation's action key, in document order, de-duplicated. */
117135
operationKeys: string[];
136+
/** Operations in the source document -- more than `operationKeys` only when trimmed. */
137+
sourceOperations: number;
118138
/** The `providerConfig.include` list, when allow-patterns were given. */
119139
include?: string[];
120140
/** Schema references replaced by a stub, by reason. */
@@ -145,6 +165,12 @@ export async function normalizeOpenApi(options: NormalizeOptions): Promise<Norma
145165
if (!Number.isInteger(maxDepth) || maxDepth < 0) {
146166
throw new NormalizeError(`--max-depth must be a non-negative integer, got ${String(options.maxDepth)}.`);
147167
}
168+
const patterns = compilePatterns(options.includePatterns);
169+
if (options.trim && patterns.length === 0) {
170+
throw new NormalizeError(
171+
'--trim keeps only the operations --include-pattern matches; give at least one --include-pattern.',
172+
);
173+
}
148174
const fromUrl = isHttpUrl(options.input);
149175
const origin = fromUrl ? options.input : resolvePath(options.cwd ?? process.cwd(), options.input);
150176
const fetchImpl = options.fetchImpl ?? fetch;
@@ -212,6 +238,15 @@ export async function normalizeOpenApi(options: NormalizeOptions): Promise<Norma
212238
// only existed to be referenced.
213239
const cuts: CutCounts = { recursive: 0, maxDepth: 0, extension: 0 };
214240
const deref = new Dereferencer(document, maxDepth, cuts);
241+
// `--trim` selects BEFORE dereferencing: a dropped operation is never walked.
242+
let sourceOperations: number | undefined;
243+
if (options.trim) {
244+
const trimmed = trimPaths(document.paths, patterns, deref);
245+
sourceOperations = trimmed.sourceOperations;
246+
if (trimmed.kept === 0) throw noMatch(sourceOperations);
247+
document = { ...document, paths: trimmed.paths };
248+
delete document.webhooks;
249+
}
215250
const out: JsonObject = {};
216251
for (const [key, value] of Object.entries(document)) {
217252
if (key === 'x-ext' || key === 'x-ext-urls') continue;
@@ -230,27 +265,9 @@ export async function normalizeOpenApi(options: NormalizeOptions): Promise<Norma
230265
const operations = listOperations(out);
231266
const operationKeys = [...new Set(operations.map((op) => op.key))];
232267
let include: string[] | undefined;
233-
if (options.includePatterns && options.includePatterns.length > 0) {
234-
const patterns = options.includePatterns.map((source) => {
235-
try {
236-
return new RegExp(source);
237-
} catch (err) {
238-
throw new NormalizeError(`--include-pattern '${source}' is not a valid regular expression: ${(err as Error).message}`);
239-
}
240-
});
241-
include = [
242-
...new Set(
243-
operations
244-
.filter((op) => patterns.some((p) => p.test(op.key) || p.test(`${op.method.toUpperCase()} ${op.path}`)))
245-
.map((op) => op.key),
246-
),
247-
];
248-
if (include.length === 0) {
249-
throw new NormalizeError(
250-
`--include-pattern matched none of the ${operationKeys.length} operation(s); a pattern is tested against ` +
251-
`each action key (an operationId, or the method_path key) and against 'METHOD /path'.`,
252-
);
253-
}
268+
if (patterns.length > 0) {
269+
include = [...new Set(operations.filter((op) => matches(patterns, op)).map((op) => op.key))];
270+
if (include.length === 0) throw noMatch(operations.length);
254271
}
255272

256273
// 7. The bytes, and their hash.
@@ -263,13 +280,80 @@ export async function normalizeOpenApi(options: NormalizeOptions): Promise<Norma
263280
sourceVersion,
264281
openapi: String(out.openapi),
265282
operationKeys,
283+
sourceOperations: sourceOperations ?? operations.length,
266284
include,
267285
cuts,
268286
refsLeft: countRefs(out),
269287
filledConsumes,
270288
};
271289
}
272290

291+
/** Compile the allow-patterns; an invalid one is a refusal naming it. */
292+
function compilePatterns(sources: readonly string[] | undefined): RegExp[] {
293+
return (sources ?? []).map((source) => {
294+
try {
295+
return new RegExp(source);
296+
} catch (err) {
297+
throw new NormalizeError(`--include-pattern '${source}' is not a valid regular expression: ${(err as Error).message}`);
298+
}
299+
});
300+
}
301+
302+
/** An allow-pattern is tested against the action key and against 'METHOD /path'. */
303+
const matches = (patterns: readonly RegExp[], op: { key: string; method: string; path: string }): boolean =>
304+
patterns.some((p) => p.test(op.key) || p.test(`${op.method.toUpperCase()} ${op.path}`));
305+
306+
const noMatch = (total: number): NormalizeError =>
307+
new NormalizeError(
308+
`--include-pattern matched none of the ${total} operation(s); a pattern is tested against ` +
309+
`each action key (an operationId, or the method_path key) and against 'METHOD /path'.`,
310+
);
311+
312+
/**
313+
* `--trim`: the `paths` object with only the operations a pattern matches. A
314+
* path item keeps its non-operation keys (`parameters`, `summary`, `servers`,
315+
* extensions); one left with no operation is dropped. A `$ref` path item is
316+
* resolved first (its operations live in the target), without walking it.
317+
*/
318+
function trimPaths(
319+
paths: Json | undefined,
320+
patterns: readonly RegExp[],
321+
deref: Dereferencer,
322+
): { paths: JsonObject; kept: number; sourceOperations: number } {
323+
const out: JsonObject = {};
324+
let kept = 0;
325+
let sourceOperations = 0;
326+
for (const [path, raw] of Object.entries(isObject(paths) ? paths : {})) {
327+
if (path.startsWith('x-')) {
328+
put(out, path, raw);
329+
continue;
330+
}
331+
const item = deref.target(raw);
332+
if (!isObject(item)) continue;
333+
const trimmed: JsonObject = {};
334+
let keptHere = 0;
335+
for (const [key, value] of Object.entries(item)) {
336+
const method = HTTP_METHODS.find((m) => m === key);
337+
if (method === undefined) {
338+
put(trimmed, key, value);
339+
continue;
340+
}
341+
if (!isObject(value)) continue;
342+
sourceOperations++;
343+
const operationId = typeof value.operationId === 'string' ? value.operationId : undefined;
344+
if (matches(patterns, { key: operationActionKey({ operationId, method, path }), method, path })) {
345+
put(trimmed, key, value);
346+
keptHere++;
347+
}
348+
}
349+
if (keptHere > 0) {
350+
put(out, path, trimmed);
351+
kept += keptHere;
352+
}
353+
}
354+
return { paths: out, kept, sourceOperations };
355+
}
356+
273357
/** Count `$ref` keys anywhere in a value. */
274358
function countRefs(node: Json): number {
275359
if (Array.isArray(node)) return node.reduce<number>((sum, item) => sum + countRefs(item), 0);
@@ -391,6 +475,27 @@ class Dereferencer {
391475
return out;
392476
}
393477

478+
/**
479+
* A `$ref` object's target, followed to the end of its chain with the
480+
* referring object's sibling keys laid over it, and nothing inside walked:
481+
* what a `$ref` path item stands for, before `--trim` selects its operations.
482+
*/
483+
target(node: Json): Json {
484+
const seen = new Set<string>();
485+
let current = node;
486+
let siblings: JsonObject = {};
487+
while (isObject(current) && typeof current.$ref === 'string') {
488+
if (seen.has(current.$ref)) {
489+
throw new NormalizeError(`the document's $ref '${current.$ref}' refers to itself outside a schema; it has no inline form.`);
490+
}
491+
seen.add(current.$ref);
492+
const { $ref, ...rest } = current;
493+
siblings = { ...rest, ...siblings };
494+
current = this.resolve($ref);
495+
}
496+
return isObject(current) ? { ...current, ...siblings } : current;
497+
}
498+
394499
/** Literal data: a `$ref` to it is resolved, and nothing inside it is walked. */
395500
private literal(node: Json): Json {
396501
const seen = new Set<string>();
@@ -537,7 +642,7 @@ function listOperations(doc: JsonObject): { key: string; method: string; path: s
537642
}
538643

539644
const USAGE =
540-
'usage: normalize-openapi --in <file | http(s) URL> --out <file.json> [--include-pattern <regex>]... ' +
645+
'usage: normalize-openapi --in <file | http(s) URL> --out <file.json> [--include-pattern <regex>]... [--trim] ' +
541646
`[--max-depth <n> (default ${DEFAULT_MAX_DEPTH})] [--server-url <url>] [--empty-consumes <media type>]`;
542647

543648
/**
@@ -547,8 +652,13 @@ const USAGE =
547652
*/
548653
export async function main(argv: readonly string[], io = { out: process.stdout, err: process.stderr }): Promise<number> {
549654
const flags: Record<string, string[]> = {};
655+
let trim = false;
550656
for (let i = 0; i < argv.length; i++) {
551657
const arg = argv[i] as string;
658+
if (arg === '--trim') {
659+
trim = true;
660+
continue;
661+
}
552662
const value = argv[i + 1];
553663
if (!/^--(in|out|include-pattern|max-depth|server-url|empty-consumes)$/.test(arg) || value === undefined) {
554664
io.err.write(`normalize-openapi: unexpected argument '${arg}'.\n${USAGE}\n`);
@@ -573,6 +683,7 @@ export async function main(argv: readonly string[], io = { out: process.stdout,
573683
input,
574684
cwd,
575685
includePatterns: flags['include-pattern'],
686+
trim,
576687
maxDepth: depthFlag === undefined ? undefined : Number(depthFlag),
577688
serverUrl: single('server-url'),
578689
emptyConsumes: single('empty-consumes'),
@@ -590,6 +701,7 @@ export async function main(argv: readonly string[], io = { out: process.stdout,
590701
sourceVersion: result.sourceVersion,
591702
openapi: result.openapi,
592703
operations: result.operationKeys.length,
704+
...(trim ? { trimmedFrom: result.sourceOperations } : {}),
593705
maxDepth: depthFlag === undefined ? DEFAULT_MAX_DEPTH : Number(depthFlag),
594706
cuts: result.cuts,
595707
refsLeft: result.refsLeft,

0 commit comments

Comments
 (0)