Skip to content

Commit deb7a89

Browse files
committed
perf: let metadata and section-pages start on their dependency's first chunk
With more than one thread, metadata and section-pages now start on the first chunk of their dependency instead of waiting for all of it, so parsing, metadata and section pages overlap. The generated files are unchanged. section-pages passes each module's entries on as they arrive and adds the chunk pages after the last of them, in module order, as if built from the whole input at once. A module's chunk pages need all of its entries, and a chunk need not hold whole modules, so groupWholeModules regroups streamed entries into batches of whole modules, and rejects a module whose entries come back after another's. Assisted-by: Claude Opus 5.5 <noreply@anthropic.com> Assisted-by: Claude Sonnet 5.5 <noreply@anthropic.com>
1 parent bad221d commit deb7a89

11 files changed

Lines changed: 348 additions & 23 deletions

File tree

‎.changeset/stream-metadata.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'@doc-kit/core': minor
3+
'@doc-kit/generator-react': minor
4+
---
5+
6+
`metadata` and `section-pages` stream their input: with more than one thread, they start on their dependency's first chunk, so parsing, metadata and section pages overlap. The generated files are unchanged. `groupWholeModules`, from `@doc-kit/core/utils/generators.mjs`, regroups streamed entries so that each batch holds whole modules.

‎docs/creating-generators.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -379,7 +379,10 @@ export async function* generate(input, worker) {
379379
- With more than one thread, `input` can be iterated only once, and passing it
380380
to `worker.stream` iterates it. If you need it twice, collect it into an
381381
array first.
382-
- The chunks are the dependency's own, in the order it yields them.
382+
- The chunks are the dependency's own, in the order it yields them. `metadata`'s
383+
hold whole files, but chunks need not hold whole modules: a generator that
384+
needs all of a module's entries at once should regroup them with
385+
`groupWholeModules` from `@doc-kit/core/utils/generators.mjs`.
383386
- Work that needs the whole input, such as sorting or writing one file, has to
384387
wait for the last chunk.
385388
- Other generators reading the same dependency are not affected: each gets

‎packages/core/src/generators/metadata/index.mjs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ export default {
1616

1717
hasParallelProcessor: true,
1818

19+
streamsInput: true,
20+
1921
markdown: {
2022
remarkPlugins: [
2123
'remark-parse',

‎packages/core/src/generators/metadata/types.d.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,10 @@ import type { Position } from 'unist';
33

44
export type Generator = GeneratorMetadata<
55
{},
6-
Generate<Array<Root>, AsyncGenerator<MetadataEntry>>,
6+
Generate<
7+
Array<Root> | AsyncIterable<Array<Root>>,
8+
AsyncGenerator<MetadataEntry>
9+
>,
710
ProcessChunk<Root, MetadataEntry, Record<string, string>>
811
>;
912

‎packages/core/src/utils/__tests__/generators.test.mjs‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import { describe, it } from 'node:test';
44
import {
55
getEntryDescription,
66
groupNodesByModule,
7+
groupWholeModules,
78
getVersionFromSemVer,
89
coerceSemVer,
910
getCompatibleVersions,
@@ -87,6 +88,97 @@ describe('groupNodesByModule', () => {
8788
});
8889
});
8990

91+
describe('groupWholeModules', () => {
92+
/**
93+
* Collects the batches regrouped from the given input, each entry as its
94+
* module and number.
95+
*
96+
* @param {Array<{ api: string }> | AsyncIterable<Array<{ api: string }>>} input
97+
* @returns {Promise<string[][]>}
98+
*/
99+
const collect = async input => {
100+
const output = [];
101+
102+
for await (const batch of groupWholeModules(input)) {
103+
output.push(batch.map(({ api, n }) => `${api}${n}`));
104+
}
105+
106+
return output;
107+
};
108+
109+
/**
110+
* Regroups the given batches, streamed in, and collects the batches it
111+
* yields.
112+
*
113+
* @param {Array<Array<{ api: string }>>} batches
114+
* @returns {Promise<string[][]>}
115+
*/
116+
const regroup = batches => {
117+
async function* input() {
118+
yield* batches;
119+
}
120+
121+
return collect(input());
122+
};
123+
124+
/**
125+
* An entry of a module, told apart from the module's others by its number.
126+
*
127+
* @param {string} api
128+
* @param {number} n
129+
* @returns {{ api: string, n: number }}
130+
*/
131+
const entry = (api, n) => ({ api, n });
132+
133+
it('passes the entries on in the same order, each batch ending once its last module is whole', async () => {
134+
const output = await regroup([
135+
[entry('a', 1), entry('a', 2), entry('b', 1)],
136+
[entry('c', 1)],
137+
]);
138+
139+
assert.deepEqual(output, [['a1', 'a2'], ['b1'], ['c1']]);
140+
});
141+
142+
it('holds back a module that continues in the next batch', async () => {
143+
const output = await regroup([
144+
[entry('a', 1), entry('b', 1)],
145+
[entry('b', 2), entry('b', 3)],
146+
[entry('b', 4), entry('c', 1)],
147+
]);
148+
149+
assert.deepEqual(output, [['a1'], ['b1', 'b2', 'b3', 'b4'], ['c1']]);
150+
});
151+
152+
it('skips empty batches', async () => {
153+
const output = await regroup([[], [entry('a', 1)], []]);
154+
155+
assert.deepEqual(output, [['a1']]);
156+
});
157+
158+
it('takes entries given as one array as a single batch', async () => {
159+
const output = await collect([entry('a', 1), entry('a', 2), entry('b', 1)]);
160+
161+
assert.deepEqual(output, [['a1', 'a2'], ['b1']]);
162+
});
163+
164+
it('rejects a module whose entries arrive apart', async () => {
165+
await assert.rejects(
166+
regroup([
167+
[entry('a', 1), entry('b', 1)],
168+
[entry('c', 1), entry('a', 2)],
169+
]),
170+
/"a" arrived apart/
171+
);
172+
});
173+
174+
it('rejects a module whose entries arrive apart in one batch too', async () => {
175+
await assert.rejects(
176+
regroup([[entry('a', 1), entry('b', 1), entry('a', 2)]]),
177+
/"a" arrived apart/
178+
);
179+
});
180+
});
181+
90182
describe('getVersionFromSemVer', () => {
91183
it('returns major.x for minor 0', () => {
92184
const version = { major: 18, minor: 0, patch: 0 };

‎packages/core/src/utils/generators.mjs‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,67 @@ export const toBatches = input => {
5959
return [input];
6060
};
6161

62+
/**
63+
* Regroups streamed entries so that each batch holds whole modules: a module
64+
* whose entries continue in the next batch is held back and passed on with
65+
* it. A module's entries must arrive together: one that comes back after
66+
* another module's is an error, wherever the batches end, as its pages would
67+
* be built without it. Entries given as one array, as a generator that streams
68+
* its input gets them with one thread, are taken as a single batch.
69+
*
70+
* @template {{ api: string }} T
71+
* @param {Array<T> | AsyncIterable<Array<T>>} batches - The entries, in
72+
* batches as they are streamed, or in one array
73+
* @returns {AsyncGenerator<Array<T>>} The same entries, in batches of whole
74+
* modules
75+
*/
76+
export async function* groupWholeModules(batches) {
77+
// The modules whose entries have all arrived
78+
const finished = new Set();
79+
80+
// The module whose entries are arriving, and those of them held back
81+
let current;
82+
let held = [];
83+
84+
for await (const batch of toBatches(batches)) {
85+
for (const { api } of batch) {
86+
if (api !== current) {
87+
if (finished.has(api)) {
88+
throw new Error(
89+
`The entries of "${api}" arrived apart: a module's entries must ` +
90+
'arrive together, in one chunk or in consecutive ones'
91+
);
92+
}
93+
94+
if (current !== undefined) {
95+
finished.add(current);
96+
}
97+
98+
current = api;
99+
}
100+
}
101+
102+
const entries = held.length > 0 ? [...held, ...batch] : batch;
103+
104+
// The module the batch ends with may continue in the next one
105+
let end = entries.length;
106+
107+
while (end > 0 && entries[end - 1].api === current) {
108+
end--;
109+
}
110+
111+
held = entries.slice(end);
112+
113+
if (end > 0) {
114+
yield entries.slice(0, end);
115+
}
116+
}
117+
118+
if (held.length > 0) {
119+
yield held;
120+
}
121+
}
122+
62123
/**
63124
* Parses the SemVer string into a Node.js-alike version
64125
*

‎packages/react/src/html/__tests__/generate.test.mjs‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,18 +5,37 @@ import { join } from 'node:path';
55
import { describe, it } from 'node:test';
66
import { pathToFileURL } from 'node:url';
77

8+
import { collectAsyncGenerator } from '@doc-kit/core/caching.mjs';
89
import { loadGenerator } from '@doc-kit/core/generators/loader.mjs';
910
import { setConfig } from '@doc-kit/core/utils/configuration/index.mjs';
1011
import { loadMarkdownPlugins } from '@doc-kit/core/utils/markdown/plugins.mjs';
1112
import { jsx, toJs } from 'estree-util-to-js';
1213

1314
import buildContent from '../../jsx-ast/utils/buildContent.mjs';
1415
import { buildNotFoundPage } from '../../jsx-ast/utils/synthetic/404.mjs';
15-
import { generate as chunk } from '../../section-pages/generate.mjs';
16+
import { generate as sectionPages } from '../../section-pages/generate.mjs';
1617
import { compile, createViteBundler } from '../bundlers/vite.mjs';
1718
import { FONTS } from '../constants.mjs';
1819
import { generate } from '../generate.mjs';
1920

21+
/**
22+
* Yields the given entries as one batch, as a dependency streaming them does.
23+
*
24+
* @param {Array<object>} entries
25+
*/
26+
async function* streamInOneBatch(entries) {
27+
yield entries;
28+
}
29+
30+
/**
31+
* Runs `section-pages` over the given entries, streamed in as one batch, and
32+
* collects its pages.
33+
*
34+
* @param {Array<object>} entries
35+
*/
36+
const chunk = entries =>
37+
collectAsyncGenerator(sectionPages(streamInOneBatch(entries)));
38+
2039
// Pages are rendered with the pipeline of `jsx-ast`
2140
await loadMarkdownPlugins(
2241
await loadGenerator(import.meta.resolve('../../jsx-ast/index.mjs'))

0 commit comments

Comments
 (0)