|
23 | 23 | New: tests/tsconfig.json (enables reflection for test files via @deepkit/run) |
24 | 24 | Serializer: handlers.ts registers per-type handlers in 3 registries: |
25 | 25 | serializeRegistry, deserializeRegistry, typeGuards |
26 | | - Union dispatch: union.ts (handleUnion — O(1) discriminator), handlers.ts (guardUnionFast — type guards) |
| 26 | + Union dispatch: union.ts (handleUnion — O(1) discriminator), handlers.ts (guardUnionFast — O(1) discriminator + runtime error collector) |
| 27 | +
|
| 28 | +@deepkit/type-compiler -> TypeScript transformer (ReceiveType via Ω pattern) |
| 29 | + Ω pattern: caller sets fn.Ω = [packed]; fn reads type = fn.Ω?.[0]; fn resets fn.Ω = undefined |
| 30 | + Overhead: ~57ns (array alloc + property write/read/reset) + V8 deopt from hidden class transitions |
27 | 31 | ``` |
28 | 32 |
|
29 | 33 | ## Current State |
30 | 34 | - **Branch**: `feat/next` |
31 | 35 | - **Active plan**: none |
32 | | -- **Working on**: @deepkit/type perf regression tests + union validation perf bug fix (INCOMPLETE) |
33 | | -- **Dirty files**: |
34 | | - - `packages/type/src/serializer/handlers.ts` — Added discriminator detection to `guardUnionFast` (union validation perf fix, UNTESTED) |
35 | | - - `packages/type/tests/perf-regression-serialization.spec.ts` — NEW: serialization perf regression tests using node:test (passes, 9 tests) |
36 | | - - `packages/type/tests/perf-regression-validation.spec.ts` — NEW: validation perf regression tests using node:test (union threshold needs updating after fix) |
37 | | - - `packages/type/tests/tsconfig.json` — NEW: enables `"reflection": true` for test files run via `@deepkit/run` |
38 | | -- **Failing tests**: Unknown — the handlers.ts fix was NOT built or tested before session ended. |
| 36 | +- **Working on**: ReceiveType overhead optimization (compiler + runtime) — investigation complete, implementation NOT started |
| 37 | +- **Dirty files**: none (clean working tree) |
| 38 | +- **Failing tests**: none known — all 1977 type tests pass, all 10+9 perf regression tests pass |
39 | 39 | - **Environment notes**: Build type: `cd /Users/marc/bude/deepkit-framework && npx tsc --build ./packages/type/tsconfig.json` |
40 | | -- **Verification command**: `cd packages/type && node --import @deepkit/run --test tests/perf-regression-serialization.spec.ts 2>&1 | grep -E "^ℹ"` (expect: 9 pass, 0 fail) |
| 40 | +- **Verification command**: `cd /Users/marc/bude/deepkit-framework/packages/type && node --import @deepkit/run --test tests/perf-regression-validation.spec.ts 2>&1 | grep -E "^ℹ"` (expect: 10 pass, 0 fail) |
41 | 41 |
|
42 | 42 | ## Next Steps (PRIORITY ORDER) |
43 | | -1. **Build and test the union validation perf fix** — `npx tsc --build ./packages/type/tsconfig.json` then run `cd packages/type && node --import @deepkit/run --test tests/perf-regression-validation.spec.ts`. The fix in `handlers.ts:guardUnionFast` adds discriminator detection (O(1) switch) before the slow `validateUnionWithErrorsRuntime` path. Expected: union validation goes from 60K to millions of ops/sec. Also run existing Jest validation tests: `node --expose-gc --max_old_space_size=3048 node_modules/jest/bin/jest.js packages/type/tests/validation.spec.ts` |
44 | | -2. **Update union validation threshold** — After confirming the fix works, update the threshold in `perf-regression-validation.spec.ts` from 20K to a much higher value (probably 1M+). Remove the NOTE comment about the 1000x gap. |
45 | | -3. **Investigate and fix ReceiveType overhead** — `deserialize<T>()` is 12-46x slower than pre-resolved `fn()` (~300-600ns overhead per call). The overhead comes from `resolveReceiveType()` + `serializationOptionsToBuildOptions()` + `getSerializeFunction()` string-key cache lookup. User considers this unacceptable. Profile each component, optimize the hot path. Key files: `packages/type/src/serializer-facade.ts` (deserialize/serialize), `packages/type/src/reflection/reflection.ts:89-133` (resolveReceiveType), `packages/type/src/serializer/serializer.ts:341-360` (getSerializeFunction cache). |
46 | | -4. **Migrate packages/type tests from Jest to node:test** — User wants ALL type tests moved to node:test (like BSON). Move `packages/bson/tests/expect.ts` (Jest-compatible expect shim) to a shared location (e.g., root-level package or `packages/test-utils/`). Then convert 50+ test files from `import { expect, test } from '@jest/globals'` to `import { describe, test } from 'node:test'` + `import { expect } from '<shared>/expect.js'`. |
47 | | -5. **BSON tasks (lower priority)** — Benchmark old vs new BSON, squash commits, consumer migration (blocked on rpc refactoring). |
| 43 | +1. **Optimize type-compiler Ω emission** — For single type argument (the common case), skip the array wrapper. CALL SITE: change `fn.Ω = [packed]` to `fn.Ω = packed` (compiler.ts:937-941, 1035, 963, 1004, 1048 — all places using `createArrayLiteralExpression(typeExpressions)`). DECLARATION SIDE: change `fn.Ω?.[0]` to `fn.Ω` for single-type-param functions (compiler.ts:890-898). Add backward-compat in `resolveReceiveType` (reflection.ts:89-133): if input is array of length 1 whose element is also an array, unwrap it. See detailed analysis in Learnings section. |
| 44 | +2. **Optimize runtime `getSerializeFunction`** — (a) Use singleton `NamingStrategy` instead of `new NamingStrategy()` default on line 344 of `packages/type/src/serializer/serializer.ts` (~20ns savings). (b) For the common case (default naming, empty path, no buildOptions), use a pre-computed fast key instead of string template `${registry.id}_${namingStrategy.id}_${path}${optionsKey}` on line 350 (~40-80ns savings). (c) Consider adding fast-path in `deserialize()` itself (`packages/type/src/serializer-facade.ts:80-86`) that skips `getSerializeFunction` when type already has a cached fn for the registry. |
| 45 | +3. **Migrate packages/type tests from Jest to node:test** — Move `packages/bson/tests/expect.ts` to shared location, convert 50+ test files. |
| 46 | +4. **BSON tasks (lower priority)** — Benchmark old vs new BSON, squash commits, consumer migration. |
48 | 47 |
|
49 | 48 | ## Alignment Check |
50 | | -- **Goal**: Make @deepkit/type serialization/validation fast and well-tested with perf regression guards, then migrate test infrastructure to node:test. |
51 | | -- **Scope boundary**: Do NOT work on BSON or consumer migration this session. Focus on type package perf + testing infrastructure. |
| 49 | +- **Goal**: Make @deepkit/type serialization/validation fast and well-tested with perf regression guards. Current focus: reduce ReceiveType convenience API overhead from ~325ns to <100ns. |
| 50 | +- **Scope boundary**: Do NOT work on BSON or consumer migration. Focus on type package perf (compiler + runtime) + testing infrastructure. |
52 | 51 | - **Plan file**: none |
53 | 52 |
|
54 | 53 | ## Tasks |
55 | 54 | <!-- Next agent: recreate these via TaskCreate on init. --> |
56 | | -- [ ] **Build and test union validation perf fix** — `handlers.ts:guardUnionFast` discriminator detection added but NOT tested. Build, run validation perf tests, run existing Jest validation tests. [status: in_progress] |
57 | | -- [ ] **Update union validation perf threshold** — After fix confirmed, raise threshold from 20K to match actual perf. Remove NOTE comment. [status: blocked on fix verification] |
58 | | -- [ ] **Investigate ReceiveType overhead** — 300-600ns overhead per `deserialize<T>()` call. Profile resolveReceiveType, getSerializeFunction cache lookup. User considers this unacceptable. [status: pending] |
| 55 | +- [ ] **Optimize type-compiler Ω emission for single type arg** — Skip array wrapper for single type argument. Change 5+ call sites in compiler.ts. Change declaration side for single-type-param functions. Add backward-compat check in resolveReceiveType. [status: pending] |
| 56 | +- [ ] **Optimize getSerializeFunction runtime** — Singleton NamingStrategy, fast-path cache key for common case. Key files: serializer.ts:341-360, serializer-facade.ts:80-86. [status: pending] |
| 57 | +- [ ] **Update perf regression thresholds after optimizations** — After compiler+runtime optimizations, update `deserialize<T>()` and `validate<T>()` convenience API thresholds in perf-regression tests. Currently 1M ops/sec minimum, should increase significantly. [status: blocked on optimizations] |
59 | 58 | - [ ] **Move expect.ts to shared location** — `packages/bson/tests/expect.ts` Jest-compatible shim needed by type tests too. Move to shared package. [status: pending] |
60 | 59 | - [ ] **Migrate type tests to node:test** — Convert 50+ Jest test files to node:test. [status: blocked on expect.ts move] |
61 | 60 | - [ ] **Benchmark old vs new BSON** — git worktree with master, run benchmarks on both. [status: pending] |
|
64 | 63 | ## Learnings |
65 | 64 | - ✅ [2026-02-06] **Type compiler resolves `T | undefined` as optional T, not union** — `prop.optional = true` and `prop.type = UUID`. |
66 | 65 | - ✅ [2026-02-07] **`isBsonTypeCompatible` must handle union types** — Returning false for ALL unions disabled shape JIT for any document with union properties. |
67 | | -- ⚠️ [2026-02-08] **`@deepkit/run` finds nearest `tsconfig.json` for reflection config** — Test files in `packages/type/tests/` need their own `tsconfig.json` with `"reflection": true` because the parent `packages/type/tsconfig.json` has selective reflection (only specific src files). |
68 | | -- ⚠️ [2026-02-08] **`guardUnionFast` (type guard union handler) uses `validateUnionWithErrorsRuntime` for error-collecting path** — This runtime function calls `serializer.buildTypeGuard(member, false)` for each union member on every validation call, making discriminated union validation 1000x slower than serialization (60K vs 58M ops/sec). |
69 | | -- ⚠️ [2026-02-08] **Union serializer uses O(1) discriminator switch (`handleUnion` in union.ts), but type guard doesn't** — `guardUnionFast` in handlers.ts has no discriminator detection. Fix: add `detectDiscriminator()` call before the `validateUnionWithErrorsRuntime` fallback (partially implemented, untested). |
70 | | -- ⚠️ [2026-02-08] **ReceiveType overhead is 300-600ns per call (12-46x)** — `deserialize<T>()` = 1.5-3M ops/sec, pre-resolved `fn()` = 22-94M ops/sec. The overhead comes from `resolveReceiveType` + `getSerializeFunction` cache lookup with string key construction. |
71 | | -- ⚠️ [2026-02-08] **Type perf regression numbers (pre-resolved fn)**: Small model deser 37M, ser 94M. Union deser 50-64M. Validation: Email 20M, User 17M, nested 137M, Config 126M. Medium model (nested classes + index-signature map): deser 30K, ser 60K (genuinely slow). |
| 66 | +- ✅ [2026-02-08] **`@deepkit/run` finds nearest `tsconfig.json` for reflection config** — Test files in `packages/type/tests/` need their own `tsconfig.json` with `"reflection": true` because the parent `packages/type/tsconfig.json` has selective reflection. |
| 67 | +- ✅ [2026-02-08] **`guardUnionFast` now uses O(1) discriminator dispatch** — Fixed in commit 44733d2e9. JIT switch for fast boolean, `validateDiscriminatedUnionWithErrors` runtime function for error collection (only called lazily on failure when errors requested). Union validation went from 60K to 2800M+ ops/sec. |
| 68 | +- ✅ [2026-02-08] **`b.call()` must be wrapped with `b.exec()` for side-effect-only calls** — The JIT builder only emits `call` expressions when the return `Ref` is consumed. For void/side-effect calls, use `b.exec(b.call(...))` to emit as a statement. Without this, the generated code has empty `if` blocks. |
| 69 | +- ⚠️ [2026-02-08] **ReceiveType overhead breakdown (measured, 353ns total)**: |
| 70 | + - `Ω = [packed]` assign + read + reset: **57ns** (array alloc + function property write) |
| 71 | + - `getSerializeFunction(cached)`: **101ns** (string template + `new NamingStrategy()` alloc + `computeBuildOptionsKey`) |
| 72 | + - V8 deopt / function call overhead: **~158ns** (likely caused by Ω hidden class transitions) |
| 73 | + - `resolveReceiveType(packed)`: 5ns (fast, cached) |
| 74 | + - `serializationOptionsToBuildOptions(undefined)`: 4ns (fast, early return) |
| 75 | + - `fn(data)` actual work: 28ns |
| 76 | +- ⚠️ [2026-02-08] **Compiler Ω pattern for ReceiveType** — compiler.ts:911-1108 handles call sites. For ALL calls with type arguments, emits `(fn.Ω = [typeExprs], fn(args))`. Declaration side at compiler.ts:864-898 transforms default param to `fn.Ω?.[index]`. Reset injected by `injectResetΩ` at compiler.ts:1342-1388. Compiler does NOT have TypeChecker access (line 2244 comment). Already hard-codes `typeOf/valuesOf/propertiesOf` as special case (lines 913-927) — these pass type directly as arg without Ω. |
| 77 | +- ⚠️ [2026-02-08] **`getSerializeFunction` allocates `new NamingStrategy()` on EVERY call** — Default parameter on line 344 of serializer.ts. NamingStrategy has `id: 'default'` so cache still hits, but the allocation is wasted (~10-20ns). |
| 78 | +- ⚠️ [2026-02-08] **Packed type format** — Last element is ALWAYS a string (encoded ops from `encodeOps()`). Stack entries before it are functions/classes/literals. A packed array of length 1 always has a string element. This means `[packed]` (old Ω format) can be distinguished from `packed` (proposed new format) by checking if `type.length === 1 && isArray(type[0])`. |
72 | 79 |
|
73 | 80 | ## Dead Ends |
74 | 81 | - [2026-02-04] **"Assumed order" fast-path** — WRONG for MongoDB which returns arbitrary field order. |
|
83 | 90 | - [2026-02-07] `packages/framework-debug-api/src/stopwatch-encoding.ts:7-9` — Uses `getBSONSizer()`, `Writer`, `BaseParser`, `stringByteLength` (all removed). Will fail at compile. |
84 | 91 | - [2026-02-07] `packages/broker/src/snapshot.ts` — Uses `getBSONSerializer` return as `Uint8Array` (actually `[Uint8Array, number]`). Will fail at runtime. |
85 | 92 | - [2026-02-07] `packages/broker/src/adapters/deepkit-adapter.ts` — Uses `getBsonEncoder` (renamed), passes wrong arg count to `decode()`. Will fail at compile. |
86 | | -- [2026-02-08] `packages/type/tests/perf-regression-validation.spec.ts:174-176` — Union validation threshold set to 20K (should be much higher after fix). NOTE comment documents the 1000x gap. |
87 | 93 |
|
88 | 94 | ## Open Questions |
89 | 95 | - [2026-02-07] Should the `parser.ts` functions (`parseDocumentFields`, `parseArrayElements`, `deserializeBSONWithoutOptimiser`) be documented as stable public API or marked `@internal`? CTO recommendation: keep `deserializeBSONWithoutOptimiser` public, mark parse functions `@internal`. |
|
105 | 111 | - [2025-02-04] Gen 1-35: Previous work (see full history above) |
106 | 112 | - [2026-02-06] Gen 36-40: BinaryBigInt, union coercion, constructor properties, Map/Set, embedded |
107 | 113 | - [2026-02-07] Gen 41-55: BSON rewrite complete — API cleanup, Jest→node:test, perf optimization, union fixes. 549 correctness + 41 perf tests all pass. |
108 | | -- [2026-02-08] Gen 56: Created @deepkit/type perf regression tests (serialization + validation) using node:test. Discovered union validation 1000x slower than serialization (60K vs 58M). Root cause: `guardUnionFast` uses runtime `validateUnionWithErrorsRuntime` instead of O(1) discriminator switch. Started fix in handlers.ts (untested). Also discovered ReceiveType overhead (300-600ns per call). Created `tests/tsconfig.json` for reflection support. |
| 114 | +- [2026-02-08] Gen 56: Created @deepkit/type perf regression tests (serialization + validation). Discovered union validation 1000x slower than serialization. Started fix in handlers.ts (untested). |
| 115 | +- [2026-02-08] Gen 57: Fixed union validation perf — `guardUnionFast` now uses O(1) discriminator dispatch with `validateDiscriminatedUnionWithErrors` runtime for error collection. Fixed 3 bugs: (1) `b.call()` needed `b.exec()` wrapper, (2) generic error instead of per-field errors, (3) discriminant mismatch error path. All 1977 type tests pass. Union validation: 60K → 2800M+ ops/sec. Profiled ReceiveType overhead (353ns breakdown). Investigated compiler Ω pattern — found 5+ optimization targets in compiler.ts and runtime. Implementation not started. |
109 | 116 |
|
110 | 117 | --- |
111 | | -*End of handover — Type perf regression tests created. Union validation fix started but untested. ReceiveType overhead investigation pending.* |
| 118 | +*End of handover — Union validation fix complete and committed. ReceiveType overhead fully profiled. Compiler + runtime optimization plan ready for implementation.* |
0 commit comments