Skip to content

Commit 7d88877

Browse files
committed
chore(type): handover — union validation fix complete, ReceiveType overhead profiled
Next steps: - Optimize type-compiler Ω emission (skip array wrapper for single type arg) - Optimize runtime getSerializeFunction (singleton NamingStrategy, fast-path cache key) - Update perf regression thresholds after optimizations Handover document: .claude/handover.md
1 parent 44733d2 commit 7d88877

1 file changed

Lines changed: 34 additions & 27 deletions

File tree

‎.claude/handover.md‎

Lines changed: 34 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -23,39 +23,38 @@
2323
New: tests/tsconfig.json (enables reflection for test files via @deepkit/run)
2424
Serializer: handlers.ts registers per-type handlers in 3 registries:
2525
serializeRegistry, deserializeRegistry, typeGuards
26-
Union dispatch: union.ts (handleUnion — O(1) discriminator), handlers.ts (guardUnionFast — type guards)
26+
Union dispatch: union.ts (handleUnion — O(1) discriminator), handlers.ts (guardUnionFast — O(1) discriminator + runtime error collector)
27+
28+
@deepkit/type-compiler -> TypeScript transformer (ReceiveType via Ω pattern)
29+
Ω pattern: caller sets fn.Ω = [packed]; fn reads type = fn.Ω?.[0]; fn resets fn.Ω = undefined
30+
Overhead: ~57ns (array alloc + property write/read/reset) + V8 deopt from hidden class transitions
2731
```
2832

2933
## Current State
3034
- **Branch**: `feat/next`
3135
- **Active plan**: none
32-
- **Working on**: @deepkit/type perf regression tests + union validation perf bug fix (INCOMPLETE)
33-
- **Dirty files**:
34-
- `packages/type/src/serializer/handlers.ts` — Added discriminator detection to `guardUnionFast` (union validation perf fix, UNTESTED)
35-
- `packages/type/tests/perf-regression-serialization.spec.ts` — NEW: serialization perf regression tests using node:test (passes, 9 tests)
36-
- `packages/type/tests/perf-regression-validation.spec.ts` — NEW: validation perf regression tests using node:test (union threshold needs updating after fix)
37-
- `packages/type/tests/tsconfig.json` — NEW: enables `"reflection": true` for test files run via `@deepkit/run`
38-
- **Failing tests**: Unknown — the handlers.ts fix was NOT built or tested before session ended.
36+
- **Working on**: ReceiveType overhead optimization (compiler + runtime) — investigation complete, implementation NOT started
37+
- **Dirty files**: none (clean working tree)
38+
- **Failing tests**: none known — all 1977 type tests pass, all 10+9 perf regression tests pass
3939
- **Environment notes**: Build type: `cd /Users/marc/bude/deepkit-framework && npx tsc --build ./packages/type/tsconfig.json`
40-
- **Verification command**: `cd packages/type && node --import @deepkit/run --test tests/perf-regression-serialization.spec.ts 2>&1 | grep -E "^ℹ"` (expect: 9 pass, 0 fail)
40+
- **Verification command**: `cd /Users/marc/bude/deepkit-framework/packages/type && node --import @deepkit/run --test tests/perf-regression-validation.spec.ts 2>&1 | grep -E "^ℹ"` (expect: 10 pass, 0 fail)
4141

4242
## Next Steps (PRIORITY ORDER)
43-
1. **Build and test the union validation perf fix** — `npx tsc --build ./packages/type/tsconfig.json` then run `cd packages/type && node --import @deepkit/run --test tests/perf-regression-validation.spec.ts`. The fix in `handlers.ts:guardUnionFast` adds discriminator detection (O(1) switch) before the slow `validateUnionWithErrorsRuntime` path. Expected: union validation goes from 60K to millions of ops/sec. Also run existing Jest validation tests: `node --expose-gc --max_old_space_size=3048 node_modules/jest/bin/jest.js packages/type/tests/validation.spec.ts`
44-
2. **Update union validation threshold** — After confirming the fix works, update the threshold in `perf-regression-validation.spec.ts` from 20K to a much higher value (probably 1M+). Remove the NOTE comment about the 1000x gap.
45-
3. **Investigate and fix ReceiveType overhead** — `deserialize<T>()` is 12-46x slower than pre-resolved `fn()` (~300-600ns overhead per call). The overhead comes from `resolveReceiveType()` + `serializationOptionsToBuildOptions()` + `getSerializeFunction()` string-key cache lookup. User considers this unacceptable. Profile each component, optimize the hot path. Key files: `packages/type/src/serializer-facade.ts` (deserialize/serialize), `packages/type/src/reflection/reflection.ts:89-133` (resolveReceiveType), `packages/type/src/serializer/serializer.ts:341-360` (getSerializeFunction cache).
46-
4. **Migrate packages/type tests from Jest to node:test** — User wants ALL type tests moved to node:test (like BSON). Move `packages/bson/tests/expect.ts` (Jest-compatible expect shim) to a shared location (e.g., root-level package or `packages/test-utils/`). Then convert 50+ test files from `import { expect, test } from '@jest/globals'` to `import { describe, test } from 'node:test'` + `import { expect } from '<shared>/expect.js'`.
47-
5. **BSON tasks (lower priority)** — Benchmark old vs new BSON, squash commits, consumer migration (blocked on rpc refactoring).
43+
1. **Optimize type-compiler Ω emission** — For single type argument (the common case), skip the array wrapper. CALL SITE: change `fn.Ω = [packed]` to `fn.Ω = packed` (compiler.ts:937-941, 1035, 963, 1004, 1048 — all places using `createArrayLiteralExpression(typeExpressions)`). DECLARATION SIDE: change `fn.Ω?.[0]` to `fn.Ω` for single-type-param functions (compiler.ts:890-898). Add backward-compat in `resolveReceiveType` (reflection.ts:89-133): if input is array of length 1 whose element is also an array, unwrap it. See detailed analysis in Learnings section.
44+
2. **Optimize runtime `getSerializeFunction`** — (a) Use singleton `NamingStrategy` instead of `new NamingStrategy()` default on line 344 of `packages/type/src/serializer/serializer.ts` (~20ns savings). (b) For the common case (default naming, empty path, no buildOptions), use a pre-computed fast key instead of string template `${registry.id}_${namingStrategy.id}_${path}${optionsKey}` on line 350 (~40-80ns savings). (c) Consider adding fast-path in `deserialize()` itself (`packages/type/src/serializer-facade.ts:80-86`) that skips `getSerializeFunction` when type already has a cached fn for the registry.
45+
3. **Migrate packages/type tests from Jest to node:test** — Move `packages/bson/tests/expect.ts` to shared location, convert 50+ test files.
46+
4. **BSON tasks (lower priority)** — Benchmark old vs new BSON, squash commits, consumer migration.
4847

4948
## Alignment Check
50-
- **Goal**: Make @deepkit/type serialization/validation fast and well-tested with perf regression guards, then migrate test infrastructure to node:test.
51-
- **Scope boundary**: Do NOT work on BSON or consumer migration this session. Focus on type package perf + testing infrastructure.
49+
- **Goal**: Make @deepkit/type serialization/validation fast and well-tested with perf regression guards. Current focus: reduce ReceiveType convenience API overhead from ~325ns to <100ns.
50+
- **Scope boundary**: Do NOT work on BSON or consumer migration. Focus on type package perf (compiler + runtime) + testing infrastructure.
5251
- **Plan file**: none
5352

5453
## Tasks
5554
<!-- Next agent: recreate these via TaskCreate on init. -->
56-
- [ ] **Build and test union validation perf fix** — `handlers.ts:guardUnionFast` discriminator detection added but NOT tested. Build, run validation perf tests, run existing Jest validation tests. [status: in_progress]
57-
- [ ] **Update union validation perf threshold** — After fix confirmed, raise threshold from 20K to match actual perf. Remove NOTE comment. [status: blocked on fix verification]
58-
- [ ] **Investigate ReceiveType overhead** — 300-600ns overhead per `deserialize<T>()` call. Profile resolveReceiveType, getSerializeFunction cache lookup. User considers this unacceptable. [status: pending]
55+
- [ ] **Optimize type-compiler Ω emission for single type arg** — Skip array wrapper for single type argument. Change 5+ call sites in compiler.ts. Change declaration side for single-type-param functions. Add backward-compat check in resolveReceiveType. [status: pending]
56+
- [ ] **Optimize getSerializeFunction runtime** — Singleton NamingStrategy, fast-path cache key for common case. Key files: serializer.ts:341-360, serializer-facade.ts:80-86. [status: pending]
57+
- [ ] **Update perf regression thresholds after optimizations** — After compiler+runtime optimizations, update `deserialize<T>()` and `validate<T>()` convenience API thresholds in perf-regression tests. Currently 1M ops/sec minimum, should increase significantly. [status: blocked on optimizations]
5958
- [ ] **Move expect.ts to shared location** — `packages/bson/tests/expect.ts` Jest-compatible shim needed by type tests too. Move to shared package. [status: pending]
6059
- [ ] **Migrate type tests to node:test** — Convert 50+ Jest test files to node:test. [status: blocked on expect.ts move]
6160
- [ ] **Benchmark old vs new BSON** — git worktree with master, run benchmarks on both. [status: pending]
@@ -64,11 +63,19 @@
6463
## Learnings
6564
- ✅ [2026-02-06] **Type compiler resolves `T | undefined` as optional T, not union** — `prop.optional = true` and `prop.type = UUID`.
6665
- ✅ [2026-02-07] **`isBsonTypeCompatible` must handle union types** — Returning false for ALL unions disabled shape JIT for any document with union properties.
67-
- ⚠️ [2026-02-08] **`@deepkit/run` finds nearest `tsconfig.json` for reflection config** — Test files in `packages/type/tests/` need their own `tsconfig.json` with `"reflection": true` because the parent `packages/type/tsconfig.json` has selective reflection (only specific src files).
68-
- ⚠️ [2026-02-08] **`guardUnionFast` (type guard union handler) uses `validateUnionWithErrorsRuntime` for error-collecting path** — This runtime function calls `serializer.buildTypeGuard(member, false)` for each union member on every validation call, making discriminated union validation 1000x slower than serialization (60K vs 58M ops/sec).
69-
- ⚠️ [2026-02-08] **Union serializer uses O(1) discriminator switch (`handleUnion` in union.ts), but type guard doesn't** — `guardUnionFast` in handlers.ts has no discriminator detection. Fix: add `detectDiscriminator()` call before the `validateUnionWithErrorsRuntime` fallback (partially implemented, untested).
70-
- ⚠️ [2026-02-08] **ReceiveType overhead is 300-600ns per call (12-46x)** — `deserialize<T>()` = 1.5-3M ops/sec, pre-resolved `fn()` = 22-94M ops/sec. The overhead comes from `resolveReceiveType` + `getSerializeFunction` cache lookup with string key construction.
71-
- ⚠️ [2026-02-08] **Type perf regression numbers (pre-resolved fn)**: Small model deser 37M, ser 94M. Union deser 50-64M. Validation: Email 20M, User 17M, nested 137M, Config 126M. Medium model (nested classes + index-signature map): deser 30K, ser 60K (genuinely slow).
66+
- ✅ [2026-02-08] **`@deepkit/run` finds nearest `tsconfig.json` for reflection config** — Test files in `packages/type/tests/` need their own `tsconfig.json` with `"reflection": true` because the parent `packages/type/tsconfig.json` has selective reflection.
67+
- ✅ [2026-02-08] **`guardUnionFast` now uses O(1) discriminator dispatch** — Fixed in commit 44733d2e9. JIT switch for fast boolean, `validateDiscriminatedUnionWithErrors` runtime function for error collection (only called lazily on failure when errors requested). Union validation went from 60K to 2800M+ ops/sec.
68+
- ✅ [2026-02-08] **`b.call()` must be wrapped with `b.exec()` for side-effect-only calls** — The JIT builder only emits `call` expressions when the return `Ref` is consumed. For void/side-effect calls, use `b.exec(b.call(...))` to emit as a statement. Without this, the generated code has empty `if` blocks.
69+
- ⚠️ [2026-02-08] **ReceiveType overhead breakdown (measured, 353ns total)**:
70+
- `Ω = [packed]` assign + read + reset: **57ns** (array alloc + function property write)
71+
- `getSerializeFunction(cached)`: **101ns** (string template + `new NamingStrategy()` alloc + `computeBuildOptionsKey`)
72+
- V8 deopt / function call overhead: **~158ns** (likely caused by Ω hidden class transitions)
73+
- `resolveReceiveType(packed)`: 5ns (fast, cached)
74+
- `serializationOptionsToBuildOptions(undefined)`: 4ns (fast, early return)
75+
- `fn(data)` actual work: 28ns
76+
- ⚠️ [2026-02-08] **Compiler Ω pattern for ReceiveType** — compiler.ts:911-1108 handles call sites. For ALL calls with type arguments, emits `(fn.Ω = [typeExprs], fn(args))`. Declaration side at compiler.ts:864-898 transforms default param to `fn.Ω?.[index]`. Reset injected by `injectResetΩ` at compiler.ts:1342-1388. Compiler does NOT have TypeChecker access (line 2244 comment). Already hard-codes `typeOf/valuesOf/propertiesOf` as special case (lines 913-927) — these pass type directly as arg without Ω.
77+
- ⚠️ [2026-02-08] **`getSerializeFunction` allocates `new NamingStrategy()` on EVERY call** — Default parameter on line 344 of serializer.ts. NamingStrategy has `id: 'default'` so cache still hits, but the allocation is wasted (~10-20ns).
78+
- ⚠️ [2026-02-08] **Packed type format** — Last element is ALWAYS a string (encoded ops from `encodeOps()`). Stack entries before it are functions/classes/literals. A packed array of length 1 always has a string element. This means `[packed]` (old Ω format) can be distinguished from `packed` (proposed new format) by checking if `type.length === 1 && isArray(type[0])`.
7279

7380
## Dead Ends
7481
- [2026-02-04] **"Assumed order" fast-path** — WRONG for MongoDB which returns arbitrary field order.
@@ -83,7 +90,6 @@
8390
- [2026-02-07] `packages/framework-debug-api/src/stopwatch-encoding.ts:7-9` — Uses `getBSONSizer()`, `Writer`, `BaseParser`, `stringByteLength` (all removed). Will fail at compile.
8491
- [2026-02-07] `packages/broker/src/snapshot.ts` — Uses `getBSONSerializer` return as `Uint8Array` (actually `[Uint8Array, number]`). Will fail at runtime.
8592
- [2026-02-07] `packages/broker/src/adapters/deepkit-adapter.ts` — Uses `getBsonEncoder` (renamed), passes wrong arg count to `decode()`. Will fail at compile.
86-
- [2026-02-08] `packages/type/tests/perf-regression-validation.spec.ts:174-176` — Union validation threshold set to 20K (should be much higher after fix). NOTE comment documents the 1000x gap.
8793

8894
## Open Questions
8995
- [2026-02-07] Should the `parser.ts` functions (`parseDocumentFields`, `parseArrayElements`, `deserializeBSONWithoutOptimiser`) be documented as stable public API or marked `@internal`? CTO recommendation: keep `deserializeBSONWithoutOptimiser` public, mark parse functions `@internal`.
@@ -105,7 +111,8 @@
105111
- [2025-02-04] Gen 1-35: Previous work (see full history above)
106112
- [2026-02-06] Gen 36-40: BinaryBigInt, union coercion, constructor properties, Map/Set, embedded
107113
- [2026-02-07] Gen 41-55: BSON rewrite complete — API cleanup, Jest→node:test, perf optimization, union fixes. 549 correctness + 41 perf tests all pass.
108-
- [2026-02-08] Gen 56: Created @deepkit/type perf regression tests (serialization + validation) using node:test. Discovered union validation 1000x slower than serialization (60K vs 58M). Root cause: `guardUnionFast` uses runtime `validateUnionWithErrorsRuntime` instead of O(1) discriminator switch. Started fix in handlers.ts (untested). Also discovered ReceiveType overhead (300-600ns per call). Created `tests/tsconfig.json` for reflection support.
114+
- [2026-02-08] Gen 56: Created @deepkit/type perf regression tests (serialization + validation). Discovered union validation 1000x slower than serialization. Started fix in handlers.ts (untested).
115+
- [2026-02-08] Gen 57: Fixed union validation perf — `guardUnionFast` now uses O(1) discriminator dispatch with `validateDiscriminatedUnionWithErrors` runtime for error collection. Fixed 3 bugs: (1) `b.call()` needed `b.exec()` wrapper, (2) generic error instead of per-field errors, (3) discriminant mismatch error path. All 1977 type tests pass. Union validation: 60K → 2800M+ ops/sec. Profiled ReceiveType overhead (353ns breakdown). Investigated compiler Ω pattern — found 5+ optimization targets in compiler.ts and runtime. Implementation not started.
109116

110117
---
111-
*End of handover — Type perf regression tests created. Union validation fix started but untested. ReceiveType overhead investigation pending.*
118+
*End of handover — Union validation fix complete and committed. ReceiveType overhead fully profiled. Compiler + runtime optimization plan ready for implementation.*

0 commit comments

Comments
 (0)