Repository navigation
chore(node): replace jest with node:test, drop 49 no-op tests - #2288
Merged
Merged
Conversation
jest cost 204 packages -- 36% of the dependency tree -- to provide eight
matchers. Node 22+ runs TypeScript test files natively, so the built-in
runner covers this suite with no toolchain at all.
packages 575 -> 321
devDeps 13 -> 10 (jest, ts-jest, @types/jest)
lib/expect.ts is a ~90 line shim over node:assert providing the matchers the
suite actually uses: toEqual, toBe, toBeDefined, toBeUndefined, toBeNull,
toHaveLength, toThrow and .rejects.toThrow. `equals` deliberately mirrors
jest's toEqual rather than assert.deepStrictEqual -- it ignores
undefined-valued properties and skips the prototype check, because the napi
bindings return class instances that tests compare against object literals.
ESM conversion needed four mechanical fixes:
- explicit file extensions on relative imports
- `from '../../'` -> `from '../../index.js'` (ESM has no directory resolution)
- __dirname -> import.meta.dirname
- EncodeOptions/InputSequence as `import type` (strip-only mode keeps value
imports at runtime, and they are types)
Also removes 49 of 139 tests whose only assertion was `toBeDefined()` or
`expect(x.constructor.name).toEqual('Normalizer')`. They asserted that a
constructor returned an object and nothing more. Every behavioural test is
kept: 90 pass, 2 skipped, same as before.
The documentation examples needed care. docs/source/*.rst pulls 37 snippets
out of examples/documentation/*.test.ts with :start-after:/:end-before: and
:dedent: 4, and those snippets show `require("tokenizers")` as the documented
API. So only the header changed there -- `var globRequire = require` became
createRequire(import.meta.url), above the first marker -- leaving all 37
regions byte-identical. Running prettier over them rewrote quotes and indent
width (which would have broken :dedent: 4 and changed the published docs), so
examples/documentation is now in .prettierignore.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
ArthurZucker
added a commit
that referenced
this pull request
Aug 5, 2026
main's #2286 dropped 8 unused devDependencies; this branch had already dropped those same 8 plus jest/ts-jest/@types/jest (#2288) and added the security resolutions. The branch's package.json is therefore a strict superset of main's changes, so both files resolve to the branch side. Note: git's auto-merge silently re-added "jest": "^29.5.0" to devDependencies, since main still lists it and the surrounding lines moved. That was dropped deliberately in #2288 and is removed here. Verified with `yarn install --immutable`.
ArthurZucker
added a commit
that referenced
this pull request
Aug 5, 2026
Picks up #2288 (jest -> node:test) and main via the base branch. This branch's package.json already contained every base change (node --test script, jest deps and the @babel/core/tar/ip-address resolutions dropped) and additionally drops eslint, so package.json, .prettierignore and yarn.lock resolve to this side. .eslintrc.yml stays deleted. Note: git's auto-merge silently re-added the six eslint devDependencies, since the base still lists them and the surrounding lines moved. Removed here. Verified with `yarn install --immutable` and `prettier . --check`.
ArthurZucker
added a commit
that referenced
this pull request
Aug 5, 2026
* chore(node): drop 8 unused devDependencies
Removes dev tooling that nothing in the repo invokes, cutting 109 packages
(687 -> 578) from yarn.lock.
- ava dead test runner; `scripts.test` is jest. The `ava` config
block and its `@swc-node/register` require went with it.
- benny, chalk only used by `scripts.bench`, which pointed at
- @swc-node/reg. `benchmark/bench.ts` -- a file that does not exist, so the
script was already broken. Dropped the script too.
- @swc/core only a peerDependency of @swc-node/register; jest transforms
via `preset: 'ts-jest'`, not swc.
- husky no `.husky/` directory, no `prepare` script -- the git hook
was never installed.
- lint-staged had an inline config block, but with husky absent nothing
ever ran it.
- npm-run-all only supplied `run-p` for the `format` script, now a plain
`&&` chain. It was the sole source of shell-quote, which
carried a CRITICAL advisory.
Resolves 2 open dependabot alerts (both shell-quote, one critical). The
remaining 31 are not reachable by deletion: tar/cacache/socks/ip-address all
hang off node-gyp <- fsevents <- jest-haste-map, and fsevents is a macOS-only
optional dep of jest, so it never installs on Linux CI at all. Clearing those
needs a jest major bump, not a dependency removal.
None of this ever shipped: `dependencies` is empty and `files` is
["index.d.ts", "index.js"].
Verified: yarn install, yarn lint, jest --listTests (6 suites), and
eslint/prettier/taplo/jest/napi binaries all resolve.
* chore(node): pin patched versions of every vulnerable transitive dep
Adds yarn `resolutions` forcing the patched release of every package with
an open Dependabot advisory in bindings/node. Takes the node manifest from
33 open alerts to 0, and the lockfile from 578 to 575 packages -- it gets
*smaller* despite two major bumps, because the newer trees dedupe better.
braces 3.0.2 -> 3.0.3
brace-expansion 1.1.13, 2.0.1 -> 1.1.18, 2.1.4
js-yaml 3.14.2, 4.1.0 -> 3.15.1, 4.3.1
minimatch 9.0.4 -> 9.0.9
glob 10.3.12 -> 10.5.0
@babel/core 7.24.3 -> 7.29.7
tar 6.2.1 -> 7.5.22 (major)
ip-address 9.0.5 -> 10.4.0 (major)
`minimatch` and `glob` needed two resolution entries each: separate
descriptors (`glob@^10.2.2` vs `glob@^10.3.10`) were resolving to different
versions, so pinning one left the other behind.
The two major bumps sit under node-gyp, which is only reachable via
fsevents -- a macOS-only optional dependency of jest. Nothing builds against
it on Linux CI.
Note this is dev tooling only: `dependencies` is empty and `files` is
["index.d.ts", "index.js"], so none of it ever shipped.
* chore(node): replace jest with node:test, drop 49 no-op tests (#2288)
jest cost 204 packages -- 36% of the dependency tree -- to provide eight
matchers. Node 22+ runs TypeScript test files natively, so the built-in
runner covers this suite with no toolchain at all.
packages 575 -> 321
devDeps 13 -> 10 (jest, ts-jest, @types/jest)
lib/expect.ts is a ~90 line shim over node:assert providing the matchers the
suite actually uses: toEqual, toBe, toBeDefined, toBeUndefined, toBeNull,
toHaveLength, toThrow and .rejects.toThrow. `equals` deliberately mirrors
jest's toEqual rather than assert.deepStrictEqual -- it ignores
undefined-valued properties and skips the prototype check, because the napi
bindings return class instances that tests compare against object literals.
ESM conversion needed four mechanical fixes:
- explicit file extensions on relative imports
- `from '../../'` -> `from '../../index.js'` (ESM has no directory resolution)
- __dirname -> import.meta.dirname
- EncodeOptions/InputSequence as `import type` (strip-only mode keeps value
imports at runtime, and they are types)
Also removes 49 of 139 tests whose only assertion was `toBeDefined()` or
`expect(x.constructor.name).toEqual('Normalizer')`. They asserted that a
constructor returned an object and nothing more. Every behavioural test is
kept: 90 pass, 2 skipped, same as before.
The documentation examples needed care. docs/source/*.rst pulls 37 snippets
out of examples/documentation/*.test.ts with :start-after:/:end-before: and
:dedent: 4, and those snippets show `require("tokenizers")` as the documented
API. So only the header changed there -- `var globRequire = require` became
createRequire(import.meta.url), above the first marker -- leaving all 37
regions byte-identical. Running prettier over them rewrote quotes and indent
width (which would have broken :dedent: 4 and changed the published docs), so
examples/documentation is now in .prettierignore.
* chore(node): drop eslint, format with prettier only (#2290)
* chore(node): replace jest with node:test, drop 49 no-op tests
jest cost 204 packages -- 36% of the dependency tree -- to provide eight
matchers. Node 22+ runs TypeScript test files natively, so the built-in
runner covers this suite with no toolchain at all.
packages 575 -> 321
devDeps 13 -> 10 (jest, ts-jest, @types/jest)
lib/expect.ts is a ~90 line shim over node:assert providing the matchers the
suite actually uses: toEqual, toBe, toBeDefined, toBeUndefined, toBeNull,
toHaveLength, toThrow and .rejects.toThrow. `equals` deliberately mirrors
jest's toEqual rather than assert.deepStrictEqual -- it ignores
undefined-valued properties and skips the prototype check, because the napi
bindings return class instances that tests compare against object literals.
ESM conversion needed four mechanical fixes:
- explicit file extensions on relative imports
- `from '../../'` -> `from '../../index.js'` (ESM has no directory resolution)
- __dirname -> import.meta.dirname
- EncodeOptions/InputSequence as `import type` (strip-only mode keeps value
imports at runtime, and they are types)
Also removes 49 of 139 tests whose only assertion was `toBeDefined()` or
`expect(x.constructor.name).toEqual('Normalizer')`. They asserted that a
constructor returned an object and nothing more. Every behavioural test is
kept: 90 pass, 2 skipped, same as before.
The documentation examples needed care. docs/source/*.rst pulls 37 snippets
out of examples/documentation/*.test.ts with :start-after:/:end-before: and
:dedent: 4, and those snippets show `require("tokenizers")` as the documented
API. So only the header changed there -- `var globRequire = require` became
createRequire(import.meta.url), above the first marker -- leaving all 37
regions byte-identical. Running prettier over them rewrote quotes and indent
width (which would have broken :dedent: 4 and changed the published docs), so
examples/documentation is now in .prettierignore.
* chore(node): drop eslint, format with prettier only
eslint and its plugins cost 211 packages -- 66% of what was left -- to
enforce import ordering and a set of TypeScript style rules. eslint 8 also
went end-of-life in October 2024, so it stops receiving security patches
while remaining the single largest source of transitive advisories here
(brace-expansion, js-yaml, minimatch, cross-spawn, ajv and flatted all
arrive through it).
packages 321 -> 110
devDeps 10 -> 4 (@napi-rs/cli, @taplo/cli, prettier, typescript)
Removed: eslint, eslint-config-prettier, eslint-plugin-import,
eslint-plugin-prettier, @typescript-eslint/eslint-plugin,
@typescript-eslint/parser, and .eslintrc.yml.
`lint` is now `prettier . --check`, which is what the eslint config mostly
delegated to anyway via plugin:prettier/recommended. What is genuinely lost
is import-order enforcement and the @typescript-eslint rules; `tsc` still
covers type errors.
Two supporting fixes so `prettier . --check` passes cleanly:
- npm/ is now prettier-ignored. Those 13 per-platform package.json stubs
are generated by napi, so formatting them only fights regeneration.
- README.md reformatted (whitespace on list items only).
Also fixes the Makefile, which was already broken on main: `check-style`
called `npm run lint-check`, a script that does not exist. `style` now
formats and `check-style` now checks, which is what the comments above them
always claimed.
Verified: yarn lint exit 0, and the suite still passes 90 with 2 skipped.
* ci: unbreak Lint TS and the cargo-readme check
Lint TS: CI runs `yarn build` before `yarn lint`, so napi regenerates
index.js and index.d.ts and prettier then checks the generated output
instead of what we committed. Ignore both, same reason `npm` is ignored.
cargo readme: CI installs cargo-readme unpinned and now resolves 3.4.0,
which strips rustdoc hidden lines (`# ...`) from code blocks. README.md
still carried them, so the generated file no longer matched. Regenerated;
the three dropped lines were rustdoc scaffolding that never belonged in
the rendered README.
Reproduced both locally (cargo-readme 3.4.0 gives CI's exact diff).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2287 (which is stacked on #2286). This PR's own diff is the top commit.
Result across the stack
yarn.lockWhy jest had to go
jest cost 204 packages — 36% of the tree — to provide eight matchers. Node 22+ runs TypeScript test files natively, so
node --testcovers this suite with no toolchain at all. Removing it also took@babel/core,fsevents,node-gyp,tar,cacacheandsockswith it.lib/expect.tsis a ~90-line shim overnode:assertfor the matchers actually used:toEqual,toBe,toBeDefined,toBeUndefined,toBeNull,toHaveLength,toThrow,.rejects.toThrow.One subtlety worth reviewing:
equalsdeliberately mirrors jest'stoEqual, notassert.deepStrictEqual. It ignoresundefined-valued properties and skips the prototype check, because the napi bindings hand back class instances (Encoding,AddedToken) that tests compare against plain object literals —deepStrictEqualwould reject those on the prototype check alone.ESM conversion needed four mechanical fixes:
from '../../'→from '../../index.js'(ESM has no directory resolution)__dirname→import.meta.dirnameEncodeOptions/InputSequence→import type(strip-only mode keeps value imports at runtime; these are types)The 49 deleted tests
Every one had
toBeDefined()orexpect(x.constructor.name).toEqual('Normalizer')as its only assertion — they asserted a constructor returned an object, nothing more. Names like "acceptsundefinedas second parameter" and "instantiates with two parameters".Every behavioural test is kept.
tokenizer(74 assertions),encoding(57) andutils(28) — offsets, word↔token mapping, negative-index slicing, truncation, padding — are untouched. 90 pass, 2 skipped, same as before.This does reduce coverage of the napi argument-marshalling surface. That was the explicit tradeoff requested; flagging it plainly so it's a conscious merge decision.
Docs: the part that nearly broke
docs/source/*.rstpulls 37 snippets out ofexamples/documentation/*.test.tsvia:start-after:/:end-before:with:dedent: 4, and those snippets showrequire("tokenizers")as the documented API.Running prettier over them rewrote quotes and re-indented 4-space → 2-space, which changes the published documentation and invalidates
:dedent: 4. I caught this by diffing every marker region, reverted it, and re-applied only a header change:var globRequire = require→createRequire(import.meta.url), which sits above the first marker.All 37 regions verified byte-identical.
examples/documentationis now in.prettierignoreso a futureyarn formatcan't silently rewrite the docs.Verification
make test(the CI path) — 90 pass, 0 fail, 2 skipped, 48 suitesyarn lint— exit 0