[ybai08]

Follow-up to #6241, which patched the generated UnicodeProperties.ts directly and was closed. As @PerBothner asked there, the fixes went to the generator first, and this PR re-imports the generated files.

The files are regenerated from PerBothner/unicode-properties at commit a58885f6811d8db9b0729ddf23a1f786494608da (npm install; make with the Unicode 18.0.0 data files from https://www.unicode.org/Public/18.0.0/). That commit includes:

• Read the trie header at the view's byteOffset PerBothner/unicode-properties#1: unicode-trie reads the trie header at the view's byteOffset. This is the same change as the open Read the unicode trie header at the view's byteOffset #6080 (for addon-unicode-graphemes: trie header is read via DataView(data.buffer) ignoring byteOffset, so a pooled base64 decode silently corrupts widths #6079), so this PR leaves unicode-trie.ts alone and that fix to Read the unicode trie header at the view's byteOffset #6080.
• Only join ZWJ and ExtPic after an ExtPic Extend* prefix (GB11) PerBothner/unicode-properties#2: GB11 only joins ZWJ × ExtPic after an ExtPic Extend* prefix.
• 487f09f: UNICODE_VERSION 18.0.0.
• Support Indic_Conjunct_Break and grapheme rule GB9c PerBothner/unicode-properties#3: Indic_Conjunct_Break data and rule GB9c, in its Unicode 18 form \p{InCB=Linker} \p{InCB=Extend}* × \p{InCB=Consonant}.

Changes

• src/third-party/UnicodeProperties.ts: replaced with out-ts/uc-properties.ts. Three local differences that the vendored file already had are kept:

  • the import path ./unicode-trie (upstream uses ./unicode-trie/index.mjs);
  • the _dec() decoder, which uses Buffer in node and atob otherwise (upstream uses window.atob);
  • the 0 | 1 |2 return type of infoToWidth.

  Apart from these, the file is identical to the generator output.

• src/third-party/unicode-trie.ts: not changed. The regenerated out-ts/unicode-trie.ts differs from it only by the byteOffset fix, which is Read the unicode trie header at the view's byteOffset #6080.

• src/third-party/tiny-inflate.ts: out-ts/tiny-inflate.ts is byte-identical to the vendored copy, so the file is not changed.

• UnicodeGraphemeProvider.ts: uses the newly exported UC.GRAPHEME_BREAK_SAW_Regional_Pair constant in place of the literal 32.

• README and the comment in UnicodeGraphemesAddon.ts: now say the data is Unicode 18. The README also gets the correct path of the generated file (src/third-party/) and lists the two helper files that come from the same out-ts directory.

• test/UnicodeGraphemesAddon.test.ts: fixes a typo. The "woman lifting weights, medium dark" case used \u{F3CB} (a private-use character) where \u{1F3CB} was meant. With the old GB11 behaviour the ZWJ after it still joined the following U+2640, so the expected width of 7 came out right by accident. With correct GB11 the typo gives 8. With \u{1F3CB}, which is what the comment describes, the width is 7 again.

• New src/UnicodeGraphemeProvider.test.ts: mocha unit tests for the provider's cluster boundaries and widths. They cover GB11, GB9c, and two Unicode 16+ data changes: U+0B53 is now Extend, and U+2630 is now East_Asian_Width=W.

Data changes

The old and new tables were decoded for every code point from U+0000 to U+10FFFF, with the old one read through the fixed trie reader. On the grapheme-break and width bits, 2444 code points differ. The largest groups are:

• 1426 code points become wide (width 3), for example the Yijing hexagrams U+4DC0..U+4DFF, the Tai Xuan Jing symbols, the Tangut components and supplement, and the newer emoji in U+1FA80..U+1FAFF.
• 707 code points lose Extended_Pictographic. Unicode 18's emoji-data.txt no longer lists many non-emoji symbols as Extended_Pictographic, for example U+2388, U+2605 and U+1F000..U+1F02B.
• 268 code points become Extend, and the rest are smaller groups that change to or from SpacingMark, Prepend or V.

The InCB bits (INCB_MASK = 0xC0) are new.

Version string: not renamed, question for maintainers

The provider is still registered as '15' and '15-graphemes', and the addon still sets unicode.activeVersion = '15-graphemes'. These strings are what users pass to term.unicode.activeVersion. The demo (demo/client/client.ts), test/playwright/TestUtils.ts and the addon benchmark all set '15-graphemes', and @xterm/addon-unicode-graphemes is published on npm (0.4.0, plus 0.5.0 betas). Renaming them would break existing callers, so this PR leaves them alone.

Should they become '18' / '18-graphemes', perhaps with the '15' names kept as aliases? If so, that could go in this PR or in a follow-up. The remaining "15" mentions are all tied to those strings: the playwright test title wcwidth V15 emoji test and its expected versions list, and the v15: 20 cells label in demo/client/components/window/testWindow.ts.

Testing

GraphemeBreakTest-18.0.0.txt (853 lines) was run through the real UnicodeGraphemeProvider.charProperties join chain, outside the repo:

	pass	fail
master, decoded through Buffer.from in node	733	120
master, decoded through atob (as in a browser)	744	109
this PR, decoded through atob (or in node with #6080 applied)	766	87

The two master rows differ because of the byteOffset bug that #6080 fixes: in node, the pooled Buffer leaves the trie header at a non-zero offset.

80 of the 87 remaining failures involve CR, LF or Control, which the generator folds into Other on purpose. The other 7 start with an InCB=Linker, for example ÷ 094D × 0915 ÷. They come from how the provider builds the state for the first code point, an xterm-side issue that is left for a separate PR.

The addon tests were run, along with the repo's build, unit and lint commands (macOS, Node 22):

• npm run build && npm run esbuild: no errors.
• npm run test-unit -- addons/addon-unicode-graphemes/out-esbuild/*.test.js: 8 passing. With master's third-party/ files and the new test file, 7 of the 8 fail. With master's UnicodeProperties.ts plus only the trie fix, 6 of the 8 fail. The 8 also pass on this branch as is, without the trie fix.
• npm run test-unit: 2411 passing.
• npm run lint: clean. npm run lint-api: clean.
• npm run test-integration -- --suite=addon-unicode-graphemes --project=Chromium: 1 passed. Before the typo fix above, it failed on the "woman lifting weights" assertion.
• npm run test-integration -- --suite=core --project=Chromium: 419 passed, 59 skipped. The core tests load this addon by default.

Not run: the Firefox and WebKit integration projects.