Fix four unicode-spoofing false-positive classes

.changeset/chilled-pugs-shave.md

@@ -0,0 +1,23 @@
+---
+'@moderation-api/unicode-spoofing': minor
+---
+
+Separate decode damage from spoofing with a new `encoding_damage` signal.
+
+U+FFFD used to count as `illegal`, so a message whose name field had been
+mangled upstream — `Hi Jos�� Luis` — reported as a spoofing attack. Across a
+115-row sample of real SMS traffic that was 61% of all flags.
+
+U+FFFD is a decoder's output, never an author's input: whatever the original
+bytes were, they are gone by the time the character exists, so it can carry no
+payload. It now reports as `encoding_damage`, which is included in `signals`
+and `words` but deliberately does **not** set `spoofed`, and the normalizer
+leaves it in place rather than silently repairing a corrupted message.
+
+`SpoofSignal` gains a member and `signals` gains a key, so an exhaustive
+`switch` or a strict `toEqual` on the signals object will need updating.
+Genuinely illegal code points (NUL, C1 controls, non-characters) are unchanged.
+
+A `SPOOFING_SIGNALS` constant is exported alongside `SPOOF_SIGNALS` — the same
+list minus `encoding_damage` — so callers can partition the two the same way
+`spoofed` does.

.changeset/great-moons-judge.md

@@ -0,0 +1,25 @@
+---
+'@moderation-api/unicode-spoofing': minor
+---
+
+Judge Latin words on UTS #39 Identifier_Status instead of skeleton luck.
+
+A word already written in Latin cannot be impersonating Latin, so the
+cross-script skeleton test measured only whether its fold happened to reach
+ASCII — which it does for ordinary European orthography, and arbitrarily.
+`æ` is a ligature that UTS #39 dissolves to `ae`, while `ø` keeps its stroke as
+a combining mark and never reaches ASCII, so `Ægir` was reported as a disguised
+word and `Ålborg` was not. Same alphabet, opposite verdicts.
+
+A Latin word now needs at least one character Unicode marks **Restricted** in
+`IdentifierStatus.txt` before its skeleton counts. `Ægir`, `Þór`, `Straße`,
+`cœur`, `ısıtır` and `Hawaiʻi` are all Allowed and pass through; `pɑypal`
+(U+0251 IPA ALPHA) and `first` (a compatibility ligature) are Restricted and are
+still caught — intra-Latin homoglyphs that no script comparison can see.
+
+The gate applies ONLY to Latin words. Cyrillic `а`/`о` are Allowed too, being
+ordinary Russian, so applying it everywhere would let `раураl` walk through.
+
+Adds `src/data/identifier-status.generated.ts` (~7 KB) and
+`scripts/generate-identifier-status.mjs`; `npm run generate:data` refreshes
+both tables.

.changeset/olive-pugs-clap.md

@@ -0,0 +1,22 @@
+---
+'@moderation-api/unicode-spoofing': minor
+---
+
+Stop reporting zero-width runs that are isolated in whitespace.
+
+A zero-width character evades a filter by splitting a word (`fr<ZWSP>ee`).
+A run with whitespace (or a string boundary) on BOTH sides splits nothing —
+the break it would create is already there — so it changes neither rendering
+nor tokenization. Rich-text editors leave these in templates constantly, and
+reporting them told senders their own newsletter was an attack.
+
+Requiring isolation on both sides is the safety of the rule. One-sided contact
+still does work: `admin<ZWSP>` renders as `admin` but compares unequal to it,
+and `<ZWSP>Valencia` glues to the front of a token the same way, so both are
+still reported. Runs longer than the new exported `ZERO_WIDTH_INERT_RUN` (4)
+are reported wherever they sit, because length alone makes a run a payload
+channel regardless of position.
+
+Scoped to genuinely zero-width, non-reordering characters (U+200B, U+200C,
+U+200D, U+2060, U+FEFF). Bidi controls, tag characters and blank glyphs are
+untouched — Trojan Source and ASCII smuggling detection are unaffected.

README.md

@@ -15,8 +15,8 @@ you the de-obfuscated text back.
 
 Zero runtime dependencies. Script classification rides the JS engine's own
 Unicode property tables (`\p{Script=…}`); the only shipped data is the
-UTS #39 confusables table, generated from the Unicode Consortium's
-`confusables.txt` and pinned to a Unicode version.
+UTS #39 security tables, generated from the Unicode Consortium's
+`confusables.txt` and `IdentifierStatus.txt` and pinned to a Unicode version.
 
 Ships as dual ESM/CommonJS with bundled type declarations. Node 20+.
 
@@ -44,6 +44,7 @@ r.signals;
 //   invisible: false,
 //   zalgo: false,
 //   illegal: false,
+//   encoding_damage: false,
 // }
 
 r.words;
@@ -75,13 +76,34 @@ a stray NUL byte:
 ```ts
 const r = analyze('НОТ busіnеss: fr\u200Bee cr̸͈͖͡ypto\u0000');
 
-r.signals; // every one of the five is true
+r.signals; // every spoofing signal is true
 r.normalized; // 'HOT business: free crypto'
 r.counts; // { wordsTotal: 4, wordsAffected: 5 }
 
 skeleton('раураl') === skeleton('paypal'); // true (UTS #39 comparison)
 ```
 
+### Broken text is not an attack
+
+`encoding_damage` is the one signal that does **not** set `spoofed`. U+FFFD is
+what a decoder emits when handed bytes it cannot read — a name mangled
+somewhere upstream, not a disguise:
+
+```ts
+const r = analyze('Hi Jos�� Luis, your appointment is confirmed.');
+
+r.signals.encoding_damage; // true
+r.spoofed; // false  <- a broken pipeline, not an attacker
+r.changed; // false  <- left alone, not silently "repaired"
+```
+
+The distinction is deliberate, and it is safe to make: whatever those bytes
+were, the decoder already destroyed them, so no payload can survive inside a
+U+FFFD for an attacker to exploit. Nothing is hidden from you — the signal and
+its word findings are still reported, so you can alert on data quality without
+it counting as a spoofing verdict. Collapsing the two means every mojibake'd
+`José` reads as an attack.
+
 ## What it catches
 
 Every example below runs as a test — see [`test/readme.test.ts`](./test/readme.test.ts),
@@ -189,7 +211,8 @@ analyze('Ship it 🎉 👨\u200D👩\u200D👧 ℹ\uFE0F').spoofed; // false —
 | `confusable_word` | Whole word is a Latin lookalike (UTS #39 skeleton resolves to ASCII) | `НОТ` → `HOT`, `ＨＯＴ`, `𝐇𝐎𝐓`        |
 | `invisible`       | Characters that render as nothing, in or between words               | zero-width, bidi, tag chars, fillers  |
 | `zalgo`           | Combining marks stacked beyond orthographic depth (≥3 per base)      | `Z̸̢̬a̛lg̕o`                               |
-| `illegal`         | Control, non-character, or replacement code points anywhere in text  | `NUL`, `U+FFFE`, `U+FFFD`             |
+| `illegal`         | Control or non-character code points anywhere in text                | `NUL`, `U+0085`, `U+FFFE`             |
+| `encoding_damage` | Decode damage — reported, but never sets `spoofed` (see above)       | `Jos��` (a mangled `José`)            |
 
 What `invisible` covers: format characters (zero-width space/joiner, word
 joiner, …), bidi controls incl. the Trojan Source overrides, tag characters,
@@ -204,14 +227,15 @@ against a name instead of a hardcoded string.
 
 | Export                           | What it is                                                                                            |
 | -------------------------------- | ----------------------------------------------------------------------------------------------------- |
-| `SPOOF_SIGNALS`                  | All five signal names — the keys of `result.signals`                                                  |
+| `SPOOF_SIGNALS`                  | All six signal names — the keys of `result.signals`                                                   |
 | `SCRIPT_NAMES`                   | Every script the classifier can name — the possible values of `dominantScript` and `words[].scripts`  |
 | `SUPPORTED_SCRIPTS`              | The subset of `SCRIPT_NAMES` this runtime's Unicode tables support (all of them, on a current engine) |
 | `PSEUDO_SCRIPTS`                 | `Common`, `Inherited`, `Unknown` — classifications that are not scripts and never appear in a finding |
 | `FORMAT_CHAR_SCRIPTS`            | Scripts exempt from `invisible` and `zalgo`, because joiners and stacked marks are their orthography  |
 | `LEGITIMATE_SCRIPT_COMBINATIONS` | Script mixes that never fire `mixed_script` (Japanese, Korean, Bopomofo-annotated Chinese)            |
 | `ZALGO_MARK_RUN`                 | Combining marks per base at which `zalgo` fires                                                       |
 | `UNICODE_VERSION`, `DATA_DATE`   | Which UTS #39 confusables table is compiled in                                                        |
+| `ZERO_WIDTH_INERT_RUN`           | Longest zero-width run still treated as inert when isolated in whitespace                             |
 
 `ScriptName` types all of these, so `expectedScripts` and any comparison
 against `dominantScript` is checked against the real list — a typo like
@@ -263,6 +287,15 @@ analyze(text, { expectedScripts: ['Cyrillic'] }); // real Russian traffic
   is Latin-dominant, other words already mix scripts, or the word's own script
   is Latin) **and** when the full UTS #39 skeleton lands in ASCII. Real words
   like `привет` contain letters with no ASCII prototype and pass through.
+- A word already written in Latin needs a UTS #39 **Restricted** character
+  before its skeleton counts. Nothing there is impersonating Latin — the word
+  IS Latin — so the skeleton test alone just asks whether the fold happens to
+  reach ASCII, which it does for ordinary European letters and does so
+  arbitrarily: `æ` is a ligature and dissolves to `ae`, while `ø` keeps its
+  stroke as a combining mark and never gets there. `Ægir`, `Þór`, `Straße`,
+  `cœur`, `ısıtır` and `Hawaiʻi` are all Allowed and pass through; `pɑypal`
+  (U+0251 IPA ALPHA) is Restricted and is still caught — an intra-Latin
+  homoglyph no script comparison can see.
 - `expectedScripts: ['Cyrillic']` marks scripts as normal for the caller's
   traffic — whole words in them are never flagged; intra-word mixing still is.
   Declaring it also sharpens detection the other way: a whole word written
@@ -299,15 +332,17 @@ platform is the rest of the answer.
 
 ## Credits
 
-- Confusables data comes from the Unicode Consortium's
-  [UTS #39](https://www.unicode.org/reports/tr39/) security tables.
+- Confusables and Identifier_Status data both come from the Unicode
+  Consortium's [UTS #39](https://www.unicode.org/reports/tr39/) security tables.
 
 ## Updating the Unicode data
 
 ```bash
 npm run generate:data                                            # latest
 node scripts/generate-confusables.mjs --version 17.0.0          # pinned
 node scripts/generate-confusables.mjs --file confusables.txt    # offline
+
+node scripts/generate-identifier-status.mjs                     # Identifier_Status
 ```
 
 .github/workflows/update-unicode-data.yml is a scheduled GitHub Actions job

package.json

@@ -54,7 +54,7 @@
     "test:coverage": "vitest run --coverage",
     "publint": "publint",
     "attw": "attw --pack .",
-    "generate:data": "node scripts/generate-confusables.mjs",
+    "generate:data": "node scripts/generate-confusables.mjs && node scripts/generate-identifier-status.mjs",
     "changeset": "changeset",
     "version": "changeset version",
     "release": "pnpm build && changeset publish"

scripts/generate-identifier-status.mjs

@@ -0,0 +1,106 @@
+#!/usr/bin/env node
+/**
+ * Regenerates src/data/identifier-status.generated.ts from the Unicode
+ * Consortium's UTS #39 Identifier_Status data.
+ *
+ * Usage:
+ *   node scripts/generate-identifier-status.mjs               # fetch /Public/security/latest/
+ *   node scripts/generate-identifier-status.mjs --version 17.0.0
+ *   node scripts/generate-identifier-status.mjs --file path/to/IdentifierStatus.txt
+ *
+ * The generated file is checked in; this script only needs to run when a new
+ * Unicode security release ships. A scheduled CI job is the intended caller.
+ *
+ * IdentifierStatus.txt lists the ALLOWED characters — those Unicode considers
+ * appropriate in identifiers. Everything absent from it is Restricted:
+ * technical, obsolete or limited-use characters that have no business in
+ * ordinary prose. That distinction is what separates "æ in a Danish surname"
+ * from "IPA alpha standing in for a latin a".
+ */
+import { readFileSync, writeFileSync, mkdirSync } from 'node:fs';
+import { dirname, join } from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const args = process.argv.slice(2);
+const getArg = (name) => {
+  const i = args.indexOf(name);
+  return i === -1 ? undefined : args[i + 1];
+};
+
+async function loadSource() {
+  const file = getArg('--file');
+  if (file) return readFileSync(file, 'utf8');
+  const version = getArg('--version') ?? 'latest';
+  const url = `https://www.unicode.org/Public/security/${version}/IdentifierStatus.txt`;
+  const res = await fetch(url);
+  if (!res.ok) throw new Error(`GET ${url} -> ${res.status}`);
+  return res.text();
+}
+
+const source = await loadSource();
+
+const versionMatch = source.match(/^#\s*Version:\s*(\S+)/m);
+const dateMatch = source.match(/^#\s*Date:\s*([\d-]+)/m);
+if (!versionMatch) throw new Error('No "# Version:" header in IdentifierStatus.txt');
+
+// Each data line is: <cp> ; Allowed  or  <cp>..<cp> ; Allowed
+const ranges = [];
+for (const line of source.split('\n')) {
+  const data = line.split('#')[0].trim();
+  if (!data) continue;
+  const [cps, status] = data.split(';').map((s) => s.trim());
+  if (status !== 'Allowed' || !cps) continue;
+  const [from, to] = cps.split('..');
+  const start = parseInt(from, 16);
+  const end = to === undefined ? start : parseInt(to, 16);
+  if (!Number.isInteger(start) || !Number.isInteger(end) || end < start) {
+    throw new Error(`Unparseable range: ${cps}`);
+  }
+  ranges.push([start, end]);
+}
+if (ranges.length < 500) {
+  throw new Error(`Suspiciously few ranges (${ranges.length}) — refusing to write`);
+}
+
+ranges.sort((a, b) => a[0] - b[0]);
+// Merge touching/overlapping ranges so the runtime binary search sees a
+// strictly increasing, non-overlapping list.
+const merged = [];
+for (const [start, end] of ranges) {
+  const last = merged[merged.length - 1];
+  if (last && start <= last[1] + 1) last[1] = Math.max(last[1], end);
+  else merged.push([start, end]);
+}
+
+// Packed as delta-encoded hex pairs: "<gap from previous end>.<length>", so the
+// common case (a short range close to the last one) stays a couple of chars.
+let cursor = 0;
+const packed = merged
+  .map(([start, end]) => {
+    const entry = `${(start - cursor).toString(36)}.${(end - start).toString(36)}`;
+    cursor = end;
+    return entry;
+  })
+  .join(';');
+const chunks = packed.match(/.{1,2000}/gs) ?? [];
+
+const __dirname = dirname(fileURLToPath(import.meta.url));
+const outPath = join(__dirname, '..', 'src', 'data', 'identifier-status.generated.ts');
+mkdirSync(dirname(outPath), { recursive: true });
+
+writeFileSync(
+  outPath,
+  `// AUTO-GENERATED by scripts/generate-identifier-status.mjs — DO NOT EDIT.
+// Source: UTS #39 IdentifierStatus.txt (https://www.unicode.org/reports/tr39/)
+// Ranges: ${merged.length} (Identifier_Status=Allowed; everything else is Restricted)
+
+export const IDENTIFIER_STATUS_VERSION = '${versionMatch[1]}';
+export const IDENTIFIER_STATUS_DATE = '${dateMatch?.[1] ?? 'unknown'}';
+
+/** Delta-encoded base-36 "<gap>.<length>" pairs — see the generator. */
+export const ALLOWED_RANGES_PACKED =
+${chunks.map((c) => `  '${c}'`).join(' +\n')};
+`,
+);
+
+console.log(`Wrote ${merged.length} allowed ranges (Unicode ${versionMatch[1]}) to ${outPath}`);