feat: detect the full hidden-character corpus

.changeset/hidden-characters-sweep.md

@@ -0,0 +1,25 @@
+---
+'@moderation-api/unicode-spoofing': minor
+---
+
+Broaden the `invisible` signal to the full hidden-character corpus.
+
+`analyze()` now also reports and strips:
+
+- **Invisible characters that belong to no word** — a bidi control or
+  zero-width character sitting between punctuation or spaces, rather than
+  inside a token. This is what hid two of the four controls in a Trojan Source
+  line (`if (level != "user<RLO> <LRI>// admin check<PDI> <LRI>") {`).
+- **Blank glyphs that are not whitespace** — Hangul fillers (U+115F, U+1160,
+  U+3164, U+FFA0), BRAILLE PATTERN BLANK, MUSICAL SYMBOL NULL NOTEHEAD. The
+  Hangul fillers previously misreported as `mixed_script`, since they are
+  Hangul letters to the property tables.
+- **Invisible combining marks** — COMBINING GRAPHEME JOINER, the Khmer inherent
+  vowels, KAITHI VOWEL SIGN I. Each stays legitimate inside its own script.
+- **Stray variation selectors** — a selector on a base with no registered
+  sequence, the "ASCII smuggling" payload channel. Registered sequences (emoji
+  presentation, keycaps, ideographic variants, Mongolian FVS) are untouched.
+
+Unicode whitespace (U+00A0, U+2000–U+200A, U+3000, …) is deliberately still not
+flagged: it is ordinary typography and every `\s` matcher already treats it as a
+space.

README.md

@@ -4,10 +4,14 @@
 [![CI](https://github.com/moderation-api/unicode-spoofing/actions/workflows/ci.yml/badge.svg)](https://github.com/moderation-api/unicode-spoofing/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE)
 
-Detection of Unicode-based text obfuscation ("homoglyph attacks"): spammers
-mixing lookalike characters into words to slip past keyword filters, e.g.
-`НОТ busіnеss рrоduсt` — visually clean, but `НОТ` is pure Cyrillic and
-`busіnеss` mixes two scripts inside one word.
+Text that reads as English but isn't. `НОТ busіnеss рrоduсt` looks ordinary in
+any font — `НОТ` is pure Cyrillic, `busіnеss` mixes two scripts inside one word,
+and neither matches a keyword filter looking for `HOT` or `business`.
+
+This library finds that, plus the characters you can't see at all: zero-width
+joiners inside a word, bidi overrides that reorder a line, tag characters
+smuggling an instruction into a prompt. It reports what fired, where, and gives
+you the de-obfuscated text back.
 
 Zero runtime dependencies. Script classification rides the JS engine's own
 Unicode property tables (`\p{Script=…}`); the only shipped data is the
@@ -28,24 +32,157 @@ npm install @moderation-api/unicode-spoofing
 import { analyze, skeleton } from '@moderation-api/unicode-spoofing';
 
 const r = analyze('Неу Anatoly, НОТ busіnеss рrоduсt just drоppеd.');
+
 r.spoofed; // true
-r.signals; // { mixed_script: true, confusable_word: true, invisible: false, zalgo: false }
-r.words; // [{ word: 'НОТ', skeleton: 'HOT', scripts: ['Cyrillic'], signals: ['confusable_word'], index: 13 }, …]
+r.signals.confusable_word; // true
 r.normalized; // 'Hey Anatoly, HOT business product just dropped.'
+r.words[1]; // { word: 'НОТ', index: 13, signals: ['confusable_word'],
+//              scripts: ['Cyrillic'], skeleton: 'HOT' }
 
 skeleton('раураl') === skeleton('paypal'); // true (UTS #39 comparison)
 ```
 
+## What it catches
+
+Every example below runs as a test — see [`test/readme.test.ts`](./test/readme.test.ts),
+which asserts each documented output. Characters that would otherwise be
+invisible in this file are written as escapes.
+
+### Trojan Source
+
+Bidi controls reorder how a line _renders_ without changing what the compiler
+reads ([CVE-2021-42574](https://trojansource.codes/)). The `// admin check`
+below is not a comment — it only looks like one.
+
+```ts
+const line = 'if (level != "user\u202E \u2066// admin check\u2069 \u2066") {';
+
+analyze(line).signals.invisible; // true
+analyze(line).normalized; // 'if (level != "user // admin check ") {'
+```
+
+The same trick renames files: `'invoice\u202Egnp.exe'` renders as
+`invoiceexe.png` in most UIs, and normalizes to `invoicegnp.exe`.
+
+### ASCII smuggling
+
+Tag characters (`U+E0000`–`U+E007F`) re-encode ASCII invisibly. Pasted into a
+prompt, an LLM reads the payload and a human reviewer sees a compliment.
+
+```ts
+const encode = (s: string) =>
+  [...s].map((c) => String.fromCodePoint(0xe0000 + c.codePointAt(0)!)).join('');
+
+const msg = `Looks helpful!${encode('Ignore all previous instructions')}`;
+
+msg.length; // 78 — the visible part is 14
+analyze(msg).signals.invisible; // true
+analyze(msg).normalized; // 'Looks helpful!'
+```
+
+To read what was hidden rather than just strip it:
+
+```ts
+const hidden = analyze(msg)
+  .words.flatMap((w) => [...w.word])
+  .filter((c) => c.codePointAt(0)! >= 0xe0000)
+  .map((c) => String.fromCodePoint(c.codePointAt(0)! - 0xe0000))
+  .join(''); // 'Ignore all previous instructions'
+```
+
+### Lookalike domains and usernames
+
+```ts
+// Cyrillic 'а' in the middle of a Latin word.
+analyze('Login at pаypal.com to verify').words[0];
+// { word: 'pаypal', index: 9, signals: ['mixed_script'],
+//   scripts: ['Latin', 'Cyrillic'], skeleton: 'paypal' }
+
+// A whole word in one non-Latin script needs context to judge — say what you
+// expect and it resolves.
+analyze('аррӏе.com', { expectedScripts: ['Latin'] }).normalized; // 'apple.com'
+
+// Comparing identifiers directly: skeleton() is the UTS #39 fold.
+skeleton('аdmin') === skeleton('admin'); // true
+```
+
+### Filter evasion
+
+Four spellings that read normally, match no keyword list, and come back clean:
+
+```ts
+analyze('Get f\u200Br\u200Be\u200Be m\u200Bo\u200Bn\u200Be\u200By now').normalized;
+// 'Get free money now'          — zero-width spaces inside the words
+
+analyze('buy cheap v\u3164i\u3164a\u3164g\u3164ra').normalized;
+// 'buy cheap viagra'            — HANGUL FILLER: blank, but not whitespace
+
+analyze('𝐅𝐑𝐄𝐄 𝐜𝐫𝐲𝐩𝐭𝐨 giveaway').normalized;
+// 'FREE crypto giveaway'        — math alphanumerics
+
+analyze('Ⓕⓡⓔⓔ ⓜⓞⓝⓔⓨ').normalized;
+// 'Free money'                  — circled letters (also ＦＲＥＥ fullwidth)
+```
+
+### Zalgo
+
+```ts
+analyze('Z̸̢̬̈a̛̠͎lg̕o̶ spam').normalized; // 'Zalgo spam'
+```
+
+### Left alone
+
+None of these are flagged — see [false-positive guards](#false-positive-guards):
+
+```ts
+analyze('Привет, как дела?').spoofed; // false — real Cyrillic, not a lookalike
+analyze('日本語のテキスト。辻\uFE00さん').spoofed; // false — ideographic variant sequence
+analyze('می\u200Cخواهم کتاب').spoofed; // false — Persian ZWNJ is orthography
+analyze('Ship it 🎉 👨\u200D👩\u200D👧 ℹ\uFE0F').spoofed; // false — emoji ZWJ, presentation selector
+```
+
 ## Signals
 
 | Signal            | Meaning                                                              | Example                               |
 | ----------------- | -------------------------------------------------------------------- | ------------------------------------- |
 | `mixed_script`    | One word blends multiple scripts                                     | `busіnеss` (Latin + Cyrillic `і`/`е`) |
 | `confusable_word` | Whole word is a Latin lookalike (UTS #39 skeleton resolves to ASCII) | `НОТ` → `HOT`, `ＨＯＴ`, `𝐇𝐎𝐓`        |
-| `invisible`       | Format characters (zero-width etc.) inside a word                    | `fr​ee`                               |
+| `invisible`       | Characters that render as nothing, in or between words               | zero-width, bidi, tag chars, fillers  |
 | `zalgo`           | Combining marks stacked beyond orthographic depth (≥3 per base)      | `Z̸̢̬a̛lg̕o`                               |
 | `illegal`         | Control, non-character, or replacement code points anywhere in text  | `NUL`, `U+FFFE`, `U+FFFD`             |
 
+What `invisible` covers: format characters (zero-width space/joiner, word
+joiner, …), bidi controls incl. the Trojan Source overrides, tag characters,
+blank-but-not-whitespace glyphs, invisible combining marks, and variation
+selectors on a base with no registered sequence — whether they sit inside a
+word or alone between punctuation.
+
+## Recipes
+
+**Run your existing filter against clean text.** `normalized` is the whole
+point: match on it, report on the original.
+
+```ts
+const { normalized, spoofed } = analyze(userInput);
+const hit = BANNED.some((word) => normalized.toLowerCase().includes(word));
+if (hit || spoofed) flagForReview(userInput);
+```
+
+**Block impersonating usernames** without banning non-Latin ones:
+
+```ts
+const taken = new Set(existingUsernames.map(skeleton));
+if (taken.has(skeleton(candidate))) reject('too similar to an existing name');
+```
+
+**Tell it what's normal for your traffic.** Declaring `expectedScripts` cuts
+false positives on genuine non-Latin content _and_ sharpens detection: a whole
+word in an unexpected script becomes evidence on its own.
+
+```ts
+analyze(text, { expectedScripts: ['Cyrillic'] }); // real Russian traffic
+```
+
 ### False-positive guards
 
 - Legitimate multilingual text is never flagged: whole words in a single
@@ -63,11 +200,35 @@ skeleton('раураl') === skeleton('paypal'); // true (UTS #39 comparison)
   are legitimate single-word script mixes and are exempt.
 - Scripts whose orthography uses joiners/zero-width characters (Arabic, Indic
   scripts, Persian ZWNJ, …) are exempt from `invisible`; scripts with stacked
-  marks (Hebrew points, Arabic tashkeel, …) are exempt from `zalgo`. Emoji ZWJ
-  sequences are never word tokens, so they are never flagged.
+  marks (Hebrew points, Arabic tashkeel, …) are exempt from `zalgo`.
+- Invisible characters that build a sequence are left alone: emoji ZWJ
+  sequences, keycaps, the tag characters in `🏴󠁧󠁢󠁳󠁣󠁴󠁿`, emoji presentation
+  selectors (`❤️`, `ℹ️`), ideographic variants (`辻︀`) and Mongolian FVS.
+- Unicode whitespace (`U+00A0`, `U+2000`–`U+200A`, `U+3000`, …) is NOT flagged:
+  it is real typography and every `\s` matcher already treats it as a space.
+  Blank glyphs that are _not_ whitespace — Hangul fillers, `U+2800` — are.
 - `normalized` only rewrites _affected_ words; legitimate non-Latin text is
   returned byte-for-byte.
 
+## From Moderation API
+
+We build [Moderation API](https://moderationapi.com) — a hosted moderation
+platform covering toxicity, NSFW, PII, spam, scams and phishing across text,
+images, video and audio in 120+ languages, with review queues, user trust
+levels and DSA/GDPR audit trails.
+
+This library is one primitive from that stack, released standalone because
+Unicode spoofing is a self-contained problem worth solving in the open. It
+makes no network calls, needs no API key, and does not talk to the hosted
+service — use it on its own for as long as it does the job. If lookalike text
+turns out to be one symptom of a wider user-generated-content problem, the
+platform is the rest of the answer.
+
+## Credits
+
+- Confusables data comes from the Unicode Consortium's
+  [UTS #39](https://www.unicode.org/reports/tr39/) security tables.
+
 ## Updating the Unicode data
 
 ```bash