Skip to content

Commit 76dfa51

Browse files
committed
tooling(cutover): js-comment-mask reads JSX children text as content, not code (objectstack#25803)
Cutover bill item 23. check-comment-mask-corpus found one file in the merged corpus where scanSource over-masks: ui/apps/console/src/pages/DocsIndex.tsx, whose JSX text <code>src/docs/*.md</code> holds a /* the scanner read as a block-comment opener, so 1,517 bytes of code from line 65 were masked as one comment. scanSource now tracks JSX on one stack of two entry kinds: an opening TAG being read (attributes are code, until its > or />) and an ELEMENT whose children are being read (text, until a tag, a closing tag or a { child expression). A tag opens where this module's position rule already allows an expression to begin (plus the keywords), in plain code or inside an attribute or child expression. It becomes an element only on evidence: its > is reached with nothing but attribute syntax on the way, and the source closes that name later on, so a type argument or assertion in JSX's position (<T,>(x) => x, <Foo>bar in a .ts file) stays code as before. Children text is flagged literal; quotes, // and /* in it open nothing. A template opened inside an entry stays innermost. Self-test: six new table rows (JSX text with /*, JSX text with // and a quote around a nested element and a child expression, a JSX comment, a self-closing tag after a spread, a generic arrow, a type assertion); the first two fail under the previous scanner. Corpus sweep: after, 18,117 files, 0 disagree, 0 unparseable (209.9 s). A first cut that read an element nested in an attribute expression as code measured 47 files fabricating, and is what the single stack replaced. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CUXv64KLMfYv3ia2sWi6te
1 parent e52c8fd commit 76dfa51

1 file changed

Lines changed: 155 additions & 2 deletions

File tree

‎scripts/js-comment-mask.mjs‎

Lines changed: 155 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -282,6 +282,30 @@ export function makeRegexRecogniser(options) {
282282
* @param {string} source
283283
* @returns {{ comment: Uint8Array, literal: Uint8Array, interpolation: Uint8Array }}
284284
*/
285+
/**
286+
* The name of a JSX tag opening at `at` (a `<`), `''` for a fragment, or null
287+
* when the next character cannot start one. Read only where the position rule
288+
* already allows an expression to begin (#25803).
289+
*/
290+
function jsxTagName(source, at) {
291+
if (source[at + 1] === '>') return '';
292+
const m = /^[A-Za-z_$][\w$.:-]*/.exec(source.slice(at + 1, at + 129));
293+
return m ? m[0] : null;
294+
}
295+
296+
/**
297+
* Every tag name this source CLOSES (`</name>`), with the offset of the last
298+
* such closer -- the evidence an opening tag is an element with children and
299+
* not a type argument or assertion that happens to sit where JSX could
300+
* (`<T,>(x) => x`, `<Foo>bar` in a `.ts` file). One pass, taken lazily the
301+
* first time a candidate tag is seen, so a file with no JSX pays nothing.
302+
*/
303+
function lastClosingTags(source) {
304+
const last = new Map();
305+
for (const m of source.matchAll(/<\/([A-Za-z_$][\w$.:-]*)?\s*>/g)) last.set(m[1] ?? '', m.index);
306+
return last;
307+
}
308+
285309
export function scanSource(source) {
286310
const n = source.length;
287311
const comment = new Uint8Array(n);
@@ -301,6 +325,28 @@ export function scanSource(source) {
301325
// closing block of this function for why they are not flagged inline.
302326
const interpolations = [];
303327

328+
// JSX (objectstack#25803, cutover bill item 23). A JSX element's CHILDREN
329+
// text is content, not code: `<code>src/docs/*.md</code>` holds a `/*` that
330+
// opens nothing, and read as code it opened a comment 1,517 bytes long.
331+
//
332+
// One stack, innermost last, of two kinds of entry:
333+
// tag an OPENING tag being read -- code, since attributes are code --
334+
// until its `>` or `/>`;
335+
// element an element whose CHILDREN are being read -- text, until a tag,
336+
// a closing tag or a `{` child expression opens.
337+
// Each carries the `{` depth open inside it (0: the tag's own syntax, or the
338+
// element's text) and the template depth it was opened at, so a template
339+
// opened inside one stays innermost. An attribute or child expression is
340+
// ordinary code, where another JSX element can open in turn.
341+
//
342+
// A tag becomes an element only on evidence: its `>` is reached with nothing
343+
// but attribute syntax on the way, and the source closes that name later on.
344+
// A type argument or an assertion in JSX's position (`<T,>(x) => x`,
345+
// `<Foo>bar` in a `.ts` file) carries no closer and stays code, as before.
346+
// Text bytes are flagged `literal`: content, the way a string's are.
347+
const jsx = [];
348+
let closers = null;
349+
304350
// A shebang is a comment to node; it is also the one line whose slashes are
305351
// neither division nor a regex.
306352
if (source.startsWith('#!')) {
@@ -344,6 +390,42 @@ export function scanSource(source) {
344390
continue;
345391
}
346392

393+
// The innermost JSX entry, when no template opened inside it since.
394+
const top = jsx.length && jsx[jsx.length - 1].templates === templates.length ? jsx[jsx.length - 1] : null;
395+
396+
// A JSX element's CHILDREN: text until a tag or a `{` expression opens.
397+
if (top && top.kind === 'element' && top.braces === 0) {
398+
const ch = source[i];
399+
if (ch === '{') {
400+
top.braces = 1;
401+
i++;
402+
prev = ''; // the expression starts fresh
403+
word = '';
404+
continue;
405+
}
406+
if (ch === '<' && source[i + 1] === '/') {
407+
// The closing tag: code, read to its `>`, and the element is done.
408+
let k = i + 2;
409+
while (k < n && /[\w$.:\s-]/.test(source[k])) k++;
410+
i = source[k] === '>' ? k + 1 : k;
411+
jsx.pop();
412+
prev = 'x'; // an element is a value
413+
word = '';
414+
continue;
415+
}
416+
const name = ch === '<' ? jsxTagName(source, i) : null;
417+
if (name !== null) {
418+
jsx.push({ kind: 'tag', name, braces: 0, templates: templates.length });
419+
i++;
420+
prev = '<';
421+
word = '';
422+
continue;
423+
}
424+
literal[i] = 1;
425+
i++;
426+
continue;
427+
}
428+
347429
const c = source[i];
348430
const next = source[i + 1];
349431

@@ -380,6 +462,49 @@ export function scanSource(source) {
380462
i++;
381463
continue;
382464
}
465+
// JSX opening tags (see the stack's comment above): the tag's own syntax,
466+
// when it is the innermost entry and no attribute expression is open.
467+
let inner = top;
468+
if (inner && inner.kind === 'tag' && inner.braces === 0) {
469+
if (c === '/' && next === '>') {
470+
jsx.pop(); // self-closing: no children
471+
i += 2;
472+
prev = 'x';
473+
word = '';
474+
continue;
475+
}
476+
if (c === '>' && source[i - 1] !== '=') {
477+
jsx.pop();
478+
i++;
479+
closers ??= lastClosingTags(source);
480+
if ((closers.get(inner.name) ?? -1) >= i) {
481+
jsx.push({ kind: 'element', name: inner.name, braces: 0, templates: templates.length });
482+
} else {
483+
prev = '>'; // not an element after all -- read on exactly as before
484+
word = '';
485+
}
486+
continue;
487+
}
488+
// Anything but attribute syntax means this was never an opening tag.
489+
if (!/[\s\w$.:=/{-]/.test(c)) {
490+
jsx.pop();
491+
inner = null;
492+
}
493+
}
494+
// A tag opens wherever an expression can begin -- this module's regex
495+
// position rule, plus the keywords -- in plain code, or in an attribute or
496+
// child expression.
497+
if (c === '<' && (inner === null || inner.braces > 0)
498+
&& (!(IDENT_CHAR.test(prev) || prev === ')' || prev === ']') || REGEX_AFTER_KEYWORD.has(word))) {
499+
const name = jsxTagName(source, i);
500+
if (name !== null) {
501+
jsx.push({ kind: 'tag', name, braces: 0, templates: templates.length });
502+
i++;
503+
prev = '<';
504+
word = '';
505+
continue;
506+
}
507+
}
383508
// A regex literal: a `/` after anything that is not a value. THIS is this
384509
// module's position rule and it stays here (#15487) -- `}`, a bare `<`/`>`
385510
// and `++`/`--` are read as regex positions, which over-masks safely and is
@@ -406,7 +531,13 @@ export function scanSource(source) {
406531
// Inside `${...}`: balance the braces, so the interpolation ends at ITS
407532
// `}` and not at one quoted inside it. A `{` or `}` in a string, regex or
408533
// comment never reaches here -- its own branch consumed it already.
409-
if (frame && frame.braces > 0) {
534+
// The innermost context owns the brace: a JSX entry's attribute or child
535+
// expression (an element's text resumes when its depth returns to 0), then
536+
// a template's interpolation.
537+
if (inner && (c === '{' || c === '}')) {
538+
if (c === '{') inner.braces++;
539+
else if (inner.braces > 0) inner.braces--;
540+
} else if (frame && frame.braces > 0) {
410541
if (c === '{') frame.braces++;
411542
else if (c === '}' && --frame.braces === 0) {
412543
interpolations.push([frame.start, i + 1, true]);
@@ -655,6 +786,12 @@ const SELF_TEST_BATTERIES = Object.freeze({
655786
'a real comment inside an interpolation': 1,
656787
'a genuine docblock is still removed': 1,
657788
'a genuine line comment is still removed': 1,
789+
'a block-comment opener inside JSX text': 1,
790+
'a quote and a // inside JSX text, around a child expression and a nested element': 1,
791+
'a JSX comment in a child expression is still a comment': 1,
792+
'a self-closing tag after a spread opens no children': 1,
793+
'a generic arrow in JSX position is code, not an element': 1,
794+
'a type assertion with no closer is code, not an element': 1,
658795
});
659796

660797
// DELETING an entry silences that battery's floor exactly as effectively as
@@ -663,7 +800,7 @@ const SELF_TEST_BATTERIES = Object.freeze({
663800
// the literal above, so the roster falls below this number; the table
664801
// cross-check in the floor block is the other half, and names WHICH label
665802
// collided.
666-
const SELF_TEST_BATTERY_FLOOR = 23;
803+
const SELF_TEST_BATTERY_FLOOR = 29;
667804

668805
// ── The SHARED RECOGNISER section's own roster and floor (#15487) ──────────
669806
//
@@ -783,6 +920,22 @@ export function selfTest() {
783920
['/** Retired: err.code = ' + "'GHOST'" + ' must never come back. */', "err.code = 'REAL';"].join('\n')],
784921
['a genuine line comment is still removed',
785922
["// err.code = 'GHOST'", "err.code = 'REAL';"].join('\n')],
923+
// JSX children text is content (objectstack#25803). The first row is the
924+
// merged tree's DocsIndex.tsx shape: read as code, its `/*` opened a
925+
// comment that ran to the GHOST docblock's closer and took REAL with it.
926+
['a block-comment opener inside JSX text',
927+
['const el = (', ' <p>Ship docs as <code className="mx-1">src/docs/*.md</code> files.</p>', ');',
928+
"err.code = 'REAL';", "/* err.code = 'GHOST' */"].join('\n')],
929+
['a quote and a // inside JSX text, around a child expression and a nested element',
930+
["const el = <div>see https://x.y/a <b>b</b> {n} don't</div>; err.code = 'REAL';", "// err.code = 'GHOST'"].join('\n')],
931+
['a JSX comment in a child expression is still a comment',
932+
["const el = <div>{/* err.code = 'GHOST' */}</div>;", "err.code = 'REAL';"].join('\n')],
933+
['a self-closing tag after a spread opens no children',
934+
["const el = <Foo {...p} />; // err.code = 'GHOST'", "err.code = 'REAL';", '</Foo>'].join('\n')],
935+
['a generic arrow in JSX position is code, not an element',
936+
["const f = <T,>(x: T) => x; // err.code = 'GHOST'", "err.code = 'REAL';", '</T>'].join('\n')],
937+
['a type assertion with no closer is code, not an element',
938+
["const a = <Foo>bar; // err.code = 'GHOST'", "err.code = 'REAL';"].join('\n')],
786939
];
787940

788941
// Both projections are driven, on every shape: they share one scanner, so a

0 commit comments

Comments
 (0)