Read a name with a macro row as a macro where an item of a file starts

A translation unit and a namespace body hold declarations and no statement
([basic.link] p1, [namespace.def] p1, GCC cp_parser_toplevel_declaration,
Clang ParseExternalDeclaration), so `name(args);` there is no call. 00e3c7a
reads such an item as a macro when the name has no lowercase letter, and the
shape of the name is its evidence. `PyDoc_STRVAR(doc, "text");`,
`TF_CALL_half(REGISTER);` and `DEFINE_int32(flag, 1, "help");` hold a
lowercase letter, and that commit named them as the rows it does not reach.

THIS COMMIT TAKES THE ROW OF THE SEED AS THE EVIDENCE IN PLACE OF THE SHAPE.
A name with a function-macro row of the seed of its own project reads as a
macro where an item starts. The row must be the function-like one, because an
object-like macro takes no arguments ([cpp.replace] p10): `printf("good");`
in 7 files of the Clang interpreter tests has an object row, and a statement
at file scope is the reading of Clang itself there. The two lookups of
00e3c7a stay: a name that a class head of the file or a type row of the seed
declares keeps its reading, and a group with the shape of a parenthesized
declarator keeps its declaration.

THE POPULATION AT 7c0d954. A census of the class walks the tree, as the
census of the gate does: an expression_statement whose first named child is a
call_expression, whose nearest ancestor that is no preprocessor node is the
translation_unit or a declaration_list. It holds 13,895 clean sites with a
callee that has a lowercase letter. 7,714 of them have a field_expression as
the callee, the `BENCHMARK(BM_Run)->Arg(0)->Unit(...);` chains that 00e3c7a
named, and 5,977 have a plain name. Of those 5,977 sites, in 764 files with
585 names, the seed of the project holds a function-macro row for 2,646.

THE HEAD CENSUS HOLDS 0 OF THE 2,646, so each one is a macro invocation now.
The treediff gives 2,660 regions of a new macro_invocation in 364 clean
files: the 2,646 sites, 2 sites of hpx swapcontext32.ipp, which the census
counts as an include fragment, and 12 sites in 5 files that the base read as
a declaration with a parenthesized declarator, because the argument is a type
keyword. Those 12 are `__usdt_def_signed(short);` and two more lines of hhvm
usdt.h, `BOOST_dynamic_bitset_is_numeric(bool);` and two more lines of the
mongo copy of boost dynamic_bitset.hpp, `REGISTER_MatMulNBits(float);` of
onnxruntime matmul_nbits.cc, `TestInitializerDataFieldSpecialized(float);`
and one more line of onnxruntime initializer_test.cc, and
`REGISTER_GPU_int32(int32_t);` and one more line of tensorflow split_v_op.cc.
A declaration of that text declares nothing, because it has no name.

THE ROWS THIS COMMIT DOES NOT REACH, BY TASK. 1,699 sites hold a name that
only another project of the corpus declares as a macro: `DEFINE_bool` 433,
`DEFINE_int32` 378, `DECLARE_bool` 157 of gflags. A seed is a fact about one
project, and a name of a dependency reaches a file through its include graph,
so those sites wait for the per-file seed deltas of tree-sitter#372 and the dependency
map of item 6 of tree-sitter#360. 1,623 sites hold a name with no row anywhere,
`PyDoc_STRVAR` 1,129 of them, whose macro is in a Python header that the
corpus does not hold, and the same include work reaches them. 38 sites of
`common();` and 38 of `specific();` in the boost type-traits tests are calls
in a function body that `TT_TEST_BEGIN(BOOST_TT_TRAIT_NAME)` opens in its
expansion, where the tree of today is right, and no row declines them.

THE BRANCH OF A CALL BEFORE A DIRECTIVE KEEPS ITS READ NOW. A first form of
this rule repaired 1,652 of the 2,646 sites, and every site that it missed
stood in a conditional group: `TF_CALL_half(DECLARE_GPU_SPECS);` in a `#if`
group of tensorflow kept its call, and the same line outside the group became
a macro invocation. `scan_macro_start` reads a call of a name with a
lowercase letter before the directive that ends a branch of a structured
group. That branch read the group, found a `;` and not a directive, and
returned false, which gave up the whole scan before the item form ran. The
branch now keeps its read of the group and lets the scan go on.

MEASURED AGAINST 7c0d954. The rule reads a seed, so every plain step measures
zero: 0 tree changes over the corpus and over the test files, and 0 changes
of the treediff. The seeded steps change 364 clean files and 64 files with an
error, of which one loses its error. No file gets a new error or more error
bytes. The ties population keeps 3,110 rows and the dedupe population 2,818,
with none risen and none fallen. The census of the clean files gives no
detector that rose. The pins do not change, 624 corpus examples pass, and the
fuzz gives no failure over 5,037 inputs.

THE CO-VALIDITY TABLE LOSES ONE ROW WITH NO TOKEN CHANGE. The table holds one
row for each stop site of the scanner, keyed by the function and the line,
and a row whose condition names no token prints that fact. `scan_macro_start`
had three such rows: the branch of a call before a directive, the read of the
group, and `if (call)`. This commit makes the first two one `if` and one
`else if`, so that function has two. The key of the comparison holds no line
number, so the count falls from 13 to 12. No parse state and no token moves,
and the co-validity of every other row is the row of the base.

Author: GrigoryEvko

src/scanner.c

@@ -2479,6 +2479,15 @@ static bool is_seed_macro_name(const Scanner *scanner, const char *name, const R
     return (kinds & TS_CPP_SEED_MACRO) != 0;
 }
 
+/// True when the seed of the project declares the name as a function-like macro. It reads no character.
+static bool is_seed_function_macro_name(const Scanner *scanner, const char *name, const Reader *reader) {
+    if (scanner == NULL || reader->word_cut) {
+        return false;
+    }
+    uint16_t kinds = seed_kinds((const TSCppSeed *)scanner->context, name, (uint32_t)strlen(name));
+    return (kinds & TS_CPP_SEED_FUNCTION_MACRO) != 0;
+}
+
 /// True when a class head of the file or the seed of the project declares the name as a type.
 ///
 /// ONE LOOKUP WITH TWO SOURCES, AND EACH SOURCE READS NO CHARACTER. `is_class_name` compares the
@@ -3840,8 +3849,16 @@ typedef enum {
 /// declares the name as a type.
 static Invocation scan_macro_invocation(Reader *reader, const char *name, size_t length, bool same_line, bool call,
                                         const Arguments *args, const Gap *gap, const bool *valid_symbols,
-                                        const Scanner *classes, bool member_macro_name, bool declared_type) {
+                                        const Scanner *classes, bool member_macro_name, bool declared_type,
+                                        bool item_only) {
     TSLexer *lexer = reader->lexer;
+    // A name that is no macro by its shape reads as a macro only where an item starts and a `;` ends it.
+    // A decline gives INVOCATION_NONE, so that the paths after this scan keep their readings: a SAL
+    // annotation with arguments, `_Pre_satisfies_(n >= sizeof(T)) static DWORD CALLBACK f(...)` of dolphin,
+    // takes the line token there, and a stop gave that file a MISSING `;`.
+    if (item_only && !(call && !gap->directive && lexer->lookahead == ';')) {
+        return INVOCATION_NONE;
+    }
     // After a directive that ends a branch of a structured group, the lookahead is not a token of the line.
     int32_t c = gap->directive ? 0 : lexer->lookahead;
     // A storage class specifier can come before a macro invocation line, and that line has its own
@@ -3901,6 +3918,9 @@ static Invocation scan_macro_invocation(Reader *reader, const char *name, size_t
             lexer->result_symbol = MACRO_ITEM_START;
             return INVOCATION_TOKEN;
         }
+        if (item_only) {
+            return INVOCATION_NONE;
+        }
         // A statement accepts the call form. A member has no call form, and it takes the call as a
         // macro invocation when the arguments cannot be a parameter list.
         if (valid_symbols[MACRO_CALL_START]) {
@@ -4179,7 +4199,15 @@ static bool scan_macro_start(Reader reader, const char *name, const char *full,
     // `Foo(x);` can hold a parameter list, and they stay declarations.
     bool member_name = member_start && !class_name && !sal && !word_in(name, GRAMMAR_WORDS) &&
                        !is_grammar_keyword(name);
-    bool invocation = (macro_name || member_name) &&
+    // A NAME WITH A FUNCTION-MACRO ROW OF THE SEED IS A MACRO WHERE AN ITEM STARTS, WHATEVER ITS SHAPE.
+    // `PyDoc_STRVAR(doc, "text");` and `TF_CALL_half(REGISTER);` hold a lowercase letter, so the shape
+    // test of a macro name declines them, and the row of the seed is the evidence in their place. The row
+    // must be the function-like one, because an object-like macro takes no arguments ([cpp.replace] p10):
+    // `printf("good");` of the Clang interpreter tests has an object row, and the file is a REPL input
+    // where a statement at file scope is the reading of Clang itself. Refer to `_macro_item`.
+    bool item_name = valid_symbols[MACRO_ITEM_START] && !macro_name && !is_grammar_keyword(name) && !class_name &&
+                     !word_in(name, GRAMMAR_WORDS) && is_seed_function_macro_name(scanner, full, &reader);
+    bool invocation = (macro_name || member_name || item_name) &&
                       (valid_symbols[MACRO_LINE_START] || valid_symbols[MACRO_BLOCK_START] ||
                        valid_symbols[MACRO_CALL_START] || valid_symbols[MACRO_ENUMERATOR_START] ||
                        valid_symbols[MACRO_LINE_AFTER_SPECIFIERS]);
@@ -4230,23 +4258,27 @@ static bool scan_macro_start(Reader reader, const char *name, const char *full,
     // THE BRANCH READS THE GROUP, AND A DECLINE GIVES THE RESULT OF THE OLD PATH. The other paths for such a name
     // are the macros before a constructor, and `scan_constructor_after_macro` gives false at the `(`. So a decline
     // after the read gives no token, as that path did.
+    Arguments args = {0};
+    bool call = false;
     if (branch_end_call && same_line) {
-        Arguments arguments = {0};
-        if (!skip_group(&reader, &arguments)) {
+        if (!skip_group(&reader, &args)) {
             return false;
         }
-        Gap after = {0};
-        skip_gap(&reader, &after);
-        if (!after.directive || after.newlines == 0) {
+        call = true;
+        gap = (Gap){0};
+        skip_gap(&reader, &gap);
+        if (gap.directive && gap.newlines > 0) {
+            lexer->result_symbol = MACRO_LINE_START;
+            return true;
+        }
+        // THE READ OF THE GROUP STAYS, AND THE SCAN GOES ON. A `return false` here gave no token at all,
+        // and the item form after it never ran: `TF_CALL_half(DECLARE_GPU_SPECS);` in a `#if` group of
+        // tensorflow kept its call, while the same line outside the group became a macro invocation.
+        if (!item_name) {
             return false;
         }
-        lexer->result_symbol = MACRO_LINE_START;
-        return true;
-    }
-    Arguments args = {0};
-    bool call = false;
-    if ((macro_name || sal || member_name || template_parameter_macro) && !gap.directive && lexer->lookahead == '(' &&
-        (same_line || length >= MACRO_MIN_BARE_LENGTH)) {
+    } else if ((macro_name || sal || member_name || template_parameter_macro || item_name) && !gap.directive &&
+               lexer->lookahead == '(' && (same_line || length >= MACRO_MIN_BARE_LENGTH)) {
         if (!skip_group(&reader, &args)) {
             return false;
         }
@@ -4312,7 +4344,7 @@ static bool scan_macro_start(Reader reader, const char *name, const char *full,
     if (invocation && (macro_name || (call && !args.pointer_declarator))) {
         Invocation result =
             scan_macro_invocation(&reader, name, length, same_line, call, &args, &gap, valid_symbols, classes,
-                                  member_start && macro_name && length >= 2 && !class_name, declared_type);
+                                  member_start && macro_name && length >= 2 && !class_name, declared_type, item_name);
         if (result == INVOCATION_TOKEN || result == INVOCATION_STOP) {
             return result == INVOCATION_TOKEN;
         }

xtask/src/scanner.rs

@@ -459,6 +459,68 @@ mod seeded {
         );
     }
 
+    /// A FUNCTION-MACRO ROW READS A NAME WITH A LOWERCASE LETTER AS A MACRO WHERE AN ITEM STARTS.
+    ///
+    /// A translation unit and a namespace body hold declarations and no statement, so `name(args);` there is
+    /// no call. 00e3c7a reads the name with no lowercase letter as a macro, and the shape of the name is its
+    /// evidence. `PyDoc_STRVAR(doc, "text");` and `TF_CALL_half(REGISTER);` hold a lowercase letter, and the
+    /// row of the seed is the evidence in their place. Refer to `scan_macro_invocation` in src/scanner.c.
+    ///
+    /// EACH LINE OF THE SOURCE NAMES AN INPUT THAT ONE TEST OF THE RULE DECIDES:
+    /// - The four lines with `PyDoc_STRVAR` and `TF_CALL_half` take the reading, at file scope, in a
+    ///   namespace body, in a linkage body and in a conditional group. In the group, a first version of this
+    ///   rule gave no token, because the branch of a call before the directive that ends a branch read the
+    ///   group and returned false. That branch now keeps its read and lets this rule run.
+    /// - `common();` keeps its call, because no row of the seed holds `common`. The corpus holds 38 such
+    ///   sites in the boost type-traits tests, where `TT_TEST_BEGIN(...)` opens a function body in its
+    ///   expansion and the call stands in that body.
+    /// - `printf("good");` keeps its call, because the row is object-like and an object-like macro takes no
+    ///   arguments ([cpp.replace] p10). The corpus holds 7 such sites, in the Clang interpreter tests.
+    /// - `Widget(x);` keeps the reading of the base, because a class head of the file declares `Widget`.
+    ///   The two lookups of 00e3c7a give that guard, and the one C++ reading of that text is a declaration
+    ///   of `x`, which no commit reads yet.
+    /// - `TF_CALL_half(*p);` keeps its declaration, because the group has the shape of a parenthesized
+    ///   declarator.
+    /// - `TF_CALL_half(REGISTER);` in a function body keeps its call, because no item starts there.
+    ///
+    /// THE GATE RUNS NO SEED, so this rule measures zero in the plain steps of the gate, and this test and a
+    /// seeded run of `xtask trees --seeds` are the evidence.
+    #[test]
+    fn a_function_macro_row_reads_a_name_with_a_lowercase_letter_as_a_macro_where_an_item_starts() {
+        const SOURCE: &str = "PyDoc_STRVAR(doc, \"text\");\nnamespace ns { TF_CALL_half(REGISTER); }\n\
+                              extern \"C\" { TF_CALL_half(REGISTER); }\n#if GOOGLE_CUDA\n\
+                              TF_CALL_half(REGISTER);\n#endif\n";
+        // Each line here keeps the tree of a parse with no seed.
+        const KEPT: &str = "common();\nprintf(\"good\");\nclass Widget { int m; };\nWidget(x);\n\
+                            TF_CALL_half(*p);\nvoid f() { TF_CALL_half(REGISTER); }\n";
+        // The rows of a seed ascend by the bytes of the name.
+        let file = SeedFile::new(
+            "PyDoc_STRVAR\tfunction-macro\nTF_CALL_half\tfunction-macro\nWidget\tfunction-macro\nprintf\tobject-macro\n",
+        );
+        let seed = Seed::read(file.path()).expect("the seed reads");
+        let unseeded = bare(SOURCE).root_node().to_sexp();
+        assert_eq!(unseeded.matches("(macro_invocation").count(), 0, "{unseeded}");
+
+        let mut parser = parser();
+        // SAFETY: `seed` lives to the end of this test.
+        unsafe { parser.set_scanner_context(seed.as_context()) };
+        let seeded = parser.parse(SOURCE, None).expect("the parse ends");
+        let sexp = seeded.root_node().to_sexp();
+        assert!(!seeded.root_node().has_error(), "{sexp}");
+        // The four sites with a function row take the reading, and no call expression stays.
+        assert_eq!(sexp.matches("(macro_invocation").count(), 4, "{sexp}");
+        assert_eq!(sexp.matches("(call_expression").count(), 0, "{sexp}");
+        // Each line of KEPT gives the tree of a parse with no seed.
+        let kept = parser.parse(KEPT, None).expect("the parse ends").root_node().to_sexp();
+        assert_eq!(kept, bare(KEPT).root_node().to_sexp(), "a line of KEPT changed with the seed");
+        assert_eq!(kept.matches("(expression_statement (call_expression").count(), 4, "{kept}");
+        // `TF_CALL_half(*p);` keeps its declaration with a parenthesized declarator.
+        assert!(
+            kept.contains("(declaration type: (type_identifier) declarator: (parenthesized_declarator (pointer_declarator declarator: (identifier))))"),
+            "{kept}"
+        );
+    }
+
     /// A MACRO ROW REACHES THE TYPE BEFORE A DECLARATOR AND A MACRO WHEN NO SOURCE DECLARES THE TYPE.
     ///
     /// `Mutex mu MOZ_UNANNOTATED;` reads as the attribute macro `Mutex`, the type `mu` and the