security: close three input-level evasions of the gate and tripwire

Attacked the inputs rather than the patterns this time. Three worked.

ONE INVALID BYTE HID A WHOLE FILE. The scanner abandoned any file that
failed to decode as UTF-8, so appending a single 0xFF made it skip every
readable line in that file, plaintext payload included. A complete
evasion costing one byte. Files are now decoded with replacement rather
than abandoned, and a NUL byte -- git's own binary heuristic -- is what
marks a file as genuinely not a review surface. Only three tracked files
reach that path today: a PNG, a Windows ETW manifest and the nomic blob.

Better still, the evasion is now its own signal: a file with a TEXT
extension that is not valid UTF-8 is reported, because a stray byte in a
.md or .c is anomalous regardless of what surrounds it.

THE TRIPWIRE MATCHED CASE-SENSITIVELY. `Scripts/evil.sh` and
`.GitHub/workflows/` walked past it, and on a case-insensitive checkout
those are the same files as the guarded ones. Now matched with POSIX
character classes rather than `${v,,}` (bash 4 only) or `tr` (external),
so the logic can be tested on any shell -- which matters for a gate
nobody can run locally the way CI runs it.

THE FILES ENDPOINT CAPS AT 3000. A pull request padded past that limit
would hide a CI change in the tail, and the gate would report green over
a change set it never saw. It now compares what the API returned against
the count the PR itself declares and REFUSES when they disagree, rather
than passing on partial data.

All three are pinned in the selftest, alongside the requirement that a
PNG with invalid UTF-8 stays silent -- the hardening must not turn every
binary into a finding.

Verified end to end against real pull requests: #1422 refused (4 guarded
paths), #1245 refused (42 guarded, 392/392 received so no truncation),
#1778 refused (2 guarded), #1678 passes.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>

Author: DeusData

.github/workflows/ci-integrity.yml

@@ -51,20 +51,48 @@ jobs:
           # Paths that define or are executed by CI. Under-covering is the
           # dangerous direction: anything CI runs can neuter a gate, so this
           # list is deliberately broad rather than precise.
+          # Matching is CASE-INSENSITIVE. `Scripts/evil.sh` and `.GitHub/`
+          # slipped past a case-sensitive test, and on a case-insensitive
+          # checkout they are the same files as the guarded ones anyway.
           is_guarded() {
+              # Character classes rather than a lowercase expansion: `${v,,}`
+              # needs bash 4, and `tr` is an external command. This form is
+              # POSIX, depends on nothing, and can be tested on any shell --
+              # which matters more than elegance for a gate nobody can run
+              # locally the way CI runs it.
               case "$1" in
-                  .github/*)             return 0 ;;
-                  scripts/*)             return 0 ;;
-                  test-infrastructure/*) return 0 ;;
-                  tests/*.sh)            return 0 ;;
-                  *)                     return 1 ;;
+                  .[Gg][Ii][Tt][Hh][Uu][Bb]/*)                     return 0 ;;
+                  [Ss][Cc][Rr][Ii][Pp][Tt][Ss]/*)                  return 0 ;;
+                  [Tt][Ee][Ss][Tt]-[Ii][Nn][Ff][Rr][Aa][Ss][Tt][Rr][Uu][Cc][Tt][Uu][Rr][Ee]/*)
+                                                                   return 0 ;;
+                  [Tt][Ee][Ss][Tt][Ss]/*.[Ss][Hh])                 return 0 ;;
+                  *)                                               return 1 ;;
               esac
           }
 
           changed="$(gh api --paginate \
               "repos/$REPO/pulls/$PR_NUMBER/files" \
               --jq '.[].filename')"
 
+          # FAIL CLOSED ON TRUNCATION. The files endpoint is capped at 3000
+          # entries, so a pull request padded past that limit could hide a CI
+          # change in the tail and this gate would never see it. Compare what
+          # we received against the count the PR itself reports, and refuse
+          # rather than pass when they disagree.
+          declared="$(gh api "repos/$REPO/pulls/$PR_NUMBER" --jq '.changed_files')"
+          received="$(printf '%s\n' "$changed" | grep -c . || true)"
+          if [ "$received" -lt "$declared" ]; then
+              echo "=== CI INTEGRITY GATE: REFUSED ==="
+              echo
+              echo "The pull request reports $declared changed files but the API"
+              echo "returned only $received. GitHub caps that endpoint at 3000"
+              echo "entries, so part of this change set is invisible to the gate"
+              echo "and a CI modification could be hiding in the tail."
+              echo
+              echo "Split the pull request into reviewable pieces."
+              exit 1
+          fi
+
           hits=""
           while IFS= read -r path; do
               [ -n "$path" ] || continue

scripts/security-injection.py

@@ -372,6 +372,41 @@ def obfuscation_findings(rel, text):
         )
 
 
+# Extensions whose contents a human or an agent reads as text. A file with one
+# of these that is NOT valid UTF-8 is itself the finding: a single stray byte
+# used to make this gate skip the entire file, payload included, which was a
+# complete one-byte evasion.
+TEXT_EXTENSIONS = frozenset({
+    ".c", ".h", ".cc", ".cpp", ".hpp", ".py", ".sh", ".bash", ".ps1", ".js",
+    ".ts", ".tsx", ".jsx", ".json", ".jsonc", ".yml", ".yaml", ".toml", ".ini",
+    ".cfg", ".md", ".txt", ".rst", ".html", ".css", ".sql", ".nix", ".mk",
+    ".cmake", ".gradle", ".rb", ".go", ".rs", ".java", ".kt", ".swift", ".php",
+    ".pl", ".lua", ".vim", ".el", ".patch", ".diff", ".man", ".xml", ".svg",
+})
+
+
+def decode_for_scan(rel, raw):
+    """Return (text, finding) for a file's bytes.
+
+    A NUL byte means genuinely binary -- git's own heuristic -- and those are
+    not a review surface, so they are skipped. Everything else is decoded with
+    replacement rather than abandoned: refusing to decode a file was itself the
+    evasion, because one invalid byte hid every readable line around it.
+    """
+    if b"\x00" in raw:
+        return None, None
+    try:
+        return raw.decode("utf-8"), None
+    except UnicodeDecodeError as exc:
+        text = raw.decode("utf-8", errors="replace")
+        if Path(rel).suffix.lower() in TEXT_EXTENSIONS:
+            return text, (
+                f"text file is not valid UTF-8 at byte {exc.start} -- a stray "
+                f"byte makes readers skip content they would otherwise show"
+            )
+        return text, None
+
+
 def scan_tree(root):
     """Walk the tree ONCE, running every tier per file.
 
@@ -386,23 +421,17 @@ def scan_tree(root):
             raw = (root / rel).read_bytes()
         except (OSError, ValueError):
             continue
-        text = None
+        text, decode_finding = decode_for_scan(rel, raw)
+        if text is None:
+            continue  # NUL bytes: genuinely binary, not a review surface
+        if decode_finding:
+            yield "phrase", rel, 1, "", decode_finding
         if not raw.isascii():
-            try:
-                text = raw.decode("utf-8")
-            except UnicodeDecodeError:
-                continue  # binary; not a review surface
             if any(classify(ord(ch)) for ch in text):
                 for line_no, line in enumerate(text.split("\n"), 1):
                     found = [(c, classify(ord(c))) for c in line if classify(ord(c))]
                     if found:
                         yield "carrier", rel, line_no, line, found
-        if text is None:
-            try:
-                text = raw.decode("utf-8")
-            except UnicodeDecodeError:
-                continue
-
         if rel.startswith("internal/cbm/vendored/grammars/") and rel.endswith("parser.c"):
             for m in _LITERAL.finditer(text):
                 lit = m.group(1)
@@ -603,6 +632,20 @@ def caught(text):
     ]:
         check(not caught(text), f"false positive on {what}")
 
+    # DECODE HARDENING. A single invalid byte used to make the gate skip an
+    # entire file, hiding every readable line around it -- a one-byte evasion.
+    text, finding = decode_for_scan("notes.md", b"hello \xff world")
+    check(text is not None, "invalid UTF-8 in a text file must still be scanned")
+    check(finding is not None, "invalid UTF-8 in a text file must be reported")
+    check(
+        "world" in (text or ""),
+        "content after an invalid byte must remain visible to the scanner",
+    )
+    _, png = decode_for_scan("image.png", b"\x89PNG\r\n\xff\xfe")
+    check(png is None, "invalid UTF-8 in a binary extension must not be reported")
+    binary, _ = decode_for_scan("blob.bin", b"a\x00b")
+    check(binary is None, "NUL bytes must be treated as binary and skipped")
+
     # A blessed line is pinned by content: changing it must invalidate the entry.
     line = 'x = "a\u200bb";'
     other = 'x = "a\u200bc";'