Fix macOS/BSD grep compatibility in security-suite hooks - #4
Open
nikolasdehor wants to merge 1 commit into
Open
Conversation
Several security-suite hooks silently stopped enforcing their checks on macOS because they rely on GNU-only shell constructs. BSD grep (the default on macOS) has no -P (PCRE) flag, and bash 3.2 (the macOS default shell) has no associative arrays. Neither failure mode is loud: the scripts either print an error to stderr while still exiting 0, or bail out early, so the checks pass without ever actually running. - grep -P is not available in BSD grep and fails with "invalid option -- P" on every invocation (prompt-injection-detector.sh, and 7 call sites in unicode-injection-scanner.sh, plus claudemd-scanner.sh). Unicode codepoint ranges are ported to `perl -C -0777 -ne`, available on both macOS and Linux, preserving the same codepoint-range syntax. The overlong-UTF-8 byte check keeps operating on raw bytes (perl without -C, since -C would decode input as UTF-8 first and change what a raw byte value matches). - The null-byte check (grep -qP against a null byte) can never work on any system, GNU or BSD: bash silently drops a null byte from a variable on assignment through command substitution, so the variable never contains the byte by the time it's checked. Replaced with a stream-size comparison (wc -c with and without stripping null bytes via tr) that never materializes the value into a bash variable. - declare -A (associative arrays) requires bash 4+; bash 3.2, the default on macOS, doesn't have it and fails immediately on the first hook invocation (output-secrets-scanner.sh, pre-commit-secrets.sh). Converted to two parallel indexed arrays (names and regexes, same index) instead of a single map. - Most critical bug, and not macOS-specific: several regex patterns start with "-----" (e.g. the Private Key pattern). Without -e, grep parses that as an unrecognized option instead of a search pattern and errors out, so private key detection never actually ran, on GNU or BSD grep. Fixed by passing patterns through -e in all four call sites that interpolate a variable into grep, including two files where no current pattern happens to start with a dash (dangerous-actions-blocker.sh, repo-integrity-scanner.sh) but the same failure mode would apply the moment one does. All fixes were tested against the real macOS default shell (bash 3.2.57) and BSD grep, with positive and negative cases for each check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Several
security-suitehooks silently stop enforcing their checks on macOS because they rely on GNU-only shell constructs that BSD grep and bash 3.2 (both macOS defaults) don't support. None of these failures are loud: the affected scripts either print an error to stderr while still exiting 0, or bail out early, so the security check simply never runs and the hook reports "all clear" regardless of the actual input.Verified against the real macOS default environment:
bash --versionreports 3.2.57, andgrep --versionreports BSD grep (no GNU extensions). All fixes below were tested there, both positive (pattern present, detected) and negative (pattern absent, not flagged) cases.Bug 1:
grep -Pis not available on BSD grepBreaks on: macOS (any hook invocation), any other BSD-grep system.
grep -Penables PCRE syntax, which is a GNU grep extension. BSD grep has no-Pflag at all and fails immediately withgrep: invalid option -- P, printing a usage message to stderr on every single hook call. This affects:prompt-injection-detector.sh(null-byte check)unicode-injection-scanner.sh(7 call sites: zero-width chars, bidi override, null byte, tag characters, overlong UTF-8, 2x homoglyph check)claudemd-scanner.sh(non-ASCII-near-keyword check)Fix: Unicode codepoint-range patterns (e.g.
[\x{200B}-\x{200D}\x{FEFF}]) are ported toperl -C -0777 -ne 'exit(/PATTERN/ ? 0 : 1)'. Perl ships by default on both macOS and Linux and accepts the exact same\x{...}codepoint-range syntax, so the regex bodies are unchanged. The overlong-UTF-8 byte check (which matches raw byte values, not decoded codepoints) uses perl without-C, since-Cwould decode input as UTF-8 first and change what a byte-range like\xC0-\xC1matches.Bug 2: the null-byte check can never work, on any system
Breaks on: every system, GNU or BSD. This is not macOS-specific.
grep -qP '\x00'(or any grep flavor's equivalent) tests a bash variable ($CONTENT) for a null byte. But bash cannot retain a real\0inside a variable at all: it's silently dropped when the variable is assigned through command substitution (VAR=$(...)). By the time$CONTENTis checked, the byte is already gone, so this check has likely never actually fired, regardless of grep flavor or OS.Fix: replaced the variable-based check with a stream-size comparison that never materializes the value into a bash variable: pipe the same jq filter through
wc -c, then again throughLC_ALL=C tr -d '\000' | wc -c, and compare the two counts. A mismatch means a null byte was present in the stream.Bug 3:
declare -Arequires bash 4+, and macOS ships bash 3.2Breaks on: macOS (default
/bin/bashis 3.2.57, from 2007; associative arrays landed in bash 4.0, 2009).output-secrets-scanner.shandpre-commit-secrets.shusedeclare -Ato map secret names to regex patterns. On bash 3.2 this fails at thedeclare -Aline itself (declare: -A: invalid option/ similar), taking down the whole hook before any secret scanning happens.Fix: converted each associative array into two parallel indexed arrays (
SECRET_NAMES/SECRET_REGEXES, andPATTERN_NAMES/PATTERN_REGEXES), iterated by shared numeric index. Indexed arrays have existed since very early bash and work identically on 3.2 and 5.x.Bug 4 (most critical, also not macOS-specific): patterns starting with
-are parsed as grep optionsBreaks on: every system, GNU or BSD, whenever a pattern happens to start with
-.This one was hiding behind Bug 3: once
declare -Ais fixed and the scripts actually run to completion, several regex patterns fail again for a different reason. The "Private Key" pattern is-----BEGIN (RSA |EC |OPENSSH )?PRIVATE KEY-----. When that string is interpolated unquoted intogrep -qiE "$pattern", grep parses the leading-----BEGIN...as an unrecognized command-line flag instead of a search pattern, and errors out instead of matching:The practical impact: private key detection in
output-secrets-scanner.shandpre-commit-secrets.shdid not work, on any platform, for as long as this pattern has existed. A staged file containing an actual RSA/EC/OpenSSH private key would pass the pre-commit hook silently.Fix: added
-ebefore the interpolated pattern in every grep call that takes a pattern from a variable (grep -qiE -e "$pattern"/grep -noE -e "$pattern").-etells grep unambiguously "this is a pattern, not an option," regardless of what it starts with. Applied this to all four call sites in the codebase that build a grep pattern from a loop variable, including two (dangerous-actions-blocker.sh,repo-integrity-scanner.sh) where no current pattern starts with-, as preventive hardening so the next added pattern doesn't reintroduce the same silent failure.How to reproduce (before this fix)
On macOS, with a plugin checkout:
How to verify (after this fix)
All example values above (
AKIA..., the RSA key block, etc.) are placeholders, not real credentials.Test plan
bash -nsyntax check on all 7 modified files/bin/bash(3.2.57) and BSD grep