Skip to content

Commit 4518c9e

Browse files
committed
test(contract): gate the published language count against the vendored grammars
The count was hand-maintained on seven separate surfaces and had drifted apart from reality AND from itself: README, docs/index.html, docs/llms.txt and the npm README said 158, while server.json, the chocolatey nuspec and package-release.sh said 159. Derived from the vendored grammar directories rather than CBM_LANG_COUNT, because the registry contains entries with no parser at all: CBM_LANG_NIM (grammar removed 2026-06-12) and CBM_LANG_OBJECTSCRIPT_EXPORT have no grammar directory, no tree_sitter function and no MANIFEST row, yet files still route to them by extension. Publishing CBM_LANG_COUNT would claim languages we do not parse. A vendored grammar directory demonstrably exists and is checksum-gated. Accepted trade-off, stated in the test: this UNDERCOUNTS languages that share one grammar (TypeScript/TSX, objectscript_udl/_routine). We publish the number we can prove rather than the larger number we cannot. The contract binds four things so it cannot go quietly vacuous: - the grammar directory count is sanity-bounded, so a layout change fails loudly instead of making every check below it trivially true - MANIFEST.md's declared figure must already agree with the directory count - every registered surface must state exactly that number, and a registered surface that states NO number fails (the wording changed, or it should be deregistered) - any UNREGISTERED file stating a language count fails, so an eighth surface cannot reintroduce the drift this exists to prevent Files that legitimately say "<N> languages" about something else — benchmark corpus subsets, probe scopes, the version-pinned winget manifests — are listed with what their number actually counts, so no exemption is unexplained. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent 095acc5 commit 4518c9e

2 files changed

Lines changed: 126 additions & 0 deletions

File tree

‎scripts/test.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -266,6 +266,9 @@ bash "$ROOT/tests/test_shell_line_endings.sh"
266266
echo "=== Step 0v: nomic blob generator contract ==="
267267
bash "$ROOT/tests/test_nomic_blob_generator_contract.sh"
268268

269+
echo "=== Step 0w: published language-count contract ==="
270+
bash "$ROOT/tests/test_language_count_contract.sh"
271+
269272
# Verify compiler supports target arch
270273
verify_compiler "$CC"
271274

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
#!/usr/bin/env bash
2+
# The published language count IS the number of vendored tree-sitter grammars,
3+
# and every public surface must agree with it.
4+
#
5+
# Why this exists: the count was hand-maintained on seven separate surfaces, so
6+
# it drifted apart from reality AND from itself — README/docs/npm said 158 while
7+
# server.json/nuspec/package-release.sh said 159.
8+
#
9+
# Why grammars and not CBM_LANG_COUNT: the registry contains entries with no
10+
# parser at all. CBM_LANG_NIM (grammar removed 2026-06-12) and
11+
# CBM_LANG_OBJECTSCRIPT_EXPORT have no grammar directory, no tree_sitter
12+
# function and no MANIFEST row, yet files still route to them by extension.
13+
# Counting them would publish languages we do not parse. A vendored grammar
14+
# directory, by contrast, demonstrably exists and is checksum-gated.
15+
#
16+
# Known and accepted: this UNDERCOUNTS languages that share one grammar
17+
# (TypeScript/TSX, objectscript_udl/_routine). We publish the number we can
18+
# prove rather than the larger number we cannot.
19+
20+
set -euo pipefail
21+
22+
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
23+
cd "$ROOT"
24+
25+
GRAMMAR_DIR="internal/cbm/vendored/grammars"
26+
27+
# --- 1. The truth: count the vendored grammar directories. ---
28+
expected=$(find "$GRAMMAR_DIR" -mindepth 1 -maxdepth 1 -type d | wc -l | tr -d ' ')
29+
30+
# A parse that silently returns nothing would make every check below vacuous.
31+
if ((expected < 100 || expected > 500)); then
32+
echo "FAIL: counted $expected grammar directories under $GRAMMAR_DIR —" \
33+
"the layout changed and this contract can no longer read it" >&2
34+
exit 1
35+
fi
36+
37+
# --- 2. MANIFEST.md must already agree; it is the human-facing ledger. ---
38+
manifest_count=$(
39+
grep -oE '^- Grammars: \*\*[0-9]+\*\*' "$GRAMMAR_DIR/MANIFEST.md" |
40+
grep -oE '[0-9]+' || true
41+
)
42+
if [[ "$manifest_count" != "$expected" ]]; then
43+
echo "FAIL: $GRAMMAR_DIR/MANIFEST.md declares ${manifest_count:-no} grammars," \
44+
"but $expected directories exist" >&2
45+
exit 1
46+
fi
47+
48+
# --- 3. Every published surface must claim exactly that number. ---
49+
SURFACES=(
50+
README.md
51+
docs/index.html
52+
docs/llms.txt
53+
pkg/npm/README.md
54+
pkg/chocolatey/codebase-memory-mcp.nuspec
55+
scripts/package-release.sh
56+
server.json
57+
)
58+
59+
# Files that legitimately say "<N> languages" about something OTHER than the
60+
# product's registry size. Each entry states what its number actually counts,
61+
# so an unexplained exemption cannot hide here.
62+
EXEMPT=(
63+
'docs/EVALUATION_PLAN.md' # historical plan, pinned to its authoring date
64+
'docs/BENCHMARK.md' # benchmark corpus subsets (63/17 scored)
65+
'scripts/clone-bench-repos.sh' # bench repo tiers (44 + 22)
66+
'tests/repro/repro_invariant_breadth.c' # breadth of one invariant (26/27)
67+
'tests/repro/repro_grammar_scripting.c' # scripting-grammar probe scope (12)
68+
'tests/test_grammar_probe_b.c' # grammar probe scope (12)
69+
'tests/test_language_count_contract.sh' # this file, which names the numbers above
70+
'pkg/winget/manifests/' # version-pinned published manifests: 0.8.1
71+
# really did ship 155, and rewriting a
72+
# released manifest would falsify it
73+
)
74+
75+
failures=0
76+
claims=0
77+
78+
for path in "${SURFACES[@]}"; do
79+
if [[ ! -f "$path" ]]; then
80+
echo "FAIL: registered surface $path does not exist — update SURFACES" >&2
81+
failures=$((failures + 1))
82+
continue
83+
fi
84+
hits=$(grep -oE '[0-9]{2,3} languages' "$path" | grep -oE '^[0-9]+' | sort -u || true)
85+
if [[ -z "$hits" ]]; then
86+
echo "FAIL: $path is registered as a language-count surface but states no" \
87+
"count — either it lost the claim (drop it from SURFACES) or the" \
88+
"wording changed and this check went vacuous" >&2
89+
failures=$((failures + 1))
90+
continue
91+
fi
92+
while IFS= read -r n; do
93+
claims=$((claims + 1))
94+
if [[ "$n" != "$expected" ]]; then
95+
echo "FAIL: $path claims $n languages, $expected grammars are vendored" >&2
96+
failures=$((failures + 1))
97+
fi
98+
done <<<"$hits"
99+
done
100+
101+
# --- 4. No UNREGISTERED file may make the claim. ---
102+
# Without this, adding an eighth surface reintroduces exactly the drift this
103+
# contract exists to prevent, and every check above would still pass.
104+
while IFS= read -r path; do
105+
[[ -z "$path" ]] && continue
106+
for known in "${SURFACES[@]}"; do
107+
[[ "$path" == "$known" ]] && continue 2
108+
done
109+
for skip in "${EXEMPT[@]}"; do
110+
[[ "$path" == "$skip"* ]] && continue 2
111+
done
112+
echo "FAIL: $path states a language count but is not registered. Add it to" \
113+
"SURFACES so it stays in step with the registry, or to EXEMPT with a" \
114+
"note saying what its number actually counts." >&2
115+
failures=$((failures + 1))
116+
done < <(git grep -IlE '[0-9]{2,3} languages' || true)
117+
118+
if ((failures > 0)); then
119+
echo "FAIL: $failures language-count contract violation(s)" >&2
120+
exit 1
121+
fi
122+
123+
echo "Language-count contract passed ($claims claims across ${#SURFACES[@]} surfaces, registry = $expected)"

0 commit comments

Comments
 (0)