Skip to content

Commit c8320ce

Browse files
committed
Merge sync/upstream-step2 into step 4: test fixes, legacy-block warning, output budget
Brings the Jenkins-green step-2 line (release-tooling DeusData#60, 904ae6b) into the fully merged step-4 tree: - 12 step-2 tests aligned with the CLI-only fork (cc9f002) - feat/legacy-mcp-block-warning: install and uninstall warn about leftover codebase-memory-mcp instruction blocks and leave them in place - feat/output-token-budget: max_output_tokens for search_code, trace_path and detect_changes, plus match_limit, source_max_lines and directory paging - lint fixes and memory-core baseline notes (904ae6b) Conflicts: - tests/test_daemon_application.c, over-budget passthrough. Kept step 2's binary-safe app_test_response_contains instead of strstr on a binary frame. Searches the whole frame and accepts either error-flag spelling: is_error from the operation result wire on this tree, isError from a passed-through body. - scripts/memory-core-baseline.txt regenerated for the merged tree. These are counts the fork already had, not new raw allocations. Validated locally: - `scripts/build-dev.sh` and the test-runner compile are clean with -Werror. - `scripts/lint.sh --ci` passes. - Vendored checksums verify.
2 parents c195cfa + 904ae6b commit c8320ce

15 files changed

Lines changed: 1187 additions & 164 deletions

‎Makefile.cbm‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -407,7 +407,7 @@ STORE_SRCS = src/store/store.c
407407
# Cypher module (new)
408408
CYPHER_SRCS = src/cypher/cypher.c
409409

410-
OPERATION_SRCS = src/operations/registry.c src/operations/json_args.c src/operations/reliability_events.c src/operations/execute.c src/operations/result_wire.c src/operations/tool_catalog.c src/operations/index_admission.c src/operations/store_host.c src/operations/session_state.c src/operations/mutation.c src/operations/adr.c src/operations/cross_repo.c src/operations/index.c src/operations/project_arg.c src/operations/index_supervisor.c src/operations/command_runner.c src/operations/read.c src/operations/coverage.c src/operations/snippet.c src/operations/search.c src/operations/trace.c src/operations/trace_ingest.c src/operations/schema.c src/operations/query.c src/operations/architecture.c src/operations/changes.c src/operations/source_search.c src/operations/file_outline.c src/operations/compare.c src/operations/compact_out.c
410+
OPERATION_SRCS = src/operations/registry.c src/operations/json_args.c src/operations/reliability_events.c src/operations/execute.c src/operations/result_wire.c src/operations/output_budget.c src/operations/tool_catalog.c src/operations/index_admission.c src/operations/store_host.c src/operations/session_state.c src/operations/mutation.c src/operations/adr.c src/operations/cross_repo.c src/operations/index.c src/operations/project_arg.c src/operations/index_supervisor.c src/operations/command_runner.c src/operations/read.c src/operations/coverage.c src/operations/snippet.c src/operations/search.c src/operations/trace.c src/operations/trace_ingest.c src/operations/schema.c src/operations/query.c src/operations/architecture.c src/operations/changes.c src/operations/source_search.c src/operations/file_outline.c src/operations/compare.c src/operations/compact_out.c
411411

412412
# Shared daemon coordination and local transport
413413
DAEMON_SRCS = \

‎docs/CONFIGURATION.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -219,7 +219,7 @@ not name is permitted.
219219

220220
## 5. Agent and Editor Integration Files
221221

222-
The CLI-first `install` path can write Codebase Memory skills and durable CLI instructions for detected agents. Hooks are a separate explicit opt-in via `codebase-memory-cli install-hooks`. Neither lifecycle surface creates, migrates, adopts, or removes MCP registrations or MCP-owned assets; existing MCP state is treated as foreign for side-by-side safety.
222+
The CLI-first `install` path can write Codebase Memory skills and durable CLI instructions for detected agents. Hooks are a separate explicit opt-in via `codebase-memory-cli install-hooks`. Neither lifecycle surface creates, migrates, adopts, or removes MCP registrations or MCP-owned assets; existing MCP state is treated as foreign for side-by-side safety. Both `install` and `uninstall` warn when an instruction file still contains a legacy `codebase-memory-mcp` managed block from the upstream MCP tool, but they leave that block in place for you to remove by hand.
223223

224224
Those target paths vary by tool and platform, so the easiest way to inspect the exact files for your machine is:
225225

‎scripts/memory-core-baseline.txt‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ internal/cbm/service_patterns.c 2
1616
internal/cbm/sqlite_writer.c 74
1717
src/cli/activation_transaction.c 80
1818
src/cli/agent_clients.c 73
19-
src/cli/cli.c 133
19+
src/cli/cli.c 134
2020
src/cli/config_edit_path.c 2
2121
src/cli/config_json_like.c 142
2222
src/cli/config_text_edit.c 72
@@ -52,7 +52,7 @@ src/git/git_context.c 18
5252
src/main.c 35
5353
src/operations/adr.c 50
5454
src/operations/architecture.c 88
55-
src/operations/changes.c 54
55+
src/operations/changes.c 56
5656
src/operations/command_runner.c 1
5757
src/operations/compact_out.c 39
5858
src/operations/compare.c 18
@@ -72,10 +72,10 @@ src/operations/result_wire.c 1
7272
src/operations/schema.c 10
7373
src/operations/search.c 13
7474
src/operations/session_state.c 12
75-
src/operations/snippet.c 24
76-
src/operations/source_search.c 87
75+
src/operations/snippet.c 26
76+
src/operations/source_search.c 96
7777
src/operations/store_host.c 11
78-
src/operations/trace.c 10
78+
src/operations/trace.c 12
7979
src/pipeline/artifact.c 38
8080
src/pipeline/fqn.c 23
8181
src/pipeline/lsp_resolve.h 4

‎src/cli/cli.c‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3283,6 +3283,40 @@ int cbm_remove_instructions(const char *path) {
32833283
: CLI_ERR;
32843284
}
32853285

3286+
/* Upstream codebase-memory-mcp wrote its own managed instruction blocks. The
3287+
* fork's CP78 side-by-side policy treats that MCP state as foreign, so install
3288+
* and uninstall never rewrite or remove it — but the block still points at MCP
3289+
* tools this CLI-only fork does not provide, so warn once per file. Read-only:
3290+
* this never modifies the file. */
3291+
#define LEGACY_MCP_MARKER_START "<!-- codebase-memory-mcp:start -->"
3292+
#define LEGACY_MCP_MARKER_END "<!-- codebase-memory-mcp:end -->"
3293+
3294+
static bool cbm_file_has_legacy_mcp_block(const char *path) {
3295+
if (!path || !path[0]) {
3296+
return false;
3297+
}
3298+
char *content = read_file_str(path, NULL);
3299+
if (!content) {
3300+
return false;
3301+
}
3302+
const char *begin = strstr(content, LEGACY_MCP_MARKER_START);
3303+
bool complete = begin != NULL &&
3304+
strstr(begin + strlen(LEGACY_MCP_MARKER_START), LEGACY_MCP_MARKER_END) != NULL;
3305+
free(content);
3306+
return complete;
3307+
}
3308+
3309+
static void cbm_warn_legacy_mcp_block(const char *path) {
3310+
if (!cbm_file_has_legacy_mcp_block(path)) {
3311+
return;
3312+
}
3313+
(void)fprintf(stderr,
3314+
" warning: %s still contains a legacy codebase-memory-mcp instruction block; "
3315+
"it refers to MCP tools that codebase-memory-cli does not provide. Remove it "
3316+
"manually if it is no longer wanted.\n",
3317+
path);
3318+
}
3319+
32863320
/* ── Codex MCP config (TOML) ─────────────────────────────────── */
32873321

32883322
#define CODEX_CMM_TABLE "mcp_servers.codebase-memory-mcp"
@@ -7922,13 +7956,17 @@ static bool install_generic_agent_config(const char *label, const char *instr_pa
79227956
if (g_install_plan) {
79237957
if (instr_path) {
79247958
plan_record(label, "instructions", instr_path);
7959+
if (cbm_file_has_legacy_mcp_block(instr_path)) {
7960+
plan_record(label, "warning", instr_path);
7961+
}
79257962
}
79267963
return true;
79277964
}
79287965
printf("%s:\n", label);
79297966
printf(" integration: CLI instructions only; hooks require `codebase-memory-cli "
79307967
"install-hooks`\n");
79317968
if (instr_path) {
7969+
cbm_warn_legacy_mcp_block(instr_path);
79327970
if (!dry_run) {
79337971
if (!prepare_config_parent(instr_path) ||
79347972
cbm_upsert_instructions(instr_path, agent_instructions_content) != CLI_OK) {
@@ -9624,6 +9662,7 @@ static char *cbm_build_install_plan_json_options(const char *home, const char *b
96249662
yyjson_mut_val *agent_files = yyjson_mut_arr(doc);
96259663
yyjson_mut_val *prompt_files = yyjson_mut_arr(doc);
96269664
yyjson_mut_val *hooks = yyjson_mut_arr(doc);
9665+
yyjson_mut_val *warnings = yyjson_mut_arr(doc);
96279666
for (int i = 0; i < plan.count; i++) {
96289667
cbm_plan_entry_t *e = &plan.items[i];
96299668
if (strcmp(e->kind, "config") == 0 || strcmp(e->kind, "mcp_config") == 0) {
@@ -9642,6 +9681,8 @@ static char *cbm_build_install_plan_json_options(const char *home, const char *b
96429681
} else if (strcmp(e->kind, "prompt") == 0) {
96439682
yyjson_mut_arr_add_strcpy(doc, prompt_files, e->path);
96449683
yyjson_mut_arr_add_strcpy(doc, instrs, e->path);
9684+
} else if (strcmp(e->kind, "warning") == 0) {
9685+
yyjson_mut_arr_add_strcpy(doc, warnings, e->path);
96459686
} else {
96469687
yyjson_mut_arr_add_strcpy(doc, instrs, e->path);
96479688
}
@@ -9652,6 +9693,7 @@ static char *cbm_build_install_plan_json_options(const char *home, const char *b
96529693
yyjson_mut_obj_add_val(doc, root, "agent_files_planned", agent_files);
96539694
yyjson_mut_obj_add_val(doc, root, "prompt_files_planned", prompt_files);
96549695
yyjson_mut_obj_add_val(doc, root, "hooks_planned", hooks);
9696+
yyjson_mut_obj_add_val(doc, root, "warnings", warnings);
96559697
yyjson_mut_obj_add_bool(doc, root, "writes_started", false);
96569698
yyjson_mut_obj_add_bool(doc, root, "network_after_install", false);
96579699
yyjson_mut_obj_add_str(doc, root, "next_safe_command",
@@ -10364,6 +10406,7 @@ static void uninstall_agent_mcp_instr(mcp_uninstall_args_t paths, bool dry_run,
1036410406
const char *instr_path = paths.instr_path;
1036510407
printf("%s: preserved MCP config by CP78 side-by-side policy\n", name);
1036610408
if (instr_path) {
10409+
cbm_warn_legacy_mcp_block(instr_path);
1036710410
if (!dry_run && cbm_remove_instructions(instr_path) != CLI_OK) {
1036810411
record_agent_config_error(true, name, "instructions_uninstall", instr_path);
1036910412
}

‎src/operations/changes.c‎

Lines changed: 94 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
#include "operations/output_budget.h"
12
#include "operations/result_wire.h"
23
#include "operations/changes.h"
34

@@ -757,6 +758,49 @@ static bool changes_read_oid(FILE *fp, char out[65], bool *oom) {
757758
return valid;
758759
}
759760

761+
/* The mandatory detect_changes metadata (base/direction/scalars) is not rows:
762+
* if it alone cannot fit, answer with a small truthful record rather than
763+
* slicing a path or an identifier. */
764+
static char *detect_budget_floor(bool legacy_json, bool engine_saturated, size_t budget_bytes) {
765+
if (!legacy_json) {
766+
cbm_sb_t sb;
767+
cbm_sb_init(&sb);
768+
cbm_tree_scalar_str(&sb, "truncation_reason", "output_budget");
769+
cbm_tree_scalar_bool(&sb, "truncated", true);
770+
cbm_tree_scalar_bool(&sb, "output_budget_floor_exceeded", true);
771+
if (engine_saturated) {
772+
cbm_tree_scalar_bool(&sb, "engine_saturated", true);
773+
}
774+
cbm_tree_scalar_int(&sb, "max_output_bytes", (long long)budget_bytes);
775+
cbm_tree_scalar_str(&sb, "hint",
776+
"mandatory detect_changes metadata exceeds the budget; raise "
777+
"max_output_tokens (no path or identifier was sliced)");
778+
return cbm_sb_finish(&sb);
779+
}
780+
yyjson_mut_doc *doc = yyjson_mut_doc_new(NULL);
781+
yyjson_mut_val *floor = doc ? yyjson_mut_obj(doc) : NULL;
782+
if (!doc || !floor) {
783+
if (doc) {
784+
yyjson_mut_doc_free(doc);
785+
}
786+
return NULL;
787+
}
788+
yyjson_mut_doc_set_root(doc, floor);
789+
yyjson_mut_obj_add_str(doc, floor, "truncation_reason", "output_budget");
790+
yyjson_mut_obj_add_bool(doc, floor, "truncated", true);
791+
yyjson_mut_obj_add_bool(doc, floor, "output_budget_floor_exceeded", true);
792+
if (engine_saturated) {
793+
yyjson_mut_obj_add_bool(doc, floor, "engine_saturated", true);
794+
}
795+
yyjson_mut_obj_add_uint(doc, floor, "max_output_bytes", budget_bytes);
796+
yyjson_mut_obj_add_str(doc, floor, "hint",
797+
"mandatory detect_changes metadata exceeds the budget; raise "
798+
"max_output_tokens (no path or identifier was sliced)");
799+
char *json = changes_doc_to_str(doc);
800+
yyjson_mut_doc_free(doc);
801+
return json;
802+
}
803+
760804
cbm_operation_result_t cbm_changes_operation_execute(const char *args,
761805
const cbm_operation_runtime_t *runtime) {
762806
char *project = changes_project_arg(args);
@@ -1081,6 +1125,9 @@ cbm_operation_result_t cbm_changes_operation_execute(const char *args,
10811125
if (changed_offset < 0) {
10821126
changed_offset = 0;
10831127
}
1128+
/* Absent max_output_tokens = no byte budget: detect_changes keeps emitting
1129+
* exactly the response it always has. */
1130+
size_t output_budget_bytes = cbm_output_budget_bytes(cbm_output_budget_tokens(args, 0));
10841131
int module_limit = changes_int_arg(args, "module_limit", CHANGES_DEFAULT_PAGE_LIMIT);
10851132
int module_offset = changes_int_arg(args, "module_offset", 0);
10861133
if (module_limit < 0) {
@@ -1257,11 +1304,15 @@ cbm_operation_result_t cbm_changes_operation_execute(const char *args,
12571304
if (imp_returned > imp_limit) {
12581305
imp_returned = imp_limit;
12591306
}
1307+
bool output_budget_hit = false;
1308+
bool output_budget_floor_exceeded = false;
1309+
1310+
render_detect_output:;
12601311
bool changed_has_more = changed_start + changed_returned < file_count;
12611312
bool impacted_has_more = want_symbols && imp_start + imp_returned < impact.visited_count;
12621313
bool module_has_more = want_symbols && module_start + module_returned < module_total;
1263-
bool response_truncated =
1264-
engine_saturated || changed_has_more || impacted_has_more || module_has_more;
1314+
bool response_truncated = engine_saturated || output_budget_hit || changed_has_more ||
1315+
impacted_has_more || module_has_more;
12651316

12661317
if (!legacy_json) {
12671318
cbm_sb_t sb;
@@ -1270,6 +1321,13 @@ cbm_operation_result_t cbm_changes_operation_execute(const char *args,
12701321
cbm_tree_scalar_bool(&sb, "working_tree_included", true);
12711322
cbm_tree_scalar_str(&sb, "merge_base", merge_base);
12721323
cbm_tree_scalar_str(&sb, "direction", direction);
1324+
if (output_budget_hit) {
1325+
cbm_tree_scalar_str(&sb, "truncation_reason", "output_budget");
1326+
cbm_tree_scalar_int(&sb, "max_output_bytes", (long long)output_budget_bytes);
1327+
}
1328+
if (output_budget_floor_exceeded) {
1329+
cbm_tree_scalar_bool(&sb, "output_budget_floor_exceeded", true);
1330+
}
12731331
if (engine_saturated) {
12741332
cbm_tree_scalar_bool(&sb, "engine_saturated", true);
12751333
}
@@ -1384,6 +1442,13 @@ cbm_operation_result_t cbm_changes_operation_execute(const char *args,
13841442
yyjson_mut_obj_add_bool(doc, root_obj, "working_tree_included", true);
13851443
yyjson_mut_obj_add_strcpy(doc, root_obj, "merge_base", merge_base);
13861444
yyjson_mut_obj_add_strcpy(doc, root_obj, "direction", direction);
1445+
if (output_budget_hit) {
1446+
yyjson_mut_obj_add_str(doc, root_obj, "truncation_reason", "output_budget");
1447+
yyjson_mut_obj_add_uint(doc, root_obj, "max_output_bytes", output_budget_bytes);
1448+
}
1449+
if (output_budget_floor_exceeded) {
1450+
yyjson_mut_obj_add_bool(doc, root_obj, "output_budget_floor_exceeded", true);
1451+
}
13871452
if (engine_saturated) {
13881453
yyjson_mut_obj_add_bool(doc, root_obj, "engine_saturated", true);
13891454
}
@@ -1479,6 +1544,33 @@ cbm_operation_result_t cbm_changes_operation_execute(const char *args,
14791544
out_str = changes_doc_to_str(doc);
14801545
yyjson_mut_doc_free(doc);
14811546
}
1547+
/* Exact serialized-size check with semantic reductions only. Low-value
1548+
* rollups and file names yield as whole sections before graph rows, and
1549+
* every continuation above remains exactly valid for the rows emitted.
1550+
* Nothing is ever byte-sliced. */
1551+
if (output_budget_bytes > 0 && out_str && strlen(out_str) > output_budget_bytes) {
1552+
output_budget_hit = true;
1553+
if (module_returned > 0) {
1554+
module_returned = 0;
1555+
} else if (changed_returned > 0) {
1556+
changed_returned = 0;
1557+
} else if (imp_returned > 0) {
1558+
/* Prefix-directory factoring can make N rows smaller than N-1, so
1559+
* probe every smaller whole-row prefix. */
1560+
imp_returned--;
1561+
} else if (!output_budget_floor_exceeded) {
1562+
output_budget_floor_exceeded = true;
1563+
} else {
1564+
free(out_str);
1565+
out_str = detect_budget_floor(legacy_json, engine_saturated, output_budget_bytes);
1566+
goto detect_output_done;
1567+
}
1568+
free(out_str);
1569+
out_str = NULL;
1570+
goto render_detect_output;
1571+
}
1572+
1573+
detect_output_done:
14821574
result = out_str ? cbm_operation_result_take(out_str, false)
14831575
: changes_error("out of memory while rendering detect_changes");
14841576
out_str = NULL;

‎src/operations/output_budget.c‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
#include "operations/output_budget.h"
2+
3+
#include "yyjson/yyjson.h"
4+
5+
#include <stdbool.h>
6+
#include <string.h>
7+
8+
int cbm_output_budget_tokens(const char *args_json, int fallback) {
9+
const char *json = args_json ? args_json : "{}";
10+
yyjson_doc *doc = yyjson_read(json, strlen(json), 0);
11+
yyjson_val *root = doc ? yyjson_doc_get_root(doc) : NULL;
12+
yyjson_val *value = yyjson_is_obj(root) ? yyjson_obj_get(root, "max_output_tokens") : NULL;
13+
bool present = value && yyjson_is_int(value);
14+
int tokens = present ? (int)yyjson_get_sint(value) : fallback;
15+
if (doc) {
16+
yyjson_doc_free(doc);
17+
}
18+
if (!present) {
19+
/* Absent/unusable argument: keep the caller's default so omitting it
20+
* never engages a budget. */
21+
return tokens;
22+
}
23+
if (tokens < CBM_OUTPUT_TOKENS_MIN) {
24+
tokens = CBM_OUTPUT_TOKENS_MIN;
25+
} else if (tokens > CBM_OUTPUT_TOKENS_MAX) {
26+
tokens = CBM_OUTPUT_TOKENS_MAX;
27+
}
28+
return tokens;
29+
}
30+
31+
size_t cbm_output_budget_bytes(int max_output_tokens) {
32+
if (max_output_tokens <= 0) {
33+
return 0;
34+
}
35+
if (max_output_tokens < CBM_OUTPUT_TOKENS_MIN) {
36+
max_output_tokens = CBM_OUTPUT_TOKENS_MIN;
37+
} else if (max_output_tokens > CBM_OUTPUT_TOKENS_MAX) {
38+
max_output_tokens = CBM_OUTPUT_TOKENS_MAX;
39+
}
40+
return (size_t)max_output_tokens * (size_t)CBM_OUTPUT_BYTES_PER_TOKEN;
41+
}

‎src/operations/output_budget.h‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
#ifndef CBM_OPERATIONS_OUTPUT_BUDGET_H
2+
#define CBM_OPERATIONS_OUTPUT_BUDGET_H
3+
4+
#include <stddef.h>
5+
6+
/* Deterministic serialized-output ceiling shared by the budgeted operations.
7+
*
8+
* `max_output_tokens` is model-neutral sizing guidance, not a tokenizer
9+
* promise: the actual cross-platform contract is a UTF-8 byte ceiling applied
10+
* only at whole semantic-unit boundaries (a row, a source line, a scalar).
11+
* No identifier is ever byte-sliced; when even the smallest meaningful unit
12+
* does not fit, the operation answers with a budget-floor record telling the
13+
* caller which continuation to use. */
14+
#define CBM_OUTPUT_BYTES_PER_TOKEN 4
15+
#define CBM_OUTPUT_TOKENS_MIN 128
16+
#define CBM_OUTPUT_TOKENS_MAX 1000000
17+
18+
/* Parse `max_output_tokens` from an operation args object. Returns `fallback`
19+
* when the key is absent or is not an integer; otherwise the requested value
20+
* clamped to [CBM_OUTPUT_TOKENS_MIN, CBM_OUTPUT_TOKENS_MAX]. A fallback <= 0
21+
* means "no budget requested". */
22+
int cbm_output_budget_tokens(const char *args_json, int fallback);
23+
24+
/* Byte ceiling for a token count. 0 means unbounded. */
25+
size_t cbm_output_budget_bytes(int max_output_tokens);
26+
27+
#endif

0 commit comments

Comments
 (0)