Skip to content

Commit 4ad6cc8

Browse files
feat(advisor): default-off tiered decision advisor with v1-safe evidence (#54)
* feat(advisor): add default-off local decisions Bind advisor evidence without changing receipt v1. Store companion records in the receipt transaction. Block model adapters until the upstream API is verified. Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(advisor): preserve v1 receipt semantics Keep gateway params digests and sign separate advisor companions. Reject input failures instead of treating them as unavailable. Ignore payload task claims and verify exported record bindings. Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(advisor): refresh corrected harness results Use the gateway state shape without spoofable task claims. Record conservative approval: 55% precision and 100% recall. Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(advisor): isolate producer and v1 exports Reserve the internal producer ID so requests cannot poison history. Keep default exports readable by released v1 verifiers. Verify leading companions against signed manifest anchors. Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(receipts): explain opt-in advisor records Document the reserved identity and signed-anchor verification. Keep the record extension outside frozen v1 defaults. Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(verify): report unchecked advisor records Default v1 exports contain no advisor evidence lines. Keep chain verification usable without hiding unchecked evidence. Require advisor records when supplied or explicitly requested. Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent b131f16 commit 4ad6cc8

61 files changed

Lines changed: 4059 additions & 59 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,11 @@ jobs:
4343
exit 1
4444
fi
4545
46+
- name: decision advisor tests
47+
env:
48+
PYTHONDONTWRITEBYTECODE: "1"
49+
run: python3 -m unittest discover -s services/decision-advisor -v
50+
4651
- name: go mod tidy drift check
4752
run: |
4853
go mod tidy

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,3 +31,7 @@ dist/
3131

3232
# Local benchmark data, profiles, and Go caches
3333
/.cache/
34+
35+
# Python service bytecode
36+
__pycache__/
37+
*.pyc

‎README.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -365,6 +365,52 @@ Set `AGENTGATE_VAULT_CACHE_TTL=0` to disable plaintext caching. Refresh sharing
365365

366366
## Configuration
367367

368+
### Optional local decision advisor
369+
370+
The advisor is off by default. Leave `AGENTGATE_ADVISOR_URL` unset.
371+
It checks only registry actions marked `risky: true`.
372+
Deterministic authorization runs first. Advisor checks happen before vault access and upstream dispatch.
373+
An advisor can require approval or deny an allowed request.
374+
It cannot turn a deterministic denial into an allow.
375+
All tiers share a 150 ms end-to-end call deadline.
376+
377+
Receipt v1 remains unchanged.
378+
Escalations use `deny` plus `advisor_escalated`.
379+
Every action receipt keeps its original params digest.
380+
An immediately following `advisor` / `decision:advise` receipt binds the evidence record.
381+
It uses the reserved producer ID `agentgate:advisor`, never an authenticated API key.
382+
Both receipts and the record commit in the same transaction.
383+
See the [binding proposal](docs/spec/proposals/advisor-evidence.md) for verification and unavailable behavior.
384+
Admin-authenticated `GET /v1/receipts/{companion_seq}/evidence` returns the record.
385+
Default JSONL exports remain byte-compatible with frozen v1 and released v0.1.x verifiers.
386+
Add `&records=1` to the export URL to include separate evidence records.
387+
The offline verifier checks record binding and the link to the preceding action receipt.
388+
Without records, this branch's verifier checks the chain and reports unchecked advisor companions.
389+
Text output includes a note, including under `--quiet`.
390+
JSON output includes `unchecked_advisor_records` when nonzero.
391+
Use `--require-advisor-records` when advisor evidence must be present.
392+
Any supplied records also enable mandatory advisor evidence checks.
393+
Request `&records=1` or use the SQLite source for that evidence.
394+
Keep the extension away from older verifiers.
395+
Record-enabled ranges can start at a companion, using their signed manifest anchor.
396+
Receipt v1 cannot prove that every risky action has a companion.
397+
398+
Service `advisor` and actions beginning `decision:` are reserved.
399+
These requests return 400 without a receipt. Registry loading also rejects the reserved service.
400+
The reservations apply even when the advisor is off.
401+
Older history with these names and ordinary agent IDs still verifies.
402+
403+
Advisor params have a 256 KiB budget and use JSON without HTML expansion.
404+
Truncated or rejected input requires approval, regardless of `allow_with_flag`.
405+
The original params still go upstream unchanged after deterministic authorization and advisory approval.
406+
Only timeouts, connection failures, and service 5xx qualify for the unavailable allow fallback.
407+
408+
The [service](services/decision-advisor/README.md) ships a deterministic heuristic for tests.
409+
Model adapters remain blocked until their upstream inference API is verified.
410+
The [Advisor results](bench/RESULTS.md#advisor) use 60 synthetic cases, not real receipt logs.
411+
These numbers do not enable the advisor or establish model quality.
412+
State and task claims are untrusted. Prompt injection cannot grant authority.
413+
368414
### Environment Variables
369415

370416
| Variable | Description | Required |
@@ -374,6 +420,11 @@ Set `AGENTGATE_VAULT_CACHE_TTL=0` to disable plaintext caching. Refresh sharing
374420
| `AGENTGATE_ADMIN_SECRET` | Secret for admin API access | Yes |
375421
| `AGENTGATE_PUBLIC_URL` | Public base URL for OAuth callbacks and admin cookie security. Default `http://localhost:8080`. | No |
376422
| `AGENTGATE_API_KEY_CACHE_TTL` | Verified-key cache duration. Default `30s`; `0` disables caching. | No |
423+
| `AGENTGATE_ADVISOR_URL` | Unset disables the advisor. Accepts loopback or `.svc.cluster.local` service origins only. | No |
424+
| `AGENTGATE_ADVISOR_UNAVAILABLE` | `allow_with_flag` by default. Also accepts `require_approval` or `deny`. | No |
425+
| `AGENTGATE_ADVISOR_APPROVAL_THRESHOLD` | Escalation probability threshold. Default `0.6`. | No |
426+
| `AGENTGATE_ADVISOR_DENY_THRESHOLD` | Denial probability threshold. Default `0.9`. Must exceed the approval threshold. | No |
427+
| `AGENTGATE_ADVISOR_CONFIDENCE_THRESHOLD` | Require approval when final confidence is lower. Default `0.8`. | No |
377428
| `AGENTGATE_RATE_KEY_RPS` | Requests per second per API key ID. Default `20`. Zero requires tenant admission disabled. | No |
378429
| `AGENTGATE_RATE_KEY_BURST` | Per-key token bucket burst. Default `40`. Zero requires tenant admission disabled. | No |
379430
| `AGENTGATE_RATE_TENANT_RPS` | Requests per second per tenant. Default `100`; `0` disables this bucket. | No |
@@ -470,6 +521,11 @@ OAuth callbacks validate encrypted state without an admin session, so they don't
470521

471522
Service configurations are YAML files in `configs/services/`. See `configs/services/google_workspace.yaml` for an example. The gateway itself loads the merged `configs/services.yaml`.
472523

524+
Each action accepts `risky: true` or `risky: false`.
525+
Omission defaults to false. Non-boolean values stop registry loading.
526+
Mark writes, deletes, payments, outbound email, and bulk exports as risky.
527+
The marker has no enforcement effect while the advisor is disabled.
528+
473529
### Upstream connections and retries
474530

475531
Each service gets a separate HTTP client, connection pool, and circuit breaker.

‎bench/RESULTS.md‎

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -371,6 +371,8 @@ Until then, no moderate-load production p99 pass is claimed.
371371

372372
# Historical Phase 0: in-process act overhead
373373

374+
For Phase 3 measurements, see [Advisor](#advisor).
375+
374376
**Verdict: FAIL overall. The `<1 ms p99` exit gate is not met across the tested load levels.**
375377

376378
Only the original concurrency-1 run with a 0 ms upstream passes.
@@ -735,3 +737,187 @@ Build, both vet commands, race tests, and tagged helper tests were rerun after t
735737
The existing closed-loop 64-request smoke matrix was rerun and passed.
736738
The full closed-loop table and CPU profile remain the original measurements, not new reruns.
737739
No `.cache/` files belong in the commit.
740+
741+
## Advisor
742+
743+
**Status: OFF by default. Model evaluation is pending.**
744+
745+
Measured on 2026-10-18, on Apple M4 with 10 CPUs and 16 GiB.
746+
Python was 3.14.6. Go was 1.26.0 on darwin/arm64.
747+
Gateway benchmarks used `GOMAXPROCS=4`.
748+
The advisor service used Python's standard library with no runtime dependencies.
749+
750+
### Dataset and method
751+
752+
The 60 labeled cases are synthetic. No real receipt logs exist yet.
753+
There are 33 escalation labels and 27 non-escalation labels.
754+
Labels were authored for this implementation, not sampled from held-out production traffic.
755+
Every state now uses the gateway's exact `service`, `action`, and `params` shape.
756+
Payload task claims were removed. The heuristic ignores them instead of increasing confidence.
757+
758+
Each configuration runs 60 warmup requests, then 1,200 measured loopback HTTP requests.
759+
Classification counts use each label once during warmup.
760+
Percentiles use nearest-rank selection.
761+
HTTP latency includes connection setup, HTTP framing, service work, response transfer, and client reads.
762+
Payload serialization and response JSON decoding are outside this interval.
763+
Gateway authorization, upstream calls, and receipt commits are also excluded.
764+
Tier latency measures backend execution only.
765+
766+
Reproduction:
767+
768+
```sh
769+
PYTHONDONTWRITEBYTECODE=1 python3 bench/advisor/evaluate.py --repeats 20
770+
```
771+
772+
### Escalation results
773+
774+
| Configuration | Precision | Recall | TP | FP | TN | FN |
775+
| --- | --- | --- | --- | --- | --- | --- |
776+
| Heuristic A | 55.00% | 100% | 33 | 27 | 0 | 0 |
777+
| Heuristic A+B test cascade | 55.00% | 100% | 33 | 27 | 0 | 0 |
778+
| Test B, all 60 cases | 55.00% | 100% | 33 | 27 | 0 | 0 |
779+
780+
Test B repeats the same heuristic on all 60 low-confidence cases.
781+
There is no separate task signal in the gateway state.
782+
The heuristic reports unknown consistency and confidence below the default 0.8 threshold.
783+
Consequently, every request requires approval.
784+
All 27 non-escalation labels become false positives.
785+
The 100% recall comes from conservative approval, not semantic understanding.
786+
These results supersede the initial 90.91% figures.
787+
This is not a Lux, Vega, or GPU result.
788+
789+
Do not use these numbers as a production safety claim.
790+
791+
### Latency
792+
793+
| Configuration | Measured calls | HTTP p50 | HTTP p99 |
794+
| --- | --- | --- | --- |
795+
| Heuristic A | 1,200 | 0.186375 ms | 0.390833 ms |
796+
| Heuristic A+B test cascade | 1,200 | 0.172042 ms | 0.379917 ms |
797+
798+
| Backend position | Measured calls | Backend p50 | Backend p99 |
799+
| --- | --- | --- | --- |
800+
| A alone | 1,200 | 0.004 ms | 0.006 ms |
801+
| A within cascade | 1,200 | 0.004 ms | 0.005 ms |
802+
| B within cascade | 1,200 | 0.002 ms | 0.003 ms |
803+
804+
The B count is 60 cases repeated 20 times.
805+
One measured run per configuration is reported.
806+
There are no repeated-run confidence intervals.
807+
These are heuristic timings, not neural-model latency claims.
808+
809+
### Disabled gateway before and after, initial implementation
810+
811+
These gateway measurements belong to the initial implementation, `bbb46ae`.
812+
They were not rerun for this review.
813+
The advisor harness above was rerun after the review fixes.
814+
815+
The existing disk-backed gateway benchmark was run at baseline `494e81be984c53d77c6b8181dbad7304a695f8a3`.
816+
The changed gateway used an unset advisor URL.
817+
Both runs used the same benchmark, concurrency 1, and a 0 ms upstream.
818+
Each run measured 5,000 requests after the benchmark's warmup.
819+
Runs alternated baseline, changed, baseline, changed, baseline, changed.
820+
821+
```sh
822+
mkdir -p .cache/tmp .cache/go-tmp .cache/go-build .cache/baseline
823+
git archive 494e81b | tar -x -C .cache/baseline
824+
export TMPDIR="$PWD/.cache/tmp" GOTMPDIR="$PWD/.cache/go-tmp"
825+
export GOCACHE="$PWD/.cache/go-build" GOMAXPROCS=4
826+
go test -C .cache/baseline -tags=bench ./bench -run '^$' \
827+
-bench '^BenchmarkHandleActE2E/delay=0s/c=1$' -benchtime=5000x -count=1
828+
go test -tags=bench ./bench -run '^$' \
829+
-bench '^BenchmarkHandleActE2E/delay=0s/c=1$' -benchtime=5000x -count=1
830+
```
831+
832+
| Run | Baseline p50 overhead | Changed p50 overhead | Baseline p99 overhead | Changed p99 overhead |
833+
| --- | --- | --- | --- | --- |
834+
| 1 | 423.8 us | 356.1 us | 6,208 us | 13,832 us |
835+
| 2 | 413.3 us | 452.6 us | 9,861 us | 11,146 us |
836+
| 3 | 449.8 us | 461.8 us | 15,466 us | 14,601 us |
837+
| Median | 423.8 us | 452.6 us | 9,861 us | 13,832 us |
838+
839+
Median throughput was 479.9 requests/s before and 460.4 requests/s after.
840+
The shared host and SQLite filesystem showed large tail variation.
841+
Median p99 increased in this sample. These runs do not establish unchanged hot-path performance.
842+
They also do not isolate the advisor branch's cost.
843+
No latency exit gate is claimed.
844+
Disabled-path tests separately check unchanged receipt preimages and zero advisor calls.
845+
846+
### Pending model runs
847+
848+
Neither inspected Kai nor Laya revision listed an ONNX artifact.
849+
Local `onnxruntime` was absent.
850+
Kai's listed weights total 1,507,642,792 bytes, above the 1.5 GB limit.
851+
Laya's root safetensors file fits the limit but has no verified ONNX runtime contract.
852+
No weights were downloaded.
853+
854+
The published vllm-sr 0.4.0 wheel did not establish a Decision 2.0 inference API.
855+
The `onnx` and `vllm-sr` adapters raise `NotImplementedError`, rather than invent calls.
856+
Kai versus Laya selection, Lux evaluation, and GPU cascade wiring remain pending.
857+
See [RUNTIME.md](../services/decision-advisor/RUNTIME.md) for artifact hashes and pinned model revisions.
858+
The advisor remains off regardless of these synthetic numbers.
859+
860+
### Evidence and validation
861+
862+
Each action receipt retains `params_sha256 = DigestParams(params)`.
863+
An adjacent `advisor` / `decision:advise` companion binds the separate evidence record through `HashRecord`.
864+
Its agent identity is the reserved `agentgate:advisor`.
865+
Matching names with ordinary agent IDs remain ordinary receipts.
866+
API requests using reserved names return 400 without writing receipts.
867+
Both receipts and the record commit in one ledger transaction.
868+
The companion's signed chain link and record identify the preceding action.
869+
Default JSONL exports remain frozen v1, without record lines.
870+
Add `&records=1` to opt into evidence records beside their companions.
871+
Both JSONL and SQLite verification check the record binding and action link.
872+
Without supplied records, verification instead checks the chain and reports the unchecked companion count.
873+
Use `--require-advisor-records` to reject missing evidence.
874+
Any supplied record lines also enable mandatory evidence verification.
875+
Leading companions use the signed manifest anchor when records are included.
876+
Escalations use existing `deny` and valid `advisor_escalated` error codes.
877+
Rejected or truncated input requires approval and records `advisor_input_rejected`.
878+
It cannot use the default unavailable allow fallback.
879+
The frozen receipt rules and version-lock test are unchanged.
880+
Bundle parsing and verification gained additive record support.
881+
V1 still cannot prove that every risky action has a companion.
882+
See the [binding proposal](../docs/spec/proposals/advisor-evidence.md) for unavailable-evidence limits.
883+
884+
| Command or check | Result |
885+
| --- | --- |
886+
| `go build ./...` | PASS |
887+
| `go vet ./...` | PASS |
888+
| `CGO_ENABLED=1 go test -race ./...` | PASS |
889+
| `go test ./pkg/receiptspec -run '^TestV1VersionLock$' -count=1` | PASS, test unchanged |
890+
| `semgrep --config auto --error --quiet ./internal ./cmd ./pkg ./services` | PASS |
891+
| `PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s services/decision-advisor -q` | PASS, 12 tests |
892+
| `PYTHONDONTWRITEBYTECODE=1 python3 bench/advisor/evaluate.py --repeats 20` | PASS, 60 labels and 2,400 measured requests |
893+
| `git diff 6e8422f --exit-code -- pkg/receiptspec/vectors_test.go` | PASS |
894+
| Default export comparison against main `494e81b`, on the same ledger | PASS, byte-identical across 4 ranges |
895+
| Main `494e81b` verifier against the new default export | PASS |
896+
| Updated verifier against an opt-in export beginning with a companion | PASS |
897+
| Updated verifier against the default export | PASS, with unchecked-companion note |
898+
| Default export with `--require-advisor-records` | FAIL as expected, missing evidence |
899+
900+
Regression tests failed before the fixes.
901+
They reproduced changed gateway digests, size bypass, 4xx allow fallback, and task-confidence spoofing.
902+
Companion tests cover atomic rollback, export round trips, and altered or missing records.
903+
Re-review tests first reproduced spoofed identities, poisoned historical names, leading-range rejection, and incompatible default exports.
904+
They pass after reserving the producer identity and making records opt-in.
905+
Final regression tests reproduced the updated verifier rejecting default exports.
906+
Chain-only mode now passes and reports unchecked companions in text and JSON.
907+
Strict mode and record-bearing bundles still enforce evidence presence.
908+
The v1 specification gained only an informative extension cross-reference in section 7.
909+
Resealed record hashes cannot bypass action sequence, entry hash, or params digest checks.
910+
The existing deployment fragments were not changed or rerendered in this review.
911+
Docker image construction was not run. No container runtime result is claimed.
912+
913+
Semgrep used its public auto profile without an application token.
914+
Its auto profile requires automatic registry metrics; disabling metrics prevents that profile from loading.
915+
No repository source was uploaded.
916+
917+
SHA-256 identifiers:
918+
919+
| File | SHA-256 |
920+
| --- | --- |
921+
| `bench/advisor/cases.jsonl` | `af1165889d7faf34563f7f3bd7a79afb2b05c3804928b0176279153e221adc0c` |
922+
| `services/decision-advisor/backends.py` | `b947297a57caaa23ce2ce60f270dd2cf0e8fdd1a51207aba78852a60cb65fb22` |
923+
| Unchanged `pkg/receiptspec/vectors_test.go` | `fbfcc8411ec707244c4fdd9e05922f5b31c12e9d92ff70e681198efcdfc846d5` |

‎bench/advisor/README.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Advisor evaluation
2+
3+
All 60 cases are synthetic. No real receipt logs exist yet.
4+
Each state uses the gateway's exact `service`, `action`, and `params` shape.
5+
Cases include 33 escalation labels and 27 non-escalation labels.
6+
They are not held-out model evaluation data.
7+
Some actions exist only in the synthetic fixtures.
8+
9+
Run from the repository root:
10+
11+
```sh
12+
PYTHONDONTWRITEBYTECODE=1 python3 bench/advisor/evaluate.py --repeats 20
13+
```
14+
15+
The harness starts a real loopback HTTP service.
16+
It runs 60 warmup requests per configuration.
17+
Warmup computes classification counts once per case.
18+
Then it measures 1,200 requests per configuration.
19+
HTTP latency includes connection setup, HTTP framing, service work, response transfer, and client reads.
20+
Payload serialization and response JSON decoding are outside the interval.
21+
It excludes gateway authorization, upstream calls, and receipt commits.
22+
Tier latency measures backend work only.
23+
Percentiles use nearest-rank selection.
24+
25+
The first configuration uses one heuristic backend at A.
26+
The second repeats that heuristic at A and B to test cascade wiring.
27+
There is no task signal in the gateway state.
28+
Payload `stated_task` fields cannot increase heuristic confidence.
29+
The heuristic's low confidence makes every request require approval under default thresholds.
30+
Consequently, all 60 cases reach B in the test cascade.
31+
This is not a Lux, Vega, or GPU benchmark.
32+
33+
Escalation uses risk, personal-data probability, and task inconsistency.
34+
The approval threshold is 0.6. The denial threshold is 0.9.
35+
Final confidence below 0.8 also requires approval.
36+
The conservative heuristic cannot distinguish the 27 acceptable requests without task context.
37+
38+
See [RESULTS.md](../RESULTS.md#advisor) for measurements and pending model runs.
39+
The advisor remains disabled regardless of these test numbers.

0 commit comments

Comments
 (0)