The reference system under test (B3) for ReleaseGateBench.
It maps one frozen release-and-deployment-context instance to a structured
promote or hold verdict with a single LLM call: render the instance into a
fixed prompt, call the model once, parse and sanitize the response.
Core is small and stateless. Dataset construction, ablation redaction, batch execution, and scoring live in ReleaseGateBench; live staging and actuation live in ReleaseGateStack.
python -m venv .venv && source .venv/bin/activate
python -m pip install -e './ReleaseGateCore[litellm]'Run from the repository root. LiteLLM supplies the provider adapter; set the API key required by the model identifier you choose.
from releasegate_core import ReleaseGateCore
gate = ReleaseGateCore(model="openai/gpt-5.5", seed=0)
output = gate.decide(instance)
print(output.decision, output.confidence)instance follows
ReleaseGateBench/schemas/instance.schema.json;
output follows schemas/agent_output.schema.json.
For network-free tests, inject any client implementing the LLMClient protocol:
gate = ReleaseGateCore(model="test-model", client=my_fake_client)instance → render locked system + instance messages → one model completion
→ tolerant JSON extraction + strict normalization
├── valid → AgentOutput
└── invalid → one bounded repair completion
├── valid → AgentOutput
└── invalid → uncertain conservative hold
Provider retry/backoff belongs to the client. JSON repair is bounded by
max_repair_retries; repeated malformed output returns hold at confidence 0.5.
This fail-closed result keeps a parser failure from becoming an automatic
promotion while preserving a complete inference run. Bench rejects records marked
as transport errors from published scoring.
Research reference implementation, not a production safety certification. Every reference run in ReleaseGateBench v1.0.0 false-promoted at least 87 of the 128 test instances whose gold decision is
hold, before selective thresholding.
With decision_threshold unset, Core uses the model's normalized decision. When
set, valid finite decision_probs are required and Core promotes exactly when
P(promote) >= decision_threshold; confidence is recomputed for the selected side
so decision and confidence cannot disagree. The Bench scorer may fit a threshold
on dev and apply it to test. Because the rule is inclusive, holding a prediction
with P(promote) == 1 requires a threshold strictly greater than 1.
The v1 system prompt in
_render_and_parse.py is a locked
research artifact and must not be edited in place — archived runs depend on its
exact semantics. It has a known security_contextual inversion, and v1
atomic-label decomposition is not valid. A corrected prompt needs a new version
and new evaluations.
python -m pip install -e './ReleaseGateCore[dev]'
python -m pytest ReleaseGateCore/testsThe default suite is offline and uses scripted model responses.
Stack vendors the dependency-free render/parse module for its Windmill worker. The vendored copy and the Core package must remain byte-identical; run the sync helper and drift test whenever that module changes:
python ReleaseGateStack/stack/windmill/lib/sync_decide.py
pytest ReleaseGateStack/tests/test_decide_sync.pyStack owns orchestration and infrastructure; Core owns only rendering, parsing, and the single decision call.
Apache-2.0; see the repository-level LICENSE.