Skip to content

Commit aa15e6a

Browse files
UniverseScriptsUniverseScripts
committed
feat: cli-v0
0 parents  commit aa15e6a

43 files changed

Lines changed: 3680 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.editorconfig‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
root = true
2+
3+
[*]
4+
charset = utf-8
5+
end_of_line = lf
6+
insert_final_newline = true
7+
indent_style = space
8+
indent_size = 4
9+
trim_trailing_whitespace = true
10+
11+
[*.{yml,yaml,json,j2,html}]
12+
indent_size = 2

‎.env.example‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
ANTHROPIC_API_KEY=PLACEHOLDER

‎.github/dependabot.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
version: 2
2+
updates:
3+
- package-ecosystem: uv
4+
directory: /
5+
schedule:
6+
interval: weekly
7+
cooldown:
8+
default-days: 7
9+
- package-ecosystem: github-actions
10+
directory: /
11+
schedule:
12+
interval: weekly
13+
cooldown:
14+
default-days: 7

‎.github/workflows/ci.yml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
8+
permissions:
9+
contents: read
10+
11+
concurrency:
12+
group: ci-${{ github.ref }}
13+
cancel-in-progress: true
14+
15+
jobs:
16+
check:
17+
runs-on: ${{ matrix.os }}
18+
timeout-minutes: 15
19+
strategy:
20+
fail-fast: false
21+
matrix:
22+
os: [ubuntu-latest, windows-latest]
23+
steps:
24+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
25+
with:
26+
persist-credentials: false
27+
- uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
28+
with:
29+
version: "0.11.21"
30+
- run: uv sync --locked
31+
- run: uv run ruff check .
32+
- run: uv run ruff format --check .
33+
- run: uv run mypy src
34+
- run: uv run pytest
35+
- name: Audit dependencies
36+
shell: bash
37+
run: |
38+
uv export --format requirements-txt --no-emit-project --quiet > "$RUNNER_TEMP/requirements.txt"
39+
uvx pip-audit@2.10.1 --require-hashes --disable-pip -r "$RUNNER_TEMP/requirements.txt"

‎.gitignore‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Secrets
2+
.env
3+
.env.*
4+
!.env.example
5+
6+
# shiftlock workspace (suites and reports contain your prompts)
7+
.shiftlock/
8+
shiftlock.yaml
9+
10+
# Python
11+
.venv/
12+
__pycache__/
13+
*.py[cod]
14+
dist/
15+
build/
16+
*.egg-info/
17+
.pytest_cache/
18+
.mypy_cache/
19+
.ruff_cache/
20+
.coverage

‎.pre-commit-config.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Works with `prek` or `pre-commit`. Hook revs are pinned to commit SHAs.
2+
repos:
3+
- repo: https://github.com/astral-sh/ruff-pre-commit
4+
rev: f12be1ebaa5351c1fc76472de98db2c3446c8253 # v0.16.10
5+
hooks:
6+
- id: ruff-check
7+
- id: ruff-format
8+
- repo: https://github.com/gitleaks/gitleaks
9+
rev: 83d9cd684c87d95d656c1458ef04895a7f1cbd8e # v8.30.1
10+
hooks:
11+
- id: gitleaks

‎.python-version‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
3.12

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Yoshio
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# shiftlock
2+
3+
Find out whether your own prompts, tool calls and JSON outputs still work before you switch LLM
4+
models: a forced deprecation, a version bump or a provider change.
5+
6+
shiftlock replays your real requests on the current model (baseline) and the replacement
7+
(candidate), runs deterministic checks, and returns one of four verdicts: **go**, **conditional**,
8+
**no-go** or **inconclusive**. It also writes an HTML report showing every case that got worse.
9+
10+
> Status: early alpha (v0.1). It supports OpenAI and Anthropic, and imports OpenAI-style JSONL.
11+
12+
## Quick start
13+
14+
```bash
15+
uv tool install . # from a clone; not on PyPI yet
16+
shiftlock init # creates shiftlock.yaml and .shiftlock/
17+
shiftlock import requests.jsonl # your own requests; nothing leaves your machine
18+
# edit shiftlock.yaml: set the baseline and candidate model IDs
19+
shiftlock estimate # calls, tokens and worst-case cost, no API calls
20+
export ANTHROPIC_API_KEY=... # and/or OPENAI_API_KEY
21+
export ANTHROPIC_WORKSPACE_ID=... # only if your key isn't scoped to a workspace
22+
shiftlock run # asks before spending, then prints the verdict
23+
shiftlock report # show the latest verdict again
24+
```
25+
26+
`examples/sample.jsonl` holds four synthetic cases you can try it on.
27+
28+
## Input
29+
30+
The input is a JSONL file in one of two OpenAI chat-completions shapes:
31+
32+
- **Batch API:** `{"custom_id": ..., "body": {"messages": [...], "tools": [...], "response_format": {...}}}`.
33+
Settings such as `temperature`, `tool_choice` and `max_completion_tokens` are replayed as
34+
given.
35+
- **Fine-tuning:** `{"messages": [...], "tools": [...]}`. A final assistant message becomes the
36+
expected answer, and its tool calls are compared exactly.
37+
38+
Anthropic runs use the same file: shiftlock translates messages, tools and JSON schemas for you.
39+
40+
## Checks
41+
42+
| Check | When it runs | Effect |
43+
|---|---|---|
44+
| `json_valid`, `schema_match` | The request had a JSON schema or JSON mode | Blocking |
45+
| `tool_calls_valid` | The request offered tools; arguments must match the tool's schema | Blocking |
46+
| `tool_call_match` | The case has an expected tool call | Blocking |
47+
| `same_tools_as_baseline` | The request offered tools, with no expected call | Advisory |
48+
| `contains`, `not_contains`, `regex`, `max_latency_ms`, `max_cost_usd` | You add them in `shiftlock.yaml` | Advisory unless `blocking: true` |
49+
50+
## Verdict rules (v1, strict)
51+
52+
1. **no-go:** any blocking check that passed on the baseline fails on the candidate. One case is
53+
enough.
54+
2. **inconclusive:**
55+
- fewer than `min_cases` usable cases (default 20);
56+
- or more than `max_error_rate` of calls failed (default 10%).
57+
- Truncated and errored outputs don't count as usable.
58+
3. **conditional:** an advisory check regressed, or cost or median latency rose by more than 20%.
59+
4. **go:** none of the above.
60+
61+
Exit codes:
62+
- `0` go
63+
- `1` no-go
64+
- `2` conditional
65+
- `3` inconclusive
66+
- `4` cost cap reached
67+
- `10` and above: errors
68+
69+
`--fail-on` chooses which verdicts fail CI.
70+
71+
## Cost
72+
73+
Customers bring their own keys, so every run spends credit on your provider account.
74+
- `shiftlock estimate` gives a worst-case figure without calling any API.
75+
- `max_cost_usd` stops the run before a call could exceed the cap.
76+
- Prices come from `src/shiftlock/data/prices.yaml`, which is hand-maintained. Each entry needs
77+
a source URL and the date it was checked.
78+
- If a model isn't listed, cost shows as "unknown" rather than being guessed. Add prices under
79+
`prices:` in `shiftlock.yaml`.
80+
81+
## Privacy
82+
83+
There is no telemetry. Keys are read only from environment variables and are redacted from all
84+
output. See [SECURITY.md](SECURITY.md).
85+
86+
## Development
87+
88+
```bash
89+
uv sync
90+
uv run ruff check . && uv run ruff format --check .
91+
uv run mypy src
92+
uv run pytest # offline; `-m live` runs the opt-in tests that spend API credit
93+
```
94+
95+
## License
96+
97+
MIT

‎SECURITY.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Security
2+
3+
## Reporting a vulnerability
4+
5+
Please report vulnerabilities privately through GitHub's
6+
[private vulnerability reporting](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing-information-about-vulnerabilities/privately-reporting-a-security-vulnerability)
7+
on this repository. Do not open a public issue.
8+
9+
## How shiftlock handles your data
10+
11+
- **API keys** are read only from `OPENAI_API_KEY` and `ANTHROPIC_API_KEY`. They are never
12+
accepted as command-line arguments or written to disk. Logs and errors redact anything that
13+
looks like a key.
14+
- **Prompts and outputs** go only to the providers you configure, using your own keys. Nothing
15+
is sent anywhere else: there is no telemetry and no update check.
16+
- **Local files** under `.shiftlock/` (suites, raw results, reports) contain your prompts. They
17+
are created owner-only (`0700`/`0600` on POSIX systems), and the directory carries its own
18+
`.gitignore` so they are not committed by accident. On Windows, rely on your user profile's
19+
permissions.
20+
- **Import** flags cases that look like they contain emails or phone numbers and asks before
21+
keeping them. This is a simple pattern check, not a guarantee; review your data before
22+
importing it.

0 commit comments

Comments
 (0)