Skip to content

Commit b252511

Browse files
authored
docs: add threat model and evaluation methodology (#3)
* docs: add security methodology and research framing * docs: add security methodology and research framing * docs: surface security research methodology
1 parent b999fcb commit b252511

3 files changed

Lines changed: 134 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,15 @@ ModelHawk/
188188
- Python docs — *"the pickle module is not secure; only unpickle data you trust."*
189189
- HuggingFace **safetensors** (the safe format) · Protect AI **modelscan** · Trail of Bits **fickling** · **picklescan**
190190

191+
192+
193+
## Security research & evaluation
194+
195+
- [Threat model](docs/THREAT_MODEL.md)
196+
- [Evaluation methodology](docs/EVALUATION.md)
197+
198+
ModelHawk treats the **no-deserialization invariant** as a first-class security property: the scanner should inspect hostile artifacts without executing them. Benchmarks therefore cover both detection quality and scanner safety.
199+
191200
## License
192201

193202
Released under the [MIT License](LICENSE).

‎docs/EVALUATION.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# ModelHawk Evaluation Methodology
2+
3+
## Evaluation objective
4+
5+
Demonstrate that ModelHawk catches supported malicious serialization patterns while avoiding execution of the inspected payload and limiting false positives on benign model artifacts.
6+
7+
## Test matrix
8+
9+
At minimum, evaluate:
10+
11+
| Category | Positive cases | Negative cases |
12+
|---|---|---|
13+
| Pickle protocol | 2, 4, 5 | benign state dicts |
14+
| Global resolution | `os/nt/posix`, `subprocess`, builtins execution | normal ML globals |
15+
| Invocation | `REDUCE`, object construction paths | non-executing opcode sequences |
16+
| Containers | raw pickle, PyTorch ZIP | safetensors |
17+
| Configuration | YAML Python tags | ordinary YAML |
18+
| Arrays | object-array pickle | numeric NumPy arrays |
19+
20+
## Safety invariant
21+
22+
The scanner must not call `pickle.load`, `pickle.loads`, `Unpickler`, or an equivalent deserializer on the inspected bytes.
23+
24+
This invariant should be tested structurally as well as behaviorally.
25+
26+
## Metrics
27+
28+
For a labeled corpus report:
29+
30+
- TP / FP / TN / FN
31+
- precision, recall and F1
32+
- false-positive rate
33+
- severity distribution
34+
- scan time by artifact type
35+
36+
Security claims should include corpus size and exact artifact-generation method.
37+
38+
## Reproducibility
39+
40+
Record:
41+
42+
- ModelHawk commit/version
43+
- Python version
44+
- operating system
45+
- corpus revision
46+
- scanner flags
47+
- artifact hashes
48+
49+
The malicious fixtures should remain harmless and deterministic. Tests should prove that scanning them does not create their payload side effects.
50+
51+
## Research questions
52+
53+
Useful future experiments include:
54+
55+
1. How much coverage improves with a complete pickle stack-machine model?
56+
2. Which gadget families remain invisible under a module allowlist?
57+
3. What is the false-positive cost of expanding coverage?
58+
4. How does static analysis compare with sandboxed dynamic loading for detection coverage?

‎docs/THREAT_MODEL.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# ModelHawk Threat Model
2+
3+
## Security objective
4+
5+
Identify code-execution-capable behavior embedded in serialized ML model artifacts without executing the artifact during inspection.
6+
7+
## Assets
8+
9+
- Developer/CI host
10+
- Model loading pipeline
11+
- Credentials and filesystem accessible to the loader
12+
- Integrity of model artifacts
13+
14+
## Attack path
15+
16+
```
17+
Attacker
18+
│
19+
▼
20+
malicious model artifact
21+
│
22+
▼
23+
model registry / download
24+
│
25+
▼
26+
torch.load / pickle.load
27+
│
28+
▼
29+
code execution
30+
```
31+
32+
ModelHawk inserts a static inspection step before the load boundary.
33+
34+
## Threats in scope
35+
36+
- Pickle `GLOBAL` / `STACK_GLOBAL` references to dangerous callables
37+
- `REDUCE`-based execution chains
38+
- Python object-construction gadget indicators
39+
- Unsafe YAML Python tags
40+
- NumPy object-array pickle payloads
41+
- Unsafe model artifacts presented as ordinary ML files
42+
43+
## Security properties
44+
45+
The scanner should:
46+
47+
1. Never deserialize the untrusted artifact.
48+
2. Produce deterministic findings for the same input.
49+
3. Preserve evidence sufficient for manual review.
50+
4. Fail CI at configured severity thresholds.
51+
5. Prefer explicit false-positive tradeoffs over pretending to prove safety.
52+
53+
## Out of scope
54+
55+
- Proving arbitrary native extensions are safe
56+
- Detecting every possible gadget chain
57+
- Runtime sandboxing
58+
- Trustworthiness of the model's learned parameters
59+
- Application-specific behavior after a safe load
60+
61+
## Key limitation
62+
63+
Static opcode analysis is necessarily bounded by the parser and classification model. A SAFE result means only that supported dangerous patterns were not detected.
64+
65+
## Defensive recommendation
66+
67+
Prefer code-free model containers such as safetensors where the deployment stack supports them, and treat untrusted pickle-compatible artifacts as executable content.

0 commit comments

Comments
 (0)