Skip to content

Commit 6f56a80

Browse files
rockyzlclaude
andcommitted
Make the vision skill honest: no fabricated image features
The old image_stub invented an observation ("mock_texture_descriptor = lamellar") — self-contradicting for a tool whose whole point is evidence honesty. Replace it with a vision hook that ships no model and returns NO features: when an image is given but no vision model is configured, image-derived features are unavailable and surface downstream as missing evidence, never as made-up data. A real vision model still plugs in here later. - skills/image_stub.py → skills/vision.py (returns []); observation_agent import updated; reproducibility skill stamp = "vision:not-configured". - workflow limitations + README + architecture updated. - New test: image-only input invents no features and forces measure_again. 20/20 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent c4d6e98 commit 6f56a80

7 files changed

Lines changed: 42 additions & 29 deletions

File tree

README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -185,7 +185,8 @@ human-in-the-loop scientific workflows. See [`docs/community-strategy.md`](docs/
185185

186186
Research prototype. Interpretations are tentative and must be confirmed by a domain
187187
expert. Literature retrieval runs over a small curated offline knowledge base (not a
188-
live literature API); image features are still mocked in this version. Do not use for
188+
live literature API). This offline build ships no vision model, so image-derived
189+
features are unavailable — reported as missing evidence, never invented. Do not use for
189190
real scientific or safety decisions.
190191

191192
## License

docs/architecture.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,8 +39,9 @@ applied to scientific reasoning.
3939
## Extensibility
4040

4141
- **Skills** (`skills/`) are pluggable capabilities. v0 ships a real offline
42-
`literature` retrieval (stdlib TF-IDF over a curated knowledge base) and a still-mocked
43-
`image_stub`; a real vision model replaces the latter without touching the spine.
42+
`literature` retrieval (stdlib TF-IDF over a curated knowledge base) and a `vision`
43+
hook that ships no model — rather than fabricate image features it returns none, so
44+
they surface as missing evidence. A real vision model plugs in without touching the spine.
4445
- **Instruments / tools** become connectors later (Phase 3: MCP-compatible). The spine
4546
never needs to know whether evidence came from a microscope, a database, or a
4647
simulation — only its `EvidenceKind` and confidence.

src/scientific_agent_lab/agents/observation_agent.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
from __future__ import annotations
44

55
from ..schemas import ObservedFeature, ScientificInput
6-
from ..skills.image_stub import features_from_image
6+
from ..skills.vision import features_from_image
77

88

99
def observe(inp: ScientificInput) -> list[ObservedFeature]:

src/scientific_agent_lab/skills/image_stub.py

Lines changed: 0 additions & 23 deletions
This file was deleted.
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
"""Vision skill — the pluggable hook for image-derived observed features.
2+
3+
By design this offline, zero-dependency build ships NO vision model, so it does
4+
NOT invent image features: a tool whose whole point is evidence honesty must not
5+
fabricate its own observations. When an image is provided but no vision model is
6+
configured, image-derived features are simply unavailable — if the question
7+
depends on them, they surface downstream as *missing evidence*, never as made-up
8+
data. A real vision / foundation model plugs in here by returning ``ObservedFeature``
9+
items with genuine provenance (that is the Phase 1-4 work in the roadmap).
10+
"""
11+
from __future__ import annotations
12+
13+
from ..schemas import ObservedFeature
14+
15+
16+
def features_from_image(image_ref: str) -> list[ObservedFeature]:
17+
# No vision model configured in this offline build → no fabricated features.
18+
return []

src/scientific_agent_lab/workflow.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ def _reproducibility(
3939
) -> ReproducibilityRecord:
4040
skills = [f"literature@{__version__}:offline-tfidf-kb"]
4141
if not inp.observations and inp.image_ref:
42-
skills.insert(0, f"image_stub@{__version__}:mock")
42+
skills.insert(0, f"vision@{__version__}:not-configured")
4343
return ReproducibilityRecord(
4444
input_sha256=_sha16(asdict(inp)),
4545
report_sha256=_sha16(report_dict),
@@ -56,7 +56,8 @@ def _reproducibility(
5656
"This is a research prototype, not a validated scientific decision system. "
5757
"Interpretations are tentative and must be confirmed by a domain expert. "
5858
"Literature retrieval runs over a small curated offline knowledge base (not a "
59-
"live literature API); image features are still mocked in this version."
59+
"live literature API). This offline build ships no vision model, so image-derived "
60+
"features are unavailable — reported as missing evidence, never invented."
6061
)
6162

6263

tests/test_workflow.py

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,3 +30,18 @@ def test_workflow_recommends_measurement_when_gap_exists():
3030
report, _ = _report()
3131
# a required measurement is missing -> agent must not ACCEPT
3232
assert report.recommended_next_measurement.action == NextAction.MEASURE_AGAIN
33+
34+
35+
def test_image_only_input_invents_no_features():
36+
# no vision model configured -> the pipeline must NOT fabricate observations;
37+
# required evidence is then honestly missing and it must not accept.
38+
inp = ScientificInput.from_dict({
39+
"question": "What phase is in this image?",
40+
"domain": "microscopy",
41+
"image_ref": "region.tif",
42+
"required": [{"name": "phase_id", "kind": "measurement", "why": "identify the phase"}],
43+
})
44+
report, _ = run_workflow(inp)
45+
assert report.observed_features == []
46+
assert report.recommended_next_measurement.action == NextAction.MEASURE_AGAIN
47+
assert any(m.name == "phase_id" for m in report.missing_evidence)

0 commit comments

Comments
 (0)