Skip to content

Commit df0000d

Browse files
authored
feat: aggressive MarkLLM detection decisions (#23)
* feat(harness): decision engine for MarkLLM text detection 5-state verdicts (DETECTED/NOT_DETECTED/INCONCLUSIVE/UNSUPPORTED/ERROR) so a negative under unknown provenance is never reported as clean. - watermark writes <input>.wm.json sidecar (key_id, config hash, scheme, implementation_commit); detect verifies it before allowing NOT_DETECTED - --key-id operator assertion, --min-tokens, --abstention-band flags - abstention band + min calibrated length qualify negatives only; a positive detector observation is always reported DETECTED - detector_verdict keeps the raw MarkLLM observation in the payload * test(harness): lock aggressive positive verdicts * feat(service): surface unresolved detector evidence * docs(service): explain aggregate detection status * no-mistakes(review): fix untrusted provenance, token gate, and detector-failure reporting * no-mistakes(document): reformat over-long line in markllm detect test * no-mistakes(document): align Gemini detector docs with disabled implementation * no-mistakes(lint): remove stray blank line failing ruff format * docs(readme): unify badge row into quiet slate/amber system - reserve amber accent for pypi version + release only - demote metadata badges to muted slate (30363d) values - restyle CI as shields workflow badge with live status color - replace social stars and classic blue license badges - match dark label chips (0f1110) across all eight badges * fix: address PR review findings - README: unsupported schemes document UNSUPPORTED verdict, not INCONCLUSIVE - detect/watermark CLI: catch OSError from config hashing, exit 3 instead of traceback - gemini detector: oversize skip reports configured=true per contract docstring - tests: environment-independent implementation_commit assertion - tests: rename detect_json to detect_literal (Python dict literal, not JSON) - tests: regression coverage for unreadable-config paths * docs: narrow ERROR verdict wording to unscoreable detector results
1 parent c88d914 commit df0000d

11 files changed

Lines changed: 1087 additions & 37 deletions

File tree

.env.example

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -70,10 +70,12 @@ WATERMARKS_SERVER_API_KEY=
7070
# WATERMARKS_SERVER_VERSION=
7171

7272
# ─── Vendor text-watermark detection ────────────────────────────
73-
# Optional Google Gemini API key. When set, the service can run Google's
74-
# official SynthID-text watermark detector via POST /detect and the
75-
# detect_before / detect_after clean options. Env only — never on argv.
76-
# Privacy: text is sent to Google only when this key is configured.
73+
# Optional Google Gemini API key. Env only — never on argv. The Gemini
74+
# SynthID-text detector is currently DISABLED: the generateContent API
75+
# exposes no DETECT_TEXT_WATERMARK task type, so /capabilities always
76+
# reports text_detectors.gemini-synthid-text as false and setting this key
77+
# enables nothing today. Kept so the wiring is ready if the task type ships;
78+
# text would then be sent to Google only when this key is configured.
7779
# WATERMARKS_GEMINI_API_KEY=
7880
# WATERMARKS_GEMINI_MODEL=gemini-2.5-flash
7981
# WATERMARKS_GEMINI_TIMEOUT=30

README.md

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,13 @@
22

33
<p align="center">
44
<a href="https://pypi.org/project/watermark-remover/"><img alt="PyPI" src="https://img.shields.io/pypi/v/watermark-remover?logo=pypi&logoColor=white&label=pypi&color=e8a33d&labelColor=0f1110"></a>
5-
<a href="https://pepy.tech/project/watermark-remover"><img alt="Downloads" src="https://img.shields.io/pepy/dt/watermark-remover?logo=python&logoColor=white&label=downloads&color=e8a33d&labelColor=0f1110"></a>
6-
<a href="https://pypi.org/project/watermark-remover/"><img alt="Python 3.10+" src="https://img.shields.io/badge/python-3.10%2B-e8a33d?logo=python&logoColor=white&labelColor=0f1110"></a>
7-
<a href="https://github.com/PyModel/watermark-remover/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/PyModel/watermark-remover/actions/workflows/ci.yml/badge.svg"></a>
85
<a href="https://github.com/PyModel/watermark-remover/releases/latest"><img alt="Release" src="https://img.shields.io/github/v/release/PyModel/watermark-remover?sort=semver&display_name=tag&label=release&color=e8a33d&labelColor=0f1110"></a>
9-
<a href="https://github.com/PyModel/watermark-remover/stargazers"><img alt="Stars" src="https://img.shields.io/github/stars/PyModel/watermark-remover?label=stars&color=e8a33d&labelColor=0f1110"></a>
10-
<a href="https://github.com/PyModel/watermark-remover/blob/main/LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-blue.svg"></a>
11-
<a href="https://hits.sh/github.com/PyModel/watermark-remover/"><img alt="Visitors" src="https://hits.sh/github.com/PyModel/watermark-remover.svg?label=visitors&color=e8a33d&labelColor=0f1110"></a>
6+
<a href="https://pypi.org/project/watermark-remover/"><img alt="Python 3.10+" src="https://img.shields.io/badge/python-3.10%2B-30363d?logo=python&logoColor=white&labelColor=0f1110"></a>
7+
<a href="https://pepy.tech/project/watermark-remover"><img alt="Downloads" src="https://img.shields.io/pepy/dt/watermark-remover?logo=python&logoColor=white&label=downloads&color=30363d&labelColor=0f1110"></a>
8+
<a href="https://github.com/PyModel/watermark-remover/actions/workflows/ci.yml"><img alt="CI" src="https://img.shields.io/github/actions/workflow/status/PyModel/watermark-remover/ci.yml?logo=github&logoColor=white&label=ci&labelColor=0f1110"></a>
9+
<a href="https://github.com/PyModel/watermark-remover/stargazers"><img alt="Stars" src="https://img.shields.io/github/stars/PyModel/watermark-remover?logo=github&logoColor=white&label=stars&color=30363d&labelColor=0f1110"></a>
10+
<a href="https://github.com/PyModel/watermark-remover/blob/main/LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-30363d?labelColor=0f1110"></a>
11+
<a href="https://hits.sh/github.com/PyModel/watermark-remover/"><img alt="Visitors" src="https://hits.sh/github.com/PyModel/watermark-remover.svg?label=visitors&color=30363d&labelColor=0f1110"></a>
1212
</p>
1313

1414
Tools for finding and removing AI provenance signals from files you own. Four channels are covered: hidden Unicode in text, statistical token watermarks, visible marks burned into images, and metadata such as C2PA, EXIF, and XMP.
@@ -296,7 +296,8 @@ Best-effort pixel-domain removal is opt-in on image cleans: `--remove-synthid` (
296296
Optional, stdlib-first, fail-soft:
297297

298298
- `score_stylometry.py` — zero-LLM stylometry (burstiness, MATTR, AI-phrase density) with confidence bands and `--explain`.
299-
- `text_detectors.py` / `detect_text_watermark.py` — Gemini's official SynthID-text detector (needs `WATERMARKS_GEMINI_API_KEY`; env only) and a MarkLLM research harness (same-scheme-config only, not a vendor oracle). Claude detection is reserved but unavailable until a public API ships.
299+
- `text_detectors.py` / `detect_text_watermark.py` — a MarkLLM research harness (same-scheme-config only, not a vendor oracle). Both vendor detectors are reserved but unavailable, and `/capabilities` always reports them `false`: Gemini's official SynthID-text detector is wired for `WATERMARKS_GEMINI_API_KEY` (env only) but the Gemini API exposes no `DETECT_TEXT_WATERMARK` task type, and Claude detection waits on a public API.
300+
- MarkLLM detection reports a document-level `verdict` (`DETECTED`, `NOT_DETECTED`, `INCONCLUSIVE`, `UNSUPPORTED`, `ERROR`) alongside the raw `detector_verdict`. Only a below-threshold score with a confirmed key/provenance match and enough scored tokens reads as `NOT_DETECTED`; unknown provenance, a near-threshold score, or a short sample produces `INCONCLUSIVE`. An unsupported scheme produces `UNSUPPORTED`, and a detector result that cannot be scored produces `ERROR`. Provenance comes from the `<output>.wm.json` sidecar the `watermark` subcommand writes, or from an operator `--key-id` assertion.
300301
- `inspect_text.py --stylometry` and `rewrite_text.py` MarkLLM before/after hooks.
301302

302303
```bash

compose.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,8 @@ services:
2727
environment:
2828
# Empty by default (no auth). Set to require `Authorization: Bearer <key>`.
2929
WATERMARKS_SERVER_API_KEY: ${WATERMARKS_SERVER_API_KEY:-}
30-
# Optional Gemini API key enabling vendor SynthID-text watermark
31-
# detection via /detect and detect_before/after (see .env.example).
30+
# Optional Gemini API key. The vendor SynthID-text detector is
31+
# currently disabled and this key enables nothing (see .env.example).
3232
WATERMARKS_GEMINI_API_KEY: ${WATERMARKS_GEMINI_API_KEY:-}
3333
WATERMARKS_GEMINI_MODEL: ${WATERMARKS_GEMINI_MODEL:-}
3434
# Optional SynthID image scorer sidecar (heavy profile).

skills/remove-ai-marks/references/service-mode.md

Lines changed: 19 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ field and writes it to the output path itself.
5252
| GET | `/health` || `{"ok": true, "version": ...}` |
5353
| GET | `/capabilities` || optional tools / backends present |
5454
| GET | `/openapi.json` || dynamically generated OpenAPI 3.0.3 spec |
55-
| POST | `/inspect` | `{"file": "<base64>", "name": "notes.md"}` | `{"ok", "kind", "suspicious", "report"}` |
55+
| POST | `/inspect` | `{"file": "<base64>", "name": "notes.md", "detect": true}` | `{"ok", "kind", "suspicious", "detection_status", "report"}` |
5656
| POST | `/detect` | `{"file": "<base64>", "name": "notes.txt"}` | `{"ok", "kind", "detections": [...]}` |
5757
| POST | `/clean` | `{"file": "<base64>", "name": "notes.md", "options": {...}}` | `{"ok", "kind", "cleaned": "<base64>", "report"}` |
5858

@@ -61,6 +61,15 @@ unrecognized formats answer `kind: "unknown"` (`/inspect`) or 400 (`/clean`).
6161
When writing a temp file for pasted text, keep a known extension (`.txt` /
6262
`.md`) in the `name` you send.
6363

64+
For `/inspect`, `detection_status` is `DETECTED`, `INCONCLUSIVE`,
65+
`NOT_DETECTED`, or `NOT_RUN`. `INCONCLUSIVE` means an available detector ran but
66+
could not rule out a watermark; it sets `suspicious: true` conservatively but is
67+
not a confirmed detection. A configured detector that ran and failed (timeout,
68+
crash, unreadable output) also aggregates to `INCONCLUSIVE`, never `NOT_RUN`.
69+
`NOT_RUN` means detection was not requested or no detector was configured.
70+
Pass `"detect": true` only with consent when a configured vendor detector may
71+
send text to its provider.
72+
6473
The machine-readable contract lives at `$WM/openapi.json` — plug it into any
6574
OpenAPI tooling instead of hand-rolling clients.
6675

@@ -92,15 +101,18 @@ default unless the user asked in-place) and summarize `report` honestly.
92101

93102
## Watermark detection before/after (when configured)
94103

95-
When `/capabilities` reports a vendor detector (`text_detectors.gemini-synthid-text`)
96-
or an image scorer (`scorers.synthid_http` / `scorers.synthid`), measure the
104+
When `/capabilities` reports a text detector (`text_detectors.markllm`) or an
105+
image scorer (`scorers.synthid_http` / `scorers.synthid`), measure the
97106
result by detecting before and after cleaning — either `POST /detect` or fold
98107
it into the clean with `{"options": {"detect_before": true, "detect_after": true}}`
99108
(returns `text_detectors.before/after` for text or `synthid_before/after` for
100-
images). Vendor detection sends text to the configured provider (Gemini) —
101-
only use it with user consent, and report the vendor's verdict honestly
102-
(Gemini = Google's official SynthID-text detector; MarkLLM is same-config-only
103-
research; Claude's detector is not public yet).
109+
images). MarkLLM is same-config-only research, not a vendor oracle; report its
110+
verdict honestly.
111+
112+
No vendor text detector is reachable today: `text_detectors.gemini-synthid-text`
113+
and `text_detectors.claude-text` always report `false` in `/capabilities` (see
114+
the detector list in the README). If one does become available, it sends text
115+
to that provider — only use it with explicit user consent.
104116

105117
## Aggregate audits (directories / websites)
106118

0 commit comments

Comments
 (0)