Problem
maigret --submit <url> currently handles only the easy case: clearweb site, no anti-bot, no JSON probe URL, no token-refresh activation, no cookie-based auth, no existing-entry update. Six TODOs in submit.py mark every place where a manual editing fallback is required. The CONTRIBUTING guide explicitly tells contributors to fall back to manual editing for any non-trivial site — this issue tracks the work to lift that limitation.
Sub-tasks
The TODOs are independent enough to land in separate PRs but share enough context to live under one issue:
- Cookies (
maigret/submit.py:218) — pass --cookies-jar-file through check_features_manually. Easy.
- status_codes (
maigret/submit.py:527) — when manual-features fallback is used, also collect HTTP status codes for the claimed and unclaimed URLs so the resulting entry can pick checkType: status_code if appropriate.
- Shared HTTP client (
maigret/submit.py:195) — replace the bespoke session.get block with SimpleAiohttpChecker so submit-mode benefits from the same SSL handling, error mapping, and retries as the main checker.
- Filter by errors (
maigret/submit.py:265) — the inline Cloudflare-detection regex should be moved into the shared errors.detect machinery (maigret/errors.py) and the dialog should be able to handle other anti-bot vendors uniformly.
- urlProbe / activation (
maigret/submit.py:475–476) — currently --submit aborts when the existing site has a urlProbe or activation block. Detect public JSON APIs alongside the HTML check and offer to populate urlProbe. Activation is a bigger lift (per-provider) — at minimum, document the gap inline.
- Update-existing (
maigret/submit.py:424) — the dialog already detects when the same domain has an existing entry but does not actually merge the new evidence into it. Implement the merge.
Code refs
maigret/submit.py:195–540
- Parallel checker in
maigret/checking.py
Problem
maigret --submit <url>currently handles only the easy case: clearweb site, no anti-bot, no JSON probe URL, no token-refresh activation, no cookie-based auth, no existing-entry update. Six TODOs insubmit.pymark every place where a manual editing fallback is required. The CONTRIBUTING guide explicitly tells contributors to fall back to manual editing for any non-trivial site — this issue tracks the work to lift that limitation.Sub-tasks
The TODOs are independent enough to land in separate PRs but share enough context to live under one issue:
maigret/submit.py:218) — pass--cookies-jar-filethroughcheck_features_manually. Easy.maigret/submit.py:527) — when manual-features fallback is used, also collect HTTP status codes for the claimed and unclaimed URLs so the resulting entry can pickcheckType: status_codeif appropriate.maigret/submit.py:195) — replace the bespokesession.getblock withSimpleAiohttpCheckerso submit-mode benefits from the same SSL handling, error mapping, and retries as the main checker.maigret/submit.py:265) — the inline Cloudflare-detection regex should be moved into the sharederrors.detectmachinery (maigret/errors.py) and the dialog should be able to handle other anti-bot vendors uniformly.maigret/submit.py:475–476) — currently--submitaborts when the existing site has aurlProbeoractivationblock. Detect public JSON APIs alongside the HTML check and offer to populateurlProbe. Activation is a bigger lift (per-provider) — at minimum, document the gap inline.maigret/submit.py:424) — the dialog already detects when the same domain has an existing entry but does not actually merge the new evidence into it. Implement the merge.Code refs
maigret/submit.py:195–540maigret/checking.py