Skip to content

Complete --submit mode: urlProbe, activation, cookies, status_codes, update-existing #2668

Description

@soxoj

Problem

maigret --submit <url> currently handles only the easy case: clearweb site, no anti-bot, no JSON probe URL, no token-refresh activation, no cookie-based auth, no existing-entry update. Six TODOs in submit.py mark every place where a manual editing fallback is required. The CONTRIBUTING guide explicitly tells contributors to fall back to manual editing for any non-trivial site — this issue tracks the work to lift that limitation.

Sub-tasks

The TODOs are independent enough to land in separate PRs but share enough context to live under one issue:

  1. Cookies (maigret/submit.py:218) — pass --cookies-jar-file through check_features_manually. Easy.
  2. status_codes (maigret/submit.py:527) — when manual-features fallback is used, also collect HTTP status codes for the claimed and unclaimed URLs so the resulting entry can pick checkType: status_code if appropriate.
  3. Shared HTTP client (maigret/submit.py:195) — replace the bespoke session.get block with SimpleAiohttpChecker so submit-mode benefits from the same SSL handling, error mapping, and retries as the main checker.
  4. Filter by errors (maigret/submit.py:265) — the inline Cloudflare-detection regex should be moved into the shared errors.detect machinery (maigret/errors.py) and the dialog should be able to handle other anti-bot vendors uniformly.
  5. urlProbe / activation (maigret/submit.py:475–476) — currently --submit aborts when the existing site has a urlProbe or activation block. Detect public JSON APIs alongside the HTML check and offer to populate urlProbe. Activation is a bigger lift (per-provider) — at minimum, document the gap inline.
  6. Update-existing (maigret/submit.py:424) — the dialog already detects when the same domain has an existing entry but does not actually merge the new evidence into it. Implement the merge.

Code refs

  • maigret/submit.py:195–540
  • Parallel checker in maigret/checking.py

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions