Background
The Gravatar postprocessor (socid_extractor/postprocessor.py) used to resolve a Gravatar vanity username by doing a blocking requests.head() on en.gravatar.com with no timeout, inside __init__.
This is wrong for two reasons:
- Postprocessors are pure HTML parsers — they must never touch the network.
- Maigret calls
extract() synchronously inside a coroutine, so a slow/hanging Gravatar request blocked the entire asyncio event loop, not just one site. A single unlucky profile could freeze a whole scan indefinitely (repro'd on a live maigret --web run).
The network call has been disabled (commented out, self.username = '' fallback) as a hotfix. gravatar_url and gravatar_email_md5_hash are still emitted offline; only the vanity-username resolution is gone.
What to do
Re-implement username resolution through maigret's existing url_mutations / --enrich mechanism, which does the extra fetch through a proper checker (timeout, proxy, TLS impersonation) instead of a bare requests call in a parser.
schemes.py already has a Gravatar scheme with url_mutations. The JSON endpoint (https://en.gravatar.com/{id}.json) returns strictly more than the old HEAD-redirect hack — username, bio, links, emails, name, location.
Two gaps to close:
- socid_extractor: the current
from regex targets gravatar.com/{username}, not the avatar form gravatar.com/avatar/{hash}. Add a rule: gravatar\.com/avatar/(?P<hash>\w{32}) → https://en.gravatar.com/{hash}.json.
- maigret:
run_url_mutations() currently mutates only results_info["url_user"]. The avatar comes from the extracted image field of the checked page, not that page's own URL — so mutate_url() needs to also run over relevant extracted URL fields (at least image).
Notes
- Consequence: username/bio/links/etc. enrichment moves from "always on" to opt-in via
--enrich. Acceptable trade — the old behavior could hang the whole scan.
- Check for regressions around Gravatar in existing tests/expectations.
Background
The
Gravatarpostprocessor (socid_extractor/postprocessor.py) used to resolve a Gravatar vanity username by doing a blockingrequests.head()onen.gravatar.comwith no timeout, inside__init__.This is wrong for two reasons:
extract()synchronously inside a coroutine, so a slow/hanging Gravatar request blocked the entire asyncio event loop, not just one site. A single unlucky profile could freeze a whole scan indefinitely (repro'd on a livemaigret --webrun).The network call has been disabled (commented out,
self.username = ''fallback) as a hotfix.gravatar_urlandgravatar_email_md5_hashare still emitted offline; only the vanity-username resolution is gone.What to do
Re-implement username resolution through maigret's existing
url_mutations/--enrichmechanism, which does the extra fetch through a proper checker (timeout, proxy, TLS impersonation) instead of a barerequestscall in a parser.schemes.pyalready has aGravatarscheme withurl_mutations. The JSON endpoint (https://en.gravatar.com/{id}.json) returns strictly more than the old HEAD-redirect hack — username, bio, links, emails, name, location.Two gaps to close:
fromregex targetsgravatar.com/{username}, not the avatar formgravatar.com/avatar/{hash}. Add a rule:gravatar\.com/avatar/(?P<hash>\w{32})→https://en.gravatar.com/{hash}.json.run_url_mutations()currently mutates onlyresults_info["url_user"]. The avatar comes from the extractedimagefield of the checked page, not that page's own URL — somutate_url()needs to also run over relevant extracted URL fields (at leastimage).Notes
--enrich. Acceptable trade — the old behavior could hang the whole scan.