Two small readers for any website, over plain HTTP, with no API key:
- Platform reader: which shop platform the site runs on (Shopify, WooCommerce, BigCommerce, Magento, PrestaShop, Squarespace; Wix and Shopline are recognised too), and that platform's public product feed: titles, URLs, prices, compare-at prices, currency, image and, where the platform says it, the catalogue size.
- Email reader: which email/SMS marketing tool the site uses (Klaviyo, Omnisend, Mailchimp, Attentive, Postscript, Privy, HubSpot and about 20 more), and how many signup forms it shows (counted for Klaviyo shops).
Detection only: the email reader never reads email content and never subscribes anybody.
pip install store-tech-collectorPython 3.9 or newer. One dependency: httpx.
Not on PyPI yet. Until the first release, install from a checkout:
pip install .
store-tech example-shop.com # both readers
store-tech -r platform --max-products 20 a.com b.com
store-tech -r email -f domains.txt --format jsonl -o out.jsonlstore-tech [DOMAIN ...] [-f FILE] [-r platform|email|both] [--max-products N] [--no-products]
[--format json|jsonl] [-o FILE] [--concurrency N] [--timeout S] [--delay S]
[--proxy URL] [--user-agent UA] [--version]
- Domains come from arguments and/or UTF-8 files (
-f, one per line,#starts a comment;-f -reads stdin). URLs are accepted; their host is used (www.is dropped). --max-products(1-250, default 100) caps the feed; each platform also has its own cap (see Limits).--no-productskeeps the counts and leaves out the product list.--proxy http://user:pass@host:port(or theSTORE_TECH_PROXYenvironment variable) is passed to httpx unchanged.--delayis the pause before every request (default 0.25 s). Requests for one domain are sequential;--concurrency(default 4) is the number of domains read in parallel.- Error messages name the option, never the value typed.
from store_tech_collector import Settings, collect
for rec in collect(["example-shop.com"], Settings(readers=("platform", "email"), max_products=20)):
print(rec["domain"], rec["platform"]["platform"], rec["platform"]["feed"])
print(rec["email"]["primaryProvider"], rec["email"]["signupForms"])collect_async(domains, settings) is the async version; read_domain(domain, settings) reads one normalised domain.
The parsers can be used on HTML you already have:
from store_tech_collector import detect_platform, detect_providers
platform, evidence = detect_platform(html, response_headers) # ("shopify", ["html:Shopify.shop", ...])
tools = detect_providers(html) # [{"name": "Klaviyo", "kind": "esp", ...}]One record per input, in input order. A reader that was not asked for has no key.
| Field | Meaning |
|---|---|
input_index |
position of the input (join key) |
domain, finalHost |
the normalised domain; the host the homepage ended on after redirects |
ok, error |
ok is true when every reader asked for gave a result; error is the first reader's error code |
requests, fetchedAt |
HTTP requests made for this domain; UTC time |
platform.platform |
shopify, woocommerce, bigcommerce, magento, prestashop, squarespace, wix or other |
platform.platformEvidence |
the markers that matched (html:..., header:..., feed:<kind>) |
platform.feed |
{kind, ok, error, productCount, catalogueTotal, currency, products[]} or null |
platform.feed.products[] |
{title, url, price, compareAtPrice, currency, imageUrl}; prices in major units |
email.providers |
every tool found, primary order: email (esp) > SMS > popup > native |
email.primaryProvider |
the first of providers, or null |
email.providerEvidence |
[{name, evidence}]: script_or_app_block, web_pixel, app, native_form |
email.signupForms |
{count, live, types[], source}; source is klaviyo_api (counted), klaviyo_unread (Klaviyo seen, file not read) or detected_only (count null) |
*.extractionTier, *.extractionTiers, *.tierAttempts |
which step answered, and every step tried with its code and time |
Feed kind: shopify_products_json, woocommerce_store_api, bigcommerce_graphql, magento_graphql,
magento_rest, prestashop_best_sales, squarespace_json, or sitemap_jsonld (a few products read from the
sitemap and product pages when the platform has no public feed).
Error codes: blocked (401/403/429/503 or a bot-wall page), http_<status>, empty, invalid_domain, a transport
error name (ConnectError, ReadTimeout, ...) or all_tiers_failed. Step codes in tierAttempts include
no_feed_route, no_feed, no_klaviyo_id and forms_api_<status>.
- One GET of the homepage per domain, shared by both readers.
- Platform: strict markers only (CDN hosts, platform headers, full globals such as
Static.SQUARESPACE_CONTEXT); a bare word on the page never counts, and one weak Shopify or Wix reference is not enough. Then the platform's own public endpoint:/products.json(+/meta.json), the WooCommerce Store API sorted by popularity, BigCommerce's storefront GraphQLbestSellingProducts(with the public storefront token the shop prints in its own homepage for its own scripts, read at run time and never stored or output), Magento GraphQL with the anonymous RESTproducts-render-infoas fallback, PrestaShop best-sales JSON, Squarespace?format=json. Without a feed: the sitemap and up to 8 product pages (JSON-LD or Open Graph product tags). - Email: vendor script hosts, app blocks and init calls in the HTML; Klaviyo also from Shopify's web-pixel list; the native Shopify newsletter form. When the page carries a Klaviyo company id, Klaviyo's public form-definition file (the one its on-site script downloads) gives the form count and types. The id is never output.
- HTTP only. No browser is used. A site that blocks plain HTTP clients, or shows a bot wall, is reported as
blockedand nothing else is tried. Tools injected later by other scripts (tag managers, app loaders) are not seen; the email reader only finds what the homepage HTML shows. - Signup forms are counted only for Klaviyo; for other tools
signupForms.countis null (detected only). - Feed caps per platform: Shopify 250, WooCommerce 100, BigCommerce 50, Magento 50; PrestaShop and Squarespace return what their first page holds. Prices follow the visitor market the site serves to this client.
catalogueTotalis only known where the platform says it (Shopify/meta.json, WooCommerceX-WP-Total, BigCommerce, Magento, PrestaShop); forsitemap_jsonldit is the number of sitemap URLs read.- One retry (after 2 s) on a transport error or 502/504; no other retries.
Read only sites you are allowed to read, respect their terms and robots rules, keep --concurrency and --delay
gentle, and do not use this tool to get past blocks. It sends an identifying User-Agent by default.
MIT