Skip to content

Commit 5d25593

Browse files
devilscrapesdevilscrapes
committed
Update alibaba-product-scraper-python
0 parents  commit 5d25593

12 files changed

Lines changed: 547 additions & 0 deletions

File tree

‎.env.example‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Your Apify API token: https://console.apify.com/settings/integrations
2+
APIFY_TOKEN=

‎.github/workflows/ci.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
pull_request:
6+
7+
permissions:
8+
contents: read
9+
10+
jobs:
11+
test:
12+
runs-on: ubuntu-latest
13+
strategy:
14+
matrix:
15+
python-version: ["3.11", "3.13"]
16+
steps:
17+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
18+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
19+
with:
20+
python-version: ${{ matrix.python-version }}
21+
- run: pip install -r requirements-dev.txt
22+
- run: ruff check .
23+
- run: ruff format --check .
24+
- run: pytest -q

‎.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
.env
2+
results.json
3+
results.csv
4+
__pycache__/
5+
.venv/
6+
.pytest_cache/
7+
.ruff_cache/

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Devil Scrapes
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
<p align="center"><a href="https://apify.com/DevilScrapes/alibaba-products-scraper"><img src="assets/banner.svg" alt="Alibaba Products Scraper — Python example" width="100%"></a></p>
2+
3+
# Alibaba Product Scraper in Python: supplier search results to JSON and CSV
4+
5+
<p align="center">
6+
<a href="https://apify.com/DevilScrapes/alibaba-products-scraper"><img alt="Run on Apify" src="https://img.shields.io/badge/Run%20on-Apify-DC2626?logo=apify&logoColor=white"></a>
7+
<img alt="Python" src="https://img.shields.io/badge/Python-3.11%2B-3776AB?logo=python&logoColor=white">
8+
<a href="LICENSE"><img alt="MIT license" src="https://img.shields.io/badge/license-MIT-0A0606"></a>
9+
<a href="https://github.com/devilscrapes/alibaba-product-scraper-python/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/devilscrapes/alibaba-product-scraper-python/actions/workflows/ci.yml/badge.svg"></a>
10+
</p>
11+
12+
This repository is a small, tested Python client for the **[Alibaba Products Scraper](https://apify.com/DevilScrapes/alibaba-products-scraper)** on Apify. Give it one or more search phrases such as `bluetooth speaker` and it returns a clean table of matching Alibaba.com offers: what the product is called, what it costs per unit, how many units the supplier wants you to buy, who the supplier is, and the link to the listing. The scraping itself runs in the cloud on Apify, so your script stays short and your machine never touches Alibaba directly. You keep the data in plain JSON and CSV, ready for pandas, a spreadsheet or your own database.
13+
14+
> **Actor:** [Alibaba Products Scraper](https://apify.com/DevilScrapes/alibaba-products-scraper) · **Publisher:** [Devil Scrapes](https://apify.com/DevilScrapes) · **Also in JavaScript:** [alibaba-product-scraper-javascript](https://github.com/devilscrapes/alibaba-product-scraper-javascript)
15+
16+
## 🔥 What you get
17+
18+
- **One row per unique product**: product ID, title, price or price range, minimum order quantity, supplier name, image URL and product URL.
19+
- **Many keywords per run**: every search term is paginated on its own, up to the limit you set.
20+
- **No repeat rows**: sponsored offers that reappear on later result pages are removed across the whole run, so you are not billed twice for one product.
21+
- **One bad term never sinks the run**: a term that fails is skipped and logged while the other terms finish.
22+
- **Blocking handled for you**: the Actor detects Alibaba's challenge pages and retries on a fresh session instead of returning an empty page as "no results".
23+
- **Honest zero results**: a term with no matching products succeeds with zero rows instead of failing the run.
24+
25+
## 🎯 Use cases
26+
27+
### Build a sourcing shortlist before you order samples
28+
29+
Run a handful of product phrases, load `results.json` into a DataFrame and sort by the lowest unit price. In a few minutes you have a ranked list of suppliers with their minimum order quantities next to each price, which is the comparison buyers usually do by hand across dozens of browser tabs.
30+
31+
### Track wholesale price movement for a category
32+
33+
Schedule the script with cron or a CI job, write each day's CSV to a dated file and compare the price ranges over time. Because every row carries the search term and a UTC timestamp, a simple `groupby` shows when a category gets cheaper or when new suppliers appear.
34+
35+
### Feed product research into your own tools
36+
37+
Agencies and e-commerce teams can pipe the rows straight into an internal catalog, a PIM or an ERP import job. The output fields are stable, so a mapping written once keeps working from run to run.
38+
39+
### Size a niche for dropshipping or private label
40+
41+
Count how many distinct suppliers list a product, how wide the price spread is and how high the typical MOQ sits. Those three numbers tell you quickly whether a niche is crowded, cheap to enter or locked behind large order sizes.
42+
43+
## ⚡ Quick start
44+
45+
You need an Apify account and an API token from [Console → Settings → Integrations](https://console.apify.com/settings/integrations). Apify's free plan includes monthly usage credit; this Actor's event prices are listed under Pricing below.
46+
47+
```bash
48+
git clone https://github.com/devilscrapes/alibaba-product-scraper-python.git
49+
cd alibaba-product-scraper-python
50+
python -m venv .venv && source .venv/bin/activate
51+
pip install -r requirements.txt
52+
export APIFY_TOKEN=your_token_here
53+
python main.py # uses input.json
54+
python main.py my-input.json --out data --max-charge 2
55+
```
56+
57+
The script starts the [Alibaba Products Scraper](https://apify.com/DevilScrapes/alibaba-products-scraper), waits for it to finish, prints a link to the run in Apify Console and saves every result to `results.json` and `results.csv`. It exits with code 1 and the run's status message if the run fails.
58+
59+
## 🧩 The core call
60+
61+
If you only need the essentials, this is the whole integration. The official `apify-client` package starts the Actor, waits for it to finish and pages through the results for you.
62+
63+
```python
64+
from apify_client import ApifyClient
65+
66+
client = ApifyClient("YOUR_APIFY_TOKEN")
67+
run = client.actor("DevilScrapes/alibaba-products-scraper").call(run_input={'searchTerms': ['bluetooth speaker'], 'maxResultsPerTerm': 20, 'proxyConfiguration': {'useApifyProxy': True}})
68+
for item in client.dataset(run.default_dataset_id).iterate_items():
69+
print(item)
70+
```
71+
72+
## 📥 Input
73+
74+
The script sends `input.json` to the Actor unchanged. The bundled file searches one term and stops at 20 products, which keeps a first test run cheap. Add more phrases to `searchTerms` to cover a whole category in a single run.
75+
76+
```json
77+
{
78+
"searchTerms": [
79+
"bluetooth speaker"
80+
],
81+
"maxResultsPerTerm": 20,
82+
"proxyConfiguration": {
83+
"useApifyProxy": true
84+
}
85+
}
86+
```
87+
88+
| Field | Type | Required | Default | Description |
89+
|---|---|---|---|---|
90+
| `searchTerms` | array | no | `["bluetooth speaker", "led strip light"]` | One or more free-text Alibaba search queries (e.g. bluetooth speaker, led strip light). Each term is fetched as its own paginated trade/search run. |
91+
| `maxResultsPerTerm` | integer | no | `100` | Cap on unique products collected per search term. Values above ~50 force multi-page pagination with run-wide duplicate removal. |
92+
| `proxyConfiguration` | object | no | `{"useApifyProxy": true}` | Apify Proxy spec — mandatory field. The Actor defaults to a RESIDENTIAL proxy pinned to the US when this is left at {"useApifyProxy": true} — the default (datacenter/auto) pool hit a served anti-bot challenge in cloud QA (2026-09-17). Override apifyProxyGroups/apifyProxyCountry here to change it. |
93+
94+
## 📤 Output
95+
96+
Every dataset row describes one product offer. Prices are kept exactly as Alibaba shows them, including ranges like `$3.50-5.20`, so nothing is lost in a currency or rounding conversion; parse them in your own code when you need numbers.
97+
98+
| Field | Type | Description |
99+
|---|---|---|
100+
| `product_id` | string | Alibaba's internal product ID. |
101+
| `search_term` | string | The search query that produced this row. |
102+
| `title` | string | Product title as listed on the search results page. |
103+
| `price` | string or null | Raw price or price-range string as served (mixed currency formats preserved). |
104+
| `moq` | string or null | Minimum order quantity string, when present. |
105+
| `supplier_name` | string or null | Supplier / company name, when present. |
106+
| `image_url` | string or null | Primary listing image URL. |
107+
| `product_url` | string | Absolute alibaba.com product detail page URL. |
108+
| `page_number` | integer | 1-indexed search results page this row was found on. |
109+
| `scraped_at` | string | UTC ISO-8601 timestamp captured when the page was fetched. |
110+
111+
## 💰 Pricing
112+
113+
| Event | Price (USD) | When it is charged |
114+
|---|---|---|
115+
| Actor start | $0.2 | Charged once when the Actor begins a run. |
116+
| Result item | $0.006 | Charged once per validated product row pushed to the dataset. |
117+
118+
Prices as configured on 2026-09-17. The [Actor page](https://apify.com/DevilScrapes/alibaba-products-scraper) always shows the current price. Runs are billed to the Apify account that owns the token. Use `--max-charge` to cap what a single run can cost.
119+
120+
## 🛠️ Working with the data in Python
121+
122+
Here is a starting point for turning the raw rows into a supplier shortlist with pandas (`pip install pandas`). It extracts the lowest price from each price string and the first number from the MOQ, then saves the 20 cheapest offers.
123+
124+
```python
125+
import json
126+
import re
127+
128+
import pandas as pd
129+
130+
rows = pd.DataFrame(json.load(open("results.json", encoding="utf-8")))
131+
132+
133+
def first_number(text):
134+
match = re.search(r"\d[\d,]*(?:\.\d+)?", text or "")
135+
return float(match.group().replace(",", "")) if match else None
136+
137+
138+
rows["price_from"] = rows["price"].map(first_number)
139+
rows["moq_units"] = rows["moq"].map(first_number)
140+
shortlist = rows.dropna(subset=["price_from"]).sort_values(["price_from", "moq_units"]).head(20)
141+
shortlist[["title", "supplier_name", "price", "moq", "product_url"]].to_csv("shortlist.csv", index=False)
142+
print(shortlist[["supplier_name", "price", "moq"]].to_string(index=False))
143+
```
144+
145+
## ❓ FAQ
146+
147+
### Do I need an Alibaba account or an Alibaba API key?
148+
149+
No. You only need an Apify API token. The Actor reads Alibaba's public search results, so there is no Alibaba login, partner program or API approval involved.
150+
151+
### Why did a term return fewer products than `maxResultsPerTerm`?
152+
153+
Alibaba repeats many sponsored offers from page to page. The Actor drops those repeats across the entire run and stops paging a term once new pages add nothing new. A smaller count means you reached the real number of distinct offers for that phrase.
154+
155+
### Can I run it from a notebook or a scheduler?
156+
157+
Yes. Import `main` and call `main.main(["input.json", "--out", "data"])` from a notebook, or run `python main.py` from cron, Airflow or a CI job. The exit code is 0 on success and 1 on a failed run, so schedulers can alert on it.
158+
159+
### Does it scrape supplier contact details or product detail pages?
160+
161+
No. It returns what Alibaba shows on its search results pages. Supplier contact data, reviews and login-only buyer prices are out of scope, and 1688.com is a separate site that this Actor does not cover.
162+
163+
## 🧯 Troubleshooting
164+
165+
- **`Set APIFY_TOKEN first`**: export the token in the same shell, or put it in a local `.env` file your shell loads. Never commit it.
166+
- **Run finished with `FAILED` or `ABORTED`**: open the printed Console link; the run log says why. Input validation errors name the field.
167+
- **Fewer rows than expected**: a `--max-charge` cap also limits how many results can be charged. Raise the cap or lower the limits in your input.
168+
- **Something looks wrong in the data**: open an issue on the [Actor page](https://apify.com/DevilScrapes/alibaba-products-scraper) so the maintainers see it.
169+
170+
## 🔗 More from Devil Scrapes
171+
172+
- [Alibaba Products Scraper on Apify Store](https://apify.com/DevilScrapes/alibaba-products-scraper)
173+
- [JavaScript version of this example](https://github.com/devilscrapes/alibaba-product-scraper-javascript)
174+
- [All Devil Scrapes Actors](https://apify.com/DevilScrapes)
175+
- [Apify web scraping cookbook](https://github.com/devilscrapes/apify-web-scraping-cookbook)
176+
177+
Related Actors:
178+
179+
- [AliExpress Products Scraper](https://apify.com/DevilScrapes/aliexpress-products-scraper) — retail prices and orders for the same products on AliExpress.
180+
- [ImportYeti Alternative Scraper](https://apify.com/DevilScrapes/importyeti-alternative-scraper) — check which suppliers actually ship to US importers.
181+
- [DirectIndustry Industrial Suppliers Scraper](https://apify.com/DevilScrapes/directindustry-industrial-suppliers-scraper) — industrial equipment suppliers outside the Alibaba ecosystem.
182+
183+
---
184+
185+
MIT licensed example code. The Actor itself runs on [Apify](https://apify.com/DevilScrapes/alibaba-products-scraper) and is maintained by [Devil Scrapes](https://apify.com/DevilScrapes). Not affiliated with Alibaba.com or Alibaba Group.

‎assets/banner.svg‎

Lines changed: 45 additions & 0 deletions
Loading

‎input.json‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{
2+
"searchTerms": [
3+
"bluetooth speaker"
4+
],
5+
"maxResultsPerTerm": 20,
6+
"proxyConfiguration": {
7+
"useApifyProxy": true
8+
}
9+
}

‎main.py‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
"""Run the Alibaba Products Scraper on Apify and save its results as JSON and CSV.
2+
3+
Actor page: https://apify.com/DevilScrapes/alibaba-products-scraper
4+
"""
5+
6+
from __future__ import annotations
7+
8+
import argparse
9+
import json
10+
import os
11+
import sys
12+
from decimal import Decimal
13+
from pathlib import Path
14+
15+
from apify_client import ApifyClient
16+
17+
ACTOR_ID = "DevilScrapes/alibaba-products-scraper"
18+
TERMINAL_OK = "SUCCEEDED"
19+
20+
21+
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
22+
parser = argparse.ArgumentParser(description="Run the Alibaba Products Scraper on Apify.")
23+
parser.add_argument("input", nargs="?", default="input.json", help="Actor input JSON file (default: input.json)")
24+
parser.add_argument("--out", default=".", help="Folder for results.json and results.csv (default: here)")
25+
parser.add_argument("--max-charge", type=Decimal, help="Stop the run once it has cost this many USD")
26+
parser.add_argument("--memory", type=int, help="Run memory in MB (default: the Actor's own setting)")
27+
return parser.parse_args(argv)
28+
29+
30+
def load_input(path: str) -> dict:
31+
try:
32+
data = json.loads(Path(path).read_text(encoding="utf-8"))
33+
except FileNotFoundError:
34+
sys.exit(f"Input file not found: {path}")
35+
except json.JSONDecodeError as error:
36+
sys.exit(f"Input file {path} is not valid JSON: {error}")
37+
if not isinstance(data, dict):
38+
sys.exit(f"Input file {path} must contain a JSON object")
39+
return data
40+
41+
42+
def main(argv: list[str] | None = None) -> int:
43+
args = parse_args(argv)
44+
token = os.environ.get("APIFY_TOKEN")
45+
if not token:
46+
sys.exit("Set APIFY_TOKEN first. Get a token at https://console.apify.com/settings/integrations")
47+
48+
run_input = load_input(args.input)
49+
client = ApifyClient(token, api_url=os.environ.get("APIFY_API_URL", "https://api.apify.com"))
50+
51+
print(f"Starting {ACTOR_ID} ...")
52+
run = client.actor(ACTOR_ID).call(
53+
run_input=run_input,
54+
max_total_charge_usd=args.max_charge,
55+
memory_mbytes=args.memory,
56+
)
57+
if run is None:
58+
sys.exit("The run could not be started.")
59+
60+
print(f"Run {run.id} finished: {run.status}")
61+
print(f"Details: https://console.apify.com/actors/runs/{run.id}")
62+
if run.status != TERMINAL_OK:
63+
print(run.status_message or "No status message.", file=sys.stderr)
64+
return 1
65+
66+
dataset = client.dataset(run.default_dataset_id)
67+
out = Path(args.out)
68+
out.mkdir(parents=True, exist_ok=True)
69+
items = json.loads(dataset.get_items_as_bytes(item_format="json", clean=True))
70+
(out / "results.json").write_text(json.dumps(items, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
71+
(out / "results.csv").write_bytes(dataset.get_items_as_bytes(item_format="csv", clean=True))
72+
73+
print(f"Saved {len(items)} items to {out / 'results.json'} and {out / 'results.csv'}")
74+
return 0
75+
76+
77+
if __name__ == "__main__":
78+
sys.exit(main())

0 commit comments

Comments
 (0)