Skip to content

Commit c1e945f

Browse files
devilscrapesdevilscrapes
committed
Medium user articles scraper in Python
0 parents  commit c1e945f

12 files changed

Lines changed: 557 additions & 0 deletions

File tree

‎.env.example‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Your Apify API token: https://console.apify.com/settings/integrations
2+
APIFY_TOKEN=

‎.github/workflows/ci.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
pull_request:
6+
7+
permissions:
8+
contents: read
9+
10+
jobs:
11+
test:
12+
runs-on: ubuntu-latest
13+
strategy:
14+
matrix:
15+
python-version: ["3.11", "3.13"]
16+
steps:
17+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
18+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
19+
with:
20+
python-version: ${{ matrix.python-version }}
21+
- run: pip install -r requirements-dev.txt
22+
- run: ruff check .
23+
- run: ruff format --check .
24+
- run: pytest -q

‎.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
.env
2+
results.json
3+
results.csv
4+
__pycache__/
5+
.venv/
6+
.pytest_cache/
7+
.ruff_cache/

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Devil Scrapes
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
<p align="center"><a href="https://apify.com/DevilScrapes/medium-user-articles-scraper"><img src="assets/banner.svg" alt="Medium Author RSS Scraper — Python example" width="100%"></a></p>
2+
3+
# Medium User Articles Scraper in Python: Medium author and publication feeds to JSON and CSV
4+
5+
<p align="center">
6+
<a href="https://apify.com/DevilScrapes/medium-user-articles-scraper"><img alt="Run on Apify" src="https://img.shields.io/badge/Run%20on-Apify-DC2626?logo=apify&logoColor=white"></a>
7+
<img alt="Python" src="https://img.shields.io/badge/Python-3.11%2B-3776AB?logo=python&logoColor=white">
8+
<a href="LICENSE"><img alt="MIT license" src="https://img.shields.io/badge/license-MIT-0A0606"></a>
9+
<a href="https://github.com/devilscrapes/medium-user-articles-scraper-python/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/devilscrapes/medium-user-articles-scraper-python/actions/workflows/ci.yml/badge.svg"></a>
10+
</p>
11+
12+
This repository is a compact, tested Python client for the **[Medium Author RSS Scraper](https://apify.com/DevilScrapes/medium-user-articles-scraper)** on Apify. Give it a list of Medium handles or publication feeds, for instance a company's engineering blog, and it returns their most recent articles as a table: headline, URL, byline, tags, publication time and, if you want it, the HTML body. Medium's feeds are fetched and parsed on Apify, so there is no XML handling in your code and no request pacing to tune. Locally you get `results.json` and `results.csv`, which is what a pandas DataFrame, a Jupyter notebook or a nightly cron archive wants to start from.
13+
14+
> **Actor:** [Medium Author RSS Scraper](https://apify.com/DevilScrapes/medium-user-articles-scraper) · **Publisher:** [Devil Scrapes](https://apify.com/DevilScrapes) · **Also in JavaScript:** [medium-user-articles-scraper-javascript](https://github.com/devilscrapes/medium-user-articles-scraper-javascript)
15+
16+
## 🔥 What you get
17+
18+
- **Accounts and publications in one list**: a handle such as `@netflixtechblog` (the `@` is optional) is turned into its Medium feed address, and a full feed URL like `https://medium.com/feed/netflix-techblog` is fetched as given.
19+
- **Rows that carry the essentials**: headline, article URL, the feed's guid as `article_id`, the byline, the tags the writer picked and the publication time converted to ISO-8601 in UTC.
20+
- **Article HTML on request**: with `includeContent` on, `content_html` holds the body from the feed, or its summary when that is all the feed provides; switch it off and the column is `null`, which keeps rows small.
21+
- **A per-account cap**: `maxArticlesPerUser` (1 to 50) trims each feed, although Medium itself rarely lists more than the ten newest posts.
22+
- **One dead handle does not stop the batch**: a feed that answers with an error is logged and skipped, and the remaining accounts are still delivered.
23+
- **Billed per article written**: a start event plus a fee per row, so a handle whose feed is empty or unreachable costs nothing extra.
24+
25+
## 🎯 Use cases
26+
27+
### Benchmark engineering blogs
28+
29+
Track a dozen company tech blogs and load the rows into a DataFrame. Monthly article counts per account and the most frequent tags show who publishes steadily and which topics, from machine learning to infrastructure, each team writes about.
30+
31+
### Build a reading corpus for analysis
32+
33+
Switch `includeContent` on and the HTML bodies give you text for keyword extraction, topic modelling or summarisation experiments in a notebook. Strip the markup with your parser of choice and keep `link` as the citation.
34+
35+
### Archive a publication over time
36+
37+
Medium's feeds only expose recent posts, so history has to be collected as it happens. Run the script daily from cron, append each `results.csv` to a growing file and deduplicate on the article URL to build the long archive the feed alone never shows.
38+
39+
### Monitor competitors' content calendars
40+
41+
Product marketers can follow rival company accounts and see within a day when a launch post, a case study or a hiring article appears, along with the tags it was filed under.
42+
43+
## ⚡ Quick start
44+
45+
You need an Apify account and an API token from [Console → Settings → Integrations](https://console.apify.com/settings/integrations). Apify's free plan includes monthly usage credit; this Actor's event prices are listed under Pricing below.
46+
47+
```bash
48+
git clone https://github.com/devilscrapes/medium-user-articles-scraper-python.git
49+
cd medium-user-articles-scraper-python
50+
python -m venv .venv && source .venv/bin/activate
51+
pip install -r requirements.txt
52+
export APIFY_TOKEN=your_token_here
53+
python main.py # uses input.json
54+
python main.py my-input.json --out data --max-charge 2
55+
```
56+
57+
The script starts the [Medium Author RSS Scraper](https://apify.com/DevilScrapes/medium-user-articles-scraper), waits for it to finish, prints a link to the run in Apify Console and saves every result to `results.json` and `results.csv`. It exits with code 1 and the run's status message if the run fails.
58+
59+
## 🧩 The core call
60+
61+
Most of `main.py` deals with arguments, output files and failure reporting. Underneath, the integration is just five lines: `apify-client` launches the Actor, waits for the run to end and iterates over every dataset item.
62+
63+
```python
64+
from apify_client import ApifyClient
65+
66+
client = ApifyClient("YOUR_APIFY_TOKEN")
67+
run = client.actor("DevilScrapes/medium-user-articles-scraper").call(run_input={'usernames': ['@netflixtechblog', '@AirbnbEng'], 'maxArticlesPerUser': 5, 'includeContent': False, 'proxyConfiguration': {'useApifyProxy': True}})
68+
for item in client.dataset(run.default_dataset_id).iterate_items():
69+
print(item)
70+
```
71+
72+
## 📥 Input
73+
74+
`input.json` goes to the Actor as is. The sample reads two official engineering accounts, Netflix Technology Blog and Airbnb Engineering, keeps five posts from each and leaves the HTML out, so the test run writes at most ten small rows. Add handles or publication feed URLs to `usernames` to watch more sources.
75+
76+
```json
77+
{
78+
"usernames": [
79+
"@netflixtechblog",
80+
"@AirbnbEng"
81+
],
82+
"maxArticlesPerUser": 5,
83+
"includeContent": false,
84+
"proxyConfiguration": {
85+
"useApifyProxy": true
86+
}
87+
}
88+
```
89+
90+
| Field | Type | Required | Default | Allowed values |
91+
|---|---|---|---|---|
92+
| `usernames` | array | yes | `["@medium"]` | — |
93+
| `maxArticlesPerUser` | integer | no | `10` | — |
94+
| `includeContent` | boolean | no | `true` | — |
95+
| `proxyConfiguration` | object | no | `{"useApifyProxy": true}` | — |
96+
97+
## 📤 Output
98+
99+
Every row is one article. `username` repeats the handle you passed, or the publication slug when you gave a feed URL. Article links come straight from the feed and include a `?source=rss` tracking query, so cut everything after `?` before using them as keys. `published` is `null` if a feed entry carries no readable date. On publication feeds, `author` is the individual writer of that piece rather than the publication.
100+
101+
| Field | Type | Always present |
102+
|---|---|---|
103+
| `username` | string | yes |
104+
| `article_id` | string or null | no |
105+
| `title` | string | yes |
106+
| `link` | string | yes |
107+
| `author` | string | no |
108+
| `content_html` | string or null | no |
109+
| `categories` | array | no |
110+
| `published` | string or null | no |
111+
| `scraped_at` | string | no |
112+
113+
## 💰 Pricing
114+
115+
| Event | Price (USD) |
116+
|---|---|
117+
| Actor start | $0.2 |
118+
| Result emitted | $0.002 |
119+
120+
Prices as configured on 2026-08-20. The [Actor page](https://apify.com/DevilScrapes/medium-user-articles-scraper) always shows the current price. Runs are billed to the Apify account that owns the token. Use `--max-charge` to cap what a single run can cost.
121+
122+
## 🛠️ Working with the data in Python
123+
124+
The pandas recipe below (`pip install pandas`) cleans the links, counts articles per account per month, lists each account's five most used tags in `top_tags.csv` and writes a tidy `articles.csv` sorted newest first.
125+
126+
```python
127+
import json
128+
129+
import pandas as pd
130+
131+
posts = pd.DataFrame(json.load(open("results.json", encoding="utf-8")))
132+
posts["published"] = pd.to_datetime(posts["published"], utc=True, errors="coerce")
133+
posts["url"] = posts["link"].str.split("?").str[0] # drop the feed's tracking query
134+
posts = posts.drop_duplicates("url")
135+
136+
cadence = (
137+
posts.dropna(subset=["published"])
138+
.assign(month=lambda df: df["published"].dt.strftime("%Y-%m"))
139+
.groupby(["username", "month"])
140+
.size()
141+
.unstack(fill_value=0)
142+
)
143+
print(cadence.to_string())
144+
145+
tags = posts.explode("categories").dropna(subset=["categories"])
146+
top_tags = tags.groupby("username")["categories"].value_counts().groupby(level=0).head(5)
147+
top_tags.rename("articles").to_csv("top_tags.csv")
148+
print(top_tags.to_string())
149+
150+
latest = posts.sort_values("published", ascending=False)
151+
latest[["username", "published", "title", "url"]].to_csv("articles.csv", index=False)
152+
```
153+
154+
## ❓ FAQ
155+
156+
### Do I need a Medium account, an integration token or a paid membership?
157+
158+
No. Only an Apify API token. The Actor reads the RSS feeds Medium publishes for every public account and publication, which do not require signing in.
159+
160+
### Why do I get only about ten articles per account?
161+
162+
Each Medium feed lists the newest posts only, typically ten. Raising `maxArticlesPerUser` cannot go past what the feed contains. To keep older items, collect on a schedule and store every run's output, as the archive use case above describes.
163+
164+
### How do I add a publication rather than a person's account?
165+
166+
Pass its feed URL, `https://medium.com/feed/<publication-slug>`. A bare handle is always expanded to `medium.com/feed/@handle`, which is the format for accounts. Profile page URLs are fetched as they are and are not feeds, so use a handle or a feed address instead.
167+
168+
### Does it return claps, responses or paywalled text?
169+
170+
No. Those numbers are not part of the feed, and member-only stories are delivered only as far as the feed itself includes them. The Actor reads the feed and nothing else.
171+
172+
## 🧯 Troubleshooting
173+
174+
- **`Set APIFY_TOKEN first`**: export the token in the same shell, or put it in a local `.env` file your shell loads. Never commit it.
175+
- **Run finished with `FAILED` or `ABORTED`**: open the printed Console link; the run log says why. Input validation errors name the field.
176+
- **Fewer rows than expected**: a `--max-charge` cap also limits how many results can be charged. Raise the cap or lower the limits in your input.
177+
- **Something looks wrong in the data**: open an issue on the [Actor page](https://apify.com/DevilScrapes/medium-user-articles-scraper) so the maintainers see it.
178+
179+
## 🔗 More from Devil Scrapes
180+
181+
- [Medium Author RSS Scraper on Apify Store](https://apify.com/DevilScrapes/medium-user-articles-scraper)
182+
- [JavaScript version of this example](https://github.com/devilscrapes/medium-user-articles-scraper-javascript)
183+
- [All Devil Scrapes Actors](https://apify.com/DevilScrapes)
184+
- [Apify web scraping cookbook](https://github.com/devilscrapes/apify-web-scraping-cookbook)
185+
186+
Related Actors:
187+
188+
- [DEV.to Scraper — Articles by Tag & Author](https://apify.com/DevilScrapes/dev-to-articles-scraper) — the same kind of article feed from DEV.to, by tag or author.
189+
- [Substack Newsletter Scraper](https://apify.com/DevilScrapes/substack-newsletter-scraper) — full Substack archives with reaction and word counts.
190+
- [Hacker News Scraper](https://apify.com/DevilScrapes/hacker-news-scraper) — see which of those posts reached Hacker News and how they scored.
191+
192+
---
193+
194+
MIT licensed example code. The Actor itself runs on [Apify](https://apify.com/DevilScrapes/medium-user-articles-scraper) and is maintained by [Devil Scrapes](https://apify.com/DevilScrapes). Not affiliated with Medium or A Medium Corporation.

‎assets/banner.svg‎

Lines changed: 39 additions & 0 deletions
Loading

‎input.json‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"usernames": [
3+
"@netflixtechblog",
4+
"@AirbnbEng"
5+
],
6+
"maxArticlesPerUser": 5,
7+
"includeContent": false,
8+
"proxyConfiguration": {
9+
"useApifyProxy": true
10+
}
11+
}

‎main.py‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
"""Run an Apify Actor and save its results as JSON and CSV.
2+
3+
Actor: Medium Author RSS Scraper
4+
5+
Actor page: https://apify.com/DevilScrapes/medium-user-articles-scraper
6+
"""
7+
8+
from __future__ import annotations
9+
10+
import argparse
11+
import json
12+
import os
13+
import sys
14+
from decimal import Decimal
15+
from pathlib import Path
16+
17+
from apify_client import ApifyClient
18+
19+
ACTOR_ID = "DevilScrapes/medium-user-articles-scraper"
20+
ACTOR_TITLE = "Medium Author RSS Scraper"
21+
TERMINAL_OK = "SUCCEEDED"
22+
23+
24+
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
25+
parser = argparse.ArgumentParser(description=f"Run the {ACTOR_TITLE} on Apify.")
26+
parser.add_argument("input", nargs="?", default="input.json", help="Actor input JSON file (default: input.json)")
27+
parser.add_argument("--out", default=".", help="Folder for results.json and results.csv (default: here)")
28+
parser.add_argument("--max-charge", type=Decimal, help="Stop the run once it has cost this many USD")
29+
parser.add_argument("--memory", type=int, help="Run memory in MB (default: the Actor's own setting)")
30+
return parser.parse_args(argv)
31+
32+
33+
def load_input(path: str) -> dict:
34+
try:
35+
data = json.loads(Path(path).read_text(encoding="utf-8"))
36+
except FileNotFoundError:
37+
sys.exit(f"Input file not found: {path}")
38+
except json.JSONDecodeError as error:
39+
sys.exit(f"Input file {path} is not valid JSON: {error}")
40+
if not isinstance(data, dict):
41+
sys.exit(f"Input file {path} must contain a JSON object")
42+
return data
43+
44+
45+
def main(argv: list[str] | None = None) -> int:
46+
args = parse_args(argv)
47+
token = os.environ.get("APIFY_TOKEN")
48+
if not token:
49+
sys.exit("Set APIFY_TOKEN first. Get a token at https://console.apify.com/settings/integrations")
50+
51+
run_input = load_input(args.input)
52+
client = ApifyClient(token, api_url=os.environ.get("APIFY_API_URL", "https://api.apify.com"))
53+
54+
print(f"Starting {ACTOR_ID} ...")
55+
run = client.actor(ACTOR_ID).call(
56+
run_input=run_input,
57+
max_total_charge_usd=args.max_charge,
58+
memory_mbytes=args.memory,
59+
)
60+
if run is None:
61+
sys.exit("The run could not be started.")
62+
63+
print(f"Run {run.id} finished: {run.status}")
64+
print(f"Details: https://console.apify.com/actors/runs/{run.id}")
65+
if run.status != TERMINAL_OK:
66+
print(run.status_message or "No status message.", file=sys.stderr)
67+
return 1
68+
69+
dataset = client.dataset(run.default_dataset_id)
70+
out = Path(args.out)
71+
out.mkdir(parents=True, exist_ok=True)
72+
items = json.loads(dataset.get_items_as_bytes(item_format="json", clean=True))
73+
(out / "results.json").write_text(json.dumps(items, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
74+
(out / "results.csv").write_bytes(dataset.get_items_as_bytes(item_format="csv", clean=True))
75+
76+
print(f"Saved {len(items)} items to {out / 'results.json'} and {out / 'results.csv'}")
77+
return 0
78+
79+
80+
if __name__ == "__main__":
81+
sys.exit(main())

0 commit comments

Comments
 (0)