Skip to content

Commit 0d58689

Browse files
devilscrapesdevilscrapes
committed
Lemmy scraper in Python
0 parents  commit 0d58689

12 files changed

Lines changed: 605 additions & 0 deletions

File tree

‎.env.example‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Your Apify API token: https://console.apify.com/settings/integrations
2+
APIFY_TOKEN=

‎.github/workflows/ci.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
pull_request:
6+
7+
permissions:
8+
contents: read
9+
10+
jobs:
11+
test:
12+
runs-on: ubuntu-latest
13+
strategy:
14+
matrix:
15+
python-version: ["3.11", "3.13"]
16+
steps:
17+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
18+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
19+
with:
20+
python-version: ${{ matrix.python-version }}
21+
- run: pip install -r requirements-dev.txt
22+
- run: ruff check .
23+
- run: ruff format --check .
24+
- run: pytest -q

‎.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
.env
2+
results.json
3+
results.csv
4+
__pycache__/
5+
.venv/
6+
.pytest_cache/
7+
.ruff_cache/

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Devil Scrapes
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
<p align="center"><a href="https://apify.com/DevilScrapes/lemmy-community-scraper"><img src="assets/banner.svg" alt="Lemmy Scraper — Posts, Comments &amp; Community Data — Python example" width="100%"></a></p>
2+
3+
# Lemmy Scraper in Python: community posts and comment threads for pandas
4+
5+
<p align="center">
6+
<a href="https://apify.com/DevilScrapes/lemmy-community-scraper"><img alt="Run on Apify" src="https://img.shields.io/badge/Run%20on-Apify-DC2626?logo=apify&logoColor=white"></a>
7+
<img alt="Python" src="https://img.shields.io/badge/Python-3.11%2B-3776AB?logo=python&logoColor=white">
8+
<a href="LICENSE"><img alt="MIT license" src="https://img.shields.io/badge/license-MIT-0A0606"></a>
9+
<a href="https://github.com/devilscrapes/lemmy-scraper-python/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/devilscrapes/lemmy-scraper-python/actions/workflows/ci.yml/badge.svg"></a>
10+
</p>
11+
12+
Here you will find a short Python script, covered by tests, that runs the **[Lemmy Community Scraper](https://apify.com/DevilScrapes/lemmy-community-scraper)** on Apify. Name a Lemmy server and one of its public communities, and you receive that community's posts, sorted the way you ask, with titles, bodies or links, vote counts and publish times; switch on comments and the replies arrive in the same table with their thread position. Lemmy is federated, so the same community can be read through different servers; this client reads it through whichever one you name. The API calls happen on Apify, and your machine only collects `results.json` and `results.csv`, ready for a DataFrame, a notebook or a corpus-building job.
13+
14+
> **Actor:** [Lemmy Scraper — Posts, Comments & Community Data](https://apify.com/DevilScrapes/lemmy-community-scraper) · **Publisher:** [Devil Scrapes](https://apify.com/DevilScrapes) · **Also in JavaScript:** [lemmy-scraper-javascript](https://github.com/devilscrapes/lemmy-scraper-javascript)
15+
16+
## 🔥 What you get
17+
18+
- **Any instance, any public community**: point `instanceUrl` at lemmy.world, lemmy.ml or a niche server, and name the community either locally (`asklemmy`) or with its home server (`memes@lemmy.world`).
19+
- **Posts and comments share one table**: `row_type` separates them, and every comment row repeats its community and post columns, so a CSV opened on its own still says where each reply lives.
20+
- **Thread structure kept**: Lemmy's dotted `comment_path` is stored as is, and `comment_parent_id` is derived from it, so a reply can be attached to its parent without another request.
21+
- **Sixteen sort orders**: from `Hot` and `New` to time-boxed tops such as `TopDay` or `TopYear`. The old bare `Top` value is refused up front, since current Lemmy servers reject it.
22+
- **Two caps you control**: `maxPosts` limits post rows and `maxCommentsPerPost` limits replies per post, with separate per-row prices for each kind.
23+
- **Errors that say what went wrong**: a misspelled community fails the run with the server's own error, a run that yields nothing exits as failed, and busy-server responses are retried with backoff first.
24+
25+
## 🎯 Use cases
26+
27+
### Measure how a community engages week by week
28+
29+
Pull `TopWeek` every Monday from cron, append the rows and chart score, upvote ratio and comment volume over time. Because subscriber and post counts ride along on each row, growth of the community itself is in the same file.
30+
31+
### Assemble a threaded text corpus
32+
33+
Comment rows carry the text, score and position in the tree. Researchers working on discourse, toxicity or summarisation can rebuild conversations from parent IDs and keep the post title as context for every reply.
34+
35+
### Compare one topic across instances
36+
37+
Run the same federated name, such as `technology@lemmy.world`, against two servers and diff the results. Different instances may have fetched different slices of a remote community, and the per-row `instance_url` keeps the sources apart.
38+
39+
### Study the Reddit-alternative migration
40+
41+
Lemmy grew largely from people leaving Reddit, which makes its communities a natural sample. Post dates, author account creation dates and bot flags make it possible to separate early adopters from later arrivals in a notebook.
42+
43+
## ⚡ Quick start
44+
45+
You need an Apify account and an API token from [Console → Settings → Integrations](https://console.apify.com/settings/integrations). Apify's free plan includes monthly usage credit; this Actor's event prices are listed under Pricing below.
46+
47+
```bash
48+
git clone https://github.com/devilscrapes/lemmy-scraper-python.git
49+
cd lemmy-scraper-python
50+
python -m venv .venv && source .venv/bin/activate
51+
pip install -r requirements.txt
52+
export APIFY_TOKEN=your_token_here
53+
python main.py # uses input.json
54+
python main.py my-input.json --out data --max-charge 2
55+
```
56+
57+
The script starts the [Lemmy Scraper — Posts, Comments & Community Data](https://apify.com/DevilScrapes/lemmy-community-scraper), waits for it to finish, prints a link to the run in Apify Console and saves every result to `results.json` and `results.csv`. It exits with code 1 and the run's status message if the run fails.
58+
59+
## 🧩 The core call
60+
61+
Reduced to essentials, the integration looks like this. The official `apify-client` package starts the run, waits on it and then streams the dataset back item by item.
62+
63+
```python
64+
from apify_client import ApifyClient
65+
66+
client = ApifyClient("YOUR_APIFY_TOKEN")
67+
run = client.actor("DevilScrapes/lemmy-community-scraper").call(run_input={'instanceUrl': 'https://lemmy.world', 'communityName': 'asklemmy', 'sort': 'TopWeek', 'maxPosts': 3, 'includeComments': True, 'maxCommentsPerPost': 2})
68+
for item in client.dataset(run.default_dataset_id).iterate_items():
69+
print(item)
70+
```
71+
72+
## 📥 Input
73+
74+
`main.py` passes `input.json` through untouched. The sample reads the top three `asklemmy` posts of the past week on lemmy.world, plus up to two comments on each, so at most nine rows are billed. Raise `maxPosts` for a real study; turn `includeComments` off when titles and scores are all you need.
75+
76+
```json
77+
{
78+
"instanceUrl": "https://lemmy.world",
79+
"communityName": "asklemmy",
80+
"sort": "TopWeek",
81+
"maxPosts": 3,
82+
"includeComments": true,
83+
"maxCommentsPerPost": 2
84+
}
85+
```
86+
87+
| Field | Type | Required | Default | Allowed values |
88+
|---|---|---|---|---|
89+
| `instanceUrl` | string | yes | `"https://lemmy.world"` | — |
90+
| `communityName` | string | yes | `"asklemmy"` | — |
91+
| `sort` | string | no | `"Hot"` | `Active`, `Controversial`, `Hot`, `MostComments`, `New`, `NewComments`, `Old`, `Scaled`, `TopAll`, `TopDay`, `TopHour`, `TopMonth`, `TopSixHour`, `TopTwelveHour`, `TopWeek`, `TopYear` |
92+
| `maxPosts` | integer | no | `100` | — |
93+
| `includeComments` | boolean | no | `false` | — |
94+
| `maxCommentsPerPost` | integer | no | `50` | — |
95+
| `useProxy` | boolean | no | `false` | — |
96+
97+
## 📤 Output
98+
99+
Rows come in two kinds, told apart by `row_type`. On post rows the `comment_*` columns are `null`; on comment rows the `author_*` columns describe the commenter, not the post author. Timestamps are ISO 8601 in UTC.
100+
101+
| Field | Type | Always present |
102+
|---|---|---|
103+
| `row_type` | string | yes |
104+
| `instance_url` | string | yes |
105+
| `community_actor_id` | string | yes |
106+
| `community_name` | string | yes |
107+
| `community_title` | string | yes |
108+
| `community_description` | string or null | no |
109+
| `community_subscribers` | integer | yes |
110+
| `community_posts_count` | integer | yes |
111+
| `community_local` | boolean | yes |
112+
| `post_id` | integer | yes |
113+
| `post_ap_id` | string | yes |
114+
| `post_url` | string | yes |
115+
| `post_title` | string | yes |
116+
| `post_body` | string or null | no |
117+
| `post_external_url` | string or null | no |
118+
| `post_score` | integer | yes |
119+
| `post_upvotes` | integer | yes |
120+
| `post_downvotes` | integer | yes |
121+
| `post_comments_count` | integer | yes |
122+
| `post_published` | string | yes |
123+
| `post_updated` | string or null | no |
124+
| `post_nsfw` | boolean | yes |
125+
| `post_featured_community` | boolean | yes |
126+
| `post_locked` | boolean | yes |
127+
| `comment_id` | integer or null | no |
128+
| `comment_ap_id` | string or null | no |
129+
| `comment_path` | string or null | no |
130+
| `comment_content` | string or null | no |
131+
| `comment_score` | integer or null | no |
132+
| `comment_published` | string or null | no |
133+
| `comment_parent_id` | integer or null | no |
134+
| `author_actor_id` | string | yes |
135+
| `author_name` | string | yes |
136+
| `author_display_name` | string or null | no |
137+
| `author_bot_account` | boolean | yes |
138+
| `author_published` | string | yes |
139+
| `scraped_at` | string | yes |
140+
141+
## 💰 Pricing
142+
143+
| Event | Price (USD) |
144+
|---|---|
145+
| Actor start | $0.2 |
146+
| Post row emitted | $0.002 |
147+
| Comment row emitted | $0.001 |
148+
149+
Prices as configured on 2026-08-20. The [Actor page](https://apify.com/DevilScrapes/lemmy-community-scraper) always shows the current price. Runs are billed to the Apify account that owns the token. Use `--max-charge` to cap what a single run can cost.
150+
151+
## 🛠️ Working with the data in Python
152+
153+
This pandas sketch (`pip install pandas`) splits posts from comments, adds an upvote ratio and an age in hours to each post, and measures how deep the sampled comment threads go. Depth comes from counting the dots in `comment_path`, so 1 means a top-level reply.
154+
155+
```python
156+
import json
157+
158+
import pandas as pd
159+
160+
rows = pd.DataFrame(json.load(open("results.json", encoding="utf-8")))
161+
posts = rows[rows["row_type"] == "post"].copy()
162+
comments = rows[rows["row_type"] == "comment"].copy()
163+
164+
votes = (posts["post_upvotes"] + posts["post_downvotes"]).clip(lower=1)
165+
posts["upvote_ratio"] = posts["post_upvotes"] / votes
166+
published = pd.to_datetime(posts["post_published"], utc=True, format="ISO8601")
167+
posts["age_hours"] = (pd.Timestamp.now(tz="UTC") - published).dt.total_seconds() / 3600
168+
169+
comments["depth"] = comments["comment_path"].str.count(r"\.")
170+
threads = comments.groupby("post_id").agg(
171+
sampled=("comment_id", "count"),
172+
deepest=("depth", "max"),
173+
replies=("comment_parent_id", lambda s: s.notna().sum()),
174+
)
175+
report = posts.set_index("post_id").join(threads)
176+
cols = ["post_title", "post_score", "upvote_ratio", "post_comments_count", "sampled", "deepest", "age_hours"]
177+
print(report[cols].sort_values("post_score", ascending=False).round(2).to_string())
178+
```
179+
180+
## ❓ FAQ
181+
182+
### Do I need a Lemmy account or an API key?
183+
184+
No. Public communities are readable without logging in, and the Actor sends no credentials. The only secret in this repository is your Apify token.
185+
186+
### Which instance should I use for a remote community?
187+
188+
The community's home server is the safest choice, so for `memes@lemmy.world` use `https://lemmy.world`. Reading through another instance works only if that server has already federated the community, and it may hold fewer posts.
189+
190+
### Can one run cover several communities?
191+
192+
Not inside the Actor; it reads one community per run. From Python, loop over a list of names, call the Actor for each and concatenate the frames; `community_actor_id` keeps them distinguishable.
193+
194+
### Why do some comment counts not match `post_comments_count`?
195+
196+
`post_comments_count` is Lemmy's total for the post, while comment rows stop at `maxCommentsPerPost`. Treat the comment rows as a sample of the discussion unless the cap is above the total.
197+
198+
## 🧯 Troubleshooting
199+
200+
- **`Set APIFY_TOKEN first`**: export the token in the same shell, or put it in a local `.env` file your shell loads. Never commit it.
201+
- **Run finished with `FAILED` or `ABORTED`**: open the printed Console link; the run log says why. Input validation errors name the field.
202+
- **Fewer rows than expected**: a `--max-charge` cap also limits how many results can be charged. Raise the cap or lower the limits in your input.
203+
- **Something looks wrong in the data**: open an issue on the [Actor page](https://apify.com/DevilScrapes/lemmy-community-scraper) so the maintainers see it.
204+
205+
## 🔗 More from Devil Scrapes
206+
207+
- [Lemmy Scraper — Posts, Comments & Community Data on Apify Store](https://apify.com/DevilScrapes/lemmy-community-scraper)
208+
- [JavaScript version of this example](https://github.com/devilscrapes/lemmy-scraper-javascript)
209+
- [All Devil Scrapes Actors](https://apify.com/DevilScrapes)
210+
- [Apify web scraping cookbook](https://github.com/devilscrapes/apify-web-scraping-cookbook)
211+
212+
Related Actors:
213+
214+
- [Mastodon Posts Scraper — Hashtags & Accounts](https://apify.com/DevilScrapes/mastodon-posts-scraper) — the microblogging side of the fediverse, by hashtag or account.
215+
- [Bluesky Scraper — Feed Posts](https://apify.com/DevilScrapes/bluesky-feed-posts) — posts from public Bluesky custom feeds in a similar flat shape.
216+
- [Hacker News Scraper](https://apify.com/DevilScrapes/hacker-news-scraper) — threaded tech discussion from a single, centralised site.
217+
218+
---
219+
220+
MIT licensed example code. The Actor itself runs on [Apify](https://apify.com/DevilScrapes/lemmy-community-scraper) and is maintained by [Devil Scrapes](https://apify.com/DevilScrapes). Not affiliated with the Lemmy project or any Lemmy instance.

‎assets/banner.svg‎

Lines changed: 64 additions & 0 deletions
Loading

‎input.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"instanceUrl": "https://lemmy.world",
3+
"communityName": "asklemmy",
4+
"sort": "TopWeek",
5+
"maxPosts": 3,
6+
"includeComments": true,
7+
"maxCommentsPerPost": 2
8+
}

0 commit comments

Comments
 (0)