Skip to content

Commit 264f509

Browse files
devilscrapesdevilscrapes
committed
SocialBlade scraper in Python
0 parents  commit 264f509

12 files changed

Lines changed: 577 additions & 0 deletions

File tree

‎.env.example‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Your Apify API token: https://console.apify.com/settings/integrations
2+
APIFY_TOKEN=

‎.github/workflows/ci.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
pull_request:
6+
7+
permissions:
8+
contents: read
9+
10+
jobs:
11+
test:
12+
runs-on: ubuntu-latest
13+
strategy:
14+
matrix:
15+
python-version: ["3.11", "3.13"]
16+
steps:
17+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
18+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
19+
with:
20+
python-version: ${{ matrix.python-version }}
21+
- run: pip install -r requirements-dev.txt
22+
- run: ruff check .
23+
- run: ruff format --check .
24+
- run: pytest -q

‎.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
.env
2+
results.json
3+
results.csv
4+
__pycache__/
5+
.venv/
6+
.pytest_cache/
7+
.ruff_cache/

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Devil Scrapes
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
<p align="center"><a href="https://apify.com/DevilScrapes/socialblade-stats-scraper"><img src="assets/banner.svg" alt="SocialBlade Stats Scraper — Python example" width="100%"></a></p>
2+
3+
# SocialBlade Scraper in Python: cross-platform account growth for pandas
4+
5+
<p align="center">
6+
<a href="https://apify.com/DevilScrapes/socialblade-stats-scraper"><img alt="Run on Apify" src="https://img.shields.io/badge/Run%20on-Apify-DC2626?logo=apify&logoColor=white"></a>
7+
<img alt="Python" src="https://img.shields.io/badge/Python-3.11%2B-3776AB?logo=python&logoColor=white">
8+
<a href="LICENSE"><img alt="MIT license" src="https://img.shields.io/badge/license-MIT-0A0606"></a>
9+
<a href="https://github.com/devilscrapes/socialblade-scraper-python/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/devilscrapes/socialblade-scraper-python/actions/workflows/ci.yml/badge.svg"></a>
10+
</p>
11+
12+
This repository is a small, tested Python client for the **[SocialBlade Stats Scraper](https://apify.com/DevilScrapes/socialblade-stats-scraper)** on Apify. Feed it a list of accounts across YouTube, Twitch, Instagram and TikTok, and it returns one row per account with the audience size, the letter grade SocialBlade assigns, a rank breakdown and, if you ask for it, a short daily history of follower movement. The heavy lifting happens on Apify, so your script stays a few dozen lines long. Locally you end up with `results.json` and `results.csv`, ready for a DataFrame, a spreadsheet or a scheduled archive.
13+
14+
> **Actor:** [SocialBlade Stats Scraper](https://apify.com/DevilScrapes/socialblade-stats-scraper) · **Publisher:** [Devil Scrapes](https://apify.com/DevilScrapes) · **Also in JavaScript:** [socialblade-scraper-javascript](https://github.com/devilscrapes/socialblade-scraper-javascript)
15+
16+
## 🔥 What you get
17+
18+
- **Four networks, one schema**: a YouTube channel and a TikTok account land in the same columns, with `follower_count` meaning subscribers on YouTube and followers everywhere else.
19+
- **Name a target two ways**: a `platform` plus `handle` pair, or a full SocialBlade profile URL, which wins if both are present. Mix networks freely in one `targets` list.
20+
- **Daily history on request**: with `includeHistory` on, `growth_history` holds up to 15 dated snapshots of followers, plus views and content count where the network has them. Switch it off for lighter rows.
21+
- **Network-specific extras stay null elsewhere**: Instagram rows add engagement rate and average likes and comments, TikTok rows add lifetime likes and the verified flag, YouTube rows add total views.
22+
- **Counts are numbers**: figures SocialBlade delivers as text are converted to integers, so sorting and arithmetic work without cleanup.
23+
- **One dead handle does not end the run**: throttled or failed requests are retried up to five times with growing pauses on a new proxy session, and a target that still fails is logged and skipped.
24+
25+
## 🎯 Use cases
26+
27+
### Compare brand accounts across networks
28+
29+
Marketing analysts often need to know where a brand's audience actually grows. Run a company's YouTube, Instagram and TikTok handles together, normalise the daily history and the network where followers rise fastest stands out in one sorted table.
30+
31+
### Vet channels before a sponsorship budget is signed
32+
33+
Agencies can load a shortlist of candidate channels, check `grade`, the `ranks` dictionary and recent momentum, then drop accounts whose following has flattened. Doing this in a notebook leaves a reproducible record of why each channel made the cut.
34+
35+
### Build a longer history than SocialBlade shows
36+
37+
Each row carries at most 15 daily snapshots. Schedule `python main.py --out snapshots/$(date +%F)` from cron every week, append the rows and drop duplicate dates, and after a quarter you hold a continuous series for every account you track.
38+
39+
### Research the creator economy
40+
41+
Students and analysts studying platform dynamics can sample many accounts per network and compare grades, follower counts and growth rates. One schema across four platforms removes most of the reshaping work before the statistics start.
42+
43+
## ⚡ Quick start
44+
45+
You need an Apify account and an API token from [Console → Settings → Integrations](https://console.apify.com/settings/integrations). Apify's free plan includes monthly usage credit; this Actor's event prices are listed under Pricing below.
46+
47+
```bash
48+
git clone https://github.com/devilscrapes/socialblade-scraper-python.git
49+
cd socialblade-scraper-python
50+
python -m venv .venv && source .venv/bin/activate
51+
pip install -r requirements.txt
52+
export APIFY_TOKEN=your_token_here
53+
python main.py # uses input.json
54+
python main.py my-input.json --out data --max-charge 2
55+
```
56+
57+
The script starts the [SocialBlade Stats Scraper](https://apify.com/DevilScrapes/socialblade-stats-scraper), waits for it to finish, prints a link to the run in Apify Console and saves every result to `results.json` and `results.csv`. It exits with code 1 and the run's status message if the run fails.
58+
59+
## 🧩 The core call
60+
61+
Leave out argument parsing and file output and very little remains: a client, a call that waits for the run, and a loop over the stored rows.
62+
63+
```python
64+
from apify_client import ApifyClient
65+
66+
client = ApifyClient("YOUR_APIFY_TOKEN")
67+
run = client.actor("DevilScrapes/socialblade-stats-scraper").call(run_input={'targets': [{'platform': 'youtube', 'handle': 'mrbeast6000'}], 'includeHistory': True, 'maxItems': 1, 'proxyConfiguration': {'useApifyProxy': True, 'apifyProxyGroups': ['RESIDENTIAL']}})
68+
for item in client.dataset(run.default_dataset_id).iterate_items():
69+
print(item)
70+
```
71+
72+
## 📥 Input
73+
74+
The script hands `input.json` to the Actor untouched. The bundled file asks for one very large public YouTube channel, the target the Actor's own daily platform test uses, with history switched on and `maxItems` at 1, so a first run stays small. Replace it with your own list, mixing platforms freely, or pass a SocialBlade profile link as `url`.
75+
76+
```json
77+
{
78+
"targets": [
79+
{
80+
"platform": "youtube",
81+
"handle": "mrbeast6000"
82+
}
83+
],
84+
"includeHistory": true,
85+
"maxItems": 1,
86+
"proxyConfiguration": {
87+
"useApifyProxy": true,
88+
"apifyProxyGroups": [
89+
"RESIDENTIAL"
90+
]
91+
}
92+
}
93+
```
94+
95+
| Field | Type | Required | Default | Allowed values |
96+
|---|---|---|---|---|
97+
| `targets` | array | yes | `[{"platform": "youtube", "handle": "mrbeast6000"}]` | — |
98+
| `includeHistory` | boolean | no | `true` | — |
99+
| `maxItems` | integer | no | `100` | — |
100+
| `proxyConfiguration` | object | no | `{"useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"]}` | — |
101+
102+
## 📤 Output
103+
104+
You get one row per account that SocialBlade could answer for. `grade` is SocialBlade's own score passed through unchanged, and `ranks` is a nested object with overall, primary, views, country and category positions, any of which may be null. `growth_history` is a list of dictionaries with a `date` and the counts for that day, and it can be empty when SocialBlade keeps no history for an account.
105+
106+
| Field | Type | Always present |
107+
|---|---|---|
108+
| `platform` | string | yes |
109+
| `handle` | string | yes |
110+
| `channel_id` | string or null | no |
111+
| `display_name` | string or null | no |
112+
| `avatar_url` | string or null | no |
113+
| `follower_count` | integer | yes |
114+
| `view_count` | integer or null | no |
115+
| `content_count` | integer or null | no |
116+
| `grade` | string | yes |
117+
| `country` | string or null | no |
118+
| `ranks` | object or null | no |
119+
| `engagement_rate` | number or null | no |
120+
| `avg_likes` | number or null | no |
121+
| `avg_comments` | number or null | no |
122+
| `total_likes` | integer or null | no |
123+
| `verified` | boolean or null | no |
124+
| `growth_history` | array | yes |
125+
| `source_url` | string | yes |
126+
| `scraped_at` | string | yes |
127+
128+
## 💰 Pricing
129+
130+
| Event | Price (USD) |
131+
|---|---|
132+
| Actor start | $0.2 |
133+
| Result emitted | $0.001 |
134+
135+
Prices as configured on 2026-10-16. The [Actor page](https://apify.com/DevilScrapes/socialblade-stats-scraper) always shows the current price. Runs are billed to the Apify account that owns the token. Use `--max-charge` to cap what a single run can cost.
136+
137+
## 🛠️ Working with the data in Python
138+
139+
The pandas snippet below (`pip install pandas`) flattens every account's daily history into one long table, then measures each account's net gain, gain per day and percentage growth across the window it covers. The result is saved to `growth_window.csv`.
140+
141+
```python
142+
import json
143+
144+
import pandas as pd
145+
146+
rows = json.load(open("results.json", encoding="utf-8"))
147+
daily = pd.json_normalize(rows, record_path="growth_history", meta=["platform", "handle"])
148+
daily["date"] = pd.to_datetime(daily["date"], utc=True)
149+
daily = daily.sort_values(["platform", "handle", "date"])
150+
151+
window = daily.groupby(["platform", "handle"]).agg(
152+
first_day=("date", "min"),
153+
last_day=("date", "max"),
154+
start=("follower_count", "first"),
155+
end=("follower_count", "last"),
156+
)
157+
window["days"] = (window["last_day"] - window["first_day"]).dt.days.clip(lower=1)
158+
window["net_gain"] = window["end"] - window["start"]
159+
window["gain_per_day"] = (window["net_gain"] / window["days"]).round(1)
160+
window["growth_pct"] = (100 * window["net_gain"] / window["start"]).round(3)
161+
162+
print(window.sort_values("growth_pct", ascending=False).to_string())
163+
window.to_csv("growth_window.csv")
164+
```
165+
166+
## ❓ FAQ
167+
168+
### Do I need SocialBlade credentials or a paid SocialBlade plan?
169+
170+
No. The Actor reads the profile data SocialBlade shows without logging in, and the only credential your script needs is an Apify API token.
171+
172+
### Are SocialBlade's estimated earnings included?
173+
174+
No. Those estimates are filled in by the browser after the page loads and are not part of the data the Actor reads, so they are left out rather than guessed.
175+
176+
### Why is a target missing from my results?
177+
178+
A handle SocialBlade does not know, or a profile that kept failing after five attempts, produces no row; the run log names it. The run's status message reports how many targets succeeded out of how many were requested.
179+
180+
### How long can a single run take?
181+
182+
The Actor stops starting new targets after about 20 minutes and saves what it has. For very large lists, split them across several runs or schedules so every account is reached.
183+
184+
## 🧯 Troubleshooting
185+
186+
- **`Set APIFY_TOKEN first`**: export the token in the same shell, or put it in a local `.env` file your shell loads. Never commit it.
187+
- **Run finished with `FAILED` or `ABORTED`**: open the printed Console link; the run log says why. Input validation errors name the field.
188+
- **Fewer rows than expected**: a `--max-charge` cap also limits how many results can be charged. Raise the cap or lower the limits in your input.
189+
- **Something looks wrong in the data**: open an issue on the [Actor page](https://apify.com/DevilScrapes/socialblade-stats-scraper) so the maintainers see it.
190+
191+
## 🔗 More from Devil Scrapes
192+
193+
- [SocialBlade Stats Scraper on Apify Store](https://apify.com/DevilScrapes/socialblade-stats-scraper)
194+
- [JavaScript version of this example](https://github.com/devilscrapes/socialblade-scraper-javascript)
195+
- [All Devil Scrapes Actors](https://apify.com/DevilScrapes)
196+
- [Apify web scraping cookbook](https://github.com/devilscrapes/apify-web-scraping-cookbook)
197+
198+
Related Actors:
199+
200+
- [YouTube Channel Videos Scraper](https://apify.com/DevilScrapes/youtube-channel-scraper) — channel metadata and recent videos straight from YouTube.
201+
- [YouTube Shorts Sponsorship Signals Scraper](https://apify.com/DevilScrapes/youtube-shorts-sponsorship-signals) — spot sponsored Shorts before you pitch a channel.
202+
- [Patreon Creator Scraper](https://apify.com/DevilScrapes/patreon-creator-scraper) — patron counts and tiers for the creator side of the economy.
203+
204+
---
205+
206+
MIT licensed example code. The Actor itself runs on [Apify](https://apify.com/DevilScrapes/socialblade-stats-scraper) and is maintained by [Devil Scrapes](https://apify.com/DevilScrapes). Not affiliated with SocialBlade.

‎assets/banner.svg‎

Lines changed: 42 additions & 0 deletions
Loading

‎input.json‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
{
2+
"targets": [
3+
{
4+
"platform": "youtube",
5+
"handle": "mrbeast6000"
6+
}
7+
],
8+
"includeHistory": true,
9+
"maxItems": 1,
10+
"proxyConfiguration": {
11+
"useApifyProxy": true,
12+
"apifyProxyGroups": [
13+
"RESIDENTIAL"
14+
]
15+
}
16+
}

0 commit comments

Comments
 (0)