Skip to content

Commit 70d4ec7

Browse files
committed
docs: update README — direct apply links, volunteer section, admin report, payments
- Add volunteer roles table with links to VOLUNTEER_ROLES.md and new issues - Add admin_report.py and admin-report.yml to project structure - Fix \"How It Works\" to reflect single daily scrape (not 7AM + 7PM) - Add Paystack + Ko-fi payment links and sponsorship offer - Clarify direct-apply-link behaviour in feature list and lifecycle
1 parent a422b1c commit 70d4ec7

1 file changed

Lines changed: 70 additions & 35 deletions

File tree

README.md

Lines changed: 70 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
> An open-source Python bot that automatically scrapes the internet for opportunities for Nigerian students — scholarships, fellowships, internships, bootcamps, and more. It updates a shared Google Spreadsheet and emails a **weekly digest** to subscribers every Sunday.
44
55
[![GitHub Issues](https://img.shields.io/github/issues/TechHub-Extensions/ScoutBot)](https://github.com/TechHub-Extensions/ScoutBot/issues)
6-
[![GitHub Stars](https://img.shields.io/github/stars/TechHub-Extensions/ScoutBot)](https://github.com/TechHub-Extensions/ScoutBot/stargazers)
6+
[![GitHub Stars](https://img.shields.io/github/stars/TechHub-Extensions/ScoutBot)](https://github.com/TechHub-Extensions/ScoutBot/stargazes)
77
[![Python 3.11](https://img.shields.io/badge/python-3.11-blue.svg)](https://www.python.org/)
88
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
99
[![ScoutBot Scrape](https://github.com/TechHub-Extensions/ScoutBot/actions/workflows/scoutbot.yml/badge.svg)](https://github.com/TechHub-Extensions/ScoutBot/actions/workflows/scoutbot.yml)
@@ -26,12 +26,12 @@ No app, no login, no fee. Fill the form once and you're on the list.
2626

2727
## What ScoutBot Does
2828

29-
- 🔍 **Scrapes 8+ live feeds daily** — Google News RSS (Nigeria + International), YouthHubAfrica, Reddit
29+
- 🔍 **Scrapes 8+ live feeds daily** — Google News RSS (Nigeria + International), YouthHubAfrica
30+
- 🔗 **Direct application links only** — every link is extracted from the article's "Apply Now" button; items with no findable apply link are dropped before entering the sheet
3031
- 📊 **Writes to two separate tabs**: Nigeria 🇳🇬 and International 🌍 — never mixed
31-
- 🤖 **AI quality scoring** — every new item scored 1–10 by Gemini; low-quality items dropped before the sheet
32-
- 🧹 **Auto-cleans daily**opportunities removed when closed, past deadline, or older than 23 days
32+
- 🤖 **AI quality scoring** — every new item scored 1–10 by Gemini 2.0 Flash; items scoring below 5 are dropped
33+
- 🧹 **Auto-cleans daily**entries removed when closed, past deadline, or older than 23 days
3334
- 📧 **One email per week** — Sunday digest with only opportunities added in the last 7 days
34-
- 🔗 **Direct application links** — links go to the actual apply page, not a blog post
3535
- 🚫 **Students only** — scholarships, fellowships, internships, bootcamps. No startup/VC content
3636
- ☁️ **Runs entirely on GitHub Actions** — no server, no Replit dependency, works 24/7 independently
3737

@@ -42,17 +42,17 @@ No app, no login, no fee. Fill the form once and you're on the list.
4242
ScoutBot uses **Google Gemini 2.0 Flash** to score and summarise every scraped opportunity before it enters the sheet.
4343

4444
```
45-
Scrape (30–40 items) → DedupePipeline → GeminiPipeline → SheetsPipeline
46-
(drop known) (score 1–10) (write tab)
47-
(drop < 5)
48-
(add AI blurb)
45+
Scrape → DedupePipeline → GeminiPipeline → SheetsPipeline
46+
(drop known) (score 1–10) (write to Nigeria/International tab)
47+
(drop < 5)
48+
(add AI blurb)
4949
```
5050

5151
**The AI does two things:**
5252
1. **Scores** each opportunity 1–10 for relevance to Nigerian students — items below 5 are dropped silently
5353
2. **Generates a 2-sentence blurb** that appears in the weekly email so subscribers instantly know if an opportunity is for them
5454

55-
**Why Gemini and not GPT-4 or Claude?**
55+
**Why Gemini and not GPT-4 or Claude?**
5656
Gemini 2.0 Flash is available on a free API tier with 1,500 requests/day — enough for ScoutBot's daily volume (typically 5–15 new items) with zero cost.
5757

5858
**Full technical documentation:** [`docs/AI_IMPLEMENTATION.md`](./docs/AI_IMPLEMENTATION.md)
@@ -64,60 +64,66 @@ Gemini 2.0 Flash is available on a free API tier with 1,500 requests/day — eno
6464
```
6565
Day 0: Opportunity posted on the web
6666
Day 0–3: Spider picks it up (MAX_POST_AGE_DAYS = 3)
67-
Day 1–3: Written to Google Sheet with Date Added stamp
67+
→ follows article to find direct "Apply Now" link
68+
→ Gemini scores it 1–10
69+
→ if score ≥ 5: written to Google Sheet
70+
Day 7: Included in Sunday weekly email digest
6871
Day 23: Hard-removed from sheet by cleanup.py (STALE_DAYS = 23)
6972
```
7073

71-
Every entry in the sheet is guaranteed to be **less than 23 days old** and from a **post no older than 3 days** when it was scraped. Nothing lingers.
72-
73-
---
74-
75-
## Why ScoutBot Exists
76-
77-
Opportunities for Nigerian students are scattered across dozens of websites with no single reliable source. ScoutBot runs quietly in the background, finds new opportunities as they appear, logs them into a shared spreadsheet, and delivers them straight to people's inboxes — once a week, clean and fresh.
78-
79-
**This is a bot, not a web app.** No dashboard, no login page, no frontend — just an automated Python system that works independently of any platform.
80-
8174
---
8275

8376
## How It Works
8477

8578
```
86-
Every day at 07:00 WAT (GitHub Actions):
87-
1. scrapy crawl opportunities → scrapes 8 RSS feeds → scores with Gemini → writes to Nigeria / International tab
79+
Every day at 07:00 WAT (GitHub Actions — scoutbot.yml):
80+
1. scrapy crawl opportunities → scrapes RSS feeds → extracts apply links
81+
→ scores each with Gemini → writes to Nigeria / International tab
8882
2. python run.py --cleanup → removes entries older than 23 days or with past deadlines
8983
90-
Every Sunday 10:00 WAT (GitHub Actions):
84+
Every Sunday 10:00 WAT (GitHub Actions — digest.yml):
9185
3. python run.py --notify → sends weekly digest (last 7 days only) to all subscribers
92-
4. python telegram_notify.py → sends Telegram channel digest
86+
87+
1st of every month 07:30 WAT (GitHub Actions — admin-report.yml):
88+
4. python admin_report.py → sends monthly stats report to project lead
9389
```
9490

9591
---
9692

93+
## Why ScoutBot Exists
94+
95+
Opportunities for Nigerian students are scattered across dozens of websites with no single reliable source. ScoutBot runs quietly in the background, finds new opportunities as they appear, extracts direct application links, scores them with AI, and delivers them straight to people's inboxes — once a week, clean and fresh.
96+
97+
**This is a bot, not a web app.** No dashboard, no login page, no frontend — just an automated Python system that works independently of any platform.
98+
99+
---
100+
97101
## Project Structure
98102

99103
```
100104
ScoutBot/
101105
├── scoutbot/
102106
│ ├── spiders/
103-
│ │ └── opportunities_spider.py ← All scraping logic + RSS parsing
107+
│ │ └── opportunities_spider.py ← All scraping + RSS + apply link extraction
104108
│ ├── pipelines.py ← DedupePipeline → GeminiPipeline → SheetsPipeline
105109
│ ├── items.py ← Scrapy item definition
106110
│ └── settings.py ← Scrapy settings + pipeline order
107111
├── notify.py ← Weekly email digest sender
108112
├── cleanup.py ← Removes expired sheet entries (23-day cap)
113+
├── admin_report.py ← Monthly stats email to project lead
109114
├── welcome.py ← Welcome email for new subscribers
110-
├── announce.py ← One-time announcement emails
111115
├── run.py ← CLI entry point
112116
├── requirements.txt
113117
├── .env.example ← Copy to .env and fill in credentials
114118
├── docs/
115-
│ └── AI_IMPLEMENTATION.md ← Full Gemini AI pipeline documentation
119+
│ ├── AI_IMPLEMENTATION.md ← Full Gemini AI pipeline documentation
120+
│ └── VOLUNTEER_ROLES.md ← Step-by-step guide for all volunteer roles
116121
├── .github/
117122
│ └── workflows/
118123
│ ├── scoutbot.yml ← Daily 07:00 WAT scrape
119124
│ ├── digest.yml ← Sunday 10:00 WAT weekly email
120-
│ ├── welcome.yml ← Monthly welcome email
125+
│ ├── admin-report.yml ← Monthly 07:30 WAT stats report
126+
│ ├── welcome.yml ← Monthly welcome email to new subscribers
121127
│ └── pytest.yml ← CI tests on every push/PR
122128
├── CHANGELOG.md ← Full project history
123129
├── CONTRIBUTING.md
@@ -127,6 +133,29 @@ ScoutBot/
127133

128134
---
129135

136+
## 🤝 Volunteer — Help Us Grow ScoutBot
137+
138+
ScoutBot is maintained by a small founding team and open-source contributors. Several roles are open to volunteers — no application required to get started.
139+
140+
**[→ Full volunteer guide: docs/VOLUNTEER_ROLES.md](./docs/VOLUNTEER_ROLES.md)**
141+
142+
| Role | Time/week | Skills needed | How to start |
143+
|------|-----------|---------------|--------------|
144+
| **Source Hunter** | 1–3 hrs | Browser, no coding | [Open issue: "volunteer: Source Hunter"](https://github.com/TechHub-Extensions/ScoutBot/issues/new) |
145+
| **Community Ambassador** | 1–2 hrs | Writing, social media | Start immediately — no approval needed |
146+
| **Issue Triager** | 30 min | Basic GitHub | Open an intro issue first |
147+
| **Data Curator** | 1–2 hrs | Google Sheets | Open an intro issue first |
148+
| **Documentation Writer** | 2–4 hrs | Markdown | Open an intro issue first |
149+
| **Source Monitor** | 30 min | Browser + GitHub | [Open issue: "volunteer: Source Monitor"](https://github.com/TechHub-Extensions/ScoutBot/issues/new) |
150+
| **Email Designer** | 2–5 hrs | HTML + CSS | Open an intro issue first |
151+
| **PR Reviewer** | 1–2 hrs | Python | Open an intro issue first |
152+
| **QA Tester** | 2–4 hrs | Python, CLI | Open an intro issue first |
153+
| **Subscriber Support** | 30 min | Gmail | Open an intro issue first |
154+
155+
To apply: open a GitHub issue titled `volunteer: interested in [Role Name]`. Include your name, location, why you want the role, and available hours.
156+
157+
---
158+
130159
## Run Locally
131160

132161
```bash
@@ -136,9 +165,11 @@ pip install -r requirements.txt
136165
cp .env.example .env
137166
# Fill in .env with your credentials (see ENGINEERING.md)
138167

139-
python run.py --scrape # Scrape only (no email)
140-
python run.py --cleanup # Remove expired entries only
141-
python run.py --notify # Send digest email only
168+
python run.py --scrape # Scrape only (no email)
169+
python run.py --cleanup # Remove expired entries only
170+
python run.py --notify # Send digest email only
171+
python run.py --dry-run # Build email preview without sending (writes email_preview.html)
172+
python admin_report.py # Send monthly stats report manually
142173
```
143174

144175
### Required `.env` variables
@@ -177,11 +208,14 @@ Once secrets are set, the bot runs on schedule with **no Replit, no VPS, no cron
177208

178209
## 💛 Support ScoutBot
179210

180-
ScoutBot is open source and free for every user. We are Nigerian students building this with zero budget.
211+
ScoutBot is open source and free for every subscriber. We are Nigerian students building this with zero budget.
212+
213+
**[→ Support on Paystack (₦1,000)](https://paystack.com/pay/scoutbot)**
214+
**[→ Support on Ko-fi (international)](https://ko-fi.com/scoutbot)**
181215

182-
**Fundraising goal:** ₦2,000,000 (~$1,500 USD) for Twitter/X advertising to reach 4,000 Nigerian students.
216+
Organisations can sponsor a featured placement in the Sunday digest for **₦5,000/placement** — reaches 500+ students directly. Email [kamsirichard1960@gmail.com](mailto:kamsirichard1960@gmail.com?subject=ScoutBot%20Sponsorship) to enquire.
183217

184-
📄 **[ScoutBot Fundraising Brief (Google Doc)](https://docs.google.com/document/d/1SqxaAg4tvuWp3LgGzqSSSw4_bxBWHmgmrQ9IyyKHtE8/edit)**
218+
📄 **[ScoutBot Fundraising Brief (Google Doc)](https://docs.google.com/document/d/1SqxaAg4tvuWp3LgGzqSSSw4_bxBWHmgmrQ9IyyKHtE8/edit)**
185219

186220
---
187221

@@ -204,6 +238,7 @@ ScoutBot is open source and welcomes contributions from developers of all skill
204238
- 🐛 **[Open an Issue](https://github.com/TechHub-Extensions/ScoutBot/issues)** — report a broken source, a bug, or a feature idea
205239
- 🔀 **Fork and submit a PR** — add sources, fix bugs, improve email design
206240
- 📣 **Share** with Nigerian student WhatsApp groups, Discord servers, Twitter/X
241+
- 🤝 **Volunteer** — see the [volunteer guide](./docs/VOLUNTEER_ROLES.md) for non-coding roles
207242

208243
**Ready to code?** Start with issues labelled [`good first issue`](https://github.com/TechHub-Extensions/ScoutBot/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
209244

0 commit comments

Comments
 (0)