Commit 341f9e6
committed
feat: spider v6 — 21 curated direct org pages, no Google News
Root cause confirmed: both Google News CBMi links AND Nigerian news
site RSS feeds are inaccessible from GitHub Actions (Cloudflare blocks
Azure datacenter IPs). The only reliable sources are direct org pages
and scholars4dev.com (no Cloudflare).
Changes:
- Removed Google News RSS entirely
- Removed Nigerian news site RSS (blocked from GitHub Actions)
- Expanded direct org list from 8 → 21 pages covering:
Nigerian govt: PTDF, NDDC, NNPC, SNEPCo, NDIC
Nigerian corporate: MTN Foundation, Tony Elumelu, Dangote, Access Bank
International: Commonwealth, Chevening, Fulbright, Mastercard Fn,
World Bank (SSL bypass added), AfDB, AU, UNDP,
British Council NG, YouthHub Africa, UNICEF, UN
- Added meta['dont_verify_ssl']=True for worldbank.org (TLS hostname
mismatch on GitHub Actions Azure DNS)
- Kept scholars4dev.com RSS as bonus (accessible, real article URLs)
- open_signals check: skip page if no apply/deadline signals found
- errback writes item with hardcoded apply_link so org is represented
even if the page is temporarily down1 parent 4b62fb1 commit 341f9e6
1 file changed
Lines changed: 295 additions & 328 deletions
0 commit comments