Skip to content

Commit 341f9e6

Browse files
committed
feat: spider v6 — 21 curated direct org pages, no Google News
Root cause confirmed: both Google News CBMi links AND Nigerian news site RSS feeds are inaccessible from GitHub Actions (Cloudflare blocks Azure datacenter IPs). The only reliable sources are direct org pages and scholars4dev.com (no Cloudflare). Changes: - Removed Google News RSS entirely - Removed Nigerian news site RSS (blocked from GitHub Actions) - Expanded direct org list from 8 → 21 pages covering: Nigerian govt: PTDF, NDDC, NNPC, SNEPCo, NDIC Nigerian corporate: MTN Foundation, Tony Elumelu, Dangote, Access Bank International: Commonwealth, Chevening, Fulbright, Mastercard Fn, World Bank (SSL bypass added), AfDB, AU, UNDP, British Council NG, YouthHub Africa, UNICEF, UN - Added meta['dont_verify_ssl']=True for worldbank.org (TLS hostname mismatch on GitHub Actions Azure DNS) - Kept scholars4dev.com RSS as bonus (accessible, real article URLs) - open_signals check: skip page if no apply/deadline signals found - errback writes item with hardcoded apply_link so org is represented even if the page is temporarily down
1 parent 4b62fb1 commit 341f9e6

1 file changed

Lines changed: 295 additions & 328 deletions

File tree

0 commit comments

Comments
 (0)