Slack Notifications When Low Server Disk Space (replace nagios)

[mekarpeles]

Feature Request

Problem / Opportunity

Proposal

Breakdown

Implementation Details (for maintainers)

Related files

Refer to this map of common Endpoints:
*

Requirements Checklist

Checklist of requirements that need to be satisfied in order for this issue to be closed:

• [ ]

Stakeholders

---

Instructions for Contributors

• Before creating a new branch or pushing up changes to a PR, please first run these commands to ensure your repository is up to date, as the pre-commit bot may add commits to your PRs upstream.

[openlibrary-bot]

Warning

This issue requires access to internal infrastructure, production systems, or institutional knowledge unavailable to community contributors. It can only be resolved by a maintainer or staff member.

⚠️ Contributors, this issue will be ready to work on once:

• Triage — maintainers will add Priority: * and Lead: @*
• Needs: Breakdown — issue lacks concrete implementation steps
• Needs: Staff / Internal — requires production access or institutional knowledge

---

Context

• The proposal isn't wired to any existing implementation yet. Two existing pieces of prior art are directly relevant:
  • scripts/monitoring/monitor.py is the continuous background monitoring framework already running on production servers (gunicorn workers, nginx logs, fail2ban jail counts, Solr updater lag, etc.). It currently reports everything to Graphite only — it has no Slack-notification path today. This is the natural home for a new disk-space job, following the same per-job pattern as monitor_fail2ban.
  • check_server_storage() in scripts/deployment/deploy.sh already checks free disk space (<2GB threshold) via SSH + df -B1 across servers — but only runs manually, once, at deploy time (added in Check server storage before olsystem deploy, too #12423 by @cdrini). It is not continuous and does not notify Slack.
• Slack-sending infrastructure already exists in the repo and doesn't need to be built from scratch: scripts/pr_slack_digest.py (send_slack_message), scripts/gh_scripts/issue_comment_bot.py, scripts/solr_builder/utils.sh's send_slack_message, and .github/workflows/pm_on_call_notification.yml (uses slackapi/slack-github-action + a SLACK_TOKEN secret) all post to Slack today.
• DB Archival Failing (out of space causing site-wide outage) #10082 (closed, P0 postmortem) is a concrete past incident where the DB ran out of disk space and took the entire site down — it's the kind of incident this proposal would catch earlier.
• No duplicate issue for a nagios replacement was found; Provisioning & Deployment Plan #4423 (Provisioning & Deployment Plan) mentions Nagios once in passing (an unrelated ImportBot migration checklist item) but doesn't cover this.

---

Issue Assessment

Open questions

• Does "replace nagios" mean fully decommissioning all nagios checks, or just adding a disk-space-specific Slack alert alongside/instead of the one nagios check that currently covers disk space? Nagios typically monitors more than disk space, and the proposal body doesn't scope this.

• What disk-usage threshold(s) and mount point(s) should trigger a Slack alert? The existing deploy-time check uses a flat 2GB-free threshold — should the continuous monitor reuse this, or use a percentage-based threshold instead?

• Which Slack channel(s) should receive these alerts, and should routing differ by server or severity?

• Should this be built as a job inside monitor.py that posts to Slack directly, or as a Grafana alert rule layered on top of the Graphite metrics monitor.py already emits? This depends on whether Grafana-to-Slack alerting is already configured for this cluster — that's internal infrastructure state not visible from this repo.

• What should the alerting behavior be to avoid repeat-notification spam (e.g. alert once per incident, nagios-style, rather than every check interval)?

• Does the repo already have Slack-sending infrastructure, or would this need a new integration? (Answer: No new integration needed — a SLACK_TOKEN secret and multiple working send-message patterns already exist, see Context above.)

• Is there existing disk-space-check code this could build on? (Answer: Partially — see check_server_storage() in Context above. It's manual/deploy-time only, not continuous, and not currently wired to Slack.)

Suggested breakdown

• Staff decision: direct Slack post from monitor.py vs. a Grafana alert rule on existing Graphite metrics
• Add a disk-space check job to scripts/monitoring/monitor.py, following the existing per-job pattern (e.g. monitor_fail2ban)
• Decide and implement the alert threshold(s) and target Slack channel
• Add debounce/state-tracking so an ongoing incident doesn't spam the channel on every check interval
• Once the new monitor is confirmed reliable, decommission the corresponding nagios check(s)

None of the above is a Good First Issue candidate — implementing and testing requires production server access and Slack app credentials.

---

🔔 Stakeholders

@cdrini — owns scripts/monitoring/ (authored the fail2ban, Solr, and py-spy monitoring jobs currently in monitor.py per merged PR history) and authored check_server_storage() in #12423, the closest existing disk-space check.

---

Full triage checklist

• Triage — correct labels applied or removed
• Problem / opportunity — problem is clear, audience identified, actionable
• Scope — issue is well-scoped; does not try to do too many things
• Justification — reasoning and impact stated; sufficient for leads to prioritize
• Success & testing criteria — defines what "done" looks like, verifiable by a contributor
• Proposal & breakdown — concrete plan present; checklist actionable for a contributor
• If bug: environment & repro steps — N/A, this is a feature request
• Risks, concerns, open questions — identified and surfaced
• Richy context — relevant code, commits, PRs reviewed; no duplicate found
• References — screenshots, designs, docs, and links present where needed

---

Note

This comment was automatically generated by Richy, Open Library's issue enrichment assistant, on behalf of @mekarpeles. This is NOT a sign-off. This issue needs lead review before work begins.