Skip to content

Shutdown hangs up to TimeoutStopSec when a poller tick is stuck on slow SNMP #481

Closed

Description

@gnacho

Observed on 2.26.1 (install.sh single-binary, systemd):

systemctl stop netpulse hung until systemd SIGKILLed it at 90s (journal: Failed with result 'timeout').

Root cause (code):

  • Poller.Stop() waits for the in-flight tick (<-p.doneCh, internal/poller/poller.go:98-103).
  • The tick runs with context.Background() and no deadline (internal/poller/poller.go:112); a live tick polling several hosts via SNMP at ~10s per timed-out request (5s timeout + 1 retry, per host/OID serial) can take minutes.
  • The 3s lifeline in cmd/netpulse/main.go:553 is armed only AFTER p.Stop()/upd.Stop() return (main.go:534-535), so it never fires while the tick is stuck.

Impact: every restart on a host with a slow/unreachable SNMP device takes 90s+SIGKILL, and anything scripted around restarts (e.g. the stable self-update helper healthcheck) sees much longer service restarts.

Suggested fix: thread a cancellable context through the tick (Stop() cancels it), and/or arm the 3s lifeline BEFORE calling p.Stop()/upd.Stop(). Also consider a per-tick deadline.

Repro: make one SNMP target blackhole (drop packets) so requests time out, then systemctl restart netpulse and watch it hit TimeoutStopSec.

Activity

  1. gnacho commented on Sep 4, 2026

    @gnacho
    OwnerAuthor

    Shipped in PR #523: Poller.Stop() now waits at most 15s for an in-flight tick and then lets the process exit, so a poll stuck on a slow/hung SNMP target can no longer hang the service stop until TimeoutStopSec.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions