Idea: network testing module with chart-based analysis

[gnacho]

Idea: network testing module with chart-based analysis

A module to run tests against routers, switch ports and WAN links, and present the results as charts (latency bars, loss pie, per-port error series, throughput over time).

Proposed architecture

Most of it builds on pieces NetPulse already has:

• Tests as collector runners, same pattern as the existing speedtest runner (NDT is already integrated via m-lab/ndt7-client-go). A new test runner could schedule and store: ICMP ping (latency, jitter, loss) towards routers and gateway, SNMP port checks on managed switches (link state, CRC errors, discards, duplex/negotiation via IF-MIB + EtherLike-MIB dot3Stats), and WAN throughput (existing NDT runner).
• Charts in the React UI with Recharts, which is already a dependency and powers the existing port/latency charts. Bars, pies and time series come almost for free.
• Go charting libraries only if we want exported reports (PNG/SVG/PDF): vicanso/go-charts (pure Go, bars/pies/lines, fast PNG/SVG, themes) or gonum/plot (scientific, exports PNG/SVG/PDF/EPS). go-chart (wcharczuk) looks abandoned and should be avoided; go-echarts generates interactive HTML but needs a browser, so it duplicates what Recharts already does.

Useful references: Gatus (Go, GPLv3) for the checks-plus-history model, pro-bing for native ICMP, showwin/speedtest-go as an Ookla alternative to NDT.

Open questions

1. Scope: WAN-only tests (like today) or also LAN throughput between the agent and the server?
2. Output: web UI only, or also downloadable/shareable reports?

Feedback welcome, especially on which tests would be most useful on real networks.

[crowedavid]

first a few thoughts on the Open questions / scope:

throughput testing really needs to be done endpoint to endpoint (usually client to server) and not between network devices or endpoint to network devices. this is especially true in consumer or low end enterprise networks since the processing power of these devices is TINY compared to their forwarding capacity - think single digit Mbps to the CPU vs 1+ Gbps forwarding on CPU+ASICs. also, if you're sourcing or terminating a test on a CPU/IO bound router it won't be able to forward as much traffic so it becomes self-defeating for most of your target market.

so, with that said, the WAN tests currently in place do exactly what you want: testing from the netpulse server to a remote server with processing capacity to handle consumer/small business class Internet speeds.

for my internal testing i have iperf3 setup as a systemd unit on the netpulse server so i can spin up an iperf3 test from my devices to do spot testing, wifi degradation checks, etc. there are android (inside "he.net - Network Tools"), macos, and linux iperf3 clients (windows too if you must 😜 ) that do a very good job and the tests are true throughput testing.

unfortunately, having netpulse arrange/orchestrate tests like that would be quite a bit outside of the netpulse primary scope. there is a large high performance R&D/science networking community working on infrastructure that does this called perfSONAR (see the iperf3 link earlier) but, again, way outside our scope. :-)

your first bullet in the Proposed architecture has some nuggets of real interest and it builds on the fantastic base you already have:

• SNMP port checks on managed switches, ethtool on openwrt:
  • link state:
    • this may not have as much utility except as added information to the log if a known device/client attached to the port loses link state. ICMP liveness checks would be the best way to know if the device/client is up/down (see below).
  • CRC errors, discards, other error counters:
    • gathering these stats is definitely useful as a lookback tool
    • it would gain utility especially if we have a decision engine that can process and, if appropriate, log, maybe alert on critical links, any rising counts.
  • duplex/negotiation:
    • this could have real utility to notice changes over time (was full, changes to half while noticing if there was a link down/up transition), goes back, continues flapping.
    • all very useful and people pay good money for systems that do this
• WAN test addition
  • iperf3 client
    • use case for me would be to perform adhoc and possibly scheduled iperf3 tests against one or more remote VPS servers
    • would need it to be configured alongside (in addition to) a normal WAN speed test option
    • iperf3 uses a full data stream (TCP/UDP/SCTP and can do multiple streams, uni or bi-directional, etc) whereas the other mainstream test protocols are sampled

another idea i've been planning to bring up and i might be best to split off to a different discussion item:

• adding client and fleet liveness testing from the netpulse server
  • arpscan is a low impact way to both check for existing client liveness as well as checking for new/unexpected attached devices outside of the DHCP served addresses (rogue devices attached to WiFi would be main check)
  • arpscan is limited to the local subnet where it is performed
  • ICMP ping as another layer of liveness check - necessary for devices on other subnets
  • IPv6 analogs for each of the above
  • less interested in long term data for local device latency, jitter, loss but could be interesting for multisite or larger networks
• add low impact port scanning/OS fingerprinting (think nmap) for fleet/client devices to check and make sure the ports you want are open and new/unexpected ports are alerted on
• log+optionally alert on:
  • fleet/client reachability (down/up) events
  • new/unknown/anomalous ports detected on fleet/clients
  • add specific remote host and service reachability/availability (light version of features in uptime kuma, uptimerobot)
    • this would benefit from the latency, jitter, loss measurements

all of the above could be done with UI and the charting base you have now. we (meaning you of course with the rest of us complaining) would need to work on the arpscan (pretty sure there are go reference implementations) and ping workers along with how to manage alerting. we'd also want to be able to mark which fleet/client devices to log+optionally alert on.

i currently do the last chunk of tasks with a mix of NetAlertX, UptimeRobot, and the Ping integration in home assistant. probably won't be rid of all of UptimeRobot since i want an independent source of reachability information separate/outside of my home and family networks. but integrating all i get from NetAlertX and most of the home assistant Ping integration into NetPulse would be great.

[gnacho]

On throughput: fully agree, and that matches how the pieces are built - the agent runs on the routers, so anything sourced or terminated there would measure their CPU, not the wire. The WAN test stays server-to-remote. The iperf3 client idea fits nicely though: the server running iperf3 against endpoints you configure (your VPS box) is exactly the adhoc testing you already do by hand, just with results stored and charted next to the NDT runs.

The counters and negotiation tracking are the part I find most interesting too: CRC/discards per port and duplex flips (especially with a link transition next to them) are cheap to poll over SNMP on the switches we already talk to, and they explain problems nothing else in the panel surfaces today. Link state alone I agree is only worth keeping as log context.

On the liveness/scanning idea: good call on the split - would you mind opening it as its own discussion? It overlaps with #1323 (first-seen client alerting) in an interesting way: arpscan/ping liveness would give that alert a second source beyond DHCP. Let's keep this thread about the test module.

No timeline promises on any of this - it stays an idea until the topology follow-ups settle.