File discriptor leak from sysinfo refresh_all() #1682

/about output:

{
  "version": "v2.9.1",
  "uiVersion": "v2.9.1",
  "commit": "8187dad",
  "deploymentId": "...",
  "updateAvailable": false,
  "latestVersion": "v2.9.1",
  "llmActive": false,
  "llmProvider": null,
  "oidcActive": false,
  "license": {
    "name": "AGPL-3.0-only",
    "version": "v1",
    "plan": "OSS",
    "deploymentInfo": "Managed"
  },
  "mode": "Standalone",
  "staging": "/dat/parseable/staging",
  "hotTier": "Disabled",
  "grpcPort": 8001,
  "store": {
    "type": "S3 bucket",
    "path": "..."
  },
  "analytics": {
    "clarityTag": null
  }
}

Description:
refresh_sys_info() calls sysinfo's System::refresh_all() on every resource-monitor tick (resource_check.rs). On Linux, refresh_all() refreshes every process and each process's tasks (threads), opening /proc//task//stat per thread. sysinfo (v0.33.1) retains an open file handle per task and does not release handles for threads that have since exited from the always-present self process. The leak therefore grows with the cumulative number of distinct thread TIDs ever created. It is more noticeable under heavy task churn, for eg: higher number of streams or slow/unavailable object storage.

Potential fixes:
The consumers of SYS_INFO only ever read memory and CPU (used_memory/total_memory/free_memory, cpus()/global_cpu_usage()); nothing reads process, disk, network, or task data. So refresh_all() can be replaced with refresh_memory()+refresh_cpu_all().

Another alternative is to move to sysinfo 0.34, which removes dead processes by default.

Observations:

Without patch

File discriptors held by the parseable process keep growing over time, and list procfs entries to non-existent processes

$ pid=$(pgrep -f '[p]arseable')
$ sudo ls /proc/$pid/fd | wc -l
323027
$ sudo ls -l /proc/$pid/fd | grep -oE '/\S+' | cut -d/ -f1-3 | sort | uniq -c | sort -rn | head -20
 319984 /proc/1375179
    313 /dat/parseable
    186 /proc/4132786
      7 /proc/3437495
(many more entries)
...

With patch (replace refresh_all with refresh_memory+refresh_cpu_all)

diff --git a/src/analytics.rs b/src/analytics.rs
index 6ceb8ba..60d7413 100644
--- a/src/analytics.rs
+++ b/src/analytics.rs
@@ -50,11 +50,12 @@ use crate::{
 const ANALYTICS_SERVER_URL: &str = "https://analytics.parseable.io:80";
 const ANALYTICS_SEND_INTERVAL_SECONDS: Interval = clokwerk::Interval::Hours(1);
 
-pub static SYS_INFO: Lazy<Mutex<System>> = Lazy::new(|| Mutex::new(System::new_all()));
+pub static SYS_INFO: Lazy<Mutex<System>> = Lazy::new(|| Mutex::new(System::new()));
 
 pub fn refresh_sys_info() {
     let mut sys_info = SYS_INFO.lock().unwrap();
-    sys_info.refresh_all();
+    sys_info.refresh_memory();
+    sys_info.refresh_cpu_all();
 }
 
 #[derive(Serialize, Deserialize)]

$ pid=$(pgrep -f '[p]arseable')
$ sudo ls /proc/$pid/fd | wc -l
1020
$ sudo ls -l /proc/$pid/fd | grep -oE '/\S+' | cut -d/ -f1-3 | sort | uniq -c | sort -rn | head -20
    537 /dat/parseable
      1 /dev/null