Skip to content

Commit c78c077

Browse files
committed
fix(subprocess): fail the tree probe on a live member it cannot measure
cbm_subprocess_tree_rss_bytes skipped any non-root process it could not measure and still returned OK with a smaller total, so a descendant the platform refused to measure silently dropped out of the RSS watchdog. Now a member is skipped only when the platform says it no longer exists: ESRCH from proc_pid_rusage on macOS, ENOENT or ESRCH for /proc/<pid>/stat on Linux, ERROR_INVALID_PARAMETER from OpenProcess (or an exit code from a handle whose memory query failed) on Windows. Any other failure fails the probe, which the supervisor already turns into resource_probe_failed after three consecutive errors. The root stays strict as before. On macOS the members now come from proc_listpids(PROC_PGRP_ONLY): the old all-process scan read every process with proc_pidinfo, which answers EPERM for every process of another user (178 of 6438 on this machine), so it could not tell a foreign process from an unreadable member and fail-closed semantics would have failed every probe there. Linux keeps its /proc scan; stat files are world-readable on a default mount, and a hidepid=1 mount now fails closed, which the doc states. A test seam makes every non-root measurement fail with one platform error. RED (old skip-any-failure decision): the live-member test returns OK where ERROR is required. GREEN with the fix; the exited-member test passes in both, so verified exit races are still ignored. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent 33ae8f3 commit c78c077

4 files changed

Lines changed: 208 additions & 62 deletions

File tree

‎docs/INDEX_RESOURCE_LIMITS.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,11 @@ quarantine a source file.
6060

6161
If RSS is enabled and three consecutive probes cannot obtain any trustworthy
6262
tree measurement while the root worker is still running, CBM fails closed with
63-
`code=resource_probe_failed`.
63+
`code=resource_probe_failed`. A probe is untrustworthy as soon as one member of
64+
the tree cannot be measured: only a process that exits between the listing and
65+
its measurement is skipped, because a total without a live member would
66+
under-report. On Linux the probe reads every `/proc/<pid>/stat` to find the
67+
tree's members, so a `/proc` mounted with `hidepid=1` makes it fail closed.
6468

6569
## Counting and failure semantics
6670

‎src/foundation/subprocess.c‎

Lines changed: 153 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -476,9 +476,53 @@ uint64_t cbm_subprocess_rss_sum_for_testing(const uint64_t *values, size_t count
476476
}
477477
return total;
478478
}
479+
480+
static atomic_int g_tree_rss_member_fault;
481+
482+
void cbm_subprocess_tree_rss_fault_members_for_testing(int platform_error) {
483+
atomic_store_explicit(&g_tree_rss_member_fault, platform_error, memory_order_release);
484+
}
485+
486+
/* Test builds can make every NON-root member's measurement fail with one
487+
* platform error (errno on POSIX, GetLastError() on Windows); 0 = real call.
488+
* Production builds compile the check out entirely. */
489+
static int cbm_tree_rss_member_fault(bool root) {
490+
return root ? 0 : atomic_load_explicit(&g_tree_rss_member_fault, memory_order_acquire);
491+
}
479492
#endif
480493

494+
/* A member that cannot be measured is skipped only when the platform says it
495+
* no longer exists (it exited between the listing and the measurement, and
496+
* holds nothing). Any other failure is a live member whose memory a total
497+
* would silently leave out, so the probe fails instead -- a watchdog that
498+
* under-reports is one that quietly stopped watching. The root is strict
499+
* either way. */
481500
#ifdef _WIN32
501+
static HANDLE cbm_tree_rss_open_member(DWORD pid, bool root, DWORD *error_out) {
502+
#ifdef CBM_ENABLE_TEST_SEAMS
503+
int fault = cbm_tree_rss_member_fault(root);
504+
if (fault != 0) {
505+
*error_out = (DWORD)fault;
506+
return NULL;
507+
}
508+
#else
509+
(void)root;
510+
#endif
511+
/* GetProcessMemoryInfo documents PROCESS_QUERY_INFORMATION |
512+
* PROCESS_VM_READ. PROCESS_QUERY_LIMITED_INFORMATION is a lesser
513+
* alternative on newer Windows; the Job Object membership list
514+
* already closes most of the pid-reuse window between listing and
515+
* opening. */
516+
HANDLE member = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, pid);
517+
*error_out = member ? ERROR_SUCCESS : GetLastError();
518+
return member;
519+
}
520+
521+
static bool cbm_tree_rss_member_exited(HANDLE member) {
522+
DWORD exit_code = 0;
523+
return GetExitCodeProcess(member, &exit_code) && exit_code != STILL_ACTIVE;
524+
}
525+
482526
static cbm_proc_tree_rss_status_t cbm_subprocess_tree_rss_platform(cbm_subprocess_t *process,
483527
uint64_t *rss_bytes) {
484528
DWORD capacity = 32;
@@ -510,33 +554,30 @@ static cbm_proc_tree_rss_status_t cbm_subprocess_tree_rss_platform(cbm_subproces
510554

511555
uint64_t total = 0;
512556
DWORD measured = 0;
513-
bool root_failed = false;
514-
for (DWORD index = 0; index < processes->NumberOfProcessIdsInList; index++) {
557+
bool failed = false;
558+
for (DWORD index = 0; index < processes->NumberOfProcessIdsInList && !failed; index++) {
515559
DWORD pid = (DWORD)processes->ProcessIdList[index];
516-
/* GetProcessMemoryInfo documents PROCESS_QUERY_INFORMATION |
517-
* PROCESS_VM_READ. PROCESS_QUERY_LIMITED_INFORMATION is a lesser
518-
* alternative on newer Windows; the Job Object membership list
519-
* already closes most of the pid-reuse window between listing and
520-
* opening. A member we cannot open is a probe failure when it is
521-
* the root. */
522-
HANDLE member = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, pid);
560+
bool root = pid == process->process_id;
561+
DWORD open_error = ERROR_SUCCESS;
562+
HANDLE member = cbm_tree_rss_open_member(pid, root, &open_error);
523563
if (!member) {
524-
root_failed = root_failed || pid == process->process_id;
564+
/* ERROR_INVALID_PARAMETER: no process has this id any more. */
565+
failed = root || open_error != ERROR_INVALID_PARAMETER;
525566
continue;
526567
}
527568
PROCESS_MEMORY_COUNTERS memory;
528569
ZeroMemory(&memory, sizeof(memory));
529570
if (GetProcessMemoryInfo(member, &memory, sizeof(memory))) {
530571
cbm_rss_add_saturated(&total, (uint64_t)memory.WorkingSetSize);
531572
measured++;
532-
} else if (pid == process->process_id) {
533-
root_failed = true;
573+
} else {
574+
failed = root || !cbm_tree_rss_member_exited(member);
534575
}
535576
CloseHandle(member);
536577
}
537578
DWORD listed = processes->NumberOfProcessIdsInList;
538579
cbm_free(CBM_MEM_CLASS_OTHER, processes);
539-
if (root_failed || (measured == 0 && listed > 0)) {
580+
if (failed || (measured == 0 && listed > 0)) {
540581
return CBM_PROC_TREE_RSS_ERROR;
541582
}
542583
if (measured == 0) {
@@ -546,56 +587,73 @@ static cbm_proc_tree_rss_status_t cbm_subprocess_tree_rss_platform(cbm_subproces
546587
return CBM_PROC_TREE_RSS_OK;
547588
}
548589
#elif defined(__APPLE__)
549-
static cbm_proc_tree_rss_status_t cbm_subprocess_tree_rss_platform(cbm_subprocess_t *process,
550-
uint64_t *rss_bytes) {
551-
int capacity = proc_listallpids(NULL, 0);
552-
if (capacity <= 0 || capacity > INT_MAX / (int)sizeof(pid_t) - 64) {
553-
return CBM_PROC_TREE_RSS_ERROR;
554-
}
555-
capacity += 64;
590+
/* The members of the contained process group, as the kernel lists them. An
591+
* all-process scan cannot decide membership: proc_pidinfo answers EPERM for
592+
* every process of another user, so "not in our group" and "in our group but
593+
* unreadable" would look the same. Returns the member count, or -1. */
594+
static int cbm_tree_rss_list_group(pid_t pgid, pid_t **pids_out) {
595+
int capacity = 64;
556596
pid_t *pids = NULL;
557-
int count = 0;
558597
for (;;) {
559598
pid_t *grown = cbm_realloc(CBM_MEM_CLASS_OTHER, pids, (size_t)capacity * sizeof(*pids));
560599
if (!grown) {
561600
cbm_free(CBM_MEM_CLASS_OTHER, pids);
562-
return CBM_PROC_TREE_RSS_ERROR;
601+
return -1;
563602
}
564603
pids = grown;
565-
count = proc_listallpids(pids, capacity * (int)sizeof(*pids));
566-
if (count <= 0) {
604+
errno = 0;
605+
int bytes =
606+
proc_listpids(PROC_PGRP_ONLY, (uint32_t)pgid, pids, capacity * (int)sizeof(*pids));
607+
if (bytes < 0 || (bytes == 0 && errno != 0)) {
567608
cbm_free(CBM_MEM_CLASS_OTHER, pids);
568-
return CBM_PROC_TREE_RSS_ERROR;
609+
return -1;
569610
}
611+
int count = bytes / (int)sizeof(*pids);
570612
if (count < capacity) {
571-
break;
613+
*pids_out = pids;
614+
return count;
572615
}
573616
if (capacity > INT_MAX / (int)sizeof(*pids) / 2) {
574617
cbm_free(CBM_MEM_CLASS_OTHER, pids);
575-
return CBM_PROC_TREE_RSS_ERROR;
618+
return -1;
576619
}
577620
capacity *= 2;
578621
}
622+
}
623+
624+
static int cbm_tree_rss_member_rusage(pid_t pid, bool root, struct rusage_info_v2 *usage) {
625+
#ifdef CBM_ENABLE_TEST_SEAMS
626+
int fault = cbm_tree_rss_member_fault(root);
627+
if (fault != 0) {
628+
errno = fault;
629+
return -1;
630+
}
631+
#else
632+
(void)root;
633+
#endif
634+
return proc_pid_rusage(pid, RUSAGE_INFO_V2, (rusage_info_t *)usage);
635+
}
636+
637+
static cbm_proc_tree_rss_status_t cbm_subprocess_tree_rss_platform(cbm_subprocess_t *process,
638+
uint64_t *rss_bytes) {
639+
pid_t *pids = NULL;
640+
int count = cbm_tree_rss_list_group(process->pgid, &pids);
641+
if (count < 0) {
642+
return CBM_PROC_TREE_RSS_ERROR;
643+
}
579644

580645
uint64_t total = 0;
581646
int measured = 0;
582-
bool root_failed = false;
583-
for (int index = 0; index < count; index++) {
584-
struct proc_bsdinfo info;
647+
bool failed = false;
648+
for (int index = 0; index < count && !failed; index++) {
585649
if (pids[index] <= 0) {
586650
continue;
587651
}
588-
if (proc_pidinfo(pids[index], PROC_PIDTBSDINFO, 0, &info, sizeof(info)) !=
589-
(int)sizeof(info)) {
590-
root_failed = root_failed || pids[index] == process->pid;
591-
continue;
592-
}
593-
if ((pid_t)info.pbi_pgid != process->pgid) {
594-
continue;
595-
}
652+
bool root = pids[index] == process->pid;
596653
struct rusage_info_v2 usage;
597-
if (proc_pid_rusage(pids[index], RUSAGE_INFO_V2, (rusage_info_t *)&usage) != 0) {
598-
root_failed = root_failed || pids[index] == process->pid;
654+
if (cbm_tree_rss_member_rusage(pids[index], root, &usage) != 0) {
655+
/* ESRCH: the member exited after the listing. */
656+
failed = root || errno != ESRCH;
599657
continue;
600658
}
601659
/* phys_footprint is what the OS charges, matching cbm_mem_charged().
@@ -606,7 +664,7 @@ static cbm_proc_tree_rss_status_t cbm_subprocess_tree_rss_platform(cbm_subproces
606664
measured++;
607665
}
608666
cbm_free(CBM_MEM_CLASS_OTHER, pids);
609-
if (root_failed) {
667+
if (failed) {
610668
return CBM_PROC_TREE_RSS_ERROR;
611669
}
612670
if (measured == 0) {
@@ -628,22 +686,61 @@ static bool cbm_proc_pid_name(const char *name) {
628686
return true;
629687
}
630688

631-
static bool cbm_linux_proc_stat(const char *pid_name, pid_t *group, int64_t *rss_pages) {
689+
typedef enum {
690+
CBM_LINUX_STAT_OK = 0,
691+
CBM_LINUX_STAT_GONE, /* the process exited after the /proc listing */
692+
CBM_LINUX_STAT_UNREADABLE, /* a process whose stat could not be read or parsed */
693+
} cbm_linux_stat_t;
694+
695+
/* /proc/<pid>/stat is world-readable, so on a default /proc mount the only
696+
* expected failure is the exit race: ENOENT at open, ESRCH at read. Anything
697+
* else (EACCES under hidepid=1, a parse failure) leaves the process's group
698+
* unknown, and the probe fails closed rather than guess it is not ours. */
699+
static cbm_linux_stat_t cbm_linux_stat_failure(int error) {
700+
return error == ENOENT || error == ESRCH ? CBM_LINUX_STAT_GONE : CBM_LINUX_STAT_UNREADABLE;
701+
}
702+
703+
static cbm_linux_stat_t cbm_linux_proc_stat_line(const char *pid_name, bool root, char *line,
704+
size_t line_size) {
632705
char path[64];
633706
int length = snprintf(path, sizeof(path), "/proc/%s/stat", pid_name);
634707
if (length <= 0 || length >= (int)sizeof(path)) {
635-
return false;
708+
return CBM_LINUX_STAT_UNREADABLE;
709+
}
710+
#ifdef CBM_ENABLE_TEST_SEAMS
711+
int fault = cbm_tree_rss_member_fault(root);
712+
if (fault != 0) {
713+
return cbm_linux_stat_failure(fault);
636714
}
715+
#else
716+
(void)root;
717+
#endif
718+
errno = 0;
637719
FILE *stat_file = cbm_fopen(path, "rb");
638720
if (!stat_file) {
639-
return false;
721+
return cbm_linux_stat_failure(errno);
640722
}
641-
char stat_line[4096];
642-
bool read = fgets(stat_line, sizeof(stat_line), stat_file) != NULL;
723+
errno = 0;
724+
bool read = fgets(line, (int)line_size, stat_file) != NULL;
725+
int read_error = errno;
643726
(void)fclose(stat_file);
644-
char *command_end = read ? strrchr(stat_line, ')') : NULL;
727+
if (!read) {
728+
return read_error == ESRCH ? CBM_LINUX_STAT_GONE : CBM_LINUX_STAT_UNREADABLE;
729+
}
730+
return CBM_LINUX_STAT_OK;
731+
}
732+
733+
static cbm_linux_stat_t cbm_linux_proc_stat(const char *pid_name, bool root, pid_t *group,
734+
int64_t *rss_pages) {
735+
char stat_line[4096];
736+
cbm_linux_stat_t status =
737+
cbm_linux_proc_stat_line(pid_name, root, stat_line, sizeof(stat_line));
738+
if (status != CBM_LINUX_STAT_OK) {
739+
return status;
740+
}
741+
char *command_end = strrchr(stat_line, ')');
645742
if (!command_end || command_end[1] != ' ') {
646-
return false;
743+
return CBM_LINUX_STAT_UNREADABLE;
647744
}
648745

649746
char *save = NULL;
@@ -656,7 +753,7 @@ static bool cbm_linux_proc_stat(const char *pid_name, pid_t *group, int64_t *rss
656753
char *end = NULL;
657754
long long value = strtoll(token, &end, 10);
658755
if (!end || *end != '\0') {
659-
return false;
756+
return CBM_LINUX_STAT_UNREADABLE;
660757
}
661758
if (field == 5) {
662759
*group = (pid_t)value;
@@ -670,7 +767,7 @@ static bool cbm_linux_proc_stat(const char *pid_name, pid_t *group, int64_t *rss
670767
token = strtok_r(NULL, " ", &save);
671768
field++;
672769
}
673-
return have_group && have_rss;
770+
return have_group && have_rss ? CBM_LINUX_STAT_OK : CBM_LINUX_STAT_UNREADABLE;
674771
}
675772

676773
static cbm_proc_tree_rss_status_t cbm_subprocess_tree_rss_platform(cbm_subprocess_t *process,
@@ -687,17 +784,18 @@ static cbm_proc_tree_rss_status_t cbm_subprocess_tree_rss_platform(cbm_subproces
687784

688785
uint64_t total = 0;
689786
int measured = 0;
690-
bool root_failed = false;
787+
bool failed = false;
691788
struct dirent *entry;
692-
while ((entry = readdir(proc)) != NULL) {
789+
while (!failed && (entry = readdir(proc)) != NULL) {
693790
if (!cbm_proc_pid_name(entry->d_name)) {
694791
continue;
695792
}
696793
pid_t group = 0;
697794
int64_t pages = 0;
698795
bool root_entry = strtol(entry->d_name, NULL, 10) == (long)process->pid;
699-
if (!cbm_linux_proc_stat(entry->d_name, &group, &pages)) {
700-
root_failed = root_failed || root_entry;
796+
cbm_linux_stat_t status = cbm_linux_proc_stat(entry->d_name, root_entry, &group, &pages);
797+
if (status != CBM_LINUX_STAT_OK) {
798+
failed = root_entry || status != CBM_LINUX_STAT_GONE;
701799
continue;
702800
}
703801
if (group != process->pgid || pages < 0) {
@@ -711,7 +809,7 @@ static cbm_proc_tree_rss_status_t cbm_subprocess_tree_rss_platform(cbm_subproces
711809
measured++;
712810
}
713811
(void)closedir(proc);
714-
if (root_failed) {
812+
if (failed) {
715813
return CBM_PROC_TREE_RSS_ERROR;
716814
}
717815
if (measured == 0) {

‎src/foundation/subprocess.h‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -159,8 +159,9 @@ cbm_proc_poll_t cbm_subprocess_poll(cbm_subprocess_t *process, cbm_proc_result_t
159159
* On macOS this is phys_footprint (what the OS holds against the process);
160160
* elsewhere it is RSS. OK returns an overflow-safe byte total, EMPTY means
161161
* the owned tree currently has no observable members, and ERROR means no
162-
* trustworthy measurement could be obtained. Individual processes that exit
163-
* during enumeration are ignored. */
162+
* trustworthy measurement could be obtained. A member that exits between the
163+
* listing and its measurement is ignored (the platform reports it gone); any
164+
* other member that cannot be measured is ERROR, never a smaller total. */
164165
cbm_proc_tree_rss_status_t cbm_subprocess_tree_rss_bytes(cbm_subprocess_t *process,
165166
uint64_t *rss_bytes);
166167
bool cbm_subprocess_root_running(const cbm_subprocess_t *process);
@@ -243,6 +244,10 @@ void cbm_subprocess_hold_parent_until_child_exits_for_testing(int spawns);
243244
* waitid does not report it yet. Test builds only. */
244245
void cbm_subprocess_observe_exiting_child_for_testing(int spawns);
245246
uint64_t cbm_subprocess_rss_sum_for_testing(const uint64_t *values, size_t count);
247+
/* Make the tree probe's measurement of every NON-root member fail with
248+
* platform_error (errno on POSIX, GetLastError() on Windows), as a live or an
249+
* exiting member would; 0 restores the real calls. Test builds only. */
250+
void cbm_subprocess_tree_rss_fault_members_for_testing(int platform_error);
246251
#endif
247252

248253
/* How the POSIX fork+exec child closed its inherited descriptors (#1484). */

0 commit comments

Comments
 (0)