feat(site-health-probe): synthetic monitoring component for NICo APIs

Adds nico-site-health-probe (#5360, sub-issue of #3723): a Go service that
runs configurable probes against NICo APIs concurrently and exposes
latency/outcome metrics for the standard collector.

Architecture (dev/k8s/site-health-probe, Go 1.26.4 / toolchain 1.26.7):
- Channel pipeline: one goroutine per probe (per-probe configurable
  interval/timeout, immediate first fire, panic recovery) produces Result
  events on a buffered channel; a Collector consumes them into a Sink.
  Probes are pure measurement (return Observations, never touch metrics),
  so future consumers — site-stats aggregation for host ingestion, instance
  creation, firmware updates (see TODOs) — attach at the Sink seam.
- Per-API packages: internal/probes/nicoapi (gRPC + SPIFFE mTLS machinery)
  and internal/probes/restapi (HTTP + Keycloak client-credentials) are
  distinct; adding a probe to one API never touches the other.
- v1 probes: gRPC machines (FindMachineIds + first-page FindMachinesByIds,
  rotation-safe cert reload), REST machines/instances (cached token,
  secrets re-read from mounted files, never logged). Every probe toggles
  independently (nico-api without REST and vice versa).
- Go stubs generated from the canonical crates/rpc/proto definitions
  (committed; Makefile proto-gen disambiguates two oneof-vs-enum-value
  name collisions Go cannot compile).
- Metrics: carbide_site_health_probe_{request_duration_milliseconds,
  requests_total,up} labeled by api (nico-api | nico-rest-api), probe,
  operation/outcome.

Security/hardening (audited: gosec 0, govulncheck clean, tests -race):
- Toolchain pinned to 1.26.7 (1.26.4's stdlib has seven reachable CVEs;
  the Grype image-scan gate reads the Go version from the binary).
- REST client refuses redirects (Go replays POST bodies on 307/308 to any
  host; the token body carries the Keycloak client_secret).
- Full TLS verification everywhere (no insecure mode exists); metrics
  server fully timeout-bounded; distroless nonroot image, digest-pinned.

Core RBAC: new SiteHealthProbe principal (nico-site-health-probe) granted
FindMachineIds + FindMachinesByIds only.

Helm: nico-site-health-probe subchart under nico-machine-a-tron (enabled by
default with the MAT chart; REST probes off until site inputs exist). The
SPIFFE Certificate and its volume render only when the gRPC probe is
enabled, so REST-only deployments don't depend on the core cert issuer.
Hardened pod (nonroot, read-only rootfs, no capabilities, seccomp
RuntimeDefault), metrics Service + gated ServiceMonitor, helm-unittest
coverage incl. per-probe toggle combinations.

CI: build-site-health-probe image job (mat-k8s-controller pattern), wired
into the aggregate jobs. Docs: metric rows in core_metrics.md; MAT README
section incl. the stale-TLS-secret reinstall note.

TODO(#5360-followup): active lifecycle probes (machine_count: 1=canary,
all=scale test) and progress p50/p95/p99 reporting.

Signed-off-by: Milad Noori <mnoori@nvidia.com>

Author: mnoori-afk

.github/workflows/ci.yaml

@@ -207,6 +207,7 @@ jobs:
               - 'rust-toolchain.toml'
               - '.github/workflows/ci.yaml'
               - 'dev/k8s/machine-a-tron-controller/**'
+              - 'dev/k8s/site-health-probe/**'
       - name: Calculate version
         id: version
         run: |
@@ -900,6 +901,60 @@ jobs:
       timeout_minutes: 30
     secrets: inherit
 
+  build-site-health-probe:
+    if: >-
+      ${{
+        !cancelled()
+        && github.event_name != 'schedule'
+        && needs.prepare.result == 'success'
+        && needs.prepare.outputs.source_files_changed == 'true'
+      }}
+    needs:
+      - prepare
+    uses: ./.github/workflows/docker-build.yml
+    with:
+      dockerfile_path: dev/k8s/site-health-probe/Dockerfile
+      context_path: dev/k8s/site-health-probe
+      image_name: ${{ needs.prepare.outputs.image_registry }}/site-health-probe
+      image_tag: ${{ needs.prepare.outputs.version }}
+      additional_tags: ${{ needs.prepare.outputs.image_registry }}/site-health-probe:${{ needs.prepare.outputs.major_minor_version }}-latest
+      platforms: linux/amd64,linux/arm64
+      runner: linux-amd64-cpu4
+      push: ${{ needs.prepare.outputs.publish_images == 'true' }}
+      load: false
+      scan: true
+      tag_latest: false
+      timeout_minutes: 30
+    secrets: inherit
+
+  test-site-health-probe:
+    name: Test Site Health Probe
+    if: >-
+      ${{
+        !cancelled()
+        && github.event_name != 'schedule'
+        && needs.prepare.result == 'success'
+        && needs.prepare.outputs.source_files_changed == 'true'
+      }}
+    needs:
+      - prepare
+    runs-on: linux-amd64-cpu4
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version-file: dev/k8s/site-health-probe/go.mod
+          cache: true
+          cache-dependency-path: dev/k8s/site-health-probe/go.sum
+
+      - name: Run tests
+        run: make -C dev/k8s/site-health-probe vet test
+
   # Like the release-container builds above, deliberately not gated on
   # `check-rest-core-proto-sync`: the tests do not read anything it produces,
   # and `core-ci-pass` still fails the merge when the protos are out of sync.
@@ -1725,6 +1780,7 @@ jobs:
             helm/charts/nico-hardware-health
             helm/charts/nico-machine-a-tron
             helm/charts/nico-machine-a-tron/charts/mat-k8s-controller
+            helm/charts/nico-machine-a-tron/charts/nico-site-health-probe
             helm/charts/nico-pxe
             helm/charts/nico-ssh-console-rs
             helm/charts/unbound
@@ -2098,6 +2154,8 @@ jobs:
       - build-release-container-aarch64
       - build-machine-a-tron
       - build-mat-k8s-controller
+      - build-site-health-probe
+      - test-site-health-probe
       - test-release-container-services
       - build-boot-artifacts-x86
       - build-boot-artifacts-bfb
@@ -2161,6 +2219,7 @@ jobs:
           IMAGES["build-release-artifacts-x86-host"]="${{ needs.build-release-artifacts-x86-host.outputs.image_ref }}"
           IMAGES["build-release-artifacts-arm-host"]="${{ needs.build-release-artifacts-arm-host.outputs.image_ref }}"
           IMAGES["build-mat-k8s-controller"]="${{ needs.build-mat-k8s-controller.outputs.image_ref }}"
+          IMAGES["build-site-health-probe"]="${{ needs.build-site-health-probe.outputs.image_ref }}"
 
           for job in "${!IMAGES[@]}"; do
             image="${IMAGES[$job]}"
@@ -2205,6 +2264,8 @@ jobs:
       - build-release-container-aarch64
       - build-machine-a-tron
       - build-mat-k8s-controller
+      - build-site-health-probe
+      - test-site-health-probe
       - test-release-container-services
       - build-boot-artifacts-x86
       - build-boot-artifacts-bfb
@@ -2282,6 +2343,8 @@ jobs:
       - merge-manifests-nvmetal-carbide
       - build-machine-a-tron
       - build-mat-k8s-controller
+      - build-site-health-probe
+      - test-site-health-probe
       - test-release-container-services
 
       # CLI images

crates/api-core/src/auth/internal_rbac_rules.rs

@@ -45,11 +45,12 @@ enum RulePrincipal {
     Flow,
     MaintenanceJobs,
     DsxExchangeConsumer,
-    Anonymous, // Permitted for everything
+    SiteHealthProbe, // synthetic read-only monitoring (#5360)
+    Anonymous,       // Permitted for everything
 }
 use self::RulePrincipal::{
     Agent, Anonymous, BmcProxy, Dhcp, Dns, DsxExchangeConsumer, Flow, ForgeAdminCLI, Health,
-    Machineatron, MaintenanceJobs, Pxe, Scout, SiteAgent, Ssh, SshRs,
+    Machineatron, MaintenanceJobs, Pxe, Scout, SiteAgent, SiteHealthProbe, Ssh, SshRs,
 };
 
 impl InternalRBACRules {
@@ -239,6 +240,7 @@ impl InternalRBACRules {
                 Ssh,
                 SshRs,
                 Flow,
+                SiteHealthProbe,
             ],
         );
         x.perm(
@@ -251,6 +253,7 @@ impl InternalRBACRules {
                 Ssh,
                 SshRs,
                 Flow,
+                SiteHealthProbe,
             ],
         );
         x.perm("FindConnectedDevicesByDpuMachineIds", vec![ForgeAdminCLI]);
@@ -1064,6 +1067,11 @@ impl RuleInfo {
                         "nico-dsx-exchange-consumer",
                         "carbide-dsx-exchange-consumer",
                     ),
+                    // New service (no legacy carbide- alias): the synthetic
+                    // monitoring probe, read-only machine queries only (#5360).
+                    RulePrincipal::SiteHealthProbe => vec![Principal::SpiffeServiceIdentifier(
+                        "nico-site-health-probe".to_string(),
+                    )],
                     RulePrincipal::Anonymous => vec![Principal::Anonymous],
                 })
                 .collect(),

dev/k8s/site-health-probe/Dockerfile

@@ -0,0 +1,31 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# Build stage. Go 1.26.7: 1.26.4's standard library carries seven
+# govulncheck-flagged CVEs reachable from this service (crypto/tls, net/http,
+# net/url, encoding/asn1 — all fixed by 1.26.6/1.26.7), and the Grype image
+# scan reads the Go version from the binary.
+# --platform=$BUILDPLATFORM + GOARCH cross-compile: the builder always runs
+# natively and emits the target architecture, instead of compiling under QEMU
+# emulation (where the Go toolchain is unreliable).
+FROM --platform=$BUILDPLATFORM golang:1.26.7-alpine@sha256:28d89ee9cc0ff9fec75c82ca201e6bf7fdf9a679d4b7b24dfa04f2bb766bb468 AS builder
+
+WORKDIR /app
+
+COPY go.mod go.sum ./
+RUN go mod download
+
+COPY . .
+ARG TARGETOS TARGETARCH
+RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH go build -trimpath -ldflags="-s -w" -o site-health-probe ./cmd/site-health-probe
+
+# Runtime stage
+FROM gcr.io/distroless/static:nonroot@sha256:f7f8f729987ad0fdf6b05eeeae94b26e6a0f613bdf46feea7fc40f7bd72953e6
+
+COPY --from=builder /app/site-health-probe /site-health-probe
+
+# Numeric UID (distroless "nonroot"): kubelet rejects named users under
+# runAsNonRoot ("cannot verify user is non-root").
+USER 65532:65532
+
+ENTRYPOINT ["/site-health-probe"]

dev/k8s/site-health-probe/Makefile

@@ -0,0 +1,73 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# nico-site-health-probe — synthetic monitoring for NICo APIs (issue #5360).
+
+MODULE      := github.com/NVIDIA/infra-controller/dev/k8s/site-health-probe
+PROTO_DIR   := ../../../crates/rpc/proto
+PB_OUT      := internal/forgepb
+IMAGE       ?= site-health-probe:dev
+
+# The gRPC proto contract is the repo-canonical crates/rpc/proto (same files the
+# Rust services compile). It is an internal, not-guaranteed API — this tool is
+# in-repo and version-locked to it, which is the supported way to consume it.
+# Only forge.proto declares a service; the rest are message dependencies.
+PROTOS := forge.proto common.proto dns.proto health.proto machine_discovery.proto \
+          measured_boot.proto mlx_device.proto scout_firmware_upgrade.proto \
+          site_explorer.proto codegen/v1/derive.proto codegen/v1/extern_path.proto
+
+# None of the protos carry option go_package (they predate Go consumers), so the
+# mapping is supplied here. Each proto package gets its own Go package dir.
+M_FLAGS := \
+  --go_opt=Mforge.proto=$(MODULE)/$(PB_OUT)/forge \
+  --go_opt=Mcommon.proto=$(MODULE)/$(PB_OUT)/common \
+  --go_opt=Mdns.proto=$(MODULE)/$(PB_OUT)/dns \
+  --go_opt=Mhealth.proto=$(MODULE)/$(PB_OUT)/health \
+  --go_opt=Mmachine_discovery.proto=$(MODULE)/$(PB_OUT)/machine_discovery \
+  --go_opt=Mmeasured_boot.proto=$(MODULE)/$(PB_OUT)/measured_boot \
+  --go_opt=Mmlx_device.proto=$(MODULE)/$(PB_OUT)/mlx_device \
+  --go_opt=Mscout_firmware_upgrade.proto=$(MODULE)/$(PB_OUT)/scout_firmware_upgrade \
+  --go_opt=Msite_explorer.proto=$(MODULE)/$(PB_OUT)/site_explorer \
+  --go_opt=Mcodegen/v1/derive.proto=$(MODULE)/$(PB_OUT)/codegenv1 \
+  --go_opt=Mcodegen/v1/extern_path.proto=$(MODULE)/$(PB_OUT)/codegenv1
+
+GRPC_M_FLAGS := $(subst --go_opt=,--go-grpc_opt=,$(M_FLAGS))
+
+.PHONY: proto-gen build test lint vet docker-build clean
+
+proto-gen:
+	rm -rf $(PB_OUT)
+	mkdir -p $(PB_OUT)
+	protoc -I $(PROTO_DIR) \
+	  --go_out=. --go_opt=module=$(MODULE) $(M_FLAGS) \
+	  --go-grpc_out=. --go-grpc_opt=module=$(MODULE) $(GRPC_M_FLAGS) \
+	  $(PROTOS)
+	# forge.proto's MachineValidationStatus has oneof fields (started,
+	# in_progress) AND nested enum values (Started, InProgress) that map to the
+	# SAME Go identifier — legal in Rust codegen, a redeclaration in Go.
+	# Disambiguate the enum-value constants to their full-path names. The two
+	# patterns below are exactly the const declaration and its single bare use
+	# (the oneof getter's default return); all other occurrences are the oneof
+	# wrapper TYPE and must keep their names.
+	perl -pi -e 's/^\tMachineValidationStatus_(Started|InProgress) MachineValidationStatus_MachineValidation\1 = 0$$/\tMachineValidationStatus_MachineValidation$${1}_$${1} MachineValidationStatus_MachineValidation$${1} = 0/' \
+	  $(PB_OUT)/forge/forge.pb.go
+	perl -pi -e 's/^\treturn MachineValidationStatus_(Started|InProgress)$$/\treturn MachineValidationStatus_MachineValidation$${1}_$${1}/' \
+	  $(PB_OUT)/forge/forge.pb.go
+
+build:
+	CGO_ENABLED=0 go build -trimpath -o bin/site-health-probe ./cmd/site-health-probe
+
+test:
+	go test -race ./...
+
+vet:
+	go vet ./...
+
+lint: vet
+	@command -v golangci-lint >/dev/null && golangci-lint run ./... || echo "golangci-lint not installed — ran go vet only"
+
+docker-build:
+	docker build -t $(IMAGE) .
+
+clean:
+	rm -rf bin