Skip to content

🌱 Add per-caller and per-server-id metrics for hcloud and robot API calls - #2166

Open
guettli wants to merge 7 commits into
mainfrom
add-hcloud-per-server-id-metric
Open

🌱 Add per-caller and per-server-id metrics for hcloud and robot API calls#2166
guettli wants to merge 7 commits into
mainfrom
add-hcloud-per-server-id-metric

Conversation

@guettli

@guettli guettli commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

We want to know which caph code is calling the hcloud/Robot APIs, and how often, so rate-limit
and traffic-volume issues (see #2163) can actually be debugged instead of guessed at. This PR
started as a hcloud-only, opt-in per-server-id metric and grew into general-purpose
instrumentation for both external APIs:

  • New: per-caller metricscaph_hcloud_api_calls_by_caller_total and
    caph_robot_api_calls_by_caller_total label every Client method call with the exact caph
    Go function that made it (via runtime.Caller, no manual bookkeeping per call site). This is
    the most direct answer to "which caph code called hcloud/Robot". Always on; cardinality is
    bounded by the number of call sites in the source, not by fleet size.
  • New: generic Robot API instrumentation — the Robot API previously had zero metrics, only
    debug logging. Added caph_robot_requests_total / caph_robot_request_duration_seconds /
    caph_robot_in_flight_requests, mirroring what hcloud-go already provides for hcloud
    (hcloud_api_requests_total etc.) for free.
  • --metric-per-server-id (renamed from --hcloud-metric-per-server-id) now drives per-
    server-id call counters on both clients: caph_hcloud_getserver_calls_total and the new
    caph_robot_getbmserver_calls_total. One Prometheus time series per distinct server, so it's
    opt-in and meant for bounded debugging runs (e.g. e2e), not permanent production use.
  • Added the Robot equivalent of the always-on per-BootState breakdown:
    caph_robot_getbmserver_calls_by_state_total, labeled by the host's ProvisioningState.
  • e2e suite now scrapes /metrics and prints per-server-id and per-state tables for both APIs
    at the end of a run (previously hcloud only). Also fixes the e2e manager's
    --metrics-bind-address override, which previously used a stale flag name
    (--metrics-bind-addr, missing "ess") and never matched, so /metrics was only reachable via
    localhost inside the pod, in production as well as e2e.
  • Added docs/caph/04-developers/07-third-party-api-metrics.md: a reference for all of the above
    (metrics table, flags, example PromQL queries) plus a "known gaps" section for follow-up work
    (no per-object correlation yet, no rate-limit-remaining gauge, no dashboards yet).
  • Added TestRecordAPICallByCallerCapturesRealCaller in both client packages, pinning down the
    runtime.Caller skip-depth the per-caller label depends on.

First step towards #2163 (reduce GetServer/GetBMServer API call volume) — we want real
per-caller and per-server-id numbers from an actual e2e provisioning run before picking a fix,
since real provisioning involves many more reconciles than any fake-client test can show.

Relates to #2163

Measured results (Test Hcloud Basic (PR) e2e run, hcloud GetServer only)

--------------------------------------------------- GetServer calls per server ID (issue #2163)
  server_id=150545759       GetServer calls=13
  server_id=150548422       GetServer calls=12
  server_id=150545163       GetServer calls=12
  server_id=150546745       GetServer calls=12
  server_id=150547759       GetServer calls=12
  server_id=150544209       GetServer calls=11
  server_id=150544795       GetServer calls=10
  server_id=150547408       GetServer calls=9
  TOTAL GetServer calls=91 across 8 server(s)

8 servers provisioned (control plane + workers) during this run, each with 9-13 GetServer calls,
91 total. This is a real baseline to measure future reductions against. The per-caller and Robot
metrics added later in this PR haven't had a fresh e2e run captured yet.

Test plan

  • go build ./..., go vet ./... pass
  • make lint passes (0 issues)
  • make test-unit passes (one unrelated, pre-existing flake reproduced-clean on 3 reruns of
    ./controllers/... in isolation: a mock ListSSHKeys() call from a stale background
    HetznerBareMetalHost reconcile bleeding into an unrelated HCloudRemediationReconciler
    spec's BeforeEach — nothing in this PR touches that code path)
  • Verified the new YAML replacement matches the actual kustomize build config/default output
    and produces the intended manager args
  • go test ./pkg/services/hcloud/client/..., ./pkg/services/baremetal/client/robot/... pass
  • e2e CI run produces the printed table with non-zero GetServer counts (see above)

Adds --hcloud-metric-per-server-id, a caph_hcloud_getserver_calls_total
Prometheus counter labeled by server_id, and enables it during e2e runs so
a summary table of GetServer calls per server (plus total) is printed at
the end of the suite. Also fixes the e2e manager's --metrics-bind-address
override, which had a stale flag name and never took effect, leaving
/metrics reachable only via localhost inside the pod.

First step towards reducing GetServer call volume during hcloud
provisioning -- we want real numbers from an actual provisioning run
before picking a fix.

Relates to #2163
# Committing as: thomas.guettler@syself.com
@github-actions github-actions Bot added size/M Denotes a PR that changes 50-200 lines, ignoring generated files. area/test Changes made in the test directory area/code Changes made in the code directory labels Jul 13, 2026
guettli added 2 commits July 13, 2026 17:03
Adds caph_hcloud_getserver_calls_by_bootstate_total, labeled by the
reconciler's BootState at call time (server.go's findServer) or a fixed
"remediation" label (remediation.go's separate GetServer call site).
Unlike the per-server-id counter, this is always on since BootState has a
small, fixed set of values, so cardinality stays bounded.

Extends the e2e summary table to also print calls grouped by BootState, so
we can see which code path drives the most GetServer calls, not just the
total per server.

Relates to #2163
# Committing as: thomas.guettler@syself.com
@janiskemper

Copy link
Copy Markdown
Contributor

@guettli doesn't hcloud give us metrics?

@guettli

guettli commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator Author

@guettli doesn't hcloud give us metrics?

@janiskemper yes, hcloud has metrics. But they do not contain the server id. Metrics in general should not contain high cardinality values like session-id, user-id, or in this case server-id.

That's why I introduced a new flag to enable that.

guettli added 2 commits July 15, 2026 11:14
That flag predates this PR and isn't part of its diff, so the doc
shouldn't discuss it or a --debug-robot-api-calls flag mirroring it.
# Committing as: thomas.guettler@syself.com
@guettli guettli changed the title Add optional per-server-id GetServer call metric for e2e measurement 🌱 Add optional per-server-id metrics Jul 15, 2026
@guettli guettli changed the title 🌱 Add optional per-server-id metrics 🌱 Add per-caller and per-server-id metrics for hcloud and robot API calls Jul 15, 2026
guettli added 2 commits July 15, 2026 11:57
caller_test.go imports it directly for reading counter/label values in
tests, so go mod tidy now lists it as direct instead of indirect. CI's
generated-files check caught this.
# Committing as: thomas.guettler@syself.com
@github-actions github-actions Bot added size/L Denotes a PR that changes 200-800 lines, ignoring generated files. and removed size/M Denotes a PR that changes 50-200 lines, ignoring generated files. labels Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/code Changes made in the code directory area/test Changes made in the test directory size/L Denotes a PR that changes 200-800 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants