🌱 Add per-caller and per-server-id metrics for hcloud and robot API calls - #2166
Open
guettli wants to merge 7 commits into
Open
🌱 Add per-caller and per-server-id metrics for hcloud and robot API calls#2166guettli wants to merge 7 commits into
guettli wants to merge 7 commits into
Conversation
Adds --hcloud-metric-per-server-id, a caph_hcloud_getserver_calls_total Prometheus counter labeled by server_id, and enables it during e2e runs so a summary table of GetServer calls per server (plus total) is printed at the end of the suite. Also fixes the e2e manager's --metrics-bind-address override, which had a stale flag name and never took effect, leaving /metrics reachable only via localhost inside the pod. First step towards reducing GetServer call volume during hcloud provisioning -- we want real numbers from an actual provisioning run before picking a fix. Relates to #2163 # Committing as: thomas.guettler@syself.com
Adds caph_hcloud_getserver_calls_by_bootstate_total, labeled by the reconciler's BootState at call time (server.go's findServer) or a fixed "remediation" label (remediation.go's separate GetServer call site). Unlike the per-server-id counter, this is always on since BootState has a small, fixed set of values, so cardinality stays bounded. Extends the e2e summary table to also print calls grouped by BootState, so we can see which code path drives the most GetServer calls, not just the total per server. Relates to #2163 # Committing as: thomas.guettler@syself.com
Contributor
|
@guettli doesn't hcloud give us metrics? |
Collaborator
Author
@janiskemper yes, hcloud has metrics. But they do not contain the server id. Metrics in general should not contain high cardinality values like session-id, user-id, or in this case server-id. That's why I introduced a new flag to enable that. |
That flag predates this PR and isn't part of its diff, so the doc shouldn't discuss it or a --debug-robot-api-calls flag mirroring it. # Committing as: thomas.guettler@syself.com
caller_test.go imports it directly for reading counter/label values in tests, so go mod tidy now lists it as direct instead of indirect. CI's generated-files check caught this. # Committing as: thomas.guettler@syself.com
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
We want to know which caph code is calling the hcloud/Robot APIs, and how often, so rate-limit
and traffic-volume issues (see #2163) can actually be debugged instead of guessed at. This PR
started as a hcloud-only, opt-in per-server-id metric and grew into general-purpose
instrumentation for both external APIs:
caph_hcloud_api_calls_by_caller_totalandcaph_robot_api_calls_by_caller_totallabel everyClientmethod call with the exact caphGo function that made it (via
runtime.Caller, no manual bookkeeping per call site). This isthe most direct answer to "which caph code called hcloud/Robot". Always on; cardinality is
bounded by the number of call sites in the source, not by fleet size.
debug logging. Added
caph_robot_requests_total/caph_robot_request_duration_seconds/caph_robot_in_flight_requests, mirroring what hcloud-go already provides for hcloud(
hcloud_api_requests_totaletc.) for free.--metric-per-server-id(renamed from--hcloud-metric-per-server-id) now drives per-server-id call counters on both clients:
caph_hcloud_getserver_calls_totaland the newcaph_robot_getbmserver_calls_total. One Prometheus time series per distinct server, so it'sopt-in and meant for bounded debugging runs (e.g. e2e), not permanent production use.
caph_robot_getbmserver_calls_by_state_total, labeled by the host'sProvisioningState./metricsand prints per-server-id and per-state tables for both APIsat the end of a run (previously hcloud only). Also fixes the e2e manager's
--metrics-bind-addressoverride, which previously used a stale flag name(
--metrics-bind-addr, missing "ess") and never matched, so/metricswas only reachable vialocalhost inside the pod, in production as well as e2e.
docs/caph/04-developers/07-third-party-api-metrics.md: a reference for all of the above(metrics table, flags, example PromQL queries) plus a "known gaps" section for follow-up work
(no per-object correlation yet, no rate-limit-remaining gauge, no dashboards yet).
TestRecordAPICallByCallerCapturesRealCallerin both client packages, pinning down theruntime.Callerskip-depth the per-caller label depends on.First step towards #2163 (reduce
GetServer/GetBMServerAPI call volume) — we want realper-caller and per-server-id numbers from an actual e2e provisioning run before picking a fix,
since real provisioning involves many more reconciles than any fake-client test can show.
Relates to #2163
Measured results (Test Hcloud Basic (PR) e2e run, hcloud GetServer only)
8 servers provisioned (control plane + workers) during this run, each with 9-13
GetServercalls,91 total. This is a real baseline to measure future reductions against. The per-caller and Robot
metrics added later in this PR haven't had a fresh e2e run captured yet.
Test plan
go build ./...,go vet ./...passmake lintpasses (0 issues)make test-unitpasses (one unrelated, pre-existing flake reproduced-clean on 3 reruns of./controllers/...in isolation: a mockListSSHKeys()call from a stale backgroundHetznerBareMetalHostreconcile bleeding into an unrelatedHCloudRemediationReconcilerspec's
BeforeEach— nothing in this PR touches that code path)kustomize build config/defaultoutputand produces the intended manager args
go test ./pkg/services/hcloud/client/...,./pkg/services/baremetal/client/robot/...passGetServercounts (see above)