Skip to content

fix(telemetry): preserve model-loading Prometheus metric name - #784

Merged
huronat merged 2 commits into
mainfrom
fix/model-load-prometheus-compatibility
Oct 17, 2026
Merged

huronat merged 2 commits into
mainfrom
fix/model-load-prometheus-compatibility

Conversation

@huronat

@huronat huronat commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Preserve the existing sie_gateway_model_loads_in_progress Prometheus family used by the chart's KEDA queries.

The canonical gauge uses unit 1. The pinned OpenTelemetry Collector 0.119.0 adds _ratio for that dimensionless gauge, so its exported name does not match the existing unsuffixed KEDA query. Clear the unit only in the local Prometheus compatibility transform, alongside the existing pinned-model compatibility override. The producer and remote OTLP branch retain the canonical name and unit 1.

Validation

  • Two local/dual-branch render regressions failed before the change and pass afterward.
  • A real cached Collector 0.119.0 regression reproduced the _ratio family before the fix. Afterward it verifies the exact unsuffixed family, three lane tuples with explicit zero/one values, producer/generation labels, and unchanged canonical remote name/unit/values.
  • Ten focused cases pass after integrating current main, including seven mocked image-prerequisite/cleanup cases. The runtime test uses only a pre-cached image, loopback ports, bounded CPU/memory/PIDs, and verifies removal of its exact test-owned container. It does not pull images or contact an external telemetry service.
  • Offline lock check, CI-scoped Ruff, formatting, and diff checks pass.
  • The full chart render suite was not run locally because the matching chart dependencies are not prepared; normal PR CI remains required.

Scope

Four production configuration/contract lines plus regression tests. No KEDA query, threshold, resource, label, freshness, receiver-identity, or canonical OTLP changes. No deployment or release publication.

Review classification

Ordinary metric compatibility repair. No authentication/authorization changes and below both very-large-change thresholds. CodeRabbit and required CI apply.

Summary by CodeRabbit

  • Telemetry
    • Prometheus exposes the model-loading gauge as sie_gateway_model_loads_in_progress without a unit suffix, preserving a consistent metric name for dashboards and queries.
    • The remote exporter continues to report the metric with its original name and unit, so its existing output remains unchanged.

@huronat
huronat requested a review from a team as a code owner October 17, 2026 11:43
@coderabbitai

coderabbitai Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Team
  • Run ID: bbf8d00e-1f31-49c1-b97c-90e50eb37059
📥 Commits

Reviewing files that changed from the base of the PR and between 5e79389 and 2769dff.

📒 Files selected for processing: 1
  • tools/ci/tests/test_collector_metric_privacy.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 6 remain after this review.

📝 Walkthrough

The change adds Prometheus compatibility mapping for sie.gateway.model_loads_in_progress. Helm-rendering checks and a collector integration test verify its metric name, unit handling, gauge samples, and remote output.

Changes: Model-loading metric compatibility

Layer / File(s) Summary
Metric mapping and rendered configuration
telemetry/contract.yaml, deploy/helm/sie-cluster/templates/_otel-collector-config.tpl, tools/ci/tests/test_helm_render.py
The contract and collector configuration map the model-loading gauge to the Prometheus family name and clear its unit. Rendered-configuration tests check the mapping, pipeline ordering, and worker trigger metadata.
Collector test setup and failure handling
tools/ci/tests/test_collector_metric_privacy.py
Helpers and tests check the cached-image prerequisite, fixture-owner container user, bounded startup diagnostics, and verified cleanup behavior.
Collector metric output integration test
tools/ci/tests/test_collector_metric_privacy.py
The integration test sends three gauge points through the rendered collector configuration. It checks Prometheus samples, remote output, and container cleanup.

Suggested reviewers: krisztian-gajdar

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 2769d

No identified issue blocks merging after normal CI checks.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 26 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: preserving the model-loading Prometheus metric name.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches: 💡 1
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

coderabbitai[bot]
coderabbitai Bot previously approved these changes Oct 17, 2026
@blacksmith-sh

blacksmith-sh Bot commented Oct 17, 2026

Copy link
Copy Markdown

Found 1 test failure on Blacksmith runners:

Failure

Test View Logs
test_model_loading_gauge_preserves_prometheus_name_and_remote_unit View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@huronat

huronat commented Oct 17, 2026

Copy link
Copy Markdown
Contributor Author

Test-only follow-up 2769dff addresses the collector fixture startup failure in CI39029792443. The private pytest directory is now accessed as its owning UID:GID instead of UID0 with all capabilities dropped. The existing15-second readiness wait, CPU/memory/PID limits, capability drop, read-only root, loopback ports and no-pull policy are unchanged. Startup failure now retains exact-container status and the last20 log lines before cleanup, with five-second diagnostic command deadlines and4096characters retained per command. Malformed diagnostic text cannot replace the original failure. The previous container logs were not retained, so this fixes the verified fixture ownership incompatibility without claiming the exact original startup error was captured. Root validation:13focused tests passed, including the pinned0.119.0 collector wire check; lock, Ruff and diff checks pass. All production changes remain the same four contract/template lines. Fresh CI39031680580 and current-head review are pending.

@huronat
huronat merged commit 04f8080 into main Oct 17, 2026
40 checks passed
@huronat
huronat deleted the fix/model-load-prometheus-compatibility branch October 17, 2026 12:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant