feat: Scrape VRAM utilization from vLLM endpoints

[yehuditkerido]

Implement the first metric: VRAM (KV cache) utilization from vLLM. This is the primary metric for routing decisions — route away from clusters with high memory pressure.

Why vLLM for VRAM?

• vllm:gpu_cache_usage_perc directly measures KV cache utilization in GPU memory
• This is what determines if a cluster can accept new requests with long contexts
• Example from requirements: "routing away from 95% VRAM to 20%"

What's Included:

• ClusterMetricsSource interface definition
• ClusterMetrics struct with VRAMUtilization field
• Prometheus text format parser using expfmt
• HTTP client with configurable timeout
• Scraper that extracts vllm:gpu_cache_usage_perc
• Controller integration: scrape metrics during reconciliation
• Expose scraped metrics on /metrics endpoint

vLLM Metrics Reference:

• Endpoint: http://<vllm-host>:8000/metrics
• Metric: vllm:gpu_cache_usage_perc (Gauge, 0.0-1.0)
• Docs: https://docs.vllm.ai/en/stable/usage/metrics/

Code Structure:

// pkg/metrics/types.go
type ClusterMetrics struct {
    ClusterName      string
    VRAMUtilization  *float64  // vllm:gpu_cache_usage_perc (0.0-1.0)
    LatencyMs        *float64  // llm_d_router_epp_request_duration (added in Issue 3)
    LastUpdated      time.Time
    Healthy          bool
    Error            error
}

// pkg/metrics/source.go
type ClusterMetricsSource interface {
    GetMetrics(ctx context.Context, cluster string) (*ClusterMetrics, error)
}

// pkg/metrics/vllm/scraper.go
type VLLMScraper struct {
    client  *http.Client
    timeout time.Duration
}

func (s *VLLMScraper) ScrapeVRAM(ctx context.Context, endpoint string) (float64, error)

Acceptance Criteria:

• ClusterMetricsSource interface defined
• vLLM scraper extracts vllm:gpu_cache_usage_perc
• Handles connection errors gracefully (sets Healthy=false)
• Controller logs VRAM utilization for each cluster
• Unit tests with mock HTTP server
• Metrics exposed at /metrics endpoint

Demo:

# Controller logs show VRAM utilization
kubectl logs -l app=weight-controller -n maas-system
# Output: "Cluster B: vram_util=0.20, healthy=true"
# Output: "Cluster C: vram_util=0.95, healthy=true"

# Prometheus can scrape controller metrics
curl http://weight-controller:8080/metrics
# Output: weight_controller_cluster_vram_utilization{cluster="B"} 0.20
# Output: weight_controller_cluster_vram_utilization{cluster="C"} 0.95

Dependencies: #1060

[yehuditkerido]

/assign

[liavweiss]

/assign

[yehuditkerido]

/unassign