Feature: LLM Observability for HTTP Service Calls

[akshat-kumar-singhal]

Problem

GoFr tracks standard HTTP metrics (app_http_service_response) for outgoing service calls, but has no awareness of LLM-specific semantics. When users make HTTP calls to LLM providers (OpenAI, Anthropic, etc.), they get generic HTTP observability but no insight into:

• Token consumption (prompt vs completion tokens)
• Per-model usage breakdown
• Token distribution patterns across requests

Proposal

Add an HTTP service decorator in a new package pkg/gofr/service/observability/ that intercepts responses from LLM APIs, extracts usage data from response bodies and headers, and records LLM-specific metrics.

Metrics

Metric	Type	Labels	Purpose
app_llm_token_usage	Counter	provider, model, token_type (prompt/completion/total)	Cumulative token consumption
app_llm_request_count	Counter	provider, model, operation, status (success/error)	Request count by outcome
app_llm_tokens_per_request	Histogram	provider, model, token_type (prompt/completion/total)	Token distribution per request (helps spot outlier prompts)

Latency is already covered by the existing app_http_service_response histogram.

Pluggable Parser Interface

A ResponseParser interface allows users to add support for any LLM provider:

type ResponseParser interface {
    ParseResponse(headers http.Header, body []byte) (*Usage, error)
}

type Usage struct {
    Model            string
    PromptTokens     int
    CompletionTokens int
    TotalTokens      int
}

Built-in parsers ship for OpenAI (also covers Azure OpenAI, Groq, Together, etc.) and Anthropic response formats. More can be added later.

User-Facing API

import "gofr.dev/pkg/gofr/service/observability"

// OpenAI-compatible provider (parser auto-selected from provider name)
a.AddHTTPService("openai", "https://api.openai.com",
    &observability.LLMConfig{Provider: "openai"},
    &service.RetryConfig{MaxRetries: 3},
)

// Custom provider with custom parser
a.AddHTTPService("custom-llm", "https://my-llm.example.com",
    &observability.LLMConfig{
        Provider: "custom",
        Parser:   &myCustomParser{},
    },
)

How It Works

The decorator follows GoFr's existing service.Options pattern (same as CircuitBreaker, Retry, RateLimiter):

1. Wraps the HTTP service in the decorator chain
2. After each response, reads the body (buffered and replaced so downstream consumers still work)
3. Calls the parser to extract token usage from headers + body
4. Records metrics and structured logs
5. Passes the response through unchanged

Structured Logging

An llmLog struct with PrettyPrint support logs each LLM call with: provider, model, operation, prompt tokens, completion tokens, total tokens, and status.

Design Decisions

• Separate package (service/observability/) rather than adding to the already-large service/ package. Other observability concerns can move here later.
• No cost tracking — token-to-cost mapping changes frequently and varies by provider. Users can derive cost from token metrics externally.
• No streaming-specific metrics (TTFT, chunk count) — can be added in a follow-up.
• No rate limit metrics — header formats vary too much across providers. Can be added later.
• Response body buffering — the decorator reads and replaces the body so downstream handlers still work normally.

Feedback Welcome

Looking for input on:

• Are the metric names and labels appropriate?
• Any additional metrics that should be tracked?
• Should the package name be different?
• Any concerns about the response body buffering approach?

[akshat-kumar-singhal]

@aryanmehrotra @Umang01-hash please share your thoughts - will implement it accordingly

[aryanmehrotra]

It's something we should definitely have but before that, I just created a discussion around how we can enable users to connect to llm and create agents etc, because gofr has a lot of inbuilt things which can help build production grade agents.

https://github.com/orgs/gofr-dev/discussions/3644 -> if you can check this out as well, and we can decide on how we should proceed.

[armorer-labs]

This looks useful, especially if the same HTTP client may be used inside agent/tool loops rather than only direct chat completions.

A few metric/label additions I would consider keeping optional so cardinality stays sane:

• agent_run_id or a sampled/redacted run correlation id, not as a default high-cardinality label on every metric, but available in structured logs or exemplars.
• tool_name / operation when the LLM call is part of a tool or agent step. That makes token drift easier to trace back to a workflow change.
• request_role: planning, tool_selection, summarization, extraction, verifier, repair. These buckets are often more useful than endpoint names once agents are involved.
• streaming=true/false and, later, ttft_ms for streaming providers. Total latency hides a lot of user-visible pain.
• parser_status: parsed, unsupported_provider, malformed_usage, missing_usage. Otherwise usage extraction failures can silently disappear.

For buffering, I would make the size cap and redaction behavior explicit. LLM responses can be large, and observability code should fail open without changing the response path unless the user opts into strict behavior.

Disclosure: I work on Armorer Labs. We see a similar pattern in agent ops: token metrics are useful, but they become much more actionable when they can be joined back to the specific run step, tool call, verifier, or retry that produced them.

[aryanmehrotra]

about agent_run_id exposing it as metrics would not be a great way - which would result in all the metrics being exposed will be for getting an overall idea, to know about how an agent processed etc trace and logs should be preferred

[aryanmehrotra]

https://github.com/gofr-dev/gofr/releases/tag/v1.58.0 - this release has almost all the features we were discussing here

[aryanmehrotra]

reopen if needed