feat(epp): add OTel span to prefix cache affinity filter decisions - #2594
Open
sudoalok wants to merge 1 commit into
Open
feat(epp): add OTel span to prefix cache affinity filter decisions#2594sudoalok wants to merge 1 commit into
sudoalok wants to merge 1 commit into
Conversation
Filter records one of five decision outcomes as a metric but emits no span, so a trace shows a request passing through the filter without showing what it decided or why. Adds a filter_prefix_cache_affinity span carrying the decision using the same outcome constants the metric records, so the trace and the metric cannot drift. Also records the candidate and sticky endpoint counts, and the TTFT penalty the load gate compares against its threshold; the penalty is recorded whether or not the gate trips. Fixes llm-d#2539 Signed-off-by: Alok Behera <alokbeherak061@gmail.com>
github-actions
Bot
added
size/M
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The prefix-cache-affinity filter already records its decision as a metric,
but there's no span for it. So when you look at a trace you can see the
request went through the filter, but not what it decided or why.
This adds a
filter_prefix_cache_affinityspan, following the samepattern as
pd_profile_handler.go.What's on the span:
llm_d.epp.filter.decision- the outcome. I'm setting this from thesame
outcome*constants thatrecordDecisionalready uses, so thespan value and the metric label are always the same thing and can't
drift apart later.
llm_d.epp.filter.candidate_endpointsandllm_d.epp.filter.sticky_endpoints- so you can see how much thefilter narrowed things down.
llm_d.epp.filter.ttft_penalty_ms- what the load gate actuallymeasured. I set this before the threshold check, so you still see the
number on traces where the gate didn't trip.
gen_ai.request.modelandgen_ai.request.id, same as the otherscheduling plugins.
Two small things while I was in there: the request param was
_so Inamed it, since I need it for the request attributes. And
bestStickyTTFT-bestNonStickyTTFTwas being calculated twice, so Ipulled it into a
penaltyvariable.I didn't add a span test -
disaggandpreciseprefixcachedon't assertspans in their unit tests either, so I followed what they do. Let me know
if you'd rather have one.
Also left out
mmobs.SpanAttributes(request)that the siblings have,since this filter doesn't deal with multimodal at all. Easy to add if you
want it for consistency.
Fixes #2539