[NA] [DOCS] alerts: you can alert on custom metrics, and the new alert form - #8086
[NA] [DOCS] alerts: you can alert on custom metrics, and the new alert form#8086aswynz wants to merge 3 commits into
Conversation
The alerts page listed the ten alert event types but never said where feedback scores come from, so the closed list of event types read as a closed list of alertable metrics. Nothing linked custom metrics to alerts in either direction. Feedback score alerts threshold on a score *name* and the query applies no filter on which system produced the score, so any score logged to a project is alertable — including custom metrics, whether they run as an online evaluation rule or are scored in the user's own pipeline and pushed via log_traces_feedback_scores / the REST API. - alerts: expand the Feedback Score Name bullet and add an "Alerting on custom metrics" section covering both paths, plus the two real limits (offline experiment metrics are not thresholdable; the window must be wider than the scoring cadence or the alert is skipped for lack of data). - custom metrics: add a production section and a What's next link, so the page a user lands on first points at alerts. - online evaluation: note that custom rule scores can drive alerts, and fix a "Custome" typo. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
documentation
⏱️ pre-commit per-hook timingNo linted files changed — nothing to run. ⏭️ 44 skipped (no matching files changed)
|
|
No test needed here. No product surface in this PR — only docs, CI, tests, lockfiles or other non-product files. Advisory, from the QA test radar. Nothing here blocks this PR, and anything it proposes is a draft for review. Re-checked after a push on 02 Sep 19:45 UTC. |
|
🌿 Preview your docs: https://opik-preview-01a063a6-c794-77c6-8318-bdc48ed6eea3.docs.buildwithfern.com/docs/opik No broken links found Unverified links (timeout / rate-limited / server error — not failing the check)• https://aistudio.google.com/apikey (401) 📌 Results for commit 6e287db |
…ow the new alert form Two changes. Custom metric content now lives on the custom metrics page, where it belongs. The alerts page keeps only a short note that custom metrics can be alerted on, plus the one alert-specific caveat (the score must be on a production trace), and links out for the rest. The two ways to get a custom metric scoring production traces, the code sample and the window guidance moved to custom_metric.mdx under "Using your custom metric in production". The step-by-step guide now follows the reworked alert form from #8099 (OPIK-8198): - Triggers come before webhook settings. - Test connection sits on the Endpoint URL row; the separate test panel is gone. Each trigger has its own "Example payload" toggle and "Test trigger" button. - The name is suggested from the first trigger and comes last; new alerts are enabled on create, and "Enable alert" is an edit-only control at the bottom of the form. - The Slack and PagerDuty setup lists follow the same order. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
aswynz
changed the title
Intro now says what an alert is for and frames setup as the two decisions a user actually makes — when to be alerted (event type plus filter conditions) and who to notify and how (the destination). The old opening led with the webhook mechanism, which is the implementation detail rather than the point. Under "Supported event types", add a table of all ten events with their event type and a one-line trigger summary. Each name links to that event's full description. The descriptions were bold text with no anchors to link to, so promote the ten names to headings; they now appear in the page nav too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2 participants
Details
Docs only. Five things, all on the alerts page except where noted:
New intro. It now says what alerts are for and frames setup as the two decisions you actually make: when do you want to be alerted (event type + filter conditions), and who to notify and how (the destination). The old opening led with webhooks, which is the mechanism, not the point. "Creating an alert" below it is unchanged.
Event type table. All ten events in one table with their event type and a one-line trigger summary; each name jumps to its full description. Those descriptions were bold text with no anchors, so the ten names are now headings — which also puts them in the page nav.
Say that custom metrics can be alerted on. A customer read this page and thought only Opik's own metrics could have thresholds. You can't tell from the page, because it lists the events but never says where feedback scores come from. Alerts match on a score name, and any score in the project counts — so custom metrics already work. Added a short note.
Keep the detail on the custom metrics page. How to get a custom metric scoring production traces is a custom-metrics topic. That content now lives in
custom_metric.mdx. The alerts page keeps only the note above, one alert-specific caveat (the score has to be on a production trace), and a link.Follow the new alert form from [OPIK-8198] [FE] alerts: put Triggers before Webhook settings #8099. The step-by-step described the old layout. Rewritten: triggers before webhook settings, Test connection next to the Endpoint URL, per-trigger "Example payload" and "Test trigger", the name suggested from your first trigger and set last, "Enable alert" edit-only. Slack and PagerDuty setup steps reordered to match.
Also fixed a
Custometypo on the online evaluation page.Change checklist
Issues
No ticket —
[NA]. Came out of a support question about alerting on custom metrics.Depends on #8099 (OPIK-8198), which ships the form changes in point 5. Please merge this after that one, or the step-by-step will describe a UI that isn't live yet.
AI-WATERMARK
AI-WATERMARK: yes
Testing
Docs only, so no test suite. What I did:
fern check→ 0 errors. The 4 warnings are pre-existing and unrelated (unauthenticated redirect check, two colour-contrast warnings, a${POSTHOG_API_HOST}placeholder).fern docs devand loaded all three pages. Confirmed each of the 10 table links points at an anchor that actually exists in the rendered page, the table renders as a table, and the cross-page links resolve. Note: the dev server logs a successful reload but keeps serving the old page — it needs a restart to check properly.MetricsAlertJob.java:440.ProjectMetricsDAO.java:980-991matches onnameonly.FeedbackDefinitionsAndScoresSelectBox.tsx:116-143.MetricsAlertJob.java:246-253. Basis for the window guidance.Test connectionon the Endpoint URL row,Example payload/Test triggerper trigger, name suggestions fromalertNameHelpers.ts(e.g.Trace errors > 5 in 5 mins,Trace feedback score: policy_compliance < 0.8),Enable alertedit-only.Not done:
create_alert_form.pngis out of date — it shows the old layout. Left in place rather than deleted; needs a re-shoot once [OPIK-8198] [FE] alerts: put Triggers before Webhook settings #8099 merges, and the screenshots from that PR would work.Documentation
This PR is the documentation change:
production/alerts/alerts.mdxevaluation/metrics/custom_metric.mdxproduction/online-evaluation/rules.mdxTwo things for a docs owner
<— the usual choice for a quality metric. I kept the existing names so the new table matches the headings; happy to rename both in a follow-up.🤖 Generated with Claude Code