Skip to content

Commit 94b7b7d

Browse files
authored
Merge pull request #12 from ding-labs/add-buildkite-annotate-notifier
feat(notifier): buildkite_annotate notifier (Wave 1 Tier-2 #2)
2 parents 6b848e8 + 76fc100 commit 94b7b7d

7 files changed

Lines changed: 451 additions & 3 deletions

File tree

ding.yaml.example

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,12 @@ notifiers:
9393
# alert-gitlab:
9494
# type: gitlab_artifact
9595
# # path: ding-alerts.md # default; relative to CWD ($CI_PROJECT_DIR in GitLab CI)
96+
#
97+
# Buildkite annotation (publishes alerts as Buildkite build annotations via
98+
# `buildkite-agent annotate`; visible at the top of the build UI):
99+
# alert-buildkite:
100+
# type: buildkite_annotate
101+
# # style: error # default; success | info | warning | error
96102

97103
rules:
98104
# Event-per-event: fires on a single reading above threshold.

docs/configuration.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -215,6 +215,25 @@ notifiers:
215215

216216
**No CI gate**: the notifier writes the file regardless of whether it's running in GitLab CI. Outside CI, it just produces a local `ding-alerts.md` — harmless. Combine with `.gitlab-ci.yml` `artifacts: { when: always, paths: [ding-alerts.md] }` to archive the file on every pipeline run (including failed jobs). See the [GitLab CI recipe](recipes/gitlab-ci.md#native-gitlab-ui-surfacing) for an end-to-end example.
217217

218+
### `type: buildkite_annotate`
219+
220+
Publishes alerts as Buildkite build annotations via `buildkite-agent annotate`. All alerts for a build land in a single rolling annotation (`--context ding --append`) shown at the top of the Buildkite job UI. Requires `buildkite-agent` on PATH (always set inside Buildkite jobs); outside Buildkite the notifier no-ops gracefully after a one-time warning.
221+
222+
```yaml
223+
notifiers:
224+
annotate:
225+
type: buildkite_annotate
226+
style: error # success | info | warning | error; default error
227+
```
228+
229+
| Field | Default | Notes |
230+
|-------|---------|-------|
231+
| `style` | `error` | Buildkite annotation style. Drives the colored badge in the build UI. |
232+
233+
**Behavior**: sync, mutex-guarded. The first `Send()` writes a `# DING Alerts` H1 header; subsequent calls append `## <rule>` sections that Buildkite's `--append` concatenates into the existing annotation body. No async queue, no retry, no metrics — failures from `buildkite-agent` (agent disconnected, body too large, etc.) are returned from `Send()` with stderr captured.
234+
235+
**No CI gate**: the notifier checks for `buildkite-agent` once at construction; outside Buildkite jobs it logs `ding: buildkite_annotate notifier: buildkite-agent not on PATH; alerts via this notifier will be no-ops` and Send becomes a no-op. See the [Buildkite recipe](recipes/buildkite.md#native-buildkite-ui-surfacing) for an end-to-end example.
236+
218237
### `type: webhook`
219238

220239
Posts a flat JSON payload to any HTTP endpoint. Useful for generic integrations (PagerDuty, custom receivers, etc.).

docs/recipes/buildkite.md

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -66,19 +66,33 @@ Use these in `match.labels` or `message` templates. See [Configuration](../confi
6666

6767
If the alert doesn't fire, check the Buildkite build log for `ding` output. Common issues: webhook URL not exposed (env hook scope, agent vs pipeline level), or `drain_timeout` shorter than the notifier retry window — see [Configuration](../configuration.md).
6868

69+
## Native Buildkite UI surfacing
70+
71+
If you want DING alerts to surface as Buildkite build annotations (visible at the top of the build UI) rather than (or alongside) external notifiers, DING ships a built-in `type: buildkite_annotate` notifier. See [`type: buildkite_annotate`](../configuration.md#type-buildkite_annotate) for the full reference.
72+
73+
Add to `ding.yaml`:
74+
75+
```yaml
76+
notifiers:
77+
annotate:
78+
type: buildkite_annotate
79+
# style: error # default; success | info | warning | error
80+
```
81+
82+
No changes to `.buildkite/pipeline.yml` are needed — the `buildkite-agent` CLI is already on PATH inside Buildkite jobs. All DING alerts from a build land in a single rolling annotation (`--context ding --append`) so the UI stays uncluttered. Outside Buildkite (e.g. local dev), the notifier no-ops gracefully after a one-time warning.
83+
6984
## Tradeoffs / known limitations
7085

7186
- **No `job` label by default.** runctx captures Buildkite's pipeline-level identifiers but not step-level (`BUILDKITE_STEP_KEY`). Add explicit `match.labels` if you need per-step rules.
7287
- **Binary download per step.** Bake DING into your agent image, or use a [`pre-command` hook](https://buildkite.com/docs/agent/v3/hooks#available-hooks) to install it once per agent.
73-
- **Annotation surface unused.** Buildkite has `buildkite-agent annotate`, the analogue of GHA's `$GITHUB_STEP_SUMMARY`. The minimal recipe doesn't use it; surfacing alerts back into the build UI would be a Tier-2 abstraction (`type: buildkite_annotate` notifier).
7488

7589
## Escalation criteria
7690

7791
This recipe is **a Tier-2 candidate** by the program's standard rubric:
7892

7993
- **Setup commands required:** 1 (`curl | tar`) — under threshold of 5
8094
- **Boilerplate lines:** ~24 — under threshold of 50
81-
- **"Gotcha" callouts:** 3 (no `job` label, binary download, no annotation surface) — over threshold of 2 → **Tier-2 candidate**
95+
- **"Gotcha" callouts:** 2 (no `job` label, binary download) — at threshold of 2
8296
- **End-to-end runnable:** yes (Buildkite has a free trial; the underlying agent is OSS and self-hostable indefinitely)
8397

84-
**Tier-2 candidate.** The structural friction is "annotations not used" — Buildkite users expect alerts to land in the build UI, not just Slack. A `type: buildkite_annotate` notifier (calling `buildkite-agent annotate --style error --context ding`) is the natural Tier-2 abstraction. Sequence it after GitLab CI's artifact notifier (similar shape).
98+
Buildkite-native alert surfacing now ships as the built-in [`type: buildkite_annotate`](../configuration.md#type-buildkite_annotate) notifier (covered in the section above). The remaining gotchas (no `job` label, binary download per step) are environmental rather than implementation gaps; the recipe stays Tier 1.

internal/config/config.go

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,8 @@ type NotifierConfig struct {
8585
EventType string `yaml:"event_type,omitempty"` // K8s Event type, "Normal" or "Warning" (default "Warning")
8686
// Fields below are specific to type: gitlab_artifact. All optional.
8787
Path string `yaml:"path,omitempty"` // artifact file path (default "ding-alerts.md")
88+
// Fields below are specific to type: buildkite_annotate. All optional.
89+
Style string `yaml:"style,omitempty"` // annotation style: success | info | warning | error (default "error")
8890
}
8991

9092
type AlertTarget struct {
@@ -320,6 +322,14 @@ func (cfg *Config) Validate() error {
320322
cfg.Notifiers[name] = nc
321323
case "gitlab_artifact":
322324
// No required fields; path defaults to "ding-alerts.md" if empty.
325+
case "buildkite_annotate":
326+
if nc.Style != "" {
327+
switch nc.Style {
328+
case "success", "info", "warning", "error":
329+
default:
330+
return fmt.Errorf("notifier %q: buildkite_annotate type requires style to be one of \"success\", \"info\", \"warning\", \"error\"", name)
331+
}
332+
}
323333
case "":
324334
return fmt.Errorf("notifier %q: type is required", name)
325335
default:
Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
package notifier
2+
3+
import (
4+
"bytes"
5+
"fmt"
6+
"log"
7+
"os/exec"
8+
"sort"
9+
"strings"
10+
"sync"
11+
"time"
12+
13+
"github.com/ding-labs/ding/internal/evaluator"
14+
)
15+
16+
// cmdRunner runs the buildkite-agent CLI. Tests inject a stub.
17+
type cmdRunner func(args []string, stdin string) error
18+
19+
func defaultCmdRunner(args []string, stdin string) error {
20+
if len(args) == 0 {
21+
return fmt.Errorf("cmdRunner: empty args")
22+
}
23+
cmd := exec.Command(args[0], args[1:]...)
24+
cmd.Stdin = strings.NewReader(stdin)
25+
var stderr bytes.Buffer
26+
cmd.Stderr = &stderr
27+
if err := cmd.Run(); err != nil {
28+
msg := strings.TrimSpace(stderr.String())
29+
if msg == "" {
30+
return err
31+
}
32+
return fmt.Errorf("%w (stderr: %s)", err, msg)
33+
}
34+
return nil
35+
}
36+
37+
// lookPath is overridable for tests.
38+
var lookPath = exec.LookPath
39+
40+
// BuildkiteAnnotateNotifier publishes DING alerts as Buildkite build
41+
// annotations by shelling out to `buildkite-agent annotate`. All alerts
42+
// for a build land in a single rolling annotation (--context ding
43+
// --append). First Send writes a `# DING Alerts` H1 header; subsequent
44+
// Sends append `## <rule>` sections only — Buildkite's --append mode
45+
// concatenates each invocation's stdin into the existing annotation body.
46+
//
47+
// Sync, mutex-guarded. Outside Buildkite (buildkite-agent not on PATH),
48+
// logs once at construction and Send becomes a no-op — same graceful-
49+
// degrade philosophy as the github_actions notifier.
50+
type BuildkiteAnnotateNotifier struct {
51+
mu sync.Mutex
52+
style string
53+
runner cmdRunner
54+
available bool // false if buildkite-agent not on PATH at construction
55+
wroteHeader bool
56+
}
57+
58+
// NewBuildkiteAnnotateNotifier constructs a notifier that publishes DING
59+
// alerts as Buildkite annotations. style defaults to "error" if empty;
60+
// validated by config.Validate. Caller is responsible for ensuring style
61+
// is one of "success", "info", "warning", "error".
62+
func NewBuildkiteAnnotateNotifier(style string) *BuildkiteAnnotateNotifier {
63+
if style == "" {
64+
style = "error"
65+
}
66+
n := &BuildkiteAnnotateNotifier{
67+
style: style,
68+
runner: defaultCmdRunner,
69+
}
70+
if _, err := lookPath("buildkite-agent"); err != nil {
71+
log.Printf("ding: buildkite_annotate notifier: buildkite-agent not on PATH; alerts via this notifier will be no-ops")
72+
n.available = false
73+
} else {
74+
n.available = true
75+
}
76+
return n
77+
}
78+
79+
// Send invokes `buildkite-agent annotate --style <style> --context ding
80+
// --append` with the rendered Markdown as stdin. No-op (returns nil)
81+
// if buildkite-agent wasn't on PATH at construction.
82+
func (n *BuildkiteAnnotateNotifier) Send(alert evaluator.Alert) error {
83+
n.mu.Lock()
84+
defer n.mu.Unlock()
85+
86+
if !n.available {
87+
return nil
88+
}
89+
90+
body := n.renderBody(alert)
91+
args := []string{"buildkite-agent", "annotate", "--style", n.style, "--context", "ding", "--append"}
92+
if err := n.runner(args, body); err != nil {
93+
return fmt.Errorf("buildkite_annotate Send: %w", err)
94+
}
95+
return nil
96+
}
97+
98+
// renderBody produces the per-invocation stdin payload. Caller must hold n.mu;
99+
// modifies n.wroteHeader.
100+
func (n *BuildkiteAnnotateNotifier) renderBody(alert evaluator.Alert) string {
101+
var b strings.Builder
102+
if !n.wroteHeader {
103+
b.WriteString("# DING Alerts\n\n")
104+
n.wroteHeader = true
105+
}
106+
fmt.Fprintf(&b, "## %s\n\n", alert.Rule)
107+
if alert.Message != "" {
108+
fmt.Fprintf(&b, "%s\n\n", alert.Message)
109+
}
110+
fmt.Fprintf(&b, "- **Metric:** `%s`\n", alert.Metric)
111+
fmt.Fprintf(&b, "- **Value:** `%v`\n", alert.Value)
112+
fmt.Fprintf(&b, "- **Fired:** `%s`\n", alert.FiredAt.Format(time.RFC3339))
113+
if alert.Count > 0 || alert.Avg != 0 || alert.Sum != 0 {
114+
fmt.Fprintf(&b, "- **Aggregates:** count=`%v` avg=`%v` min=`%v` max=`%v` sum=`%v`\n",
115+
alert.Count, alert.Avg, alert.Min, alert.Max, alert.Sum)
116+
}
117+
if len(alert.Labels) > 0 {
118+
b.WriteString("- **Labels:**\n")
119+
// Stable order so concatenated annotation diffs cleanly between Sends.
120+
keys := make([]string, 0, len(alert.Labels))
121+
for k := range alert.Labels {
122+
keys = append(keys, k)
123+
}
124+
sort.Strings(keys)
125+
for _, k := range keys {
126+
fmt.Fprintf(&b, " - `%s`: `%s`\n", k, alert.Labels[k])
127+
}
128+
}
129+
b.WriteString("\n")
130+
return b.String()
131+
}

0 commit comments

Comments
 (0)