Skip to content

Commit 01b24bd

Browse files
anton-107Isaac
andauthored
aitools: add install funnel telemetry (#7469)
Stacked on #7468. Stacked on: aitools: bound best-effort telemetry auth ## Summary Every `databricks aitools install` run now logs an `aitools_command_event` funnel next to the existing `aitools_install_event`, so the telemetry shows where runs stop: ```text install RunE STARTED (every run past flag parsing) validate flags, --agents names --> FINISHED FAILURE INVALID_FLAGS | UNKNOWN_AGENT STEP FLAGS_VALIDATED --path dump --> STEP SKILLS_DUMPED, FINISHED SUCCESS (0 skills written: FINISHED NOTHING_TO_DO) scope prompt --> FINISHED CANCELLED USER_ABORTED | FAILURE PROMPT_FAILED STEP SCOPE_RESOLVED picker --> FINISHED CANCELLED (nothing checked) no agent detected: FINISHED NOTHING_TO_DO STEP AGENTS_SELECTED confirm prompt --> FINISHED CANCELLED (declined) STEP PLAN_CONFIRMED executePlan STEP SKILLS_INSTALLED (skills group installed) STEP PLUGINS_INSTALLED (at least one plugin installed) FINISHED SUCCESS | FAILURE <category> ``` Each event carries `command`, `stage`, `interactive`, `install_mode` (DEFAULT / SKILLS_ONLY / PATH) and, once known, `scope`. FINISHED also carries `outcome`, `error_category` and the targeted `agents`. Every event of a run shares the entry's `cmd_exec_id`. The CLI sets no calling-agent field on these events: the upload's User-Agent already has `agent/<name>`, the calling agent's Databricks tooling (`aitools-agent/...`) and `cmd-exec-id`. `error_category` comes from the existing `classifyInstallError`. Failures in flag validation, agent lookup and prompts are tagged with a category where they happen (a prompt's own failure is PROMPT_FAILED whatever its type, except a Ctrl-C abort; an error before the prompt opens keeps its own category). `Error()` / `Unwrap()` pass through, so the user still sees the same message: ```diff classifyInstallError(err) *installer.BlockedError / *installer.SkillError -> their category + *categorizedError (withCategory, promptError) -> the tagged category stdlibErrorCategory(err) ``` Some runs get an explicit outcome instead of the one their return value implies: - Returns nil but installed nothing (e.g. the only detected agent's plugin install failed, which is a warning): FAILURE with the first failing agent's category. This includes a non-interactive run whose detected agents are all skipped for the scope (UNSUPPORTED_SCOPE). The interactive picker never offers skipped agents, so the same machine finishes NOTHING_TO_DO there. - Fails only because a named agent failed or was skipped (`--agents`): the category comes from that agent's outcome (e.g. UNSUPPORTED_SCOPE), not UNCATEGORIZED. - `--output json` failure: classifies the real error, not the `ErrAlreadyPrinted` sentinel. - `--project --global` and `--agents ","`: INVALID_FLAGS, not PROMPT_FAILED / UNKNOWN_AGENT. Runs that end before PLAN_CONFIRMED (a flag error, a cancelled prompt, no agent detected, a `--path` dump) upload their events only if the background telemetry auth has already resolved; they never wait for it. Runs past PLAN_CONFIRMED keep the bounded wait. An unknown `--agents` name now fails before the scope prompt instead of after it (changelog fragment `.nextchanges/cli/aitools-install-agents-before-scope-prompt.md`). Otherwise no change to output, prompts, exit codes or error text. ## Evidence - **Before** (non-test code of this PR reverted, test files kept): acceptance `cmd/aitools/telemetry/install` fails; the upload has only `aitools_install_event`: ```text --- FAIL: TestAccept/cmd/aitools/telemetry/install -{"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}} -{"aitools_command_event":{..."stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED"...}} -... SCOPE_RESOLVED, AGENTS_SELECTED, PLAN_CONFIRMED, SKILLS_INSTALLED ... -{"aitools_command_event":{..."stage":"FINISHED","outcome":"SUCCESS",...,"agents":["CLAUDE_CODE"],"scope":"GLOBAL"}} ``` **After:** the same test passes. `install-failure` pins the failure paths past PLAN_CONFIRMED: missing skill, manifest 404 / 500, named agent skipped for the scope. Runs that end earlier upload only if auth already resolved, which depends on timing, so they are covered by unit tests instead. - Unit tests in `cmd/aitools/funnel_test.go` (for the funnel code in `funnel.go`) cover every early return and each success path (e.g. `TestInstallFunnelEarlyExits`, `TestInstallFunnelAbortedScopePromptKeepsNamedAgents`, `TestInstallFunnelPathDumpSuccess`, `TestInstallFunnelPathDumpWithNoSkillToWrite`, `TestInstallFunnelPluginSuccess`, `TestInstallFunnelInteractiveConfirm`, `TestInstallFunnelDetectedPluginFailureIsNotSuccess`, `TestInstallFunnelSkillsFailureThroughJSON`); they and `TestAitoolsEventsWireKeys` pass with `-count=1`. - `TestInstallUnknownAgentFailsBeforeScopePrompt`: with prompts supported, `--agents bogus` fails without showing the scope prompt (fails if `resolveAgentNames` moves back below the prompt). - The upload wait: `TestInstallEarlyExitDoesNotWaitForTelemetryAuth` (a flag error returns at once while auth is still resolving; fails after 10 s with the wait put back) and `TestInstallPastPlanConfirmedWaitsForTelemetryAuth` (auth that finishes only after the install step still reaches the upload; fails without the wait). - `go build ./...`, `./task test-exp-aitools`, `go test ./libs/telemetry/... ./cmd/root/...`, `./task lint-q`, `./task fmt-q`, `./task checks`, and `go vet` with `GOOS=windows` / `GOOS=darwin` pass. ## Merge Danger **Door:** two-way Telemetry only. Reverting removes the events; nothing on disk or in the install state changes. **Blast Radius:** telemetry - More events per `aitools install` upload (about eight instead of one), sent at exit and only when auth resolved. Runs that stop before the plan is confirmed (flag errors, a cancelled prompt, no agent detected) and `--path` dumps logged no event before, so they made no telemetry request. They now upload only if the background auth has already resolved when they end, and never wait for it, so an unreachable or slow-to-authenticate workspace adds nothing to them. When auth has resolved, the upload itself can still take up to its timeout (`DATABRICKS_CLI_TELEMETRY_TIMEOUT`, default 3 s) against a slow workspace. The flip side: a run that ends in milliseconds (e.g. a flag error) usually ends before auth resolves, so its events are often dropped; a run that stopped at a prompt usually had time to resolve auth. The funnel therefore under-counts the fastest early exits, and their STARTED events, relative to slower ones; the code comment next to the rule says the same. Runs past the plan confirmation keep the bounded auth wait. - Not covered: the funnel trusts the skills installer, which reports no per-agent delivery. So two cases still finish SUCCESS with SKILLS_INSTALLED although an agent got no skill: every skill was filtered out (experimental, or below the minimum CLI version; #7481, further up this stack, makes the installer report this and the funnel finish NOTHING_TO_DO), or the installer could not write an agent's destination (it warns and continues). `--output json` and `aitools_install_event` report these runs as installed today too. The `--path` dump knows how many skills it wrote, so a dump that writes none finishes NOTHING_TO_DO. A run killed by Ctrl-C outside a prompt, or rejected by flag parsing, uploads no funnel events. - Not tested here: interactive prompts on Windows (the funnel paths are covered by unit tests with mocked prompts and compile-checked for Windows). This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
1 parent a29c957 commit 01b24bd

20 files changed

Lines changed: 1253 additions & 89 deletions
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
* `databricks aitools install` now rejects an unknown `--agents` name before it shows the scope prompt, instead of after. ([#7469](https://github.com/databricks/cli/pull/7469))
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Cloud = false
Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
2+
=== a skill missing from the manifest fails after the plan is confirmed
3+
4+
>>> [CLI] aitools install --skills-only --scope=global --agents=cursor --skills=nonexistent
5+
Installing Databricks skills for Cursor...
6+
Using skills version test-ref
7+
Fetching skills manifest...
8+
Error: skill "nonexistent" not found
9+
10+
>>> print_telemetry
11+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
12+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
13+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
14+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
15+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
16+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"FAILURE","error_category":"SKILL_NOT_FOUND","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CURSOR"],"scope":"GLOBAL"}}
17+
{"command":"aitools_install","exit_code":1,"aitools_install_event":{"agents":["CURSOR"],"scope":"GLOBAL","error_category":"SKILL_NOT_FOUND"}}
18+
19+
=== a manifest the server does not have is MANIFEST_NOT_FOUND
20+
21+
>>> DATABRICKS_SKILLS_REF=missing-ref [CLI] aitools install --skills-only --scope=global --agents=cursor
22+
Installing Databricks skills for Cursor...
23+
Using skills version missing-ref
24+
Fetching skills manifest...
25+
Error: skills manifest at databricks-agent-skills@missing-ref: not found
26+
27+
>>> print_telemetry
28+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
29+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
30+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
31+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
32+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
33+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"FAILURE","error_category":"MANIFEST_NOT_FOUND","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CURSOR"],"scope":"GLOBAL"}}
34+
{"command":"aitools_install","exit_code":1,"aitools_install_event":{"agents":["CURSOR"],"scope":"GLOBAL","error_category":"MANIFEST_NOT_FOUND"}}
35+
36+
=== any other HTTP error status from the manifest fetch is not classified yet (the error is untyped)
37+
38+
>>> DATABRICKS_SKILLS_REF=broken-ref [CLI] aitools install --skills-only --scope=global --agents=cursor
39+
Installing Databricks skills for Cursor...
40+
Using skills version broken-ref
41+
Fetching skills manifest...
42+
Error: failed to fetch manifest: HTTP 500
43+
44+
>>> print_telemetry
45+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
46+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
47+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
48+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
49+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
50+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"FAILURE","error_category":"UNCATEGORIZED","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CURSOR"],"scope":"GLOBAL"}}
51+
{"command":"aitools_install","exit_code":1,"aitools_install_event":{"agents":["CURSOR"],"scope":"GLOBAL","error_category":"UNCATEGORIZED"}}
52+
53+
=== a named agent skipped for the scope takes its category from the agent's outcome
54+
55+
>>> [CLI] aitools install --skills-only --scope=project --agents=cursor
56+
Skipped Cursor: does not support project-scoped skills
57+
Error: Cursor: does not support project-scoped skills
58+
59+
>>> print_telemetry
60+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
61+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
62+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"PROJECT"}}
63+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"PROJECT"}}
64+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"PROJECT"}}
65+
{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"FAILURE","error_category":"UNSUPPORTED_SCOPE","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CURSOR"],"scope":"PROJECT"}}
66+
{"command":"aitools_install","exit_code":1,"aitools_install_event":{"agents":["CURSOR"],"scope":"PROJECT","error_category":"TYPE_UNSPECIFIED","agent_results":[{"agent":"CURSOR","error_category":"UNSUPPORTED_SCOPE"}]}}
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Isolate HOME so parallel aitools tests don't race on a shared ~/.databricks.
2+
sethome home
3+
mkdir -p "${USERPROFILE:-$HOME}/.cursor"
4+
5+
# Only runs past the plan confirmation are here: a run that ends earlier (a
6+
# flag error, an unknown agent, a --path dump) uploads only if auth has already
7+
# resolved, so whether its events show up depends on timing. The funnel unit
8+
# tests in cmd/aitools cover those runs.
9+
10+
title "a skill missing from the manifest fails after the plan is confirmed"
11+
musterr trace $CLI aitools install --skills-only --scope=global --agents=cursor --skills=nonexistent
12+
trace print_telemetry
13+
14+
title "a manifest the server does not have is MANIFEST_NOT_FOUND"
15+
musterr trace DATABRICKS_SKILLS_REF=missing-ref $CLI aitools install --skills-only --scope=global --agents=cursor
16+
trace print_telemetry
17+
18+
title "any other HTTP error status from the manifest fetch is not classified yet (the error is untyped)"
19+
musterr trace DATABRICKS_SKILLS_REF=broken-ref $CLI aitools install --skills-only --scope=global --agents=cursor
20+
trace print_telemetry
21+
22+
title "a named agent skipped for the scope takes its category from the agent's outcome"
23+
musterr trace $CLI aitools install --skills-only --scope=project --agents=cursor
24+
trace print_telemetry
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
[[Server]]
2+
Pattern = "GET /missing-ref/manifest.json"
3+
Response.StatusCode = 404
4+
5+
[[Server]]
6+
Pattern = "GET /broken-ref/manifest.json"
7+
Response.StatusCode = 500

‎acceptance/cmd/aitools/telemetry/install/output.txt‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,14 @@ Fetching skills manifest...
88
Installed 1 skill.
99

1010
>>> print_telemetry
11+
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
12+
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
13+
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
14+
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
15+
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
16+
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SKILLS_INSTALLED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
1117
{"command":"aitools_install","exit_code":0,"aitools_install_event":{"agents":["CLAUDE_CODE"],"scope":"GLOBAL","error_category":"TYPE_UNSPECIFIED"}}
18+
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"SUCCESS","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CLAUDE_CODE"],"scope":"GLOBAL"}}
1219

1320
=== databricks-cli auth with a valid cached token: the event is uploaded
1421

@@ -19,7 +26,14 @@ Fetching skills manifest...
1926
Installed 1 skill.
2027

2128
>>> print_telemetry
29+
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
30+
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
31+
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
32+
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
33+
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
34+
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SKILLS_INSTALLED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
2235
{"command":"aitools_install","exit_code":0,"aitools_install_event":{"agents":["CLAUDE_CODE"],"scope":"GLOBAL","error_category":"TYPE_UNSPECIFIED"}}
36+
{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"SUCCESS","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CLAUDE_CODE"],"scope":"GLOBAL"}}
2337

2438
=== databricks-cli auth with an expired cached token: no refresh, nothing is uploaded
2539

‎acceptance/cmd/aitools/telemetry/install/test.toml‎

Lines changed: 0 additions & 29 deletions
This file was deleted.
Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,36 @@
1-
# Telemetry payloads of the install hint.
1+
# Telemetry payloads of the aitools commands and of the install hint.
2+
# The mock server replaces raw.githubusercontent.com (skills manifest and files).
3+
Env.DATABRICKS_SKILLS_BASE_URL = "$DATABRICKS_HOST"
4+
Env.DATABRICKS_CLI_GITHUB_API_URL = "$DATABRICKS_HOST"
5+
Env.DATABRICKS_SKILLS_REF = "test-ref"
26
RecordRequests = true
37

48
Ignore = [
59
"home",
610
]
11+
12+
[[Server]]
13+
Pattern = "GET /test-ref/manifest.json"
14+
Response.Body = '''
15+
{
16+
"version": "2",
17+
"updated_at": "2026-01-01T00:00:00Z",
18+
"skills": {
19+
"databricks-test": {
20+
"version": "1.0.0",
21+
"description": "Test skill",
22+
"files": ["SKILL.md"],
23+
"repo_dir": "skills"
24+
}
25+
}
26+
}
27+
'''
28+
29+
[[Server]]
30+
Pattern = "GET /test-ref/skills/databricks-test/SKILL.md"
31+
Response.Body = '''---
32+
name: databricks-test
33+
---
34+
35+
# Test skill
36+
'''

‎cmd/aitools/categorize.go‎

Lines changed: 34 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,38 @@ import (
1414
"github.com/databricks/cli/libs/telemetry/protos"
1515
)
1616

17+
// categorizedError tags an error with a telemetry category at the site that
18+
// knows what failed (flag validation, agent lookup, a prompt). The tag wins
19+
// over the standard library types. Error() and Unwrap() pass through, so the
20+
// user-visible message is unchanged.
21+
type categorizedError struct {
22+
err error
23+
category protos.AitoolsErrorCategory
24+
}
25+
26+
func (e *categorizedError) Error() string { return e.err.Error() }
27+
func (e *categorizedError) Unwrap() error { return e.err }
28+
29+
// withCategory tags a non-nil err with category.
30+
func withCategory(err error, category protos.AitoolsErrorCategory) error {
31+
return &categorizedError{err: err, category: category}
32+
}
33+
34+
// promptError tags a non-nil prompt failure. A user abort (Ctrl-C) keeps its own
35+
// classification so the funnel reports it as CANCELLED, not as a failure.
36+
// Any other error of a prompt step is PROMPT_FAILED, even one with an OS
37+
// error type: the prompt library wraps a failure to open the terminal as an
38+
// *fs.PathError, and FILESYSTEM would point at the skill files instead.
39+
func promptError(err error) error {
40+
if errors.Is(err, huh.ErrUserAborted) {
41+
return err
42+
}
43+
return withCategory(err, protos.AitoolsErrorCategoryPromptFailed)
44+
}
45+
1746
// classifyInstallError maps an aitools error to its telemetry category without
18-
// reading any error text. Typed installer errors win, then standard library
19-
// error types.
47+
// reading any error text. Typed installer errors win, then categories tagged at
48+
// the failure site, then standard library error types.
2049
func classifyInstallError(err error) protos.AitoolsErrorCategory {
2150
if err == nil {
2251
return protos.AitoolsErrorCategoryUnspecified
@@ -28,6 +57,9 @@ func classifyInstallError(err error) protos.AitoolsErrorCategory {
2857
if skill, ok := errors.AsType[*installer.SkillError](err); ok {
2958
return skillErrorCategory(skill)
3059
}
60+
if tagged, ok := errors.AsType[*categorizedError](err); ok {
61+
return tagged.category
62+
}
3163
return stdlibErrorCategory(err)
3264
}
3365

‎cmd/aitools/categorize_test.go‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,10 @@ func TestClassifyInstallErrorGenericCauses(t *testing.T) {
114114
{"filesystem on rename", fmt.Errorf("replace skill: %w", &os.LinkError{Op: "rename", Old: "/a", New: "/b", Err: syscall.EBUSY}), protos.AitoolsErrorCategoryFilesystem},
115115
{"filesystem syscall", fmt.Errorf("resolve cwd: %w", os.NewSyscallError("getwd", syscall.ENOENT)), protos.AitoolsErrorCategoryFilesystem},
116116
{"user abort", huh.ErrUserAborted, protos.AitoolsErrorCategoryUserAborted},
117+
{"tagged", withCategory(errors.New("bad flags"), protos.AitoolsErrorCategoryInvalidFlags), protos.AitoolsErrorCategoryInvalidFlags},
118+
{"prompt failure", promptError(errors.New("no tty")), protos.AitoolsErrorCategoryPromptFailed},
119+
{"prompt abort keeps abort", promptError(huh.ErrUserAborted), protos.AitoolsErrorCategoryUserAborted},
120+
{"typed error beats tag", withCategory(&installer.BlockedError{Reason: installer.ReasonCLINotOnPath}, protos.AitoolsErrorCategoryInvalidFlags), protos.AitoolsErrorCategoryCLINotOnPath},
117121
{"plain text stays uncategorized", errors.New("something"), protos.AitoolsErrorCategoryUncategorized},
118122
}
119123
for _, tc := range cases {
@@ -123,6 +127,13 @@ func TestClassifyInstallErrorGenericCauses(t *testing.T) {
123127
}
124128
}
125129

130+
func TestWithCategoryKeepsMessage(t *testing.T) {
131+
cause := errors.New("cannot use --skills-only with --path")
132+
err := withCategory(cause, protos.AitoolsErrorCategoryInvalidFlags)
133+
assert.Equal(t, cause.Error(), err.Error())
134+
assert.ErrorIs(t, err, cause)
135+
}
136+
126137
// A corrupt install state must not be blamed on the remote manifest. It stays
127138
// UNCATEGORIZED until the installer returns a typed state error, which should
128139
// change this to STATE_INVALID.

0 commit comments

Comments
 (0)