Skip to content

Commit 0f407e7

Browse files
anton-107Isaac
andauthored
aitools: log install-hint telemetry and bound telemetry uploads (#7467)
## Summary The CLI logs a new `aitools_hint_event` each time it shows a recommendation to install Databricks AI tools, so hints shown can be compared with installs that follow. It also bounds the whole telemetry upload, including client construction and authentication, by the upload timeout. ```diff MaybeHint(ctx, cmd) # non-aitools command run by Claude Code without the tools cmdio.LogString(hintText) + logHintShown(INSTALL_TEXT) runCreate (apps init) flags mode, skills missing - cmdio.LogString("Tip: ...") + PrintSkillsTip # prints the same tip + logHintShown(APPS_INIT_TIP) otherwise: RecommendSkillsInstall, skills missing no prompt possible - cmdio.LogString("Tip: ...") + PrintSkillsTip # APPS_INIT_TIP + logHintShown(APPS_INIT_PROMPT) # before asking, so a declined prompt counts AskYesOrNo(...) ``` Wire shape (one entry per hint shown): ```json {"aitools_hint_event": {"hint_kind": "INSTALL_TEXT"}} ``` - The event carries only the hint kind. The command it was shown on is the entry's `execution_context.command`. The calling agent is not a field: the SDK already sends `agent/<name>` in the User-Agent of the telemetry upload. - `AitoolsHintKind` also declares `CLAUDE_CODE_NATIVE` and `AGENT_INSTALL_TEXT`, which nothing emits yet. - One `apps init` run can log two events (`INSTALL_TEXT` from the root hint and `APPS_INIT_TIP`), so count per `hint_kind`. - The `apps init` tip and prompt now also stay silent when every detected agent has the databricks plugin in its own plugin manifest, for every project or for the working directory, not only when the shared skills are installed (`HasDatabricksToolingInstalled`, reusing `databricksPluginActiveIn`). A detected agent without either still gets the recommendation. Upload timeout (`libs/telemetry/logger.go`). The SDK resolves the config (including host metadata discovery) and fetches tokens with `context.Background()`, so before this change the upload timeout bounded only the HTTP requests. A slow token endpoint or host could hold the command's exit far past the timeout. ```diff Upload(ctx) - client.New(cfg) # unbounded: config resolution, host metadata ctx = WithTimeout(uploadTimeout) # 3s, DATABRICKS_CLI_TELEMETRY_TIMEOUT + within(ctx, newAPIClient(cfg)) # client construction gives up at the deadline + AuthVisitor = authenticate # every request, SDK retries included: + within(ctx, cfg.Authenticate) # gives up at the deadline for 3 attempts: attempt(ctx, ...) # requests stay on the caller's goroutine ``` An upload whose events are all hint events uses a 1s timeout instead of the 3s default (`uploadTimeout`), since most commands that show a hint would otherwise not upload at all. `DATABRICKS_CLI_TELEMETRY_TIMEOUT`, when set, applies to every upload. A hinted command whose own auth failed still tries the upload, so it runs the credential chain again at exit, bounded by the same 1s. On timeout the goroutine started by `within` is abandoned; a panic in it is returned as an error and logged like any upload failure. `cmd/root` runs `Upload` as the last step before the process exits; the ssh agent shim runs it just before exec. If that exec fails, `cmd/root` uploads again on the same logger; while an abandoned goroutine of the first upload still runs, that second `Upload` is skipped, so it does not use the config concurrently. Supporting changes: ```text libs/telemetry/logger.go BufferedEvents(ctx): read buffered events from tests in other packages libs/testserver/handlers.go POST /telemetry-ext accepts every log in the request acceptance/cmd/aitools/telemetry/ print_telemetry helper + `hint` and `apps-init-tip` cases libs/aitools/agents/hint_test.go uses testutil.SetAgentEnv instead of its own agent env list ``` ## Evidence Upload timeout, with `Upload` temporarily switched back to `client.New(cfg)` (the token source, host metadata resolver and token refresh block until the test ends), then at this PR's head: - **Before:** ```text --- FAIL: TestTelemetryUploadTimeoutBoundsTokenAcquisition (10.01s) Upload did not return after its timeout --- FAIL: TestTelemetryUploadTimeoutBoundsTokenRefreshOnRetry (10.01s) Upload did not return after its timeout --- FAIL: TestTelemetryUploadTimeoutBoundsConfigResolution (10.01s) Upload did not return after its timeout ``` **After:** ```text --- PASS: TestTelemetryUploadTimeoutBoundsTokenAcquisition (0.10s) # timeout 0.1s --- PASS: TestTelemetryUploadTimeoutBoundsTokenRefreshOnRetry (2.00s) # timeout 2s --- PASS: TestTelemetryUploadTimeoutBoundsConfigResolution (0.10s) # timeout 0.1s ``` Hint-only timeout, at this PR's head: ```text --- PASS: TestUploadTimeout # hint-only 1s, mixed 3s, env overrides both --- PASS: TestTelemetryUploadHintOnlyUsesShortTimeout (1.00s) # stalled server: gives up after 1s --- PASS: TestTelemetryUploadHintOnlyWithUnresolvedConfig # never-authenticated config (bundle dir): still uploads --- PASS: TestTelemetryUploadHintOnlyWithFailingAuthIsBounded # preset auth type, failing credentials: gives up within 1s --- PASS: TestTelemetryUploadSkipsWhileEarlierUploadRuns # second Upload skipped while the first's goroutine runs --- PASS: TestWithinReturnsPanicAsError ``` The last three fail with their checks removed. Hint event, at this PR's head (these tests assert the buffered and uploaded events; they were not re-run here with the call sites reverted): ```text --- PASS: TestMaybeHint_ClaudeNotInstalledHints # [INSTALL_TEXT]; the silent and throttled cases log nothing --- PASS: TestRecommendSkillsInstallNonInteractive # [APPS_INIT_TIP] --- PASS: TestRecommendSkillsInstallInteractiveDecline # [APPS_INIT_PROMPT], prompt declined --- PASS: TestAitoolsEventsWireKeys # {"aitools_hint_event":{"hint_kind":"INSTALL_TEXT"}} --- PASS: TestAccept/cmd/aitools/telemetry/hint # jobs list uploads one event; version and an installed plugin upload none --- PASS: TestAccept/cmd/aitools/telemetry/apps-init-tip # INSTALL_TEXT + APPS_INIT_TIP; with the plugin installed, no hint, tip or event ``` `apps init` guard: `TestRecommendSkillsInstallSkipsWhenPluginInstalled` and the plugin case of `apps-init-tip` fail with the old skills-only guard and pass at this head. `TestHasDatabricksToolingInstalled` covers the plugin for every agent, for one of two agents, for another project only, shared skills, and neither; the one-of-two and other-project cases fail with an any-agent, any-project check. Also passing at this head: `go build ./...`, `./task test-exp-aitools`, `go test ./libs/telemetry/... ./libs/aitools/... ./libs/testserver/ ./cmd/apps/... ./cmd/aitools/... ./cmd/air/... ./cmd/environments/... ./experimental/ssh/...`, the full acceptance suite (`go test ./acceptance`), `./task lint-q`, `./task fmt-q`, `./task checks`, `./task check-changelog`, and `go vet` with `GOOS=windows` and `GOOS=darwin`. In `./cmd/root`, `TestWorkspaceClientOrPromptRejectsPATOnSPOGFromConfigFile` fails locally both with and without this change. ## Merge Danger **Door:** two-way The event is additive and reverting stops sending it. No upload to a real workspace was run here, only the acceptance test server and a marshal test that pins the wire keys. **Blast Radius:** telemetry - Commands that show a hint now make one telemetry upload at exit. When the hint is the only event, the upload waits at most 1s if `/telemetry-ext` is slow or unreachable; commands that also log other events keep the 3s default. Before, most such commands (`current-user me`, `jobs list`, ...) logged nothing and skipped the upload. Hint and tip texts are unchanged. - Every command that uploads telemetry (any command that logged an event and resolved auth, and the ssh agent shim) now returns from the upload within the timeout even when token fetch or host metadata discovery hangs. It then goes on without uploading and logs the timeout (`telemetry upload failed: ...`, info level in `cmd/root`, debug in the shim); before, it waited for the SDK call to finish. Exit codes are unchanged: upload errors were and are only logged. The upload also abandons a U2M (`databricks-cli`) token refresh that is still running at the deadline: if the access token expired during a long command and the refresh outlasts the rest of the upload budget (3s, or 1s for a hint-only upload), the process exits before the refreshed token is stored. If the server already rotated the refresh token, the next command fails to refresh until `databricks auth login`. Before, `Upload` waited and the refreshed token was saved. The CLI already takes the same risk elsewhere: the U2M token source in `libs/auth/credentials.go` uses `auth.WithAsyncRefresh`, so a background refresh can also be cut short when a command exits. - The install hint prints at most once per Claude Code session per hour (once per command without `CLAUDE_CODE_SESSION_ID`, or when the CLI cache is disabled or not writable). The `apps init` tip and prompt print on every run where a detected agent has neither the shared skills nor the plugin for the working directory. - Hint counts are a lower bound: events upload only when the command resolved auth (not for `version`, `completion`, `configure` or most `auth` subcommands), the upload succeeds and telemetry is enabled. Because the install hint prints once per session per hour, a session whose hinted command resolves no auth logs no install hint event for that hour (documented in `libs/telemetry/protos/README.md`). `APPS_INIT_*` events are also logged for humans who have an agent config dir; the event has no calling-agent field to tell them apart. An accepted `APPS_INIT_PROMPT` and a declined one look the same: the install the prompt starts logs no event. - Changelog fragments: `.nextchanges/cli/telemetry-upload-timeout.md`, `.nextchanges/cli/apps-init-plugin-tip.md`. The acceptance cases use bash and `jq` and were not run on Windows here; Windows and macOS were only cross-compiled and vetted. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
1 parent 60c0428 commit 0f407e7

32 files changed

Lines changed: 819 additions & 87 deletions
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
* `apps init` no longer suggests installing skills when every detected coding agent already has the Databricks plugin. ([#7467](https://github.com/databricks/cli/pull/7467))
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
* Bound the whole telemetry upload at exit, including config resolution and authentication, by its timeout, so a stalled auth or telemetry endpoint no longer keeps a command from exiting. ([#7467](https://github.com/databricks/cli/pull/7467))

‎acceptance/cmd/aitools/telemetry/apps-init-tip/home/.claude/.gitkeep‎

Whitespace-only changes.
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Cloud = false
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
2+
=== apps init with flags prints the tip: one hint-shown event per hint printed
3+
4+
>>> [CLI] apps init --template ./template --skip-install --name app-tip
5+
Databricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install
6+
✔ Creating project
7+
8+
✔ Project created successfully!
9+
10+
Location: [TEST_TMP_DIR]/app-tip
11+
Files: 3
12+
13+
Tip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them.
14+
15+
>>> print_telemetry
16+
{"command":"apps_init","exit_code":0,"aitools_hint_event":{"hint_kind":"INSTALL_TEXT"}}
17+
{"command":"apps_init","exit_code":0,"aitools_hint_event":{"hint_kind":"APPS_INIT_TIP"}}
18+
19+
=== plugin installed: no hint, no tip, no event
20+
21+
>>> [CLI] apps init --template ./template --skip-install --name app-plugin
22+
✔ Creating project
23+
24+
✔ Project created successfully!
25+
26+
Location: [TEST_TMP_DIR]/app-plugin
27+
Files: 3
28+
29+
30+
>>> print_telemetry
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"plugins":{"databricks@claude-plugins-official":[{"scope":"user","version":"0.2.9"}]}}
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# Isolate HOME so the real machine's agents and skills are not seen. The
2+
# committed home/.claude, a Claude Code config dir without Databricks skills, is
3+
# what makes apps init print its tip.
4+
sethome home
5+
export CLAUDECODE=1
6+
unset CLAUDE_CODE_SESSION_ID
7+
unset CLAUDE_CONFIG_DIR
8+
9+
title "apps init with flags prints the tip: one hint-shown event per hint printed"
10+
trace $CLI apps init --template ./template --skip-install --name app-tip > LOG.apps-init
11+
trace print_telemetry
12+
rm -rf app-tip
13+
14+
title "plugin installed: no hint, no tip, no event"
15+
sethome ./plugin-home
16+
trace $CLI apps init --template ./template --skip-install --name app-plugin > LOG.apps-init-plugin
17+
trace print_telemetry
18+
rm -rf app-plugin
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
command: ['npm', 'run', 'start']
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"version": "1.0",
3+
"plugins": {}
4+
}
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
bundle:
2+
name: {{.projectName}}

0 commit comments

Comments
 (0)