Repository navigation
Commit 0f407e7
aitools: log install-hint telemetry and bound telemetry uploads (#7467)
## Summary
The CLI logs a new `aitools_hint_event` each time it shows a
recommendation to install Databricks AI tools, so hints shown can be
compared with installs that follow. It also bounds the whole telemetry
upload, including client construction and authentication, by the upload
timeout.
```diff
MaybeHint(ctx, cmd) # non-aitools command run by Claude Code without the tools
cmdio.LogString(hintText)
+ logHintShown(INSTALL_TEXT)
runCreate (apps init)
flags mode, skills missing
- cmdio.LogString("Tip: ...")
+ PrintSkillsTip # prints the same tip
+ logHintShown(APPS_INIT_TIP)
otherwise: RecommendSkillsInstall, skills missing
no prompt possible
- cmdio.LogString("Tip: ...")
+ PrintSkillsTip # APPS_INIT_TIP
+ logHintShown(APPS_INIT_PROMPT) # before asking, so a declined prompt counts
AskYesOrNo(...)
```
Wire shape (one entry per hint shown):
```json
{"aitools_hint_event": {"hint_kind": "INSTALL_TEXT"}}
```
- The event carries only the hint kind. The command it was shown on is
the entry's `execution_context.command`. The calling agent is not a
field: the SDK already sends `agent/<name>` in the User-Agent of the
telemetry upload.
- `AitoolsHintKind` also declares `CLAUDE_CODE_NATIVE` and
`AGENT_INSTALL_TEXT`, which nothing emits yet.
- One `apps init` run can log two events (`INSTALL_TEXT` from the root
hint and `APPS_INIT_TIP`), so count per `hint_kind`.
- The `apps init` tip and prompt now also stay silent when every
detected agent has the databricks plugin in its own plugin manifest, for
every project or for the working directory, not only when the shared
skills are installed (`HasDatabricksToolingInstalled`, reusing
`databricksPluginActiveIn`). A detected agent without either still gets
the recommendation.
Upload timeout (`libs/telemetry/logger.go`). The SDK resolves the config
(including host metadata discovery) and fetches tokens with
`context.Background()`, so before this change the upload timeout bounded
only the HTTP requests. A slow token endpoint or host could hold the
command's exit far past the timeout.
```diff
Upload(ctx)
- client.New(cfg) # unbounded: config resolution, host metadata
ctx = WithTimeout(uploadTimeout) # 3s, DATABRICKS_CLI_TELEMETRY_TIMEOUT
+ within(ctx, newAPIClient(cfg)) # client construction gives up at the deadline
+ AuthVisitor = authenticate # every request, SDK retries included:
+ within(ctx, cfg.Authenticate) # gives up at the deadline
for 3 attempts: attempt(ctx, ...) # requests stay on the caller's goroutine
```
An upload whose events are all hint events uses a 1s timeout instead of
the 3s default (`uploadTimeout`), since most commands that show a hint
would otherwise not upload at all. `DATABRICKS_CLI_TELEMETRY_TIMEOUT`,
when set, applies to every upload. A hinted command whose own auth
failed still tries the upload, so it runs the credential chain again at
exit, bounded by the same 1s.
On timeout the goroutine started by `within` is abandoned; a panic in it
is returned as an error and logged like any upload failure. `cmd/root`
runs `Upload` as the last step before the process exits; the ssh agent
shim runs it just before exec. If that exec fails, `cmd/root` uploads
again on the same logger; while an abandoned goroutine of the first
upload still runs, that second `Upload` is skipped, so it does not use
the config concurrently.
Supporting changes:
```text
libs/telemetry/logger.go BufferedEvents(ctx): read buffered events from tests in other packages
libs/testserver/handlers.go POST /telemetry-ext accepts every log in the request
acceptance/cmd/aitools/telemetry/ print_telemetry helper + `hint` and `apps-init-tip` cases
libs/aitools/agents/hint_test.go uses testutil.SetAgentEnv instead of its own agent env list
```
## Evidence
Upload timeout, with `Upload` temporarily switched back to
`client.New(cfg)` (the token source, host metadata resolver and token
refresh block until the test ends), then at this PR's head:
- **Before:**
```text
--- FAIL: TestTelemetryUploadTimeoutBoundsTokenAcquisition (10.01s)
Upload did not return after its timeout
--- FAIL: TestTelemetryUploadTimeoutBoundsTokenRefreshOnRetry (10.01s)
Upload did not return after its timeout
--- FAIL: TestTelemetryUploadTimeoutBoundsConfigResolution (10.01s)
Upload did not return after its timeout
```
**After:**
```text
--- PASS: TestTelemetryUploadTimeoutBoundsTokenAcquisition (0.10s) #
timeout 0.1s
--- PASS: TestTelemetryUploadTimeoutBoundsTokenRefreshOnRetry (2.00s) #
timeout 2s
--- PASS: TestTelemetryUploadTimeoutBoundsConfigResolution (0.10s) #
timeout 0.1s
```
Hint-only timeout, at this PR's head:
```text
--- PASS: TestUploadTimeout # hint-only 1s, mixed 3s, env overrides both
--- PASS: TestTelemetryUploadHintOnlyUsesShortTimeout (1.00s) # stalled server: gives up after 1s
--- PASS: TestTelemetryUploadHintOnlyWithUnresolvedConfig # never-authenticated config (bundle dir): still uploads
--- PASS: TestTelemetryUploadHintOnlyWithFailingAuthIsBounded # preset auth type, failing credentials: gives up within 1s
--- PASS: TestTelemetryUploadSkipsWhileEarlierUploadRuns # second Upload skipped while the first's goroutine runs
--- PASS: TestWithinReturnsPanicAsError
```
The last three fail with their checks removed.
Hint event, at this PR's head (these tests assert the buffered and
uploaded events; they were not re-run here with the call sites
reverted):
```text
--- PASS: TestMaybeHint_ClaudeNotInstalledHints # [INSTALL_TEXT]; the silent and throttled cases log nothing
--- PASS: TestRecommendSkillsInstallNonInteractive # [APPS_INIT_TIP]
--- PASS: TestRecommendSkillsInstallInteractiveDecline # [APPS_INIT_PROMPT], prompt declined
--- PASS: TestAitoolsEventsWireKeys # {"aitools_hint_event":{"hint_kind":"INSTALL_TEXT"}}
--- PASS: TestAccept/cmd/aitools/telemetry/hint # jobs list uploads one event; version and an installed plugin upload none
--- PASS: TestAccept/cmd/aitools/telemetry/apps-init-tip # INSTALL_TEXT + APPS_INIT_TIP; with the plugin installed, no hint, tip or event
```
`apps init` guard: `TestRecommendSkillsInstallSkipsWhenPluginInstalled`
and the plugin case of `apps-init-tip` fail with the old skills-only
guard and pass at this head. `TestHasDatabricksToolingInstalled` covers
the plugin for every agent, for one of two agents, for another project
only, shared skills, and neither; the one-of-two and other-project cases
fail with an any-agent, any-project check.
Also passing at this head: `go build ./...`, `./task test-exp-aitools`,
`go test ./libs/telemetry/... ./libs/aitools/... ./libs/testserver/
./cmd/apps/... ./cmd/aitools/... ./cmd/air/... ./cmd/environments/...
./experimental/ssh/...`, the full acceptance suite (`go test
./acceptance`), `./task lint-q`, `./task fmt-q`, `./task checks`,
`./task check-changelog`, and `go vet` with `GOOS=windows` and
`GOOS=darwin`. In `./cmd/root`,
`TestWorkspaceClientOrPromptRejectsPATOnSPOGFromConfigFile` fails
locally both with and without this change.
## Merge Danger
**Door:** two-way
The event is additive and reverting stops sending it. No upload to a
real workspace was run here, only the acceptance test server and a
marshal test that pins the wire keys.
**Blast Radius:** telemetry
- Commands that show a hint now make one telemetry upload at exit. When
the hint is the only event, the upload waits at most 1s if
`/telemetry-ext` is slow or unreachable; commands that also log other
events keep the 3s default. Before, most such commands (`current-user
me`, `jobs list`, ...) logged nothing and skipped the upload. Hint and
tip texts are unchanged.
- Every command that uploads telemetry (any command that logged an event
and resolved auth, and the ssh agent shim) now returns from the upload
within the timeout even when token fetch or host metadata discovery
hangs. It then goes on without uploading and logs the timeout
(`telemetry upload failed: ...`, info level in `cmd/root`, debug in the
shim); before, it waited for the SDK call to finish. Exit codes are
unchanged: upload errors were and are only logged. The upload also
abandons a U2M (`databricks-cli`) token refresh that is still running at
the deadline: if the access token expired during a long command and the
refresh outlasts the rest of the upload budget (3s, or 1s for a
hint-only upload), the process exits before the refreshed token is
stored. If the server already rotated the refresh token, the next
command fails to refresh until `databricks auth login`. Before, `Upload`
waited and the refreshed token was saved. The CLI already takes the same
risk elsewhere: the U2M token source in `libs/auth/credentials.go` uses
`auth.WithAsyncRefresh`, so a background refresh can also be cut short
when a command exits.
- The install hint prints at most once per Claude Code session per hour
(once per command without `CLAUDE_CODE_SESSION_ID`, or when the CLI
cache is disabled or not writable). The `apps init` tip and prompt print
on every run where a detected agent has neither the shared skills nor
the plugin for the working directory.
- Hint counts are a lower bound: events upload only when the command
resolved auth (not for `version`, `completion`, `configure` or most
`auth` subcommands), the upload succeeds and telemetry is enabled.
Because the install hint prints once per session per hour, a session
whose hinted command resolves no auth logs no install hint event for
that hour (documented in `libs/telemetry/protos/README.md`).
`APPS_INIT_*` events are also logged for humans who have an agent config
dir; the event has no calling-agent field to tell them apart. An
accepted `APPS_INIT_PROMPT` and a declined one look the same: the
install the prompt starts logs no event.
- Changelog fragments: `.nextchanges/cli/telemetry-upload-timeout.md`,
`.nextchanges/cli/apps-init-plugin-tip.md`. The acceptance cases use
bash and `jq` and were not run on Windows here; Windows and macOS were
only cross-compiled and vetted.
This pull request and its description were written by Isaac.
Co-authored-by: Isaac <no-reply@databricks.com>1 parent 60c0428 commit 0f407e7
32 files changed
Lines changed: 819 additions & 87 deletions
File tree
- .nextchanges/cli
- acceptance/cmd/aitools/telemetry
- apps-init-tip
- home/.claude
- plugin-home/.claude/plugins
- template
- hint
- installed-home/.claude/plugins
- cmd
- apps
- root
- libs
- aitools/agents
- telemetry
- protos
- testserver
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
Whitespace-only changes.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
0 commit comments