Repository navigation
aitools: log install-hint telemetry and bound telemetry uploads #7467
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| * `apps init` no longer suggests installing skills when every detected coding agent already has the Databricks plugin. ([#7467](https://github.com/databricks/cli/pull/7467)) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| * Bound the whole telemetry upload at exit, including config resolution and authentication, by its timeout, so a stalled auth or telemetry endpoint no longer keeps a command from exiting. ([#7467](https://github.com/databricks/cli/pull/7467)) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Cloud = false |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
|
|
||
| === apps init with flags prints the tip: one hint-shown event per hint printed | ||
|
|
||
| >>> [CLI] apps init --template ./template --skip-install --name app-tip | ||
| Databricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install | ||
| ✔ Creating project | ||
|
|
||
| ✔ Project created successfully! | ||
|
|
||
| Location: [TEST_TMP_DIR]/app-tip | ||
| Files: 3 | ||
|
|
||
| Tip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them. | ||
|
|
||
| >>> print_telemetry | ||
| {"command":"apps_init","exit_code":0,"aitools_hint_event":{"hint_kind":"INSTALL_TEXT"}} | ||
| {"command":"apps_init","exit_code":0,"aitools_hint_event":{"hint_kind":"APPS_INIT_TIP"}} | ||
|
|
||
| === plugin installed: no hint, no tip, no event | ||
|
|
||
| >>> [CLI] apps init --template ./template --skip-install --name app-plugin | ||
| ✔ Creating project | ||
|
|
||
| ✔ Project created successfully! | ||
|
|
||
| Location: [TEST_TMP_DIR]/app-plugin | ||
| Files: 3 | ||
|
|
||
|
|
||
| >>> print_telemetry |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| {"plugins":{"databricks@claude-plugins-official":[{"scope":"user","version":"0.2.9"}]}} |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| # Isolate HOME so the real machine's agents and skills are not seen. The | ||
| # committed home/.claude, a Claude Code config dir without Databricks skills, is | ||
| # what makes apps init print its tip. | ||
| sethome home | ||
| export CLAUDECODE=1 | ||
| unset CLAUDE_CODE_SESSION_ID | ||
| unset CLAUDE_CONFIG_DIR | ||
|
|
||
| title "apps init with flags prints the tip: one hint-shown event per hint printed" | ||
| trace $CLI apps init --template ./template --skip-install --name app-tip > LOG.apps-init | ||
| trace print_telemetry | ||
| rm -rf app-tip | ||
|
|
||
| title "plugin installed: no hint, no tip, no event" | ||
| sethome ./plugin-home | ||
| trace $CLI apps init --template ./template --skip-install --name app-plugin > LOG.apps-init-plugin | ||
| trace print_telemetry | ||
| rm -rf app-plugin | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| command: ['npm', 'run', 'start'] |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| { | ||
| "version": "1.0", | ||
| "plugins": {} | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| bundle: | ||
| name: {{.projectName}} |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| [[Repls]] | ||
| # Fix path with reverse slashes in the output for Windows. | ||
| Old = 'TEST_TMP_DIR]\\app-tip' | ||
| New = 'TEST_TMP_DIR]/app-tip' | ||
|
|
||
| [[Repls]] | ||
| Old = 'TEST_TMP_DIR]\\app-plugin' | ||
| New = 'TEST_TMP_DIR]/app-plugin' |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| {"plugins":{"databricks@claude-plugins-official":[{"scope":"user","version":"0.2.9"}]}} |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Cloud = false |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
|
|
||
| === hint printed on an authenticated command: one hint-shown event is uploaded | ||
|
|
||
| >>> [CLI] jobs list | ||
| Databricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install | ||
|
|
||
| >>> print_telemetry | ||
| {"command":"jobs_list","exit_code":0,"aitools_hint_event":{"hint_kind":"INSTALL_TEXT"}} | ||
|
|
||
| === hint printed on a command that resolves no auth: nothing is uploaded | ||
|
|
||
| >>> [CLI] version | ||
| Databricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install | ||
| Databricks CLI v[CLI_VERSION] | ||
|
|
||
| >>> print_telemetry | ||
|
|
||
| === plugin installed: no hint, no event | ||
|
|
||
| >>> [CLI] jobs list | ||
|
|
||
| >>> print_telemetry |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # Isolate HOME so the real machine's plugins and skills are not seen. | ||
| sethome home | ||
| export CLAUDECODE=1 | ||
| unset CLAUDE_CODE_SESSION_ID | ||
| # A host CLAUDE_CONFIG_DIR would move the plugin file away from the isolated HOME. | ||
| unset CLAUDE_CONFIG_DIR | ||
|
|
||
| title "hint printed on an authenticated command: one hint-shown event is uploaded" | ||
| trace $CLI jobs list | ||
| trace print_telemetry | ||
|
|
||
| title "hint printed on a command that resolves no auth: nothing is uploaded" | ||
| trace $CLI version | ||
| trace print_telemetry | ||
|
|
||
| title "plugin installed: no hint, no event" | ||
| sethome ./installed-home | ||
| trace $CLI jobs list | ||
| trace print_telemetry |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| # A host that disables telemetry or sets a zero upload timeout would upload | ||
| # nothing, so these cases would record no events. | ||
| unset DATABRICKS_CLI_DISABLE_TELEMETRY DATABRICKS_CLI_TELEMETRY_TIMEOUT | ||
|
|
||
| # print_telemetry prints each telemetry event uploaded since the last call, one | ||
| # per line: the command it ran under, its exit code, and the event without the | ||
| # rest of the execution context (timings and versions vary). print_requests.py | ||
| # clears the recorded requests, so the next call sees only the next command's | ||
| # upload. It fails when no request was recorded, hence the file check. | ||
| print_telemetry() { | ||
| if [ -f "$OUT_REQUESTS" ]; then | ||
| print_requests.py --oneline //telemetry-ext | jq -c '.body.protoLogs[] | fromjson | ||
| | .entry.databricks_cli_log | ||
| | {command: .execution_context.command, exit_code: .execution_context.exit_code} | ||
| + (del(.execution_context))' | ||
| fi | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| # Telemetry payloads of the install hint. | ||
| RecordRequests = true | ||
|
|
||
| Ignore = [ | ||
| "home", | ||
| ] |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1913,8 +1913,8 @@ func runCreate(ctx context.Context, opts createOptions) (runErr error) { | |
| // Recommend skills installation if coding agents are detected without skills. | ||
| // In flags mode, only print a hint — never prompt interactively. | ||
| if flagsMode { | ||
| if !agents.HasDatabricksSkillsInstalled(ctx) { | ||
| cmdio.LogString(ctx, "Tip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them.") | ||
| if !agents.HasDatabricksToolingInstalled(ctx) { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [nit]
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added |
||
| agents.PrintSkillsTip(ctx) | ||
| } | ||
| } else if err := agents.RecommendSkillsInstall(ctx, installer.InstallAllSkills); err != nil { | ||
| log.Warnf(ctx, "Skills recommendation failed: %v", err) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,13 +2,14 @@ package agents | |
|
|
||
| import ( | ||
| "context" | ||
| "path/filepath" | ||
| "time" | ||
|
|
||
| "github.com/databricks/cli/libs/cache" | ||
| "github.com/databricks/cli/libs/cmdio" | ||
| "github.com/databricks/cli/libs/env" | ||
| "github.com/databricks/cli/libs/log" | ||
| "github.com/databricks/cli/libs/telemetry" | ||
| "github.com/databricks/cli/libs/telemetry/protos" | ||
| "github.com/databricks/databricks-sdk-go/useragent" | ||
| "github.com/spf13/cobra" | ||
| ) | ||
|
|
@@ -49,8 +50,9 @@ const hintSessionTTL = time.Hour | |
| // tooling when Claude Code drives the CLI without it installed. It is a no-op | ||
| // for human callers, other agents, aitools commands themselves, when there is | ||
| // no output stream, once the tooling is present, and when the session was | ||
| // hinted within the last hintSessionTTL. Best-effort: it writes only to stderr | ||
| // and never fails the command. | ||
| // hinted within the last hintSessionTTL. Best-effort: it never fails the | ||
| // command. Besides the stderr line, it logs a telemetry event for each hint | ||
| // shown, which makes a command that resolved auth upload telemetry at exit. | ||
| func MaybeHint(ctx context.Context, cmd *cobra.Command) { | ||
| if useragent.AgentProvider() != claudeAgentProvider { | ||
| return | ||
|
|
@@ -72,6 +74,7 @@ func MaybeHint(ctx context.Context, cmd *cobra.Command) { | |
| return | ||
| } | ||
| cmdio.LogString(ctx, hintText) | ||
| logHintShown(ctx, protos.AitoolsHintKindInstallText) | ||
| recordSessionHint(ctx) | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [nice to have]
So the session contributes zero hint events even though the agent saw the hint. That skews "hints shown vs installs" low. Please document this session-level loss next to the "uploads only when the command resolved auth" note in
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Documented it in |
||
| } | ||
|
|
||
|
|
@@ -110,12 +113,7 @@ func claudeToolingInstalled(ctx context.Context) bool { | |
| // The canonical dir always holds a real databricks* subdir after a skills | ||
| // install, regardless of whether the agent copy is a symlink or a copy. The | ||
| // legacy dir is checked too so users of an older install aren't hinted. | ||
| home, err := env.UserHomeDir(ctx) | ||
| if err != nil { | ||
| return false | ||
| } | ||
| return HasDatabricksSkillsIn(filepath.Join(home, CanonicalSkillsDir)) || | ||
| HasDatabricksSkillsIn(filepath.Join(home, LegacySkillsDir)) | ||
| return hasSharedSkills(ctx) | ||
| } | ||
|
|
||
| // alreadyHintedThisSession reports whether this Claude session was hinted | ||
|
|
@@ -142,3 +140,17 @@ func recordSessionHint(ctx context.Context) { | |
| c := cache.NewCache(ctx, hintCacheComponent, hintSessionTTL, nil) | ||
| cache.Put(ctx, c, session, true) | ||
| } | ||
|
|
||
| // logHintShown records that a recommendation to install Databricks AI tools of | ||
| // the given kind was shown. Call it once per hint shown; for a prompt, call it | ||
| // before asking so a declined prompt still counts. The upload's | ||
| // ExecutionContext.Command names the command the hint was shown on. | ||
| // | ||
| // The event is buffered and uploaded by cmd/root at exit, which needs resolved | ||
| // auth: a hint shown on a command that never resolves auth (e.g. `databricks | ||
| // version`) is not uploaded. | ||
| func logHintShown(ctx context.Context, kind protos.AitoolsHintKind) { | ||
| telemetry.Log(ctx, protos.DatabricksCliLog{ | ||
| AitoolsHintEvent: &protos.AitoolsHintEvent{HintKind: kind}, | ||
| }) | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[nit] Under the every-agent rule, the "plugin installed" case passes only if no other agent is detected.
sethomeisolates HOME and USERPROFILE, but OpenCode and Goose find their config dirs throughAPPDATA(Windows) orXDG_CONFIG_HOME, Gemini throughGEMINI_CLI_HOME, Pi throughPI_CODING_AGENT_DIR, and Goose also throughGOOSE_PATH_ROOT. The test inherits all of these from the host. On a Windows machine with OpenCode installed, or on Linux withXDG_CONFIG_HOMEset and OpenCode or Goose installed, this case prints the tip and fails. Could you unset these variables inscript.prepare(or pointAPPDATAat a directory under$HOME), next to the existing telemetry env cleanup?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed. I'll fix it in a follow-up PR once the stack lands. The fix: in
acceptance/cmd/aitools/telemetry/script.prepare, next to the existing telemetry env cleanup, unset the agent config-dir variables (XDG_CONFIG_HOME,APPDATA,GEMINI_CLI_HOME,PI_CODING_AGENT_DIR,GOOSE_PATH_ROOT) or point them under$HOME, so only the test's own agents are detected.