Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .nextchanges/cli/apps-init-plugin-tip.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* `apps init` no longer suggests installing skills when every detected coding agent already has the Databricks plugin. ([#7467](https://github.com/databricks/cli/pull/7467))
1 change: 1 addition & 0 deletions .nextchanges/cli/telemetry-upload-timeout.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* Bound the whole telemetry upload at exit, including config resolution and authentication, by its timeout, so a stalled auth or telemetry endpoint no longer keeps a command from exiting. ([#7467](https://github.com/databricks/cli/pull/7467))
Empty file.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
30 changes: 30 additions & 0 deletions acceptance/cmd/aitools/telemetry/apps-init-tip/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@

=== apps init with flags prints the tip: one hint-shown event per hint printed

>>> [CLI] apps init --template ./template --skip-install --name app-tip
Databricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install
✔ Creating project

✔ Project created successfully!

Location: [TEST_TMP_DIR]/app-tip
Files: 3

Tip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them.

>>> print_telemetry
{"command":"apps_init","exit_code":0,"aitools_hint_event":{"hint_kind":"INSTALL_TEXT"}}
{"command":"apps_init","exit_code":0,"aitools_hint_event":{"hint_kind":"APPS_INIT_TIP"}}

=== plugin installed: no hint, no tip, no event

>>> [CLI] apps init --template ./template --skip-install --name app-plugin
✔ Creating project

✔ Project created successfully!

Location: [TEST_TMP_DIR]/app-plugin
Files: 3


>>> print_telemetry
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"plugins":{"databricks@claude-plugins-official":[{"scope":"user","version":"0.2.9"}]}}
18 changes: 18 additions & 0 deletions acceptance/cmd/aitools/telemetry/apps-init-tip/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Isolate HOME so the real machine's agents and skills are not seen. The
# committed home/.claude, a Claude Code config dir without Databricks skills, is
# what makes apps init print its tip.
sethome home
export CLAUDECODE=1
unset CLAUDE_CODE_SESSION_ID
unset CLAUDE_CONFIG_DIR

title "apps init with flags prints the tip: one hint-shown event per hint printed"
trace $CLI apps init --template ./template --skip-install --name app-tip > LOG.apps-init
trace print_telemetry
rm -rf app-tip

title "plugin installed: no hint, no tip, no event"
sethome ./plugin-home
trace $CLI apps init --template ./template --skip-install --name app-plugin > LOG.apps-init-plugin
trace print_telemetry
rm -rf app-plugin
Comment on lines +14 to +18

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Under the every-agent rule, the "plugin installed" case passes only if no other agent is detected. sethome isolates HOME and USERPROFILE, but OpenCode and Goose find their config dirs through APPDATA (Windows) or XDG_CONFIG_HOME, Gemini through GEMINI_CLI_HOME, Pi through PI_CODING_AGENT_DIR, and Goose also through GOOSE_PATH_ROOT. The test inherits all of these from the host. On a Windows machine with OpenCode installed, or on Linux with XDG_CONFIG_HOME set and OpenCode or Goose installed, this case prints the tip and fails. Could you unset these variables in script.prepare (or point APPDATA at a directory under $HOME), next to the existing telemetry env cleanup?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I'll fix it in a follow-up PR once the stack lands. The fix: in acceptance/cmd/aitools/telemetry/script.prepare, next to the existing telemetry env cleanup, unset the agent config-dir variables (XDG_CONFIG_HOME, APPDATA, GEMINI_CLI_HOME, PI_CODING_AGENT_DIR, GOOSE_PATH_ROOT) or point them under $HOME, so only the test's own agents are detected.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
command: ['npm', 'run', 'start']
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"version": "1.0",
"plugins": {}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
bundle:
name: {{.projectName}}
8 changes: 8 additions & 0 deletions acceptance/cmd/aitools/telemetry/apps-init-tip/test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[[Repls]]
# Fix path with reverse slashes in the output for Windows.
Old = 'TEST_TMP_DIR]\\app-tip'
New = 'TEST_TMP_DIR]/app-tip'

[[Repls]]
Old = 'TEST_TMP_DIR]\\app-plugin'
New = 'TEST_TMP_DIR]/app-plugin'
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"plugins":{"databricks@claude-plugins-official":[{"scope":"user","version":"0.2.9"}]}}
1 change: 1 addition & 0 deletions acceptance/cmd/aitools/telemetry/hint/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
22 changes: 22 additions & 0 deletions acceptance/cmd/aitools/telemetry/hint/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@

=== hint printed on an authenticated command: one hint-shown event is uploaded

>>> [CLI] jobs list
Databricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install

>>> print_telemetry
{"command":"jobs_list","exit_code":0,"aitools_hint_event":{"hint_kind":"INSTALL_TEXT"}}

=== hint printed on a command that resolves no auth: nothing is uploaded

>>> [CLI] version
Databricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install
Databricks CLI v[CLI_VERSION]

>>> print_telemetry

=== plugin installed: no hint, no event

>>> [CLI] jobs list

>>> print_telemetry
19 changes: 19 additions & 0 deletions acceptance/cmd/aitools/telemetry/hint/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Isolate HOME so the real machine's plugins and skills are not seen.
sethome home
export CLAUDECODE=1
unset CLAUDE_CODE_SESSION_ID
# A host CLAUDE_CONFIG_DIR would move the plugin file away from the isolated HOME.
unset CLAUDE_CONFIG_DIR

title "hint printed on an authenticated command: one hint-shown event is uploaded"
trace $CLI jobs list
trace print_telemetry

title "hint printed on a command that resolves no auth: nothing is uploaded"
trace $CLI version
trace print_telemetry

title "plugin installed: no hint, no event"
sethome ./installed-home
trace $CLI jobs list
trace print_telemetry
17 changes: 17 additions & 0 deletions acceptance/cmd/aitools/telemetry/script.prepare
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# A host that disables telemetry or sets a zero upload timeout would upload
# nothing, so these cases would record no events.
unset DATABRICKS_CLI_DISABLE_TELEMETRY DATABRICKS_CLI_TELEMETRY_TIMEOUT

# print_telemetry prints each telemetry event uploaded since the last call, one
# per line: the command it ran under, its exit code, and the event without the
# rest of the execution context (timings and versions vary). print_requests.py
# clears the recorded requests, so the next call sees only the next command's
# upload. It fails when no request was recorded, hence the file check.
print_telemetry() {
if [ -f "$OUT_REQUESTS" ]; then
print_requests.py --oneline //telemetry-ext | jq -c '.body.protoLogs[] | fromjson
| .entry.databricks_cli_log
| {command: .execution_context.command, exit_code: .execution_context.exit_code}
+ (del(.execution_context))'
fi
}
6 changes: 6 additions & 0 deletions acceptance/cmd/aitools/telemetry/test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Telemetry payloads of the install hint.
RecordRequests = true

Ignore = [
"home",
]
4 changes: 2 additions & 2 deletions cmd/apps/init.go
Original file line number Diff line number Diff line change
Expand Up @@ -1913,8 +1913,8 @@ func runCreate(ctx context.Context, opts createOptions) (runErr error) {
// Recommend skills installation if coding agents are detected without skills.
// In flags mode, only print a hint — never prompt interactively.
if flagsMode {
if !agents.HasDatabricksSkillsInstalled(ctx) {
cmdio.LogString(ctx, "Tip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them.")
if !agents.HasDatabricksToolingInstalled(ctx) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] apps init now skips its skills tip and its "Install skills now?" prompt when a detected agent has the databricks plugin, which users will notice. .agents/skills/pr-checklist/SKILL.md lists changed command behavior as needing a .nextchanges fragment, but the only fragment added covers the upload timeout. Please add one, e.g. .nextchanges/cli/apps-init-plugin-tip.md: "* apps init no longer suggests installing skills when a detected coding agent has the Databricks plugin. (#7467)"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added .nextchanges/cli/apps-init-plugin-tip.md with your wording and the PR link. In 29d7bbc.

agents.PrintSkillsTip(ctx)
}
} else if err := agents.RecommendSkillsInstall(ctx, installer.InstallAllSkills); err != nil {
log.Warnf(ctx, "Skills recommendation failed: %v", err)
Expand Down
3 changes: 2 additions & 1 deletion cmd/root/root.go
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,8 @@ func New(ctx context.Context) *cobra.Command {
cmd.SetContext(ctx)

// Recommend installing Databricks AI tooling to Claude Code when it is
// driving the CLI without the tooling installed (best-effort, stderr only).
// driving the CLI without the tooling installed (best-effort: a stderr
// line and a telemetry event that is uploaded at exit).
agents.MaybeHint(ctx, cmd)
return nil
}
Expand Down
30 changes: 21 additions & 9 deletions libs/aitools/agents/hint.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,14 @@ package agents

import (
"context"
"path/filepath"
"time"

"github.com/databricks/cli/libs/cache"
"github.com/databricks/cli/libs/cmdio"
"github.com/databricks/cli/libs/env"
"github.com/databricks/cli/libs/log"
"github.com/databricks/cli/libs/telemetry"
"github.com/databricks/cli/libs/telemetry/protos"
"github.com/databricks/databricks-sdk-go/useragent"
"github.com/spf13/cobra"
)
Expand Down Expand Up @@ -49,8 +50,9 @@ const hintSessionTTL = time.Hour
// tooling when Claude Code drives the CLI without it installed. It is a no-op
// for human callers, other agents, aitools commands themselves, when there is
// no output stream, once the tooling is present, and when the session was
// hinted within the last hintSessionTTL. Best-effort: it writes only to stderr
// and never fails the command.
// hinted within the last hintSessionTTL. Best-effort: it never fails the
// command. Besides the stderr line, it logs a telemetry event for each hint
// shown, which makes a command that resolved auth upload telemetry at exit.
func MaybeHint(ctx context.Context, cmd *cobra.Command) {
if useragent.AgentProvider() != claudeAgentProvider {
return
Expand All @@ -72,6 +74,7 @@ func MaybeHint(ctx context.Context, cmd *cobra.Command) {
return
}
cmdio.LogString(ctx, hintText)
logHintShown(ctx, protos.AitoolsHintKindInstallText)
recordSessionHint(ctx)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nice to have] recordSessionHint marks the session as hinted even when this command's event can never upload. Take a Claude session whose first non-aitools command resolves no auth, e.g. databricks version or auth profiles:

  • the hint prints;
  • the event is dropped at exit, because there is no ConfigUsed;
  • for the next hour, later commands in that session print no hint and log nothing.

So the session contributes zero hint events even though the agent saw the hint. That skews "hints shown vs installs" low. Please document this session-level loss next to the "uploads only when the command resolved auth" note in protos/README.md, or decide whether the metric can accept it before dashboards depend on it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documented it in libs/telemetry/protos/README.md, right after the "uploads only when the command resolved auth" sentence: because the hint prints once per Claude Code session per hour, a session whose hinted command resolves no auth (e.g. databricks version) logs no install hint event for that hour. Merge Danger says the same. In 0326a2e.

}

Expand Down Expand Up @@ -110,12 +113,7 @@ func claudeToolingInstalled(ctx context.Context) bool {
// The canonical dir always holds a real databricks* subdir after a skills
// install, regardless of whether the agent copy is a symlink or a copy. The
// legacy dir is checked too so users of an older install aren't hinted.
home, err := env.UserHomeDir(ctx)
if err != nil {
return false
}
return HasDatabricksSkillsIn(filepath.Join(home, CanonicalSkillsDir)) ||
HasDatabricksSkillsIn(filepath.Join(home, LegacySkillsDir))
return hasSharedSkills(ctx)
}

// alreadyHintedThisSession reports whether this Claude session was hinted
Expand All @@ -142,3 +140,17 @@ func recordSessionHint(ctx context.Context) {
c := cache.NewCache(ctx, hintCacheComponent, hintSessionTTL, nil)
cache.Put(ctx, c, session, true)
}

// logHintShown records that a recommendation to install Databricks AI tools of
// the given kind was shown. Call it once per hint shown; for a prompt, call it
// before asking so a declined prompt still counts. The upload's
// ExecutionContext.Command names the command the hint was shown on.
//
// The event is buffered and uploaded by cmd/root at exit, which needs resolved
// auth: a hint shown on a command that never resolves auth (e.g. `databricks
// version`) is not uploaded.
func logHintShown(ctx context.Context, kind protos.AitoolsHintKind) {
telemetry.Log(ctx, protos.DatabricksCliLog{
AitoolsHintEvent: &protos.AitoolsHintEvent{HintKind: kind},
})
}
67 changes: 32 additions & 35 deletions libs/aitools/agents/hint_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,46 +8,22 @@ import (
"path/filepath"
"testing"

"github.com/databricks/cli/internal/testutil"
"github.com/databricks/cli/libs/cmdio"
"github.com/databricks/cli/libs/env"
"github.com/databricks/cli/libs/log"
"github.com/databricks/databricks-sdk-go/useragent"
"github.com/databricks/cli/libs/telemetry"
"github.com/databricks/cli/libs/telemetry/protos"
"github.com/spf13/cobra"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)

// agentEnvVars are the env vars useragent.AgentProvider() inspects. Kept in
// sync with listKnownAgents() in the SDK plus the AGENT/AI_AGENT fallbacks
// (mirrors acceptance/acceptance_test.go).
var agentEnvVars = []string{
"AGENT", "AI_AGENT", "AMP_CURRENT_THREAD_ID", "ANTIGRAVITY_AGENT",
"AUGMENT_AGENT", "CLAUDECODE", "CLINE_ACTIVE", "CODEX_CI", "COPILOT_CLI",
"CURSOR_AGENT", "GEMINI_CLI", "GOOSE_TERMINAL", "KIRO", "OPENCLAW_SHELL",
"OPENCODE", "VSCODE_AGENT", "WINDSURF_AGENT",
}

// clearAgentEnv unsets every agent-detection env var (restoring them after the
// test) so the host agent driving this test binary does not leak into
// AgentProvider(). t.Setenv cannot unset, hence the manual save/restore.
func clearAgentEnv(t *testing.T) {
for _, v := range agentEnvVars {
if orig, ok := os.LookupEnv(v); ok {
t.Cleanup(func() { os.Setenv(v, orig) }) //nolint:usetesting // restoring, not setting fresh
os.Unsetenv(v) //nolint:usetesting // t.Setenv cannot unset
}
}
}

// setClaudeAgent makes useragent.AgentProvider() report Claude Code for this
// test. It clears the ambient agent env, sets CLAUDECODE, and resets the SDK's
// process-global cache, so these tests must not run in parallel. The cleanup
// clears the cache again so the value never leaks into another test.
// test, with the host's environment cleared. These tests must not run in
// parallel (see testutil.SetAgentEnv).
func setClaudeAgent(t *testing.T) {
clearAgentEnv(t)
t.Setenv("CLAUDECODE", "1")
useragent.ClearCache()
t.Cleanup(useragent.ClearCache)
testutil.SetAgentEnv(t, "CLAUDECODE", "1")
}

// claudeCtx returns a context with a captured stderr buffer and an isolated
Expand All @@ -72,22 +48,35 @@ func writeCanonicalSkill(t *testing.T, home, name string) {
require.NoError(t, os.MkdirAll(filepath.Join(home, ".databricks", "aitools", "skills", name), 0o755))
}

// hintKinds returns the kinds of the hint-shown events buffered on ctx,
// failing on any other event so a stray event cannot hide behind the filter.
func hintKinds(t *testing.T, ctx context.Context) []protos.AitoolsHintKind {
var out []protos.AitoolsHintKind
for _, e := range telemetry.BufferedEvents(ctx) {
require.NotNil(t, e.AitoolsHintEvent, "unexpected non-hint event: %+v", e)
out = append(out, e.AitoolsHintEvent.HintKind)
}
return out
}

func TestMaybeHint_HumanCallerSilent(t *testing.T) {
// No agent env: AgentProvider() == "", so no hint even with nothing installed.
clearAgentEnv(t)
useragent.ClearCache()
t.Cleanup(useragent.ClearCache)
testutil.SetAgentEnv(t)
ctx, stderr := cmdio.NewTestContextWithStderr(t.Context())
ctx = env.WithUserHomeDir(ctx, t.TempDir())
ctx = telemetry.WithNewLogger(ctx)
MaybeHint(ctx, regularCmd())
assert.Empty(t, stderr.String())
assert.Empty(t, hintKinds(t, ctx))
}

func TestMaybeHint_ClaudeNotInstalledHints(t *testing.T) {
setClaudeAgent(t)
ctx, stderr := claudeCtx(t, t.TempDir())
ctx = telemetry.WithNewLogger(ctx)
MaybeHint(ctx, regularCmd())
assert.Contains(t, stderr.String(), hintText)
assert.Equal(t, []protos.AitoolsHintKind{protos.AitoolsHintKindInstallText}, hintKinds(t, ctx))
}

func TestMaybeHint_PluginInstalledSilent(t *testing.T) {
Expand All @@ -97,9 +86,11 @@ func TestMaybeHint_PluginInstalledSilent(t *testing.T) {
// The core regression: a plugin install writes only Claude's manifest, no
// skills subdir. Detection must treat it as installed.
writePluginManifest(t, home, `{"plugins":{"databricks@claude-plugins-official":[{"scope":"user"}]}}`)
ctx = telemetry.WithNewLogger(ctx)

MaybeHint(ctx, regularCmd())
assert.Empty(t, stderr.String())
assert.Empty(t, hintKinds(t, ctx))
}

func TestClaudeToolingInstalled_LogsPluginVersion(t *testing.T) {
Expand Down Expand Up @@ -191,21 +182,25 @@ func TestMaybeHint_NonDatabricksSkillStillHints(t *testing.T) {
func TestMaybeHint_AIToolsCommandSilent(t *testing.T) {
setClaudeAgent(t)
ctx, stderr := claudeCtx(t, t.TempDir())
ctx = telemetry.WithNewLogger(ctx)

parent := &cobra.Command{Use: aiToolsCommandName}
child := &cobra.Command{Use: "install"}
parent.AddCommand(child)

MaybeHint(ctx, child)
assert.Empty(t, stderr.String())
assert.Empty(t, hintKinds(t, ctx))
}

func TestMaybeHint_NoIONoPanic(t *testing.T) {
setClaudeAgent(t)
// Bare context without cmdio installed: cobra resolves --help and bare
// invocations before PersistentPreRunE runs, so HasIO can be false here.
ctx := env.WithUserHomeDir(t.Context(), t.TempDir())
ctx = telemetry.WithNewLogger(ctx)
assert.NotPanics(t, func() { MaybeHint(ctx, regularCmd()) })
assert.Empty(t, hintKinds(t, ctx))
}

func TestMaybeHint_ThrottledPerSession(t *testing.T) {
Expand All @@ -217,15 +212,17 @@ func TestMaybeHint_ThrottledPerSession(t *testing.T) {
ctx = env.WithUserHomeDir(ctx, home)
ctx = env.Set(ctx, "DATABRICKS_CACHE_DIR", cacheDir)
ctx = env.Set(ctx, claudeSessionEnv, "session-abc")
return ctx, stderr
return telemetry.WithNewLogger(ctx), stderr
}

ctx1, stderr1 := newCtx()
MaybeHint(ctx1, regularCmd())
assert.Contains(t, stderr1.String(), hintText)
assert.Equal(t, []protos.AitoolsHintKind{protos.AitoolsHintKindInstallText}, hintKinds(t, ctx1))

// Same session id, second command: must stay quiet.
// Same session id, second command: must stay quiet and log nothing.
ctx2, stderr2 := newCtx()
MaybeHint(ctx2, regularCmd())
assert.Empty(t, stderr2.String())
assert.Empty(t, hintKinds(t, ctx2))
}
Loading
Loading