Skip to content

Add authenticated AIR feedback command - #7254

Open
delbalso wants to merge 2 commits into
mainfrom
codex/air-agent-feedback
Open

delbalso wants to merge 2 commits into
mainfrom
codex/air-agent-feedback

Conversation

@delbalso

@delbalso delbalso commented Oct 29, 2026 •

Copy link
Copy Markdown

Changes

Add databricks air feedback with workspace authentication, human/agent attribution, optional run and agent context, contact opt-in, and a JSON receipt. Include command help, acceptance coverage, and a changelog entry.

Databricks employees can review staging and production reports at go/air-cli-feedback. Supporting schema: https://github.com/databricks-eng/universe/pull/2915550. Dashboard, Insights, skill, and documentation: https://github.com/databricks-eng/universe/pull/2915780. Collector and migration: https://github.com/databricks-eng/fast-feedback-notebooks/pull/49. Release after backend metadata rollout.

Why

Let users and coding agents report actionable AIR product friction through their existing workspace login, with enough context for product and engineering teams to investigate.

Tests

Before rebasing onto current main, full format, lint, and checks passed. Root unit suite: 12,519 tests (132 skipped); tools unit suite: 73 tests. Focused feedback and help acceptance tests passed. The broader local acceptance suite hit AppKit/bundle timeouts and was stopped after approximately 21 minutes; no clean full local acceptance pass is claimed. A subsequent bounded run passed feedback, help, and config-help (3/3). GitHub CI passed Linux and macOS tests in both deployment modes, lint, generated-file validation, and regression reporting on the previous head.

The rebase preserved both feedback commits without patch changes. Local retesting stopped before execution because the corporate Go proxy blocked upstream golang.org/x/net@v0.62.0 under its recently-published policy. Fresh CI was triggered on the rebased head; see the checks below for its results.

Two real staging submissions were traced from CLI receipts to raw feedback text and existing fields. New typed metadata remains pending schema deployment.

This PR was written with Codex.

@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Oct 29, 2026
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 6af851af8 | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

2 tests: regression=2 unreleased=0 coverage=0 failing=0

test branch main (aed6702) latest (v1.23.0)
TestAccept/air/feedback ✅ ❌ ❌
TestAccept/air/help ✅ ❌ ❌
TestAccept/air/feedback: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: Diff:
        --- air/feedback/output.txt
        +++ /tmp/TestAcceptairfeedback849732613/002/output.txt
        @@ -1,28 +1,31 @@
         
         === agent report with a receipt
         >>> [CLI] air feedback --message Logs omitted the failure reason --reporter agent --agent-name codex --run-id 123 -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "data": {
        -    "received": true,
        -    "feedback_id": "[UUID]"
        -  }
        -}
        +Error: unknown flag: --message
         
        -=== human report
        ->>> [CLI] air feedback --message The workaround succeeded
        -Feedback received. Reference: [UUID]
        +Usage:
        +  databricks air [command]
         
        -=== validation errors
        ->>> musterr [CLI] air feedback --message     -o json
        -true
        +Available Commands:
        +  cancel          Cancel one or more runs
        +  convert-to-dabs Convert an AIR run YAML into a Databricks Asset Bundle
        +  get             Show status, configuration, and timing details for a specified run
        +  images          Manage container images for AI Runtime
        +  list            List active runs for the current user
        +  logs            Stream or fetch logs for a run
        +  pools           View provisioned GPU pools
        +  run             Submit a training workload from a YAML config
         
        ->>> musterr [CLI] air feedback --message Logs omitted the failure reason --reporter robot -o json
        -true
        +Flags:
        +  -h, --help   help for air
         
        ->>> musterr [CLI] air feedback --message Logs omitted the failure reason --agent-name codex -o json
        -true
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
        ->>> musterr [CLI] air feedback unexpected --message Logs omitted the failure reason -o json
        -true
        +Use "databricks air [command] --help" for more information about a command.
        +
        +
        +Exit code: 1

latest (v1.23.0): same output as main (aed6702)

TestAccept/air/help: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: Diff:
        --- air/help/output.txt
        +++ /tmp/TestAcceptairhelp1530177345/002/output.txt
        @@ -11,7 +11,6 @@
         Available Commands:
           cancel          Cancel one or more runs
           convert-to-dabs Convert an AIR run YAML into a Databricks Asset Bundle
        -  feedback        Send feedback about AI Runtime to Databricks
           get             Show status, configuration, and timing details for a specified run
           images          Manage container images for AI Runtime
           list            List active runs for the current user
        @@ -214,36 +213,30 @@
         
         === feedback help
         >>> [CLI] air feedback --help
        -Send feedback about AI Runtime using your workspace authentication.
        +Run and manage AI Runtime training workloads on Databricks serverless GPU compute.
         
        -Include the command, observed problem, expected behavior, and any workaround
        -in --message. Use --reporter agent for reports written by an agent.
        +These commands are under active development.
         
        -Sends your message, CLI version, reporter type, and optional run ID and agent
        -name to Fast Feedback. The service associates it with your workspace and
        -authenticated principal. No transcripts, source files, or logs are collected.
        -Do not include credentials or sensitive data. Follow-up contact is opt-in.
        -
        -This is an explicit submission, independent of automatic usage telemetry.
        -Success confirms receipt by the collector; processing into review tools is
        -asynchronous.
        -
         Usage:
        -  databricks air feedback [flags]
        +  databricks air [command]
         
        -Examples:
        -  databricks air feedback --message "air logs omitted the failure reason" --reporter agent --agent-name codex --run-id 123 -o json
        +Available Commands:
        +  cancel          Cancel one or more runs
        +  convert-to-dabs Convert an AIR run YAML into a Databricks Asset Bundle
        +  get             Show status, configuration, and timing details for a specified run
        +  images          Manage container images for AI Runtime
        +  list            List active runs for the current user
        +  logs            Stream or fetch logs for a run
        +  pools           View provisioned GPU pools
        +  run             Submit a training workload from a YAML config
         
         Flags:
        -      --agent-name string   Agent name; requires --reporter agent
        -      --can-be-contacted    Allow follow-up about this feedback
        -  -h, --help                help for feedback
        -      --message string      Feedback message (1–10,000 characters)
        -      --reporter string     Who wrote the feedback: human or agent (default "human")
        -      --run-id string       Related job run ID
        +  -h, --help   help for air
         
         Global Flags:
               --debug            enable debug logging
           -o, --output type      output type: text or json (default text)
           -p, --profile string   ~/.databrickscfg profile
           -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks air [command] --help" for more information about a command.
        
FAIL

latest (v1.23.0): same output as main (aed6702)

@delbalso
delbalso force-pushed the codex/air-agent-feedback branch from b40f37c to 6af851a Compare October 29, 2026 17:01
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 6af851a

Run: 40917487010

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 281 17 5:09
✅​ aws-windows-latest-4core-16gb 283 15 6:24
✅​ azure linux-2core-8gb 280 17 5:20
✅​ azure-windows-latest-4core-16gb 282 15 5:51
✅​ gcp linux-2core-8gb 281 17 5:28
✅​ gcp-windows-latest-4core-16gb 283 15 4:45
Top 6 slowest tests (at least 2 minutes):
duration env testname
6:16 aws-windows-latest-4core-16gb TestAccept
5:44 azure-windows-latest-4core-16gb TestAccept
4:40 gcp-windows-latest-4core-16gb TestAccept
3:39 azure linux-2core-8gb TestAccept
3:39 gcp linux-2core-8gb TestAccept
3:28 aws linux-2core-8gb TestAccept

@delbalso
delbalso marked this pull request as ready for review October 29, 2026 17:42
@delbalso
delbalso requested review from a team as code owners October 29, 2026 17:42
@delbalso
delbalso requested a review from rclarey October 29, 2026 17:42
@maggiewang-db

Copy link
Copy Markdown
Contributor

This PR depends on backend changes that need to be merged and released before it merges:

  1. https://github.com/databricks-eng/universe/pull/2915550: new FastFeedbackLog fields. Deploy it to the workspace telemetry handler and Lumberjack, and confirm the new fields appear in the staging feedback logs.
  2. https://github.com/databricks-eng/fast-feedback-notebooks/pull/49: collector migration and Jira sync. Run the migration, then deploy the reader and sync.
  3. https://github.com/databricks-eng/universe/pull/2915780: ingestion and dashboard changes. The agent skill and public docs in that PR should land only after a CLI release that includes this command, so it may be worth splitting.

Please hold this PR until those are live.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-assisted AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants