[Release] Release v1.18.0

## Release v1.18.0

### CLI

 * The AI Runtime commands have moved to `databricks air`. The previous `databricks experimental air` path now directs users to the new command. ([#6722](#6722))
 * Write local state, cache, and config files atomically so an interrupted or concurrent write cannot corrupt them. ([#6708](#6708))
 * Deprecate `--region` in `databricks auth docker configure` ahead of its removal in the next release, infer the Artifact Registry region when it is omitted, and add `databricks auth docker host --profile <name>` to show the profile's registry host and credential-helper status. ([#6782](#6782))
 * Return `UNAUTHENTICATED` instead of `INVALID_REFRESH_TOKEN` when `databricks auth token --output json` cannot refresh a cached U2M token. ([#6731](#6731))
 * Retry the current-user (SCIM `Me`) lookup on transient HTTP 500 responses so a temporarily-unavailable backend no longer fails bundle commands outright. ([#6766](#6766))
 * Preserve workspace-file and volume access for SSH server descendants when the bootstrap notebook exits and the server survives. ([#6645](#6645))
 * `ssh connect` and `ssh setup` now accept a `--keep-detached-processes` flag to keep processes detached from the SSH session (`tmux`, `setsid`, `nohup`) running after the tunnel shuts down. Teardown then terminates only the tunnel's own process group, and the bootstrap job run is held open while any detached process is still running, so the survivors keep their `/Workspace` and `/Volumes` access. A held-open run also suppresses cluster autotermination, so the flag is off by default, is bounded by `--server-timeout`, and is dedicated-cluster only. Without it, the server now logs a warning naming the detached processes it is about to destroy, instead of sweeping them silently. ([#6387](#6387))

### Bundles

 * direct: Allow clearing a catalog's or schema's `custom_max_retention_hours` by removing it from configuration. ([#6792](#6792))
 * direct: Allow clearing a genie space's `description` and a secret's `comment` by removing them from configuration. ([#6789](#6789))
 * Fix direct-engine deploy recreating an MLflow experiment on every deploy when its `trace_location` was set out-of-band. ([#6787](#6787))
 * direct: Store a Genie space's `serialized_space` in state as a content hash instead of its full contents. ([#6707](#6707))
 * Fix `bundle deploy` failing with "Invalid python file reference" for jobs that use `git_source` with a `spark_python_task` on the direct engine. ([#6751](#6751))
 * Fixed the direct engine mishandling UC grants that combine `ALL_PRIVILEGES` with a privilege it does not imply (`MANAGE`, `READ_METADATA`, `EXTERNAL_USE_SCHEMA`, `EXTERNAL_USE_LOCATION`): such privileges were dropped when granted and left behind when revoked, so the deployment never converged. ([#6733](#6733), [#6743](#6743))
 * Don't fail migration if clean up actions fail. ([#6772](#6772))
 * Ignore the backend-provided `spark.sql.ansi.enabled: "true"` pipeline configuration default when detecting direct-engine drift. ([#6816](#6816))
 * Fix recreating a postgres synced table sometimes failing with a 409 ALREADY_EXISTS error while the previous table is still being deleted. ([#6728](#6728))
 * Direct engine no longer recreates a resource when an immutable field the config omits was populated by the backend. ([#6790](#6790))

### Dependency Updates

 * Bump dependencies with known vulnerabilities. ([#6723](#6723))
 * Bump `github.com/databricks/databricks-sdk-go` from v0.178.0 to v0.182.0. ([#6817](#6817))
 * Bump the Databricks Terraform provider from v1.132.0 to v1.134.0. ([#6818](#6818))

Author: deco-sdk-tagging[bot]