Repository navigation
Fix AWS Spark Python task acceptance test access mode - #7187
Open
shreyas-goenka wants to merge 3 commits into
Open
shreyas-goenka wants to merge 3 commits into
shreyas-goenka wants to merge 3 commits into
Conversation
Contributor
|
Commit: Regression test reportSelected 1 changed tests (limit=20, 0 not selected) 2 tests: regression=0 unreleased=0 coverage=2 failing=0
|
Collaborator
Integration test reportCommit: 87e25b1
Top 15 slowest tests (at least 2 minutes):
|
denik
requested changes
Oct 27, 2026
denik
left a comment
denik
left a comment
Contributor
There was a problem hiding this comment.
Should we change the config? it seems like a backend issue. I have ES ticket with jobs team. We can use known failures for now.
janniklasrose
removed their request for review
November 9, 2026 09:36
| node_type_id: $NODE_TYPE_ID | ||
| instance_pool_id: $TEST_INSTANCE_POOL_ID | ||
| num_workers: 2 | ||
| data_security_mode: USER_ISOLATION |
Contributor
There was a problem hiding this comment.
Nothing here says why the mode is pinned: the AWS test account enabled admin protection for no-isolation shared clusters, and the test that actually needs this is run/spark_python_task, which copies this template; deploy/simple itself never runs the job. Without that, a later trim of the "simple" fixture or a switch to $DATA_SECURITY_MODE like the library fixtures use would quietly bring the AWS failure back. Can you add a one-line comment above it with the reason and the dependent test?
This was referenced Nov 11, 2026
denik
added
the
test-only-fix
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The AWS test account enabled admin protection for no-isolation shared clusters on October 22 at 14:43 UTC. The CI principal is a workspace admin, and the test left cluster access mode unspecified. Workspace file uploads and API reads succeed, but this protection prevents the driver from reading the Python file. The original test passed at 12:38 UTC and failed by 19:09 UTC that day; an account audit event confirms the setting enablement between those runs.
Set USER_ISOLATION (standard access mode) in the existing shared cluster template. This restores the task's workspace-file access while retaining the security protection. The final diff is one added line.
The restriction is documented: https://docs.databricks.com/aws/en/admin/account-settings/no-isolation-shared
Validation: