Repository navigation
fix(spark): allow directory paths in submit_deps - #6368
Merged
rsareddy0329 merged 1 commit intoOct 5, 2026
Merged
Conversation
PySparkProcessor/SparkJarProcessor rejected a local directory passed via submit_py_files / submit_jars / submit_files: _stage_submit_deps required os.path.isfile(dep_path) and raised "is not a valid local file" for a directory. Users could only list individual files. Accept a directory as well: copy its tree into the staging tempdir (via shutil.copytree) so its contents are uploaded to the dependency input channel and mounted into the Spark container, alongside the existing single-file handling. Update the error message to mention directories. Add a unit test that a directory is staged (not rejected). Relates-to aws#2447
jam-jee
approved these changes
Oct 5, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available: Relates to #2447
Description of changes:
PySparkProcessor/SparkJarProcessorrejected a local directory passed viasubmit_py_files/submit_jars/submit_files._stage_submit_depsrequiredos.path.isfile(dep_path)and raisedsubmit_deps path ... is not a valid local filefor a directory, so users could only enumerate individual files.
This accepts a directory as well: its tree is copied into the staging temp dir (via
shutil.copytree) so its contents are uploaded to the dependency input channel andmounted into the Spark container, alongside the existing single-file handling. The
error message now mentions directories.
Testing:
test_stage_submit_deps_accepts_directory— a local directory isstaged (input channel created, contents uploaded) rather than rejected. Verified it
fails on unfixed
master-v2and passes with this change.test_stage_submit_depscases still pass (11 passed).flake8andblack -l 100clean on changed files.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.