Option to change configuration relating python version(3)

[abdulnyctale]

Is there any possibility relating submitting a custom jar for script runner or anything else to change the version of python after bootstrap and before spark job?

[RikHeijdens]

Hey @abdulnyctale, custom jars to be used by your Spark job can be supplied using the --submit-args argument as demonstrated here. Does that not provide the functionality you're looking for?

Currently sparksteps does not allow end-users to specify a Python version. This would be a nice addition to the library. Implementing this in sparksteps should be fairly straightforward, basically what we need to do is expose a commandline option that allows the python version to be specified, and if it is specified we should update the value of PYSPARK_PYTHON appropriately for the spark-env classification when launching the cluster. If this is something you might be interested in contributing, feel free to shoot in a PR.

[abdulnyctale]

@RikHeijdens I wanted the custom jars for Amazon .sh script runner as i wanted to run scipt to change python version used by spark which was not possible in bootstrap.

I have modified sparkstep to do that, will certainty contact you for contribution and make fork and push those. Though i dont find what is PR and how to contact

[abdulbasitds]

@RikHeijdens i have forked the branch and have added the option for configuration from my personal profile, can you confirm if you got the pull request?