Skip to content

Pipeline TypeError: can only concatenate list (not "NoneType") to list Using SourceCode #5518

Closed

Description

@CongaJAMM

PySDK Version

  • PySDK V2 (2.x)
  • PySDK V3 (3.3.1)

Describe the bug
Forces a parameter for an optional argument in a SourceCode object when using the Pipeline class upsert method.

To reproduce
A clear, step-by-step set of instructions to reproduce the bug.
The provided code need to be complete and runnable, if additional data is needed, please include them in the issue.
Conda environment with this requirements.yaml file content:
name: aws_env
channels:

  • pytorch
  • nvidia
  • conda-forge
  • defaults
    dependencies:
  • python=3.12
  • pip=25.3
  • docker=20.10.9
  • imbalanced-learn=0.14.0
  • ipykernel=7.1.0
  • ipywidgets=8.1.8
  • numpy=2.4.0
  • pandas=2.3.2
  • pillow=10.4.0
  • pyarrow=21.0.0
  • pytorch=2.5.1
  • pytorch-cuda=12.4
  • s3fs=2025.10.0
  • scikit-learn=1.7.2
  • seaborn=0.13.2
  • pip:
    • sagemaker==3.3.1

Code -
def main():

src = SourceCode(
source_dir=str(
SOURCE_DIR
), # Everything inside the source_dir will be copied to /opt/ml/input/data/code/**
entry_script="weather_pred/training/train.py", # Relative to the source_dir
# command="python train.py epochs=1",
ignore_patterns=[
".env",
".git",
"pycache",
".DS_Store",
".cache",
".ipynb_checkpoints",
"*.egg-info",
],
)
# ADD THIS LINE to satisfy the internal hashing utility
compute = Compute(
instance_type=TRAIN_INSTANCE_TYPE,
instance_count=TRAIN_INSTANCE_COUNT,
volume_size_in_gb=TRAIN_VOLUME_SIZE_GB,
)
my_output_config = OutputDataConfig(
s3_output_path=TRAIN_S3_OUTPUT_PATH, # Configure path to output artifacts
compression_type="GZIP", # Options: 'GZIP' (default) or 'NONE'
# kms_key_id="your-kms-key-arn", # Optional: for encryption at rest
)

trainer = ModelTrainer(
    role=SAGEMAKER_ROLE,
    sagemaker_session=pipeline_session,
    output_data_config=my_output_config,
    base_job_name="local-trainer",
    source_code=src,
    compute=compute,
    training_image=TRAIN_IMAGE_URI,
    training_input_mode="File",
    environment=env_vars,
)

# Receive the TrainingJobRequest object (indicates everything required to run the training job)
train_step_args = trainer.train()

# Look into cache configs later
# my_train_cache_config = CacheConfig(enable_caching=True, expire_after="P1D")

# Define the Training Step
training_step = TrainingStep(
    name="TrainWeatherModelStep",
    step_args=train_step_args,
    display_name="Training-Phase",
    description="Executing the defined modeltrainer wrapper to build and train an Agent",
    cache_config=None,
    depends_on=None,
    retry_policies=None,
)

# Initialize the Pipeline
pipeline = Pipeline(
    name=PIPELINE_NAME,
    # parameters=None,
    # pipeline_experiment_config=None,
    # mlflow_config=None,
    # steps=[training_step, model_step],
    steps=[training_step],
    sagemaker_session=pipeline_session,
    # pipeline_definition_config=None,
)

# This step either creates a new pipeline in SageMaker or updates an existing one with the same name.
pipeline.upsert(role_arn=SAGEMAKER_ROLE)   # <<== PROBLEM HERE

# This command kicks off the actual execution of the pipeline in SageMaker. From this point, SageMaker will orchestrate the execution of each step, managing resources and data flow between steps.
execution = pipeline.start()

Expected behavior
Expected to run without any specific requirements and rely on just the container image's dependencies.

Screenshots or logs
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in _run_module_as_main:198 │
│ in _run_code:88 │
│ │
│ /mnt/c/Users/brianperez/Desktop/DEV/aws/Weather-Pred-AWS/src/weather_pred/orchestrate/ml-pipelin │
│ e-cloud.py:463 in │
│ │
│ 460 │
│ 461 │
│ 462 if name == "main": │
│ ❱ 463 │ main() │
│ 464 │
│ │
│ /mnt/c/Users/brianperez/Desktop/DEV/aws/Weather-Pred-AWS/src/weather_pred/orchestrate/ml-pipelin │
│ e-cloud.py:387 in main │
│ │
│ 384 │ ) │
│ 385 │ │
│ 386 │ # This step either creates a new pipeline in SageMaker or updates an existing one wi │
│ ❱ 387 │ pipeline.upsert(role_arn=SAGEMAKER_ROLE) │
│ 388 │ │
│ 389 │ # This command kicks off the actual execution of the pipeline in SageMaker. From thi │
│ 390 │ execution = pipeline.start() │
│ │
│ /home/congajamm/miniconda3/envs/aws_env/lib/python3.12/site-packages/sagemaker/mlops/workflow/pi │
│ peline.py:333 in upsert │
│ │
│ 330 │ │ │ # after fetching the config. │
│ 331 │ │ │ raise ValueError("An AWS IAM role is required to create or update a Pipeline │
│ 332 │ │ try: │
│ ❱ 333 │ │ │ response = self.create(role_arn, description, tags, parallelism_config) │
│ 334 │ │ except ClientError as ce: │
│ 335 │ │ │ error_code = ce.response["Error"]["Code"] │
│ 336 │ │ │ error_message = ce.response["Error"]["Message"] │
│ │
│ /home/congajamm/miniconda3/envs/aws_env/lib/python3.12/site-packages/sagemaker/core/telemetry/te │
│ lemetry_logging.py:168 in wrapper │
│ │
│ 165 │ │ │ │ │ caught_ex = e │
│ 166 │ │ │ │ finally: │
│ 167 │ │ │ │ │ if caught_ex: │
│ ❱ 168 │ │ │ │ │ │ raise caught_ex │
│ 169 │ │ │ │ │ return response # pylint: disable=W0150 │
│ 170 │ │ │ else: │
│ 171 │ │ │ │ logger.debug( │
│ │
│ /home/congajamm/miniconda3/envs/aws_env/lib/python3.12/site-packages/sagemaker/core/telemetry/te │
│ lemetry_logging.py:139 in wrapper │
│ │
│ 136 │ │ │ │ start_timer = perf_counter() │
│ 137 │ │ │ │ try: │
│ 138 │ │ │ │ │ # Call the original function │
│ ❱ 139 │ │ │ │ │ response = func(*args, **kwargs) │
│ 140 │ │ │ │ │ stop_timer = perf_counter() │
│ 141 │ │ │ │ │ elapsed = stop_timer - start_timer │
│ 142 │ │ │ │ │ extra += f"&x-latency={round(elapsed, 2)}" │
│ │
│ /home/congajamm/miniconda3/envs/aws_env/lib/python3.12/site-packages/sagemaker/mlops/workflow/pi │
│ peline.py:197 in create │
│ │
│ 194 │ │ tags = format_tags(tags) │
│ 195 │ │ tags = _append_project_tags(tags) │
│ 196 │ │ tags = self.sagemaker_session.append_sagemaker_config_tags(tags, PIPELINE_TAGS │
│ ❱ 197 │ │ kwargs = self._create_args(role_arn, description, parallelism_config) │
│ 198 │ │ update_args( │
│ 199 │ │ │ kwargs, │
│ 200 │ │ │ Tags=tags, │
│ │
│ /home/congajamm/miniconda3/envs/aws_env/lib/python3.12/site-packages/sagemaker/mlops/workflow/pi │
│ peline.py:220 in _create_args │
│ │
│ 217 │ │ Returns: │
│ 218 │ │ │ A keyword argument dict for calling create_pipeline. │
│ 219 │ │ """ │
│ ❱ 220 │ │ pipeline_definition = self.definition() │
│ 221 │ │ kwargs = dict( │
│ 222 │ │ │ PipelineName=self.name, │
│ 223 │ │ │ RoleArn=role_arn, │
│ │
│ /home/congajamm/miniconda3/envs/aws_env/lib/python3.12/site-packages/sagemaker/mlops/workflow/pi │
│ peline.py:448 in definition │
│ │
│ 445 │ │ │ sagemaker_session=self.sagemaker_session, │
│ 446 │ │ │ steps=self.steps, │
│ 447 │ │ │ pipeline_definition_config=self.pipeline_definition_config, │
│ ❱ 448 │ │ ).build() │
│ 449 │ │ │
│ 450 │ │ request_dict = { │
│ 451 │ │ │ "Version": self._version, │
│ │
│ /home/congajamm/miniconda3/envs/aws_env/lib/python3.12/site-packages/sagemaker/mlops/workflow/_s │
│ teps_compiler.py:402 in build │
│ │
│ 399 │ │ if self._build_count > 1: │
│ 400 │ │ │ raise RuntimeError("Cannot build a pipeline more than once with the same com │
│ 401 │ │ │
│ ❱ 402 │ │ return self._initialize_queue_and_build(self._input_steps) │
│ 403 │
│ │
│ /home/congajamm/miniconda3/envs/aws_env/lib/python3.12/site-packages/sagemaker/mlops/workflow/_s │
│ teps_compiler.py:386 in _initialize_queue_and_build │
│ │
│ 383 │ │ │ if isinstance(step, ConditionStep): │
│ 384 │ │ │ │ compiled_steps.append(self._build_condition_step(step)) │
│ 385 │ │ │ else: │
│ ❱ 386 │ │ │ │ compiled_steps.append(self._build_step(step)) │
│ 387 │ │ │
│ 388 │ │ self._set_serialize_output_to_json_flag(compiled_steps) │
│ 389 │ │ return compiled_steps │
│ │
│ /home/congajamm/miniconda3/envs/aws_env/lib/python3.12/site-packages/sagemaker/mlops/workflow/_s │
│ teps_compiler.py:319 in _build_step │
│ │
│ 316 │ │ │ pipeline_name=self.pipeline_name, │
│ 317 │ │ │ step_name=step.name, │
│ 318 │ │ │ sagemaker_session=self.sagemaker_session, │
│ ❱ 319 │ │ │ code_hash=get_code_hash(step), │
│ 320 │ │ │ config_hash=get_config_hash(step), │
│ 321 │ │ │ pipeline_definition_config=self.pipeline_definition_config, │
│ 322 │ │ │ upload_runtime_scripts=self.upload_runtime_scripts, │
│ │
│ /home/congajamm/miniconda3/envs/aws_env/lib/python3.12/site-packages/sagemaker/core/workflow/uti │
│ lities.py:177 in get_code_hash │
│ │
│ 174 │ │ │ source_dir = source_code.source_dir │
│ 175 │ │ │ requirements = source_code.requirements │
│ 176 │ │ │ entry_point = source_code.entry_script │
│ ❱ 177 │ │ │ return get_training_code_hash(entry_point, source_dir, requirements) │
│ 178 │ return None │
│ 179 │
│ 180 │
│ │
│ /home/congajamm/miniconda3/envs/aws_env/lib/python3.12/site-packages/sagemaker/core/workflow/uti │
│ lities.py:251 in get_training_code_hash │
│ │
│ 248 │ │ if source_dir: │
│ 249 │ │ │ source_dir_url = urlparse(source_dir) │
│ 250 │ │ │ if source_dir_url.scheme == "" or source_dir_url.scheme == "file": │
│ ❱ 251 │ │ │ │ return hash_files_or_dirs([source_dir] + dependencies) │
│ 252 │ │ elif entry_point: │
│ 253 │ │ │ entry_point_url = urlparse(entry_point) │
│ 254 │ │ │ if entry_point_url.scheme == "" or entry_point_url.scheme == "file": │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: can only concatenate list (not "NoneType") to list
(aws_env) congajamm@CONNB441:/mnt/c/Users/brianperez/Desktop/DEV/aws/Weather-Pred-AWS$

System information
A description of your system. Please provide:

  • SageMaker Python SDK version:3.3.1
  • Framework name (eg. PyTorch) or algorithm (eg. KMeans):PyTorch
  • Framework version: Torchserve version: 0.12.0
  • Python version:3.12.12
  • CPU or GPU:CPU
  • Custom Docker image (Y/N):N

Additional context
Running on Windows 11 with Debian WSL 2 Linux.
(aws_env) congajamm@CONNB441:/mnt/c/Users/brianperez/Desktop/DEV/aws/Weather-Pred-AWS$ python -m src.weather_pred.orchestrate.ml-pipeline-cloud

Activity

  1. ravees commented on Feb 9, 2026

    @ravees

    Facing the same issue when passing source code with requirements.txt. The same error is seen even when the requirements is not explicitly set with requirements.txt file is present in the source location.
    source_code=SourceCode(
    source_dir='pipeline_code/scripts',
    entry_script='train.py',
    requirements="requirements.txt"
    )

  2. dgallitelli commented on Feb 21, 2026

    @dgallitelli

    Workaround

    The bug is in sagemaker/core/workflow/utilities.py at line 177 where get_code_hash() passes requirements (a str or None - the path to requirements.txt) to get_training_code_hash() which expects dependencies: List[str].

    Monkey-patch fix

    Add this at the top of your pipeline script before importing other sagemaker modules:

    import sagemaker.core.workflow.utilities as _workflow_utils
    
    _original_get_training_code_hash = _workflow_utils.get_training_code_hash
    
    def _patched_get_training_code_hash(entry_point, source_dir, dependencies):
        """Patched version that handles string/None dependencies."""
        if dependencies is None:
            dependencies = []
        elif isinstance(dependencies, str):
            dependencies = [dependencies] if dependencies else []
        return _original_get_training_code_hash(entry_point, source_dir, dependencies)
    
    _workflow_utils.get_training_code_hash = _patched_get_training_code_hash

    Suggested fix in SDK

    In sagemaker/core/workflow/utilities.py, line 177 should convert requirements to a list before passing:

    # Current (broken):
    return get_training_code_hash(entry_point, source_dir, requirements)
    
    # Fixed:
    deps = [requirements] if requirements else []
    return get_training_code_hash(entry_point, source_dir, deps)

    Tested with SDK v3.4.1 on Python 3.14.

  3. doublecompile commented on Mar 12, 2026

    @doublecompile

    This PR seems to address this bug: #5618

  4. doublecompile commented on Mar 20, 2026

    @doublecompile

    Now that the PR is merged and 3.6.0 was released, this bug no longer appears. I think we can consider this issue resolved.

  5. 1 remaining item

  6. mohamedzeidan2021 commented on Sep 1, 2026

    @mohamedzeidan2021
    Collaborator

    Fixed in #5618, released in v3.6.0. The failure was in get_training_code_hash, where the requirements value (a path string or None) was concatenated to a list; it's now guarded so a SourceCode with requirements unset (or set to a requirements.txt path) hashes correctly through pipeline.upsert(). This matches the confirmation above that the error no longer appears on 3.6.0.

    Please upgrade to 3.6.0 or later. Closing as resolved — thanks for the detailed repro and to @dgallitelli for the interim workaround.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions