Add Python 3.13 support and prepare for 3.14

Python 3.13:
- pyproject.toml: requires-python ">=3.10,<3.14", add the 3.13 classifier;
  regenerate uv.lock accordingly.
- PythonVersion enum and default image prefixes: add 3.13 (and 3.14) in
  flytekit core, the sqlalchemy plugin and the openai batch plugin.
- CI (pythonbuild.yml): run 3.13 on pull requests and in the nightly
  matrix, enable the extras (tensorflow/pytorch) tests on 3.13, exclude
  pandas<2 and plugins whose dependencies do not support 3.13 yet.
- dev-requirements.in: tensorflow>=2.21 and torch>=2.6 on 3.13; lift the
  protobuf<5 dev pin on 3.13 only (tensorflow 2.21 needs protobuf>=6, and
  the issue behind the pin, flyteorg/flyte#5448, was fixed upstream).
- Publish py3.13 images in pythonpublish.yml and build_image.yml.

Fixes that also unblock Python 3.14 (not enabled yet, see PR):
- type_engine: on 3.14 typing.Union *is* types.UnionType, so registering
  UnionTransformer for both raised "already registered" at import time.
- test_unions: accept the "A | B" rendering of unions in the error message.

Unrelated to the interpreter version but required for the extras tests to
pass with a current torch (and already failing in the nightly 3.11 run):
- pytorch extras: torch>=2.6 defaults torch.load(weights_only=True), which
  cannot load the whole nn.Module / checkpoint objects flytekit stores.
  Pass weights_only=False where the keyword exists.

Closes flyteorg/flyte#6993

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q3fT13jays4BfX7WiY9XY6
Signed-off-by: schnalf <erik.bochinski@gmail.com>

Author: bochinski

.github/workflows/build_image.yml

@@ -28,7 +28,7 @@ jobs:
     needs: get-version
     strategy:
       matrix:
-        python-version: ["3.10", "3.11", "3.12"]
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
     steps:
       - uses: actions/checkout@v4
         with:

.github/workflows/pythonbuild.yml

@@ -27,9 +27,9 @@ jobs:
       - name: Set Python versions for run
         run: |
           if [[ ${{ github.event_name }} == "schedule" ]]; then
-            echo "python_versions=[\"3.10\",\"3.11\",\"3.12\"]" >> $GITHUB_ENV
+            echo "python_versions=[\"3.10\",\"3.11\",\"3.12\",\"3.13\"]" >> $GITHUB_ENV
           else
-            echo "python_versions=[\"3.12\"]" >> $GITHUB_ENV
+            echo "python_versions=[\"3.12\",\"3.13\"]" >> $GITHUB_ENV
           fi
 
   build:
@@ -139,6 +139,9 @@ jobs:
             pandas: "pandas>=2.0.0"
           - pandas: "pandas<2.0.0"
             python-version: "3.12"
+          # pandas 1.x has no wheels for python 3.13
+          - pandas: "pandas<2.0.0"
+            python-version: "3.13"
 
     steps:
       - uses: actions/checkout@v4
@@ -389,6 +392,27 @@ jobs:
           # Segmentation fault on python 3.12: https://github.com/flyteorg/flyte/issues/5020
           - python-version: 3.12
             plugin-names: "flytekit-kf-pytorch"
+          # Python 3.13: the exclusions below mirror the python 3.12 ones (same dependency limitations) and should
+          # be revisited together with them.
+          - python-version: 3.13
+            plugin-names: "flytekit-airflow"
+          - python-version: 3.13
+            plugin-names: "flytekit-kf-pytorch"
+          - python-version: 3.13
+            plugin-names: "flytekit-modin"
+          - python-version: 3.13
+            plugin-names: "flytekit-onnx-pytorch"
+          - python-version: 3.13
+            plugin-names: "flytekit-ray"
+          - python-version: 3.13
+            plugin-names: "flytekit-vaex"
+          # The tensorflow releases that provide python 3.13 wheels require protobuf>=6, which conflicts with the
+          # "protobuf<5" pin used below for the plugin tests.
+          - python-version: 3.13
+            plugin-names: "flytekit-mlflow"
+          # flytekitplugins-duckdb declares python_requires<3.13 (duckdb<=1.0.0 pin)
+          - python-version: 3.13
+            plugin-names: "flytekit-duckdb"
     steps:
       - uses: actions/checkout@v4
       - name: "Clear action cache"

.github/workflows/pythonpublish.yml

@@ -82,6 +82,7 @@ jobs:
           - "3.10"
           - "3.11"
           - "3.12"
+          - "3.13"
     steps:
       - uses: actions/checkout@v4
         with:

dev-requirements.in

@@ -23,12 +23,17 @@ pytest-icdiff
 
 # Tensorflow is not available for python 3.12 yet: https://github.com/tensorflow/tensorflow/issues/62003
 tensorflow<=2.15.1; python_version<'3.12'
+# tensorflow>=2.21 ships python 3.13 wheels (linux x86_64/aarch64, macos arm64, windows) but requires protobuf>=6,
+# see the protobuf constraint below.
+tensorflow>=2.21.0; python_version>='3.13'
 # Newer versions of torch bring in nvidia dependencies that are not present in windows, so
 # we put this constraint while we do not have per-environment requirements files
 torch<=1.12.1; python_version<'3.11'
 # pytorch 2 supports python 3.11
 # pytorch 2 does not support 3.12 yet: https://github.com/pytorch/pytorch/issues/110436
 torch; python_version<'3.12'
+# pytorch>=2.6 ships python 3.13 wheels
+torch>=2.6.0; python_version>='3.13'
 pydantic
 
 # TODO: Currently, the python-magic library causes build errors on Windows due to its dependency on DLLs for libmagic.
@@ -40,7 +45,9 @@ python-magic; (platform_system=='Darwin' or platform_system=='Linux')
 # Google released a new major version of the protobuf library and once that started being used in the ecosystem at large,
 # including `googleapis-common-protos` we started seeing errors in CI, so let's constrain that for now.
 # The issue to support protobuf 5 is being tracked in https://github.com/flyteorg/flyte/issues/5448.
-protobuf<5
+# The root cause was fixed in googleapis-common-protos>=1.63.2. On python 3.13 the pin is lifted, because the
+# tensorflow releases that support 3.13 require protobuf>=6 (flytekit itself does not cap protobuf).
+protobuf<5; python_version<'3.13'
 types-protobuf<5
 
 types-croniter

flytekit/configuration/default_images.py

@@ -13,6 +13,8 @@ class PythonVersion(enum.Enum):
     PYTHON_3_10 = (3, 10)
     PYTHON_3_11 = (3, 11)
     PYTHON_3_12 = (3, 12)
+    PYTHON_3_13 = (3, 13)
+    PYTHON_3_14 = (3, 14)
 
 
 class DefaultImages(object):
@@ -26,6 +28,8 @@ class DefaultImages(object):
         PythonVersion.PYTHON_3_10: "cr.flyte.org/flyteorg/flytekit:py3.10-",
         PythonVersion.PYTHON_3_11: "cr.flyte.org/flyteorg/flytekit:py3.11-",
         PythonVersion.PYTHON_3_12: "cr.flyte.org/flyteorg/flytekit:py3.12-",
+        PythonVersion.PYTHON_3_13: "cr.flyte.org/flyteorg/flytekit:py3.13-",
+        PythonVersion.PYTHON_3_14: "cr.flyte.org/flyteorg/flytekit:py3.14-",
     }
 
     @classmethod

flytekit/core/type_engine.py

@@ -2810,7 +2810,9 @@ def _register_default_type_transformers():
     if sys.version_info >= (3, 10):
         from types import UnionType
 
-        TypeEngine.register(UnionTransformer(), [UnionType])
+        # Since Python 3.14, ``typing.Union`` is an alias of ``types.UnionType`` (both spellings produce the
+        # same object), so registering it a second time as an additional type would raise.
+        TypeEngine.register(UnionTransformer(), [UnionType] if UnionType is not typing.Union else None)
     else:
         TypeEngine.register(UnionTransformer())
     TypeEngine.register(DictTransformer())

flytekit/extras/pytorch/checkpoint.py

@@ -9,6 +9,7 @@
 
 from flytekit.core.context_manager import FlyteContext
 from flytekit.core.type_engine import TypeEngine, TypeTransformer, TypeTransformerFailedError
+from flytekit.extras.pytorch.native import _torch_load
 from flytekit.models.core import types as _core_types
 from flytekit.models.literals import Blob, BlobMetadata, Literal, Scalar
 from flytekit.models.types import LiteralType
@@ -119,7 +120,7 @@ def to_python_value(
             map_location = torch.device("cpu")
 
         # load checkpoint from a file
-        return typing.cast(PyTorchCheckpoint, torch.load(local_path, map_location=map_location))
+        return typing.cast(PyTorchCheckpoint, _torch_load(local_path, map_location=map_location))
 
     def guess_python_type(self, literal_type: LiteralType) -> Type[PyTorchCheckpoint]:
         if (

flytekit/extras/pytorch/native.py

@@ -1,4 +1,6 @@
+import inspect
 import pathlib
+import typing
 from typing import Type, TypeVar
 
 import torch
@@ -12,6 +14,16 @@
 T = TypeVar("T")
 
 
+def _torch_load(path: str, map_location: typing.Any) -> typing.Any:
+    """
+    Load a full object with ``torch.load``. flytekit serializes whole tensors/modules/checkpoints, so ``weights_only``
+    (the default since torch 2.6) must be disabled. The keyword does not exist before torch 1.13, hence the check.
+    """
+    if "weights_only" in inspect.signature(torch.load).parameters:
+        return torch.load(path, map_location=map_location, weights_only=False)
+    return torch.load(path, map_location=map_location)
+
+
 class PyTorchTypeTransformer(TypeTransformer[T]):
     def get_literal_type(self, t: Type[T]) -> LiteralType:
         return LiteralType(
@@ -63,7 +75,7 @@ def to_python_value(self, ctx: FlyteContext, lv: Literal, expected_python_type:
             map_location = torch.device("cpu")
 
         # load pytorch tensor/module from a file
-        return torch.load(local_path, map_location=map_location)
+        return _torch_load(local_path, map_location=map_location)
 
 
 class PyTorchTensorTransformer(PyTorchTypeTransformer[torch.Tensor]):

plugins/flytekit-openai/flytekitplugins/openai/batch/task.py

@@ -64,6 +64,8 @@ class OpenAIFileDefaultImages(DefaultImages):
         PythonVersion.PYTHON_3_10: "cr.flyte.org/flyteorg/flytekit:py3.10-openai-batch-",
         PythonVersion.PYTHON_3_11: "cr.flyte.org/flyteorg/flytekit:py3.11-openai-batch-",
         PythonVersion.PYTHON_3_12: "cr.flyte.org/flyteorg/flytekit:py3.12-openai-batch-",
+        PythonVersion.PYTHON_3_13: "cr.flyte.org/flyteorg/flytekit:py3.13-openai-batch-",
+        PythonVersion.PYTHON_3_14: "cr.flyte.org/flyteorg/flytekit:py3.14-openai-batch-",
     }
 
 

plugins/flytekit-sqlalchemy/flytekitplugins/sqlalchemy/task.py

@@ -26,6 +26,8 @@ class SQLAlchemyDefaultImages(DefaultImages):
         PythonVersion.PYTHON_3_10: "cr.flyte.org/flyteorg/flytekit:py3.10-sqlalchemy-",
         PythonVersion.PYTHON_3_11: "cr.flyte.org/flyteorg/flytekit:py3.11-sqlalchemy-",
         PythonVersion.PYTHON_3_12: "cr.flyte.org/flyteorg/flytekit:py3.12-sqlalchemy-",
+        PythonVersion.PYTHON_3_13: "cr.flyte.org/flyteorg/flytekit:py3.13-sqlalchemy-",
+        PythonVersion.PYTHON_3_14: "cr.flyte.org/flyteorg/flytekit:py3.14-sqlalchemy-",
     }