Allow dependency numpy to be >= 2.0.0

Describe the feature you'd like
In June 2024, numpy 2.0.0 was released. sagemaker-python-sdk depends on numpy>=1.9.0,<2.0. This creates a dependency hell for me, as I have dependencies in my python package that depend on numpy >= 2.0.0.

You could either enforce numpy >= 2.0.0 and make new releases of the package incompatible with numpy < 2.0.0 or keep supporting the currently supported numpy versions, but also add those >= 2.0.0. I.e. depending on whether or not there are breaking changes with numpy >= 2.0.0 in your code base, establish different code paths depending on the installed version of numpy.

How would this feature be used? Please describe.

Ensuring I can resolve dependencies in Python packages that have both this SDK as well as other dependencies with a requirement for numpy >= 2.0.

Describe alternatives you've considered

I don't think there is an alternative, as in the long run, this problem will get worse as more and more other packages depend on numpy >= 2.0.0. I am surprised no one has opened an issue until now.

Additional context
I am also opening a support case with AWS Premium Support.

Is there anything holding up removal of the pin? From a quick scan, I would think it is basically a few tiny replacements for things like np.NaN which ruff check path/to/code/ --select NPY201 --fix will just do.
There are a few uses np.int which might be fishy if (and only if!) this code ever runs on windows (otherwise, it may be fishy, but there is no change in NumPy 2).

@ellisonbg

@ellisonbg do you know who we should talk to about NumPy 2 support here?

Any status on this when we could expect this support to be added?

Any updates here? We're in the same situation.

@nargokul

Hi @nargokul, I see this pr #4955 is merged. Would this be part of the next release?

@nargokul

Seems like pr #4963 reverted the change for numpy. Was it intentional? @nargokul

same

@knikure

@knikure and @zhaoqizqwang, you reviewed the PR. numpy compatibility is a big issue for many people here. Any idea?

@nargokul

@nargokul Seems like numpy update PR was actively closed, with a currently open PR existing. Any blocker for the update?

There are compatibility challenges in upgrading Amazon SageMaker Python SDK with NumPy 2.0 since pandas library currently lacks support for NumPy 2.0. We will incorporate the latest NumPy version once pandas provides the necessary support. Your patience is appreciated as we await this external dependency.

The update from Pandas has a caveat mentioned below in https://pandas.pydata.org/pandas-docs/stable/whatsnew/v2.2.2.html

One major caveat is that arrays created with numpy 2.0’s new StringDtype will convert to object dtyped arrays upon Series/DataFrame creation. Full support for numpy 2.0’s StringDtype is expected to land in pandas 3.0.

This causes regression in SageMaker PySDK functionalities and hence we will need to wait for pandas 3.0 to make this update.

Please reach out to our support team if you have any further inquiries.

@mufaddal-rohawala

since pandas library currently lacks support for NumPy 2.0.

Pandas has reasonable support in newer versions since the first release of NumPy (version 2.2.2). There might be pandas versions which miss a numpy <2 pin making it possible to accidentally install incompatible versions.

But, unless you have another dependency that still enforces pandas<2.2.2 that should not stop you from updating.

The update from Pandas has a caveat mentioned below [StringDType not supported]

Sorry, but there seems to be a misunderstanding here: The new StringDType in NumPy should not be relevant. It is new and simply doesn't affect existing code.
(If this somehow is a problem, it should be a 3rd party, i.e. one that immediately started using StringDType, dependency problem that I don't think you should worry about.)

Please reach out to our support team if you have any further inquiries.

@mufaddal-rohawala it would be helpful if there was an issue to discuss the exact problem. If this seems like a NumPy/pandas related difficulty NumPy and pandas maintainers are certainly here to help. Please do reach out to me (or pandas maintainers, e.g. with an issue there).

+1

@rsareddy0329

Thank you for your continued patience. The team remains actively engaged on this issue. We are working toward a resolution soon and will continue to share updates here as the work advances.

Please continue to share any specific use cases or concerns in this thread.

Are we able to get another public update next week please?

I think the majority are mostly concerned with the length of time this has taken, the dependency conflicts this creates in applications, and that we're now using EOL numpy with some outstanding CVEs.

Can you please explain to us why this is taking so long? Apparently it must be complicated, but I don't see where that complexity comes from exactly. There are people with significant contributions to numpy and pandas in this thread who offered their help in resolving this issue.

@rsareddy0329 @nargokul are you able to share where the complexity comes from? From the error patterns, this appears to be downstream integration challenges rather than core sagemaker-python-sdk compatibility issues - is that accurate? Understanding the technical blockers would help the community better appreciate the timeline.

Thanks for the continued work on this.

Apparently AWS Support wants to provide a progress update here but since this hasn’t really happened I think l it’s ok to share what I know. The problems are downstream container dependencies that need to be updated as well, and the progress is „predominantly positive“, whatever that means.

Mind clarifying this? #5199

It looks like all integration testing passed, and the above was merged into the main branch - would you mind confirming the timeline / next steps here?

It looks like you changed the numpy constraint from being pinned exactly to version 1.26.4 using ==1.26.4, to now supporting a range from 1.26.4 up to (but not including) 2.3.3 using >=1.26.4,<2.3.3, which includes numpy 2.x versions. Can you confirm you have fixed the dependency issues and all the updates to other packages (scipy, pandas, scikit-learn, tensorflow, etc.) are correct and working as intended?

~~Looks like v2.253.0 adds support for numpy 2.0 cheers 🥳~~

~~I'll test it early next week.~~

@des1-gner

Finally. Thanks everyone, incl. @des1-gner from AWS Support to escalate the case. And as usually, resolved issues are not closed by AWS staff (so I'll do it as an OP). If you are unhappy with how AWS is handling community feedback, maybe upvote my issue on it here in the SDK (#4380) or forward it to your account manager / premium support.

The issue also mentions a repo specifically trying to collect community feedback: https://github.com/aws/amazon-sagemaker-feedback, which also seems abandoned by AWS employees. It's going great with AWS's 'customer obsession' for SageMaker.

I see that v2.253.0 was yanked from PyPI due to "Compatibility issues with numpy 2.0 upgrade". Can anyone elaborate further on that and when we might expect the next version on PyPI?

@des1-gner

We now have a new release with reverted numpy 2.0 support (https://github.com/aws/sagemaker-python-sdk/releases/tag/v2.253.1) based on #5307 and no information anywhere on why it was reverted. @des1-gner can you comment?

@nargokul

Finally. Thanks everyone, incl. @des1-gner from AWS Support to escalate the case. And as usually, resolved issues are not closed by AWS staff (so I'll do it as an OP). If you are unhappy with how AWS is handling community feedback, maybe upvote my issue on it here in the SDK (#4380) or forward it to your account manager / premium support.

The issue also mentions a repo specifically trying to collect community feedback: https://github.com/aws/amazon-sagemaker-feedback, which also seems abandoned by AWS employees. It's going great with AWS's 'customer obsession' for SageMaker.

I am not sure of the state of https://github.com/aws/amazon-sagemaker-feedback I'll see if I can get anyone to comment on it.

We now have a new release with reverted numpy 2.0 support (https://github.com/aws/sagemaker-python-sdk/releases/tag/v2.253.1) based on #5307 and no information anywhere on why it was reverted. @des1-gner can you comment?

Unfortunately, I do not have much further insight other than the generic notice here: https://pypi.org/project/sagemaker/2.253.0/ about it having compatibility issues.

@nargokul @rsareddy0329 could you comment on whether this is coming in the next release, and why this had to be pulled?

It looks like the work is again in progress in #5311

This really should be fixed already, Numpy 2.0 is OLD at this point! Even PyTorch Captum, which relies heavily on NumPy, much more than SageMaker SDK, made the migration recently (meta-pytorch/captum#1581). I would much rather see some legacy pipelines broken (again, about TWO YEARS old at this point!) and required to properly modernize, and require Numpy 2+ for new versions. This is holding our entire tech stack back for all ML projects, since they rely on SageMaker SDK.

Looks like v2.254.0 adds support for numpy 2.0 cheers 🥳

Looks like it was also released here: https://pypi.org/project/sagemaker/#history

Navigation Menu

Repository navigation

Allow dependency numpy to be >= 2.0.0 #4882

15 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Allow dependency numpy to be >= 2.0.0 #4882

Description

Activity

seberg commented on Oct 8, 2024

jakirkham commented on Oct 25, 2024

victoriarouton commented on Nov 4, 2024

radoshi commented on Dec 10, 2024

aaravind100 commented on Dec 11, 2024

wickeat commented on Dec 16, 2024

amorisot commented on Dec 19, 2024

lorenzwalthert commented on Dec 30, 2024

wickeat commented on Jan 10, 2025

mufaddal-rohawala commented on Jan 23, 2025

seberg commented on Jan 23, 2025

Yc-Chen commented on Mar 7, 2025

15 remaining items

des1-gner commented on Oct 4, 2025

lorenzwalthert commented on Oct 4, 2025

des1-gner commented on Oct 10, 2025

aaravind100 commented on Oct 10, 2025

lorenzwalthert commented on Oct 11, 2025

kristoferm94 commented on Oct 13, 2025

lorenzwalthert commented on Oct 14, 2025

des1-gner commented on Oct 14, 2025

smejkyz commented on Oct 17, 2025

j-adamczyk commented on Oct 22, 2025

smejkyz commented on Oct 29, 2025

des1-gner commented on Oct 29, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions