fix: don't report a 405 from a non-Connect endpoint as an SDK crash (FLYTE-SDK-81) - #1525
Merged
Merged
Conversation
…FLYTE-SDK-81) Every Connect RPC the SDK issues is a POST -- connectrpc sends GET only when the caller passes use_get=True to execute_unary, which the SDK never does, and the stream variants are hardcoded to POST. A Connect handler always accepts POST on its own procedure path, so a 405 Method Not Allowed proves the request was answered by something that routes no Connect procedures at all: a proxy, ingress or misconfigured endpoint. That is endpoint configuration, not an SDK bug. Adds the 405 phrase to _is_non_connect_endpoint_response alongside the existing 2xx-not-200 and text/* discriminators. Deliberately only 405: 400 (FLYTE-SDK-7C) and 500 (FLYTE-SDK-64) produce the identical UNKNOWN/bare-phrase shape and remain real signal. fixes FLYTE-SDK-81 Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
cosmicBboy
approved these changes
Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
FLYTE-SDK-81 — 4 events, all from one host in a ~17 minute window on 2026-09-02, release 2.6.10:
The endpoint answered a Connect RPC with HTTP 405. connectrpc maps only a handful of statuses (401/403/404/429/502/503/504) onto Connect codes; everything else falls through to
ConnectWireError.from_http_status, which yieldsCode.UNKNOWNwith the bare stdlib reason phrase as the whole message — the same shape_is_non_connect_endpoint_responsealready recognises for the 2xx class.The event pins the raise to
_protocol_connect.py:246, the branch belowif response_content_type == CONNECT_UNARY_CONTENT_TYPE_JSON: return— so the 405 response was not even in Connect's own JSON error format.Why a 405 is unambiguously not an SDK bug
Every Connect RPC the SDK issues is a POST. connectrpc's
execute_unarysends GET only when the caller opts in withuse_get=True;execute_client_stream/execute_server_stream/execute_bidi_streamhardcodehttp_method="POST". There is nouse_getanywhere in this repo.A Connect handler always accepts POST on its own procedure path. So a 405 cannot be the backend rejecting our choice of method — it proves the POST was answered by something that routes no Connect procedures at all: a proxy, an ingress, a static-file host, a load balancer default backend, or simply a misconfigured endpoint. That is endpoint configuration the SDK cannot recover from, exactly like the 2xx case (FLYTE-SDK-77/78) and the
text/*case (FLYTE-SDK-7A/6P).Because the whole filter rests on that premise,
test_sdk_never_sends_a_connect_getpins it: it walks the AST of every module undersrc/flyte/and fails if any call site ever passesuse_get. If that changes, a 405 could become a legitimate backend rejection and this filter would start hiding a real bug — the test says so in its docstring.Fix
Add the 405 phrase to
_is_non_connect_endpoint_response, beside the existing 2xx andtext/*discriminators.Deliberately only 405. I ran the discriminator across the whole unresolved corpus (75 non-Go issues) before writing it. Four issues carry the bare-HTTP-phrase shape:
Filtering the status class rather than the individual status would have silenced FLYTE-SDK-64, the largest genuine backend signal in the project. 405 is the only status in the corpus that is provably a routing answer.
Tests
5 new tests in
tests/flyte/test_sentry.py, using the existing_wire_error_for_statushelper so they exercise the realConnectWireError.from_http_statuspath rather than a hand-built error:__cause__of aRuntimeSystemError: Upload failed for …— is not reportedMethod Not Allowedcarryingdetailsstill reports, since a real Connect JSON error body always yields details andfrom_http_statusnever doesuse_getAST guard described aboveThe two behavioural tests were verified failing on clean
mainwith the exact production signature (Expected 'init' to not have been called. Called 1 times.); the three guards pass on both sides by design. Full file: 79 passed.ruff format/ruff check/mypy/ty/check-docstringsall clean.fixes FLYTE-SDK-81