Skip to content

fix: don't report a 405 from a non-Connect endpoint as an SDK crash (FLYTE-SDK-81) - #1525

Merged
cosmicBboy merged 1 commit into
mainfrom
fix/sentry-81-method-not-allowed
Sep 3, 2026
Merged

fix: don't report a 405 from a non-Connect endpoint as an SDK crash (FLYTE-SDK-81)#1525
cosmicBboy merged 1 commit into
mainfrom
fix/sentry-81-method-not-allowed

Conversation

@EngHabu

@EngHabu EngHabu commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Problem

FLYTE-SDK-81 — 4 events, all from one host in a ~17 minute window on 2026-09-02, release 2.6.10:

RuntimeSystemError: Upload failed for /var/folders/.../fastc34306b3….tar.gz
  (org='flyte', project='flytesnacks', domain='development'): Method Not Allowed
  └── ConnectError: Method Not Allowed

The endpoint answered a Connect RPC with HTTP 405. connectrpc maps only a handful of statuses (401/403/404/429/502/503/504) onto Connect codes; everything else falls through to ConnectWireError.from_http_status, which yields Code.UNKNOWN with the bare stdlib reason phrase as the whole message — the same shape _is_non_connect_endpoint_response already recognises for the 2xx class.

The event pins the raise to _protocol_connect.py:246, the branch below if response_content_type == CONNECT_UNARY_CONTENT_TYPE_JSON: return — so the 405 response was not even in Connect's own JSON error format.

Why a 405 is unambiguously not an SDK bug

Every Connect RPC the SDK issues is a POST. connectrpc's execute_unary sends GET only when the caller opts in with use_get=True; execute_client_stream / execute_server_stream / execute_bidi_stream hardcode http_method="POST". There is no use_get anywhere in this repo.

A Connect handler always accepts POST on its own procedure path. So a 405 cannot be the backend rejecting our choice of method — it proves the POST was answered by something that routes no Connect procedures at all: a proxy, an ingress, a static-file host, a load balancer default backend, or simply a misconfigured endpoint. That is endpoint configuration the SDK cannot recover from, exactly like the 2xx case (FLYTE-SDK-77/78) and the text/* case (FLYTE-SDK-7A/6P).

Because the whole filter rests on that premise, test_sdk_never_sends_a_connect_get pins it: it walks the AST of every module under src/flyte/ and fails if any call site ever passes use_get. If that changes, a 405 could become a legitimate backend rejection and this filter would start hiding a real bug — the test says so in its docstring.

Fix

Add the 405 phrase to _is_non_connect_endpoint_response, beside the existing 2xx and text/* discriminators.

Deliberately only 405. I ran the discriminator across the whole unresolved corpus (75 non-Go issues) before writing it. Four issues carry the bare-HTTP-phrase shape:

Issue Events Status Verdict
FLYTE-SDK-81 4 405 Method Not Allowed filtered by this PR
FLYTE-SDK-7C 2 400 Bad Request untouched — a 400 could be a genuinely malformed request from us
FLYTE-SDK-64 276 500 Internal Server Error untouched — real backend signal
FLYTE-SDK-6S 1 500 Internal Server Error untouched

Filtering the status class rather than the individual status would have silenced FLYTE-SDK-64, the largest genuine backend signal in the project. 405 is the only status in the corpus that is provably a routing answer.

Tests

5 new tests in tests/flyte/test_sentry.py, using the existing _wire_error_for_status helper so they exercise the real ConnectWireError.from_http_status path rather than a hand-built error:

  • a bare 405 is not reported
  • the real FLYTE-SDK-81 shape — the 405 as the __cause__ of a RuntimeSystemError: Upload failed for … — is not reported
  • 400 / 406 / 409 / 410 / 500 / 501 all still report (parametrized; covers 7C and 64)
  • a Method Not Allowed carrying details still reports, since a real Connect JSON error body always yields details and from_http_status never does
  • the use_get AST guard described above

The two behavioural tests were verified failing on clean main with the exact production signature (Expected 'init' to not have been called. Called 1 times.); the three guards pass on both sides by design. Full file: 79 passed. ruff format / ruff check / mypy / ty / check-docstrings all clean.

fixes FLYTE-SDK-81

…FLYTE-SDK-81)

Every Connect RPC the SDK issues is a POST -- connectrpc sends GET only when the
caller passes use_get=True to execute_unary, which the SDK never does, and the
stream variants are hardcoded to POST. A Connect handler always accepts POST on
its own procedure path, so a 405 Method Not Allowed proves the request was
answered by something that routes no Connect procedures at all: a proxy, ingress
or misconfigured endpoint. That is endpoint configuration, not an SDK bug.

Adds the 405 phrase to _is_non_connect_endpoint_response alongside the existing
2xx-not-200 and text/* discriminators. Deliberately only 405: 400 (FLYTE-SDK-7C)
and 500 (FLYTE-SDK-64) produce the identical UNKNOWN/bare-phrase shape and remain
real signal.

fixes FLYTE-SDK-81

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@EngHabu EngHabu added the sentry-fix Fix for an issue surfaced by Sentry label Sep 3, 2026
@cosmicBboy
cosmicBboy merged commit 8db1a58 into main Sep 3, 2026
64 checks passed
@cosmicBboy
cosmicBboy deleted the fix/sentry-81-method-not-allowed branch September 3, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sentry-fix Fix for an issue surfaced by Sentry

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants