Skip to content

Commit 9cb3058

Browse files
authored
fix: survive a broken pyOpenSSL install (FLYTE-SDK-7T) (#1488)
Sentry: [FLYTE-SDK-7T](https://unionai.sentry.io/issues/7693268977/) — `AttributeError: module 'lib' has no attribute 'GEN_EMAIL'`, 2 events, release 2.5.6. ## What happens `flyte deploy` dies during client initialisation, before it does any work: ``` flyte/cli/_deploy.py invoke flyte/cli/_common.py initialize_config -> init flyte/_initialize.py init -> _initialize_client flyte/remote/_client/controlplane.py <module> flyte/remote/_client/auth/_session.py <module> <- from OpenSSL import SSL, crypto OpenSSL/SSL.py <module> OpenSSL/crypto.py X509Extension AttributeError: module 'lib' has no attribute 'GEN_EMAIL' ``` `_session.py` imported pyOpenSSL at module scope, and `controlplane.py` imports `_session`, so importing pyOpenSSL was on the path of **every command that builds a client**. That import is unusually fragile. pyOpenSSL binds names out of the `cryptography` C bindings *at class-body time* — `X509Extension` reads `_lib.GEN_EMAIL` while the module is still executing — so an incompatible pyOpenSSL/cryptography pair fails during **import**, with an `AttributeError`, not the `ImportError` an optional-dependency guard would normally catch. The reporting host is an Ubuntu box running out of `/usr/local/lib/python3.10/dist-packages`, the classic shape for this: a distro-packaged pyOpenSSL sitting alongside a newer pip-installed `cryptography`. Two things went wrong: the user's `flyte deploy` failed with a message naming a module they never imported and giving no hint what to reconcile, and the SDK reported its own crash to Sentry, where a broken third-party install in someone's environment is not an SDK bug. ## The fix pyOpenSSL is only *used* by `_bootstrap_ssl_from_server`, i.e. only when `insecure_skip_verify` is set. Nothing else in the module touches it. So the import failure is held at module scope rather than propagated, and converted at the one call site that needs the library: ```python _PYOPENSSL_IMPORT_ERROR: BaseException | None = None try: from OpenSSL import SSL, crypto except (ImportError, AttributeError) as _e: SSL = None crypto = None _PYOPENSSL_IMPORT_ERROR = _e ``` `_bootstrap_ssl_from_server` then raises an `InitializationError` naming both packages and the command that reconciles them, chained to the original `AttributeError`. `InitializationError` is already on `_is_user_error`'s allow-list, so this class of report stops reaching Sentry — same treatment `EndpointUnreachable` got a few lines below in #1387. `AttributeError` is deliberately in the `except` tuple alongside `ImportError`; catching only `ImportError` would not have caught this crash at all. Note the module-level `SSL` / `crypto` names are kept rather than moved into a function-local import, so the eight existing tests that `patch(f"{_SESSION_MOD}.SSL")` keep working untouched. ## Verification The three new tests execute the real module body into a fresh module object with `import OpenSSL` raising the production `AttributeError`, so they exercise the actual module-level guard without reloading — and therefore without disturbing the live `_session` other modules imported from. On clean `origin/main` all three fail, reproducing the Sentry signature exactly: ``` src/flyte/remote/_client/auth/_session.py:12: in <module> from OpenSSL import SSL, crypto E AttributeError: module 'lib' has no attribute 'GEN_EMAIL' ``` On this branch all three pass. They cover: the module imports despite the broken install; the cold path converts it to a user-kind `InitializationError` with the cause chained and both package names in the message, raised *before* any socket connect is attempted; and `_sentry.capture_exception` drops it. `tests/flyte/remote` + `tests/flyte/test_sentry.py`: 657 passed. Full `tests/flyte`: 3631 passed, 21 failed — the same 21 that fail on clean `origin/main` in this environment (verified by running both and diffing; no test fails only on this branch). ruff, mypy and `check-docstrings` clean. ## Left alone deliberately I enumerated every third-party package that importing `controlplane` pulls in, to see whether 7T had siblings. pyOpenSSL is the only one that is imported but not needed by the failing command — everything else on that list (`pyqwest`, `obstore`, `pydantic_core`, `cryptography` itself) is genuinely used, so guarding its import would only move the failure rather than remove it. Worth flagging separately, though: that same import pulls in **pandas and pyarrow**, which `flyte deploy` has no use for. That is an import-cost issue rather than a crash, with no Sentry evidence behind it, so it is not touched here. fixes FLYTE-SDK-7T --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
1 parent 34f44f9 commit 9cb3058

2 files changed

Lines changed: 114 additions & 1 deletion

File tree

src/flyte/remote/_client/auth/_session.py

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@
99
from urllib.parse import urlparse
1010

1111
import pyqwest
12-
from OpenSSL import SSL, crypto
1312

1413
from flyte._logging import logger
1514
from flyte._utils.org_discovery import hostname_from_url
@@ -21,6 +20,23 @@
2120
get_async_proxy_authenticator,
2221
)
2322

23+
# pyOpenSSL is a hard dependency but it is only *used* on the insecure_skip_verify
24+
# path below, and its import is unusually fragile: it binds names out of the
25+
# `cryptography` C bindings at class-body time, so a mismatched pyOpenSSL/cryptography
26+
# pair fails during import with an `AttributeError` such as
27+
# `module 'lib' has no attribute 'GEN_EMAIL'` rather than a clean `ImportError`.
28+
# Importing it unguarded meant that mismatch took down every command that builds a
29+
# client -- `flyte deploy` died at `_initialize_client` with a bare AttributeError
30+
# naming a module the user never imported (FLYTE-SDK-7T). Hold the failure instead and
31+
# report it from the one function that needs pyOpenSSL.
32+
_PYOPENSSL_IMPORT_ERROR: BaseException | None = None
33+
try:
34+
from OpenSSL import SSL, crypto
35+
except (ImportError, AttributeError) as _e: # pragma: no cover - depends on the install
36+
SSL = None # type: ignore[assignment] # ty: ignore[invalid-assignment]
37+
crypto = None # type: ignore[assignment] # ty: ignore[invalid-assignment]
38+
_PYOPENSSL_IMPORT_ERROR = _e
39+
2440
_USE_PYQWEST_DNS_RESOLVER_ENV = "_FLYTE_USE_PYQWEST_DNS_RESOLVER"
2541
_TRUE_ENV_VALUES = frozenset({"1", "true", "yes", "on"})
2642

@@ -88,6 +104,19 @@ def _bootstrap_ssl_from_server(endpoint: str) -> bytes:
88104
"""
89105
from flyte.errors import InitializationError
90106

107+
if _PYOPENSSL_IMPORT_ERROR is not None:
108+
# A broken pyOpenSSL install is the user's environment, not an SDK bug, and the
109+
# raw AttributeError names neither package involved.
110+
raise InitializationError(
111+
"PyOpenSSLUnavailable",
112+
"user",
113+
f"Could not import pyOpenSSL, which is needed to retrieve the server's TLS "
114+
f"certificate chain when insecure_skip_verify is enabled: "
115+
f"{_PYOPENSSL_IMPORT_ERROR}. This usually means the installed pyOpenSSL and "
116+
f"cryptography versions are incompatible - reinstall them together with "
117+
f"`pip install --upgrade pyOpenSSL cryptography`.",
118+
) from _PYOPENSSL_IMPORT_ERROR
119+
91120
hostname = hostname_from_url(endpoint)
92121
parts = hostname.rsplit(":", 1)
93122
if len(parts) == 2 and parts[1].isdigit():

tests/flyte/remote/test_session.py

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -409,6 +409,90 @@ def test_closes_socket_if_connection_setup_fails(self):
409409
mock_sock.close.assert_called_once()
410410

411411

412+
class TestPyOpenSSLImportFailure:
413+
"""A pyOpenSSL/cryptography version mismatch must not take down unrelated commands.
414+
415+
pyOpenSSL binds names out of the cryptography C bindings at class-body time, so an
416+
incompatible pair fails during import with `AttributeError: module 'lib' has no
417+
attribute 'GEN_EMAIL'` (FLYTE-SDK-7T) instead of a clean ImportError.
418+
"""
419+
420+
_BOOM_MESSAGE = "module 'lib' has no attribute 'GEN_EMAIL'"
421+
422+
def _load_module_with_broken_pyopenssl(self):
423+
"""Execute the real module body with `import OpenSSL` raising.
424+
425+
Loaded into a fresh module object rather than reloaded in place, so the live
426+
`_session` (and the classes other modules imported from it) stays untouched.
427+
"""
428+
import builtins
429+
import importlib.util
430+
431+
from flyte.remote._client.auth import _session
432+
433+
boom = AttributeError(self._BOOM_MESSAGE)
434+
real_import = builtins.__import__
435+
436+
def fake_import(name, *args, **kwargs):
437+
if name == "OpenSSL":
438+
raise boom
439+
return real_import(name, *args, **kwargs)
440+
441+
spec = importlib.util.spec_from_file_location(
442+
"flyte.remote._client.auth._session_broken_pyopenssl_probe", _session.__file__
443+
)
444+
module = importlib.util.module_from_spec(spec)
445+
with patch.object(builtins, "__import__", fake_import):
446+
spec.loader.exec_module(module)
447+
return module, boom
448+
449+
def test_module_imports_despite_broken_pyopenssl(self):
450+
"""Importing the session module is what `flyte deploy` does; it must survive."""
451+
module, boom = self._load_module_with_broken_pyopenssl()
452+
453+
assert module._PYOPENSSL_IMPORT_ERROR is boom
454+
assert module.SSL is None
455+
assert module.crypto is None
456+
# The rest of the module is intact -- only the cold path is degraded.
457+
assert module.normalize_rpc_endpoint("example.com", insecure=True) == "http://example.com"
458+
459+
def test_bootstrap_reports_broken_pyopenssl_as_user_error(self):
460+
from flyte.errors import InitializationError
461+
462+
module, boom = self._load_module_with_broken_pyopenssl()
463+
464+
with patch(f"{_SESSION_MOD}.socket") as mock_socket:
465+
with pytest.raises(InitializationError) as exc_info:
466+
module._bootstrap_ssl_from_server("https://example.com:443")
467+
468+
# Raised before any connection is attempted -- nothing to clean up.
469+
mock_socket.create_connection.assert_not_called()
470+
471+
err = exc_info.value
472+
assert err.kind == "user"
473+
assert err.__cause__ is boom
474+
message = str(err)
475+
assert "pyOpenSSL" in message
476+
assert "cryptography" in message
477+
assert self._BOOM_MESSAGE in message
478+
479+
def test_broken_pyopenssl_error_is_not_reported_to_sentry(self):
480+
"""The point of the conversion: a broken install is the environment, not a bug."""
481+
from unittest import mock as _mock
482+
483+
from flyte import _sentry
484+
from flyte.errors import InitializationError
485+
486+
module, _ = self._load_module_with_broken_pyopenssl()
487+
488+
with pytest.raises(InitializationError) as exc_info:
489+
module._bootstrap_ssl_from_server("https://example.com:443")
490+
491+
with _mock.patch.object(_sentry, "init") as init_mock:
492+
_sentry.capture_exception(exc_info.value)
493+
init_mock.assert_not_called()
494+
495+
412496
class TestClientSetSessionConfig:
413497
def test_exposes_session_config(self):
414498
from flyte.remote._client.controlplane import ClientSet

0 commit comments

Comments
 (0)