Env-var control of in-cluster auth mode (ExternalCommand without EAGER_API_KEY)

[EngHabu]

Problem

A customer does not want the control plane issuing eager API keys to their task pods. They mint their own, more narrowly scoped tokens and want the SDK to authenticate with ExternalCommand, which we already support and prototyped with them.

init_in_cluster resolved credentials in this order:

1. an explicit api_key= argument
2. a mounted config file, if the pod has UCTL_CONFIG / FLYTECTL_CONFIG set
3. the injected _UNION_EAGER_API_KEY / EAGER_API_KEY

So the only way to opt out of the injected key was to mount a config file. There was no env-var-only route — even though every credentials config entry already derives an env var (admin.authType → FLYTE_ADMIN_AUTHTYPE).

Two things were also quietly broken along the way, and both had to be fixed for the env-var route to work end to end.

What this does

A task pod can now carry:

FLYTE_AUTH_TYPE=ExternalCommand
FLYTE_AUTH_COMMAND="/usr/local/bin/mint-token --audience flyte"

and authenticate by running that command, with no API key issued to it at all.

• FLYTE_AUTH_COMMAND takes a shell-quoted command line, or a JSON array when an argument contains spaces. Its stdout is the access token, re-run on refresh.
• Setting FLYTE_AUTH_TYPE disables the injected-key fallback outright rather than layering on top of it, so a cluster still injecting a key mid-migration cannot have the two silently disagree.
• The endpoint still comes from the injected _U_EP_OVERRIDE, and now also accepts FLYTE_ADMIN_ENDPOINT — dropping the API key drops the endpoint it used to decode to.
• FLYTE_AUTH_PROXY_COMMAND is picked up independently of the auth type.
• An explicit api_key= argument still wins, and logs that it is ignoring the env vars instead of doing so quietly.
• ExternalCommand with no command raises at init, rather than deferring to an AuthenticationError on the first RPC that names neither env var.

The entries are read via ConfigEntry.read() with no config file, which consults only the environment. That deliberately keeps the pod off resolve_config_path, whose git rev-parse subprocess is wasted work in a task pod.

Two bugs fixed on the way

FLYTE_ADMIN_COMMAND was unusable even on the existing config-file path. ConfigEntry applied no transform to env values, so a list-typed entry came back as the raw string and asyncio.create_subprocess_exec(*self._cmd) spread it one argument per character. Same shape for the proxy command and scopes. YAML already yields a list and is unaffected; the new transforms are idempotent on that path.

create_remote_controller accepted command and threw it away. It took command, proxy_command, http_proxy_url and client_config and never forwarded them to ControllerClient. Since init_in_cluster builds the SDK client and the controller from the same kwargs, auth_type="ExternalCommand" would have authenticated the client correctly while leaving the controller holding the auth mode with nothing to run — surfacing only as AuthenticationError: Command cannot be empty for command authenticator on the controller's first RPC. The controller is what enqueues and watches child actions, so eager tasks would have failed at the point of spawning work rather than at init.

rpc_retries is deliberately still not forwarded there: the controller has never installed the retry interceptor, and adding it would change retry behavior rather than fix a drop.

On the env var names

These names are normally derived, not chosen: get_env_name() does FLYTE_{SWITCH_UPPERCASED}, so admin.authType → FLYTE_ADMIN_AUTHTYPE. That spelling leaks flyte-1 flyteadmin naming into something a customer has to type into a pod spec, and read_from_env's docstring already called the scheme provisional.

Renaming outright would cost more than the wart. The scheme's one real virtue is that env var and config key are derivable from each other both ways; changing 2 of the 11 admin.* entries leaves no rule, just a lookup table with exceptions. The derived names are also public surface predating this feature — they are what the config-file path has always used.

So the preferred name is now explicit rather than implicit. YamlConfigEntry takes an aliases tuple checked ahead of the derived name; get_env_name() reports the preferred one (docs, error messages) and env_names() gives the precedence order.

setting	preferred	still accepted
admin.authType	FLYTE_AUTH_TYPE	FLYTE_ADMIN_AUTHTYPE
admin.command	FLYTE_AUTH_COMMAND	FLYTE_ADMIN_COMMAND
admin.proxyCommand	FLYTE_AUTH_PROXY_COMMAND	FLYTE_ADMIN_PROXYCOMMAND

Nothing that worked before stops working. When two names for one setting disagree the higher-precedence one wins and the conflict is logged rather than dropped silently — that ambiguity is how a migration goes wrong quietly; identical values are not reported.

Scoped deliberately to the three auth entries this feature documents. FLYTE_ADMIN_ENDPOINT keeps its derived name, since the endpoint is a platform setting rather than an auth one. Renaming the rest of the admin.* family is the scheme-wide change the code comment anticipates and wants its own discussion.

Known gap

The opt-in Rust controller (_F_USE_RUST_CONTROLLER=1) reads the injected API key directly and does not honor these vars. Anyone on it cannot drop the key yet. Called out in the init_in_cluster docstring.

Testing

• 18 new tests in tests/user_api/test_init_in_cluster_env_auth.py covering parsing (shlex, JSON array, quoted args), precedence (explicit auth type beats the injected key; explicit api_key beats the env vars; no env vars means unchanged legacy behavior), the endpoint fallback, the loud-failure case, the controller forwarding, and the alias mechanism (preferred name wins, derived name still honored, conflicts reported, non-aliased entries unchanged).
• Verified end to end that the env vars produce an AsyncCommandAuthenticator which runs the command and returns its stdout as the token.
• Full suite failure set is byte-identical before and after this change. ruff and mypy clean.

Note for reviewers running tests inside a Flyte task pod: _U_EP_OVERRIDE and _UNION_EAGER_API_KEY are live in that environment and leak into several tests, including a pre-existing one in test_init_in_cluster_fallback.py. Unset them for a clean run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUo1yWWgg9hxxgni4hm8XR