Skip to content

fix: treat a blank project/domain as unset (FLYTE-SDK-3A) - #1444

Open
EngHabu wants to merge 2 commits into
mainfrom
fix/sentry-3a-empty-project
Open

fix: treat a blank project/domain as unset (FLYTE-SDK-3A)#1444
EngHabu wants to merge 2 commits into
mainfrom
fix/sentry-3a-empty-project

Conversation

@EngHabu

@EngHabu EngHabu commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

What

flyte run --project "$PROJECT" --domain "$DOMAIN" with unset shell variables hands the CLI empty strings. Two things then went wrong:

  1. CLIConfig.init used project if project is not None else self.config.task.project, so a blank value overrode the config file instead of falling back to it.
  2. require_project_and_domain — the guard that exists precisely to catch a missing project — checked cfg.project is None, so the blank sailed straight through.

The blank project then reached CreateUploadLocation, where the backend answered failed to validate project: invalid_argument: id is required. That came back as a RuntimeSystemError and was reported to Sentry as an SDK crash rather than as the user's missing configuration.

The Sentry event shows the shape exactly — note both project and domain are empty, which is what you get from -p "$P" -d "$D" with both unset:

RuntimeSystemError: Upload failed for /var/folders/.../fast9b7b178a....tar.gz
  (org='flyte', project='', domain=''): failed to validate project: invalid_argument: id is required

The fix

Both halves, because either alone leaves a bad outcome:

  • CLIConfig.init treats a blank --project/--domain as not provided and falls back to the config file. This matches what init_from_config already does (project or cfg.task.project) and what the config layer's set_if_exists already does (it filters empty strings out of TaskConfig.auto). Without this half, the common case — a CI script with an unset variable — still fails when it should have just used the config file.
  • require_project_and_domain, current_project, current_domain reject blank as well as None. Without this half, a user with nothing configured anywhere still gets the opaque backend error. With it they get the existing InitializationError("ProjectNotConfigured", "user", ...), which is on _is_user_error's allow-list and so is filtered out of Sentry.

The shared blank_to_none helper also strips surrounding whitespace, so my-project keeps working rather than becoming a new failure mode.

I deliberately left the in-cluster tctx.action.project path in current_project alone — it is populated by the backend, there is no Sentry evidence for it, and widening the change there would mask rather than surface a backend problem.

Verification

20 of the new tests fail on origin/main and pass here (the remainder are regression guards for behavior that must not change — explicit values still override the config file).

Full tests/flyte is identical to the clean-origin/main baseline in this sandbox: 3597 passed, 7 failed on both sides, the 7 being the known pre-existing local failures (test_ls_files_loaded_modules_*, test_image_with_secrets, test_retrieve_degrades_when_keyring_not_installed, the two test_launch_remote_*, test_real_build). tests/cli + tests/user_api + tests/flyte/config: 999 passed. ruff format/check, check_docstring_style.py (491 files clean) and mypy all clean.

fixes FLYTE-SDK-3A

Sentry: https://unionai.sentry.io/issues/7491437173/

Note on the Sentry grouping

FLYTE-SDK-3A is a mixed group: Sentry buckets everything that crashes at the same _upload_single_file else-branch, so its 9 events are four unrelated root causes:

when release shape owner
08-13, 08-14 2.5.18 failed to validate project: invalid_argument: id is required this PR
07-10 2.4.4 nginx 302 Found HTML page backend
05-29 2.3.6 permission denied (org/identity) user config
05-19 2.3.2 cross org calls are not allowed user config

This PR fixes only the first, which is the only shape still firing and the only one on a current release. The other three are aged out and are not SDK bugs. I've kept fixes FLYTE-SDK-3A so the group gets marked resolved-in-next-release; if one of the other shapes recurs, Sentry's regression detection will reopen it.

@EngHabu EngHabu added the sentry-fix Fix for an issue surfaced by Sentry label Aug 16, 2026
@EngHabu

EngHabu commented Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

Self-review follow-up — pushed 07bb579, please read this before reviewing the diff.

The first commit had a regression. Normalizing the blank inside CLIConfig.init was too low in the stack: flyte get secret, flyte create secret and flyte delete secret all call cfg.init(project="", domain="") deliberately — the empty string is their org-level scope sentinel (there's a # todo: remove this hack about it in _create.py). blank_to_none rewrote that to None, which then fell back to the config file:

cfg.init(project='', domain='')   # user config has project=flytesnacks/development
  before this PR : project=''            -> org-level        ✅
  first commit   : project='flytesnacks' -> project-scoped   ❌

Two consequences: secret listings silently change scope, and --cluster-pool breaks outright for anyone with a project in their config file, because Secret._resolve_scope raises when a request carries both a cluster pool and a project/domain.

Fix: only a value typed on the command line is ambiguous, so the normalization moves to a click callback on the shared PROJECT_OPTION/DOMAIN_OPTION — plus the two flyte rerun declares itself, which bypass the shared ones and would otherwise have lost the 3A fix. A caller passing project="" in Python now means it. The _initialize.py half (require_project_and_domain, current_project, current_domain) is unchanged and still guards a blank arriving from the config file or flyte.init(project=""); none of the secret commands go through those.

Verified with a scratchpad repro exercising cfg.init('', '') end-to-end against both revisions (a plain test would have failed at collection on the old revision, since it imports the new blank_option_to_none, and proved nothing):

org-level scope preserved 3A blank still falls back
origin/main YES — (this is the bug)
first commit NO YES
07bb579 YES YES

tests/cli + tests/user_api: 999 passed. The 3A tests now drive the command through click so the callback actually runs, and TestOrgLevelSecretScopeIsPreserved pins the sentinel so this can't regress again.

`flyte run --project "$PROJECT" --domain "$DOMAIN"` with unset shell variables
hands the CLI empty strings. `CLIConfig.init` only fell back to the config file on
`None`, so the blank values overrode it, and `require_project_and_domain` -- the guard
that exists precisely to catch this -- checked `is None` and let them through.

The blank project then reached CreateUploadLocation, where the backend answered
"failed to validate project: invalid_argument: id is required". That surfaced as a
RuntimeSystemError and was reported to Sentry as an SDK crash rather than as the
user's missing configuration.

Two halves:
- `CLIConfig.init` now treats a blank --project/--domain as not provided and falls
  back to the config file, matching what `init_from_config` already does with
  `project or cfg.task.project` and what the config layer's `set_if_exists` does.
- `require_project_and_domain`, `current_project` and `current_domain` reject blank
  as well as None, so when nothing supplies a project the user gets the existing
  user-kind InitializationError (filtered from Sentry) instead of an opaque
  backend error.

fixes FLYTE-SDK-3A

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Follow-up to the previous commit, which normalized the blank inside
`CLIConfig.init`. That was too low in the stack: `flyte get secret`,
`flyte create secret` and `flyte delete secret` all call `init` with a
literal empty project/domain to select the *org-level* scope, and the
normalization rewrote it to None, which then fell back to the config
file. The listing would have silently changed scope, and `--cluster-pool`
would have started failing outright, since `Secret._resolve_scope`
rejects a request that carries both a cluster pool and a project/domain.

Only a value typed on the command line is ambiguous, so the
normalization moves to a click callback on the shared PROJECT/DOMAIN
options (and on the two `flyte rerun` declares itself, which bypass
those). A caller that passes project="" in Python now means it.

fixes FLYTE-SDK-3A

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@EngHabu
EngHabu force-pushed the fix/sentry-3a-empty-project branch from 07bb579 to 198bd00 Compare September 3, 2026 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sentry-fix Fix for an issue surfaced by Sentry

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant