Skip to content

auth-providers: OIDC callback hardening - #10931

Merged
ArtyomSavchenko merged 11 commits into
hcengineering:developfrom
MichaelUray:feat/oidc-hardening
Oct 13, 2026
Merged

ArtyomSavchenko merged 11 commits into
hcengineering:developfrom
MichaelUray:feat/oidc-hardening

Conversation

@MichaelUray

@MichaelUray MichaelUray commented Jun 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Hardens OIDC startup and callback handling, and makes ADMIN_EMAILS parsing
safe while preserving existing login-id configurations by default.

  • Retry OIDC issuer discovery and strategy registration with capped backoff so
    a temporary IdP outage cannot leave the service permanently without the
    oidc Passport strategy.
  • Return 503 with Retry-After while OIDC login is initializing.
  • Make every OIDC callback failure redirect to /login rather than returning a
    500; synchronous Passport failures and rejected middleware promises are
    handled as well as callback errors and missing users.
  • Pass an email to the shared provider-login path only after the provider has
    confirmed it: OIDC and Google use email_verified, while GitHub requires a
    primary, verified address from its emails API.
  • Abort a pending OIDC registration retry during account-service shutdown.
  • Normalize ADMIN_EMAILS, ignore empty values, flag malformed email-shaped
    values, and make ADMIN_EMAILS_STRICT=TRUE equivalent to true.

Compatibility and operational notes

ADMIN_EMAILS entries that are not valid email-shaped values are retained by
default for compatibility with deployments using login IDs such as admin.
They are warned about at startup. Set ADMIN_EMAILS_STRICT=true (case
insensitive) to drop them.

The admin decision is made by the shared account login helper, which both the
MongoDB and PostgreSQL account paths use; this is not a PostgreSQL-only code
path.

The callback query markers (oidc_callback_failed and oidc_no_account) are
diagnostic markers. The current login UI does not render them yet; surfacing a
user-facing message belongs in a small LoginApp.svelte follow-up.

Verification

  • pods/authProviders: openid.test.ts, providerEmailVerification.test.ts
  • server/account: adminEmails.test.ts

@huly-github-staging

Copy link
Copy Markdown

Connected to Huly®: UBERF-16534

…500)

Production-blocking: every OIDC callback (bogus or real code) returned
500 Internal Server Error, even with valid sessions. No log was emitted
between `try auth via openid` and the 500 — passport strategy threw
silently before `handleProviderAuth` could log.

Per Codex plan-review amendments:
- Switch to explicit-callback passport.authenticate variant so strategy
  errors surface as err/info/status args instead of being swallowed.
- Privacy-safe diagnostic shape (no raw code/state/tokens/user). Fields:
  stage, errMessage/Name/Stack, hasUser, infoSummary, statusCode,
  hasSession + sessionKeys (names only), cookie + state + code
  PRESENCE (lengths only), host, forwardedProto.
- Permanent invalid-callback guard: ALL failures → 302 to /login with
  ?error=oidc_callback_failed, never 500. Same wrap covers
  handleProviderAuth throws.

Image-tag: hardcoreeng/account:integration-wac-2026-06-21-codex-r7.

Next step (P0-T2): user attempts login, we capture the actual error
from the diagnostic, then ship a surgical fix in r8.

Signed-off-by: Michael Uray <michael.uray@gmail.com>
Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
Codex plan-review Critical: server/account/src/admin.ts:15-18 built the
ADMIN_EMAILS set without normalizing, AND isAdminEmail() didn't normalize
the input. Today value is clean so isAdmin works, but a future env-value
like ADMIN_EMAILS=" Michael@Uray.io " would silently break admin
detection (Set lookup would miss the lowercased input).

Mongo backend already has this hardening (collections/mongo.ts:1141-1147).
Postgres + this helper didn't.

Now BOTH sides normalize: env split -> trim -> toLowerCase -> filter
non-empty + warn on no-@ entries (Codex Optional: warn not reject --
too sharp for a latent fix). Lookup also trim().toLowerCase().

Standalone upstream-PR-able per Codex Q-I3.

Tests cover whitespace, case-insensitive env, case-insensitive lookup,
invalid-shape warn, empty/unset env, multi-entry.

Signed-off-by: Michael Uray <michael.uray@gmail.com>
Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
…amendment)

Codex E4 amendment (optional, low-priority): hasCookieHeader is
sufficient to diagnose missing session-cookie issues; the
cookieHeaderLen byte count adds no operational value and is one more
PII-adjacent surface to keep an eye on. Drop it from both diagnostic
paths in pods/authProviders/src/openid.ts.

Signed-off-by: Michael Uray <michael.uray@gmail.com>
Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
…TH-2)

Bei jedem fehlgeschlagenen OIDC-Callback wurde ein umfangreicher Diag-Block
(errStack/sessionKeys/host/…) auf error-Level geloggt -> anonymes Log-
Flooding via wiederholter invalider Callbacks. Normalbetrieb loggt jetzt nur
eine knappe Fehlerzeile; der volle Diag-Block nur mit OIDC_DEBUG=true.

Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
… allowed (L-AUTH-3)

Ein Tippfehler wie ADMIN_EMAILS=admin,michel@... legte bislang einen
unbeabsichtigten Admin-Identifier an (Eintrag ohne @ wurde still behalten).
Jetzt werden @-lose Eintraege per Default verworfen (fail-closed) und nur
bei ADMIN_EMAILS_ALLOW_LOGIN_ID=true als Login-ID akzeptiert; Warnung stets.

Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
…L-AUTH-4)

handleProviderAuth liefert '' bei einem nicht aufloesbaren Login (z.B. kein
Account + Signup disabled). Bisher erfolgte dann kein Redirect und next()
lief ins Leere -> haengende/leere Antwort. Jetzt fail-closed: explizites
302 auf /login?error=oidc_no_account.

Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
The account service registered the OIDC passport strategy only once at
startup via a single fire-and-forget Issuer.discover(). A transient issuer
outage in that one boot moment left the 'oidc' strategy permanently
unregistered, producing 500 "Unknown authentication strategy" on every
OpenID login until a manual container restart (observed: ~10 days).

Make registration self-healing:

- Extract registerOidcStrategyWithRetry(), which retries the ENTIRE
  registration chain (discover -> new Client -> new Strategy -> passport.use)
  with capped exponential backoff (5s, x2, cap 60s), unbounded. The loop only
  returns after passport.use succeeds, so a failure at client or strategy
  construction (e.g. temporarily incomplete issuer metadata during an IdP
  upgrade) is retried too instead of permanently disabling OIDC. discover and
  sleep are injectable for tests; passport is a structural { use } interface.
  It never rejects: the success log runs failure-safe AFTER passport.use and
  returns immediately, so a throwing logger can never re-enter the loop and
  register the strategy a second time; the failure warn is likewise guarded so
  a throwing sink cannot break the never-rejects contract.

- Gate log flooding via shouldWarnOnAttempt(): warn on the first 5 attempts
  and every 60th thereafter (~24 warn lines/day at the 60s cap instead of
  ~1440), following the branch's L-AUTH-2 rationale; error stacks only under
  OIDC_DEBUG. Success logs once with the attempt count.

- Wire it fire-and-forget in registerOpenid() and flip an oidcReady flag on
  success. The .then body is a plain boolean flip that cannot throw; a trailing
  .catch(() => {}) is belt-and-suspenders so this fire-and-forget chain can
  never surface an unhandled rejection. Routes stay registered synchronously.

- During the pending window, /auth/openid responds 503 with Retry-After: 5
  and a short text body instead of a 500 or a redirect (the login app ignores
  an error query param, and L-AUTH-4 only covers the callback path).

Add the package's first tests: the retry helper in isolation (schedule, 60s
cap, whole-attempt retry incl. client-construction failure, warn gating,
OIDC_DEBUG both ways, a throwing success log that must not re-register, and a
throwing failure warn that must not reject) and the registerOpenid wiring
(synchronous route registration, 503 pending window, exactly-once strategy
registration).

Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
Comment thread server/account/src/admin.ts Outdated
…t mode opt-in

Review feedback on hcengineering#10931: dropping entries without "@" by default would
silently revoke admin rights on upgrade for deployments that use login-id
style values. The repo's own dev/tests/ws-tests compose files ship
ADMIN_EMAILS=admin,${PLATFORM_ADMIN_EMAILS} and tests/prepare-pg.sh creates
that `admin` account.

Restore the original semantics as the default: every non-empty entry is
kept (trimmed + lowercased), and entries without "@" are logged once at
startup so a typo stays visible. The fail-closed behaviour becomes an
explicit opt-in via ADMIN_EMAILS_STRICT=true. ADMIN_EMAILS_ALLOW_LOGIN_ID,
introduced earlier on this branch and never released, is removed so there
is a single switch.

Empty entries stay dropped. This is a deliberate tightening compared to
the original `new Set(ADMIN_EMAILS.split(','))`, which put '' into the set
for any empty entry (ADMIN_EMAILS='', ',admin', 'admin,', 'admin,,ops@...')
and so made isAdminEmail('') true. loginOrSignUpWithProvider passes '' when
the provider returns no email, so such a login received an admin token.
isAdminEmail('') is now always false.

Tests: default keeps `admin` and warns, strict drops, only the exact value
"true" enables strict mode, no warning for clean values, empty entries in
every position never make an empty email an admin, trim/lowercase coverage
unchanged. Each test saves, clears and restores ADMIN_EMAILS and
ADMIN_EMAILS_STRICT so none depends on the outer environment.

Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
@MichaelUray MichaelUray changed the title auth-providers: OIDC callback hardening (3 commits) auth-providers: OIDC callback hardening Sep 29, 2026
@MichaelUray

Copy link
Copy Markdown
Contributor Author

@ArtyomSavchenko friendly ping on this one: the ADMIN_EMAILS behaviour is back to the original semantics with the strict mode as opt-in (see my reply above), and the branch still merges cleanly into develop. Let me know if you'd like the OIDC retry part split out or changed.

@ArtyomSavchenko

Copy link
Copy Markdown
Member

@MichaelUray Please take a look at review comments:

Blocking

1. Possible request hang in the callback wrapper

The wrapper is:

await new Promise<void>((resolve) => {
  passport.authenticate('oidc', { failureRedirect: loginUrl }, (err, user, info, status) => {
    ...
    resolve()
  })(ctx, async () => {})
})
  • The promise returned by the middleware call is discarded, and the executor only ever calls resolve.
  • If the middleware rejects without invoking the callback, the request hangs forever and the rejection goes unhandled.
  • Because nothing can reject, the outer try/catch is partly dead code.
    This matters most while the strategy is still being registered. /auth/openid/callback has no oidcReady check, so a callback that arrives before registration finishes hits passport with an unregistered 'oidc' strategy ("Unknown authentication strategy").

Suggested fix: pass reject through and attach .catch(reject) to the middleware call (or restructure to await authenticate(...) with a callback that records the result). Alternatively, give the callback route the same readiness check as the login route.

2. The headline fix has no test

Every test in openid.test.ts covers the retry helper and the 503 wiring. Nothing covers the callback guarantees:

  • all failures → 302 /login?error=oidc_callback_failed, never 500
  • the oidc_no_account redirect
  • the path where handleProviderAuth throws
    A test with a fake passport that (a) calls back with an error, (b) calls back with no user, and (c) throws would pin this down.

Non-blocking

  • Admin lookup. parseAdminEmails / isAdminEmail are well covered, and keeping no-@ entries by default addresses ArtyomSavchenko's comment, with ADMIN_EMAILS_STRICT as an opt-in. Two things to confirm:
    • Callers should pass a verified email. Case-insensitive matching widens what matches, and a bare admin entry matches whatever a provider puts in that field.
    • The PR says Postgres lacked the hardening, but the diff only touches the shared helper. Make sure the Postgres path really goes through it.
  • Weak @ check. bar@@ counts as an email and is kept silently (test 4 asserts this). includes('@') is a loose test if the goal is typo visibility.
  • ADMIN_EMAILS_STRICT parsing. Only the exact string 'true' enables it; TRUE or 1 are silently ignored (test 4e locks this in). An operator who sets TRUE will believe they enabled fail-closed mode. Consider case-insensitive parsing or a warning on unrecognized values.
  • Error-level logging on anonymous input. Each bad or bogus callback logs at error. The verbose fields are gated behind OIDC_DEBUG, but a scanner still produces one error line per request. warn, or a simple rate limit, would be safer.
  • Duplicated diagnostics. The base and debug diagnostic objects are built twice (authenticate callback and catch). A small buildOidcDiag(ctx, err, info?, status?, user?) helper would remove the drift risk.
  • ?error= is ignored by the UI. As the PR itself notes, the login app only reads navigateUrl and token, so oidc_callback_failed and oidc_no_account currently show a plain login page. Fine as a marker, but a follow-up in LoginApp.svelte would help users.
  • Retry loop. The design is reasonable (unbounded, no transient/permanent classification, gated warns, never-rejects contract). Two small additions:
    • The setTimeout sleep has no abort or shutdown hook, which matters for graceful shutdown and tests.
    • With a permanently wrong issuer URL, users get 503 indefinitely; the hourly warn is the only signal.
  • registerOpenid called-once invariant. It is enforced only by a comment. A second call would create a second retry loop and duplicate routes.

…s review

Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
… on shutdown; more callback tests

Signed-off-by: Michael Uray <michaeluray@users.noreply.github.com>
@MichaelUray

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review. The latest commits address the requested
changes.

  1. Google now passes an email to handleProviderAuth only when Google has
    confirmed email_verified. GitHub retains the metadata from its emails API
    and passes an email only when it is both primary and verified; profiles
    without that fetched metadata are treated as unverified. The provider's
    social identifier remains unchanged.
  2. OIDC callback coverage now exercises both a synchronous
    passport.authenticate() throw and a rejected middleware promise. Both are
    handled by the callback guard and redirect to
    /login?error=oidc_callback_failed.
  3. handleProviderAuth catches its own failures and returns an empty string,
    so that outcome redirects to /login?error=oidc_no_account; the callback
    test reflects that production behavior.
  4. The account-service shutdown path now aborts the OIDC registration retry
    loop through an AbortSignal, with coverage for that wiring.

The existing callback diagnostics and guarded logging remain in place. The
error query values are diagnostic markers; displaying a user-facing message in
the login UI remains a separate follow-up.

Verification:

  • pods/authProviders: targeted Jest tests for OIDC and provider email
    verification
  • pods/authProviders: formatter

@ArtyomSavchenko
ArtyomSavchenko merged commit 1870da7 into hcengineering:develop Oct 13, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants