Skip to content

InClusterConfigLoader token refresh uses timezone-naive datetime.now(), stalls on backward clock jumps (DST) #2768

Open

Description

@radiohead-lab

What happened (please include outputs or screenshots):

InClusterConfigLoader schedules the projected service-account token re-read using timezone-naive datetime.datetime.now() (local wall-clock time). In kubernetes/config/incluster_config.py (v37.0.1):

# line ~47
self._token_refresh_period = datetime.timedelta(minutes=1)
# line ~96
if self.token_expires_at <= datetime.datetime.now():
# lines ~108-109
self.token_expires_at = datetime.datetime.now() + self._token_refresh_period

When the local wall clock jumps backwards — e.g. a DST fall-back while the container runs with a non-UTC TZ (such as TZ=Europe/Stockholm), an NTP step, or a manual clock change — the stored token_expires_at (computed before the jump) stays in the perceived future. The refresh condition token_expires_at <= now() never becomes true for up to the duration of the backward jump, so _read_token_file() is not called. Meanwhile the kubelet rotates the projected token on disk, the client keeps sending the stale cached bearer token, and the API server responds with 401 Unauthorized until wall-clock time catches up.

Observed at a CEST→CET fall-back: a ~46-minute burst of 401 Unauthorized responses, self-recovering once local time passed the stale token_expires_at.

What you expected to happen:

Token refresh timing should be immune to wall-clock discontinuities. Using a monotonic clock (time.monotonic()) for the refresh deadline — or a timezone-aware UTC datetime — would make the refresh fire on the intended cadence regardless of DST/NTP/manual wall-clock changes.

How to reproduce it (as minimally and precisely as possible):

  1. Run a pod using in-cluster config with a non-UTC TZ that observes DST (e.g. TZ=Europe/Stockholm).
  2. Let the client cache a token shortly before a DST fall-back (clock about to move back 1h).
  3. At the transition, the wall clock jumps back 1h; token_expires_at now sits ~1h in the "future".
  4. kubelet rotates the SA token on disk; the client does not re-read it.
  5. Requests to the API server return 401 Unauthorized until the wall clock reaches the pre-jump token_expires_at.

Equivalent without waiting for a real DST event: step the host/container clock backward after a token read and observe that _read_token_file() is not invoked by refresh_api_key_hook until the wall clock catches up.

Anything else we need to know?:

Suggested fix: compute the refresh deadline from time.monotonic() instead of datetime.datetime.now() (monotonic never goes backwards), or use timezone-aware UTC. The change is confined to _read_token_file() (store deadline) and the _refresh_api_key hook (compare against the same clock). This is a latent defect affecting any client using in-cluster config with a non-UTC TZ during a backward wall-clock change; it is not specific to any one service.

Environment:

  • Kubernetes version (kubectl version): N/A — client-side timing defect, reproducible against any API server
  • OS (e.g., MacOS 10.13.6): Linux container with TZ=Europe/Stockholm (any non-UTC, DST-observing zone)
  • Python version (python --version): Python 3.13
  • Python client version (pip list | grep kubernetes): kubernetes==37.0.1 (latest release; code path unchanged in current master)

Activity

  1. OKAN-SOFTWARE commented on Nov 3, 2026

    @OKAN-SOFTWARE

    Hi,

    I've reproduced the issue with in-cluster token refresh when the system clock moves backward.

    I've prepared a focused fix that uses time.monotonic() to schedule token refreshes, along with a regression test that reproduces the original failure.

    Validation:

    • Confirmed the regression test fails before the fix.
    • All 13 targeted tests pass after the fix.
    • Python syntax and Git whitespace checks pass.

    Before opening a pull request, I'd appreciate confirmation that this approach aligns with the project's expectations.

    I'm also happy to complete the CNCF Contributor License Agreement through EasyCLA when prompted.

    Thanks for your time!

  2. roycaihw commented on Nov 5, 2026

    @roycaihw
    Member

    PR: #2771

  3. radiohead-lab commented on Nov 8, 2026

    @radiohead-lab
    Author

    PR: #2771

    Hi @roycaihw
    Thanks very much for handling this issue.
    If the commit will be merged and fix will be released in next version then this ticket can be closed.

    @radiohead-lab

  4. tatumjakhiya26 commented on Nov 9, 2026

    @tatumjakhiya26

    While switching to UTC is an improvement, time.monotonic() is strictly preferred here to completely decouple the refresh logic from system clock adjustments. If we stick to datetime.datetime.now(datetime.timezone.utc), have we considered how token_expires_at behaves if the container is paused or suspended during a virtual machine migration?

  5. radiohead-lab commented on Nov 9, 2026

    @radiohead-lab
    Author

    Thanks @tatumjakhiya26 — agreed, and that's the direction the PR took: #2771 ("Use monotonic time for in-cluster token refresh") uses time.monotonic() for the refresh deadline rather than UTC datetime, which fully decouples the refresh from wall-clock adjustments (DST, NTP steps, manual changes). So the monotonic approach you're advocating is the one being merged, not the UTC-datetime alternative.

    On the pause/suspend-during-VM-migration edge case, for completeness:

    • UTC datetime.now(timezone.utc): tracks wall time, so it keeps advancing across a pause. On resume, now() has moved forward by the pause duration, token_expires_at is already in the past, and the next refresh check re-reads the token immediately. Safe, but only because wall time advanced.
    • time.monotonic(): cannot go backward, but whether it advances during a full system suspend is platform-dependent. If it's frozen during the pause, the deadline is reached late by at most the pause duration — a bounded, self-correcting delay, never an indefinite stall.

    In both cases the behavior is bounded by the 1-minute refresh period, and the _refresh_api_key hook is invoked on essentially every API call, so in practice the token is re-read within seconds of resume. Crucially, neither clock reintroduces the original failure mode: a backward wall-clock jump can no longer push the deadline into a false future. Monotonic is the safer default precisely because it removes that class of bug entirely, with only a benign bounded-delay trade-off in the rare suspend case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions