Repository navigation
Description
What happened (please include outputs or screenshots):
InClusterConfigLoader schedules the projected service-account token re-read using timezone-naive datetime.datetime.now() (local wall-clock time). In kubernetes/config/incluster_config.py (v37.0.1):
# line ~47
self._token_refresh_period = datetime.timedelta(minutes=1)
# line ~96
if self.token_expires_at <= datetime.datetime.now():
# lines ~108-109
self.token_expires_at = datetime.datetime.now() + self._token_refresh_periodWhen the local wall clock jumps backwards — e.g. a DST fall-back while the container runs with a non-UTC TZ (such as TZ=Europe/Stockholm), an NTP step, or a manual clock change — the stored token_expires_at (computed before the jump) stays in the perceived future. The refresh condition token_expires_at <= now() never becomes true for up to the duration of the backward jump, so _read_token_file() is not called. Meanwhile the kubelet rotates the projected token on disk, the client keeps sending the stale cached bearer token, and the API server responds with 401 Unauthorized until wall-clock time catches up.
Observed at a CEST→CET fall-back: a ~46-minute burst of 401 Unauthorized responses, self-recovering once local time passed the stale token_expires_at.
What you expected to happen:
Token refresh timing should be immune to wall-clock discontinuities. Using a monotonic clock (time.monotonic()) for the refresh deadline — or a timezone-aware UTC datetime — would make the refresh fire on the intended cadence regardless of DST/NTP/manual wall-clock changes.
How to reproduce it (as minimally and precisely as possible):
- Run a pod using in-cluster config with a non-UTC
TZthat observes DST (e.g.TZ=Europe/Stockholm). - Let the client cache a token shortly before a DST fall-back (clock about to move back 1h).
- At the transition, the wall clock jumps back 1h;
token_expires_atnow sits ~1h in the "future". - kubelet rotates the SA token on disk; the client does not re-read it.
- Requests to the API server return
401 Unauthorizeduntil the wall clock reaches the pre-jumptoken_expires_at.
Equivalent without waiting for a real DST event: step the host/container clock backward after a token read and observe that _read_token_file() is not invoked by refresh_api_key_hook until the wall clock catches up.
Anything else we need to know?:
Suggested fix: compute the refresh deadline from time.monotonic() instead of datetime.datetime.now() (monotonic never goes backwards), or use timezone-aware UTC. The change is confined to _read_token_file() (store deadline) and the _refresh_api_key hook (compare against the same clock). This is a latent defect affecting any client using in-cluster config with a non-UTC TZ during a backward wall-clock change; it is not specific to any one service.
Environment:
- Kubernetes version (
kubectl version): N/A — client-side timing defect, reproducible against any API server - OS (e.g., MacOS 10.13.6): Linux container with
TZ=Europe/Stockholm(any non-UTC, DST-observing zone) - Python version (
python --version): Python 3.13 - Python client version (
pip list | grep kubernetes): kubernetes==37.0.1 (latest release; code path unchanged in currentmaster)
Activity
OKAN-SOFTWARE commentedon Nov 3, 2026
Hi,
I've reproduced the issue with in-cluster token refresh when the system clock moves backward.
I've prepared a focused fix that uses
time.monotonic()to schedule token refreshes, along with a regression test that reproduces the original failure.Validation:
Before opening a pull request, I'd appreciate confirmation that this approach aligns with the project's expectations.
I'm also happy to complete the CNCF Contributor License Agreement through EasyCLA when prompted.
Thanks for your time!
roycaihw commentedon Nov 5, 2026
PR: #2771
radiohead-lab commentedon Nov 8, 2026
Hi @roycaihw
Thanks very much for handling this issue.
If the commit will be merged and fix will be released in next version then this ticket can be closed.
@radiohead-lab
tatumjakhiya26 commentedon Nov 9, 2026
While switching to UTC is an improvement,
time.monotonic()is strictly preferred here to completely decouple the refresh logic from system clock adjustments. If we stick todatetime.datetime.now(datetime.timezone.utc), have we considered howtoken_expires_atbehaves if the container is paused or suspended during a virtual machine migration?radiohead-lab commentedon Nov 9, 2026
Thanks @tatumjakhiya26 — agreed, and that's the direction the PR took: #2771 ("Use monotonic time for in-cluster token refresh") uses
time.monotonic()for the refresh deadline rather than UTC datetime, which fully decouples the refresh from wall-clock adjustments (DST, NTP steps, manual changes). So the monotonic approach you're advocating is the one being merged, not the UTC-datetime alternative.On the pause/suspend-during-VM-migration edge case, for completeness:
datetime.now(timezone.utc): tracks wall time, so it keeps advancing across a pause. On resume,now()has moved forward by the pause duration,token_expires_atis already in the past, and the next refresh check re-reads the token immediately. Safe, but only because wall time advanced.time.monotonic(): cannot go backward, but whether it advances during a full system suspend is platform-dependent. If it's frozen during the pause, the deadline is reached late by at most the pause duration — a bounded, self-correcting delay, never an indefinite stall.In both cases the behavior is bounded by the 1-minute refresh period, and the
_refresh_api_keyhook is invoked on essentially every API call, so in practice the token is re-read within seconds of resume. Crucially, neither clock reintroduces the original failure mode: a backward wall-clock jump can no longer push the deadline into a false future. Monotonic is the safer default precisely because it removes that class of bug entirely, with only a benign bounded-delay trade-off in the rare suspend case.