Bug description
Jupyterhub aborts startup if managed services are not reachable. When a hub managed service is set up following the example in docs, the default network policies prevent the hub process from reaching the service. As a result, the hub's requests to check whether the service is up or not, times out. And then the hub aborts startup because it assumes the managed service failed to start.
In the current documented example of setting up a managed service, we only allow incoming traffic from pods with the label hub.jupyter.org/network-access-hub: "true" but the hub pod doesn't have this label. So traffic from the hub process to the managed service running at hub:<port> is blocked by network policy enforcement.
One possible fix would be to add a separate podSelector block in the service documentation example to allow traffic from the hub pod as well. This is what I ended up doing in 2i2c-org/infrastructure#8113. Or we could also consider whether it makes sense to add the hub.jupyter.org/network-access-hub: "true" label to the hub pod itself.
How to reproduce
Here's my local config with a Colima k8s cluster:
colima start \
--network-address \
--kubernetes \
--k3s-arg="--flannel-backend=none" \
--k3s-arg="--disable-network-policy"
Note that Colima uses flannel by default which doesn't enforce network policies. so using calico instead of flannel to make sure we enforce network policies.
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.0/manifests/calico.yaml
jupyterhub_values.yaml following the structure in the example. Instead of a fastapi service, just using python http.server for simplicity:
hub:
config:
JupyterHub:
authenticator_class: dummy
DummyAuthenticator:
password: "password"
services:
test-service:
url: http://hub:9999
command:
- python3
- -m
- http.server
- "9999"
- --bind
- "0.0.0.0"
display: true
oauth_no_confirm: true
service:
extraPorts:
- name: test-service
port: 9999
targetPort: 9999
networkPolicy:
ingress:
- ports:
- port: 9999
from:
- podSelector:
matchLabels:
hub.jupyter.org/network-access-hub: "true"
proxy:
chp:
networkPolicy:
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: jupyterhub
app.kubernetes.io/component: hub
ports:
- port: 9999
Install the hub chart:
helm upgrade --install jupyterhub jupyterhub/jupyterhub -n jhub -f jupyterhub-values.yaml
Check that the service is not reachable through hub:9999 url but is reachable through localhost:9999 from the hub pod:
kubectl exec -it -n jhub hub-5cd58cbd9c-rpsbz -- bash
jovyan@hub-5cd58cbd9c-rpsbz:/srv/jupyterhub$ curl localhost:9999
<!DOCTYPE HTML>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href="jupyterhub.sqlite">jupyterhub.sqlite</a></li>
</ul>
<hr>
</body>
</html>
jovyan@hub-5cd58cbd9c-rpsbz:/srv/jupyterhub$ curl hub:9999
^C
(this one hangs)
Note in the logs that the hub complains about not being able to reach the managed service:
kubectl logs hub-5cd58cbd9c-rpsbz -n jhub -p
Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
Loading pass-through config section hub.config.DummyAuthenticator
Loading pass-through config section hub.config.JupyterHub
[I 2026-04-16 09:16:03.627 JupyterHub app:3364] Running JupyterHub version 5.4.4
[I 2026-04-16 09:16:03.627 JupyterHub app:3394] Using Authenticator: jupyterhub.auth.DummyAuthenticator-5.4.4
[I 2026-04-16 09:16:03.627 JupyterHub app:3394] Using Spawner: kubespawner.spawner.KubeSpawner-7.0.0
[I 2026-04-16 09:16:03.627 JupyterHub app:3394] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-5.4.4
/usr/local/lib/python3.12/site-packages/jupyter_events/schema.py:68: JupyterEventsVersionWarning: The `version` property of an event schema must be a string. It has been type coerced, but in a future version of this library, it will fail to validate. Please update schema: https://schema.jupyter.org/jupyterhub/events/server-action
validate_schema(_schema)
[W 2026-04-16 09:16:03.672 JupyterHub auth:1537] DummyAuthenticator.password is deprecated in JupyterHub 5.3. Use SharedPasswordAuthenticator.user_password instead.
[W 2026-04-16 09:16:03.675 JupyterHub auth:1541] Using testing authenticator DummyAuthenticator! This is not meant for production!
[I 2026-04-16 09:16:03.687 JupyterHub app:2931] Creating service jupyterhub-idle-culler without oauth.
[I 2026-04-16 09:16:03.690 JupyterHub app:2893] Creating service test-service with oauth_client_id=service-test-service
[I 2026-04-16 09:16:03.692 JupyterHub provider:661] Creating oauth client service-test-service
[I 2026-04-16 09:16:03.701 JupyterHub app:2667] Adding API token for service: test-service
[W 2026-04-16 09:16:03.714 JupyterHub app:3230] Allowing service test-service to complete OAuth without confirmation on an authorization web page
[I 2026-04-16 09:16:03.718 JupyterHub app:3434] Initialized 0 spawners in 0.002 seconds
[I 2026-04-16 09:16:03.721 JupyterHub metrics:425] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2026-04-16 09:16:03.721 JupyterHub metrics:425] Found 0 active users in the last ActiveUserPeriods.seven_days
[I 2026-04-16 09:16:03.721 JupyterHub metrics:425] Found 0 active users in the last ActiveUserPeriods.thirty_days
[I 2026-04-16 09:16:03.721 JupyterHub app:3721] Not starting proxy
[I 2026-04-16 09:16:03.733 JupyterHub app:3757] Hub API listening on http://:8081/hub/
[I 2026-04-16 09:16:03.733 JupyterHub app:3759] Private Hub API connect url http://hub:8081/hub/
[I 2026-04-16 09:16:03.733 JupyterHub app:3633] Starting managed service jupyterhub-idle-culler
[I 2026-04-16 09:16:03.733 JupyterHub service:423] Starting service 'jupyterhub-idle-culler': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10']
[I 2026-04-16 09:16:03.734 JupyterHub service:136] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=3600 --cull-every=600 --concurrency=10
[I 2026-04-16 09:16:03.734 JupyterHub app:3633] Starting managed service test-service at http://hub:9999
[I 2026-04-16 09:16:03.734 JupyterHub service:423] Starting service 'test-service': ['python3', '-m', 'http.server', '9999', '--bind', '0.0.0.0']
[I 2026-04-16 09:16:03.735 JupyterHub service:136] Spawning python3 -m http.server 9999 --bind 0.0.0.0
[I 2026-04-16 09:16:03.884 JupyterHub log:192] 200 GET /hub/api/ (jupyterhub-idle-culler@::1) 3.75ms
[I 2026-04-16 09:16:03.891 JupyterHub log:192] 200 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@::1) 5.95ms
127.0.0.1 - - [16/Apr/2026 09:17:36] "GET / HTTP/1.1" 200 -
[C 2026-04-16 09:19:24.467 JupyterHub app:3665] Cannot connect to test-service service managed
Notice that the hub restarts every 3 minutes or so failing this service uptime check:
kubectl get pods -n jhub
NAME READY STATUS RESTARTS AGE
continuous-image-puller-qhc24 1/1 Running 0 13m
hub-5cd58cbd9c-rpsbz 1/1 Running 3 (3m27s ago) 13m
proxy-776cdc7f6d-p8c67 1/1 Running 0 13m
user-scheduler-95c7c9f9c-269wr 1/1 Running 0 13m
user-scheduler-95c7c9f9c-rbrq4 1/1 Running 0 13m
and eventually enters CrashLoopBackoff state:
kubectl get pods -n jhub
NAME READY STATUS RESTARTS AGE
continuous-image-puller-qhc24 1/1 Running 0 29m
hub-5cd58cbd9c-rpsbz 0/1 CrashLoopBackOff 6 (83s ago) 29m
proxy-776cdc7f6d-p8c67 1/1 Running 0 29m
user-scheduler-95c7c9f9c-269wr 1/1 Running 0 29m
user-scheduler-95c7c9f9c-rbrq4 1/1 Running 0 29m
Expected behaviour
The hub process should be able to reach the hub managed service and start up check for the service should pass.
Actual behaviour
The hub process isn't able to reach the hub managed service and the hub keeps crashing.
Your personal set up
- Version(s): jupyterhub 5.4.4
Full environment
jovyan@hub-5cd58cbd9c-rpsbz:/srv/jupyterhub$ pip freeze
aiohappyeyeballs==2.6.1
aiohttp==3.13.3
aiosignal==1.4.0
alembic==1.18.4
annotated-types==0.7.0
arrow==1.4.0
attrs==26.1.0
bcrypt==5.0.0
cachetools==7.0.5
certifi==2026.2.25
certipy==0.2.2
cffi==2.0.0
charset-normalizer==3.4.6
cryptography==46.0.6
escapism==1.1.0
fqdn==1.5.1
frozenlist==1.8.0
google-auth==2.49.1
google-auth-oauthlib==1.3.0
greenlet==3.3.2
idna==3.11
isoduration==20.11.0
Jinja2==3.1.6
jsonpointer==3.1.1
jsonschema==4.26.0
jsonschema-specifications==2025.9.1
jupyter-events==0.12.0
jupyterhub==5.4.4
jupyterhub-firstuseauthenticator==1.1.1
jupyterhub-hmacauthenticator==1.0
jupyterhub-idle-culler==1.4.0
jupyterhub-kubespawner==7.0.0
jupyterhub-ldapauthenticator==2.0.2
jupyterhub-ltiauthenticator==1.6.3
jupyterhub-nativeauthenticator==1.3.0
jupyterhub-tmpauthenticator==1.0.0
kubernetes_asyncio==35.0.1
lark==1.3.1
ldap3==2.9.1
Mako==1.3.10
MarkupSafe==3.0.3
multidict==6.7.1
mwoauth==0.4.0
oauthenticator==17.4.0
oauthlib==3.3.1
onetimepass==1.0.1
packaging==26.0
pamela==1.2.0
prometheus_client==0.24.1
propcache==0.4.1
psycopg2==2.9.11
py-spy==0.4.1
pyasn1==0.6.3
pyasn1_modules==0.4.2
pycparser==3.0
pycurl==7.45.7
pydantic==2.12.5
pydantic_core==2.41.5
PyJWT==2.12.1
PyMySQL==1.1.2
python-dateutil==2.9.0.post0
python-json-logger==4.0.0
python-slugify==8.0.4
PyYAML==6.0.3
referencing==0.37.0
requests==2.33.0
requests-oauthlib==2.0.0
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rfc3987-syntax==1.1.0
rpds-py==0.30.0
ruamel.yaml==0.19.1
six==1.17.0
SQLAlchemy==2.0.48
sqlalchemy-cockroachdb==2.0.3
statsd==4.0.1
text-unidecode==1.3
tornado==6.5.5
traitlets==5.14.3
typing-inspection==0.4.2
typing_extensions==4.15.0
tzdata==2025.3
uri-template==1.3.0
urllib3==2.6.3
webcolors==25.10.0
yarl==1.23.0
Configuration
Logs
Bug description
Jupyterhub aborts startup if managed services are not reachable. When a hub managed service is set up following the example in docs, the default network policies prevent the hub process from reaching the service. As a result, the hub's requests to check whether the service is up or not, times out. And then the hub aborts startup because it assumes the managed service failed to start.
In the current documented example of setting up a managed service, we only allow incoming traffic from pods with the label
hub.jupyter.org/network-access-hub: "true"but the hub pod doesn't have this label. So traffic from the hub process to the managed service running athub:<port>is blocked by network policy enforcement.One possible fix would be to add a separate
podSelectorblock in the service documentation example to allow traffic from thehubpod as well. This is what I ended up doing in 2i2c-org/infrastructure#8113. Or we could also consider whether it makes sense to add thehub.jupyter.org/network-access-hub: "true"label to the hub pod itself.How to reproduce
Here's my local config with a Colima k8s cluster:
Note that Colima uses flannel by default which doesn't enforce network policies. so using calico instead of flannel to make sure we enforce network policies.
jupyterhub_values.yamlfollowing the structure in the example. Instead of a fastapi service, just using python http.server for simplicity:Install the hub chart:
Check that the service is not reachable through
hub:9999url but is reachable throughlocalhost:9999from the hub pod:Note in the logs that the hub complains about not being able to reach the managed service:
Notice that the hub restarts every 3 minutes or so failing this service uptime check:
and eventually enters
CrashLoopBackoffstate:Expected behaviour
The hub process should be able to reach the hub managed service and start up check for the service should pass.
Actual behaviour
The hub process isn't able to reach the hub managed service and the hub keeps crashing.
Your personal set up
Full environment
Configuration
# jupyterhub_config.pyLogs