bug: Worker deployment missing readinessProbe

[sareetamugde-arch]

Describe the bug

Description

The worker deployment template (charts/langfuse/templates/worker/deployment.yaml) has a livenessProbe but no readinessProbe.

Problem

Without a readinessProbe, during rollouts or pod restarts, Kubernetes keeps the worker pod in Service endpoints immediately after the
container starts — before it has established connections to ClickHouse, PostgreSQL, and Redis. This causes connection refused errors
for requests routed to the not-yet-ready pod.

The web deployment (charts/langfuse/templates/web/deployment.yaml) already has a readinessProbe, but the worker does not.

Suggested fix

Add a readinessProbe to the worker deployment template:

readinessProbe:
httpGet:
path: /api/health
port: http
initialDelaySeconds: 30
periodSeconds: 15
timeoutSeconds: 10
failureThreshold: 3
successThreshold: 1

Environment

• Chart version: langfuse-k8s (latest main branch)
• Kubernetes: 1.28+

To reproduce

To reproduce

1. Deploy langfuse using the helm chart with worker replicas >= 1
2. Trigger a rollout (helm upgrade or kubectl rollout restart)
3. During rollout, the new worker pod receives traffic immediately before ClickHouse/PostgreSQL connections are established
4. Observe connection refused errors in worker logs during the first 20-30 seconds after container start

Chart version and values.yaml configuration

Chart version and values.yaml configuration

Chart: langfuse-k8s (latest main branch)
Default values.yaml — no readinessProbe configured for worker

Relevant section in templates/worker/deployment.yaml:

• livenessProbe exists (exec: ps aux | grep node)
• readinessProbe is missing

For comparison, templates/web/deployment.yaml already has readinessProbe configured.

Kubernetes Status

During rollout, new worker pod shows Running/Ready immediately after container starts,
even though internal connections (ClickHouse, PostgreSQL, Redis) are not yet established.

This is because without readinessProbe, kubelet marks the pod Ready as soon as
all containers are running — it has no health check to gate readiness.

Logs

Worker logs during rollout (first 20-30s after start):

ClickHouse query failed with non-retryable error: upstream connect error or disconnect/reset
before headers. retried and the latest reset reason: remote connection failure,
transport failure reason: delayed connect error: Connection refused

ClickhouseWriter.flush observations upstream connect error or disconnect/reset before headers.

Additional information

The web deployment already has a readinessProbe on /api/public/ready. The worker exposes /api/health which can serve the same purpose. Suggested fix is to add a readinessProbe with httpGet on /api/health port http.

Are you interested to contribute a fix for this bug?

Yes

[dosubot]

Hi @sareetamugde-arch! I'm Dosu and I'm helping the Langfuse team.

I can confirm this bug — the worker deployment template currently only has a livenessProbe (hitting /api/health on port http) and no readinessProbe [1]. The web deployment, by contrast, already has a readinessProbe configured on /api/public/ready with configurable values [2].

Your suggested fix aligns well with how the web deployment handles it. To keep it consistent with the existing patterns in this chart, the implementation should ideally make the probe parameters configurable via values.yaml (e.g. .Values.langfuse.worker.readinessProbe.*), matching how the web deployment and the existing worker livenessProbe are structured.

A couple of notes that may be helpful for your contribution:

• PR fix: set HOSTNAME=0.0.0.0 by default in worker deployment #329 (merged) fixed the worker's health-check HTTP server to properly listen on 0.0.0.0:3030, which is a prerequisite for HTTP probes to work correctly on the worker [3].
• PR feat(chart): add optional startupProbe for web and worker #371 (open) is adding optional startupProbe support for both web and worker — you may want to coordinate with that PR to avoid conflicts [4].

Since you've indicated interest in contributing a fix, a PR adding a readinessProbe block to charts/langfuse/templates/worker/deployment.yaml (mirroring the web deployment's pattern with Helm-configurable defaults) would be a clean solution.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[sareetamugde-arch]

Hi @sareetamugde-arch! I'm Dosu and I'm helping the Langfuse team.

I can confirm this bug — the worker deployment template currently only has a livenessProbe (hitting /api/health on port http) and no readinessProbe [1]. The web deployment, by contrast, already has a readinessProbe configured on /api/public/ready with configurable values [2].

Your suggested fix aligns well with how the web deployment handles it. To keep it consistent with the existing patterns in this chart, the implementation should ideally make the probe parameters configurable via values.yaml (e.g. .Values.langfuse.worker.readinessProbe.*), matching how the web deployment and the existing worker livenessProbe are structured.

A couple of notes that may be helpful for your contribution:

• PR fix: set HOSTNAME=0.0.0.0 by default in worker deployment #329 (merged) fixed the worker's health-check HTTP server to properly listen on 0.0.0.0:3030, which is a prerequisite for HTTP probes to work correctly on the worker [3].
• PR feat(chart): add optional startupProbe for web and worker #371 (open) is adding optional startupProbe support for both web and worker — you may want to coordinate with that PR to avoid conflicts [4].

Since you've indicated interest in contributing a fix, a PR adding a readinessProbe block to charts/langfuse/templates/worker/deployment.yaml (mirroring the web deployment's pattern with Helm-configurable defaults) would be a clean solution.

thanks for confirming. will raise a PR for this soon. will add readinessProbe to worker deployment matching the web pattern with
configurable values via values.yaml. will also check PR #371 to avoid conflicts.

[sareetamugde-arch]

Hi @Steffen911 , please check this bug and any upcoming fixes coming up for this?

[Steffen911]

@sareetamugde-arch The worker container should never receive HTTP requests outside of the health checks as it's a pure async-queue processor. Therefore, there also shouldn't be any service connection.
Can you add more detail on why this feels like a requirement to you? As of now, this setup is deliberately chosen.

[sareetamugde-arch]

@sareetamugde-arch The worker container should never receive HTTP requests outside of the health checks as it's a pure async-queue processor. Therefore, there also shouldn't be any service connection. Can you add more detail on why this feels like a requirement to you? As of now, this setup is deliberately chosen.

@Steffen911 :
the worker exposes /api/health on port 3030 which returns 200 only after ClickHouse/PostgreSQL/Redis connections are established. our
issue was during rolling updates — without readinessProbe, kubernetes marks the new pod Ready immediately (since all containers are
running), terminates the old pod, but the new worker hasn't connected to clickhouse yet. jobs picked up during that window fail with
connection refused and get dropped.

after adding readinessProbe on /api/health, kubernetes waits for the health endpoint to return 200 before marking pod Ready and
terminating the old pod. this fixed the issue — worker restarts stopped completely.

even though no Service routes external traffic to the worker, the Ready condition still controls rollout behavior (old pod
termination timing). is there a reason this was deliberately excluded, or would you be open to adding it as optional/configurable?

[Steffen911]

@sareetamugde-arch If you feel that this is useful, I'd be happy to accept a PR to add this.