[Bug]: Code Interpreter file deletion has never worked: `deleteCodeEnvFile` calls a route that does not exist (404)

[Spyros-Vl]

What happened?

Severity: High urgency, unbounded consequence. Nothing visibly breaks for users; storage grows without limit and no layer reports a problem.

---

1. What we observed

We went looking for something else entirely — a burst of authorisation errors after a tenant migration — and found that the object store on a six-week-old production deployment had grown to 13 GiB across 29,177 objects, with no obvious owner.

Investigating from the Code Interpreter side:

Deletions are being attempted, constantly. Over 24 hours on a two-node deployment:

DELETE requests received      501 (node A) / 490 (node B)

None of them ever succeed. file_server logs File deleted successfully on every successful removal. That line appears zero times, on either node, ever:

successful deletions          0

So objects only accumulate. On a freshly built deployment with light real traffic:

Sep 1  09:00   2 objects
Sep 2  09:24  40 objects, 1.7 MiB      (~19 hours)

The client is retrying, not giving up. The same object paths recur roughly six times each in a 24-hour window.

And deleting by hand confirmed the objects were genuinely unreferenced. Removing everything older than the session TTL cleared 27,124 of the 29,177 objects with no user-visible effect.

2. What the issue is

deleteCodeEnvFile issues:

DELETE /v1/sessions/:session_id/objects/:fileId?kind=<kind>&id=<id>

The Code Interpreter service registers that path as GET only. Its delete route is at a different path:

path	methods registered on /v1
/sessions/:session_id/objects/:fileId	GET only — router.get(…, fetchLimiter, sessionAuth, …)
/files/:session_id/:fileId	DELETE — router.delete(…, fetchLimiter, sessionAuth, …)

/sessions/:session_id/objects/:fileId does accept DELETE — but only on the internal file_server service, which is not exposed on /v1. The public /v1/files/... route proxies to it. It looks as though the client was written against the internal path rather than the public one.

So every deletion returns 404 before authorisation is even evaluated.

A second, independent bug sits behind this one

Fixing the route is necessary but not sufficient. There is a second defect that the 404 currently masks, and it will surface the moment the path is corrected:

• code-interpreter gates every delete behind a Redis session key with SESSION_CACHE_TTL, default 86400s (24 hours), which is not refreshed by use.
• LibreChat does not consider a file eligible for deletion until its retention window elapses — DEFAULT_RETENTION_HOURS = 24 * 30 in tempChatRetention.ts, i.e. 30 days by default.

So on vanilla upstream defaults, every retention-swept file is already 29 days past the point at which it could be authorised for deletion. The delete is guaranteed to fail, on every attempt, forever.

And packages/api/src/files/sweep.ts has no backoff, no give-up and no retry cap: a file that fails to delete stays flagged expired and is refetched and retried on every hourly run indefinitely.

These two bugs are sequential:

	today	after the route fix alone
response	404 — no such route	403 — session key expired
reaches sessionAuth?	no	yes
deletes anything?	no	still no
visible to the client?	no (404 read as "already gone")	yes, but retried forever

Correcting the path is still the right first change — it turns a silent failure into a diagnosable one — but the retention/TTL incompatibility has to be addressed too or nothing is actually deleted.

(Credit: the retention and sweep findings in this section come from the GN LibreChat team's review of danny-avila/LibreChat; we verified the route and TTL behaviour against a running deployment.)

3. What this error is causing

An unbounded storage leak in every deployment. Nothing else garbage-collects. 13 GiB on a six-week-old instance; on that host the object store sat on the OS disk, which reached 80% and would eventually have taken the service down.

The files become permanently undeletable, not merely undeleted. This is the part that makes it more than untidiness. The Code Interpreter authorises deletion with the same session key as download — session:<session_id> in Redis, TTL SESSION_CACHE_TTL, default 86400s — and that key is not refreshed by use. Once it expires 24 hours after upload, the object cannot be read, used as an execution input, or deleted, by anyone, through any route. So every file missed by this bug is stranded for the lifetime of the deployment.

The failure is silent at every layer. deleteCodeEnvFile treats 404 as already-gone, so the client concludes success and clears its state. The service logs nothing unusual, because a 404 on an unrouted path is not an error condition. The operator sees only a disk slowly filling with no attribution.

It hides the real signal. Because the client believes deletion succeeded, there is no retry backlog, no error metric, and nothing to alert on. We only found it by measuring the bucket.

4. Why this is not a deployment or configuration problem

We checked this specifically before filing, because "your setup is wrong" is the obvious first response.

The route has never existed. Searching the entire upstream history of service/src/service/router.ts for any commit that added or removed a DELETE on /sessions/:session_id/objects/... returns nothing. router.delete('/files/:session_id/:fileId') has been present since the initial public release. So this is not a breaking change on the interpreter side that the client is lagging behind — the client has never targeted a route that existed.

Nothing in our configuration can influence the path. The only Code Interpreter settings on the LibreChat side are LIBRECHAT_CODE_BASEURL, CODEAPI_AUTH_PROVIDER, CODEAPI_JWT_*. The base URL is correct and demonstrably working — executions, uploads and downloads against the same base URL all succeed. The delete path is hardcoded relative to it.

It is not a fork divergence. We compared our fork of the interpreter against upstream LibreChat-AI/code-interpreter (formerly ClickHouse/code-interpreter): service/src/service/router.ts is byte-identical. Both expose DELETE only at /files/:session_id/:fileId. Independently confirmed line-for-line by a second reviewer.

It is not proxying or networking. The 404 was reproduced from on the host, against 127.0.0.1:3112, with no load balancer or reverse proxy in the path. The control call to /v1/files/... from the same shell returns 403, so the token, the router and sessionAuth are all functioning.

5. Suggested fix

Three changes are needed. The first alone will not fix deletion.

1. Point deleteCodeEnvFile at the route that exists (danny-avila/LibreChat):

DELETE /v1/files/:session_id/:fileId?kind=<kind>&id=<id>

Query parameters are unchanged — sessionAuth reads kind/id/version from the query string on both routes.

2. Reconcile the retention window with the auth TTL (danny-avila/LibreChat). A 30-day default retention cannot delete anything from a service whose delete authorisation expires at 24 hours. Either the sweep has to run inside the auth window, or the two defaults need to be made aware of each other. As shipped they are mutually incompatible.

3. Provide a fallback when the session key has expired (LibreChat-AI/code-interpreter). Today an object whose session:<id> key has lapsed cannot be read, used as an execution input, or deleted — by anyone, through any route. There is no independent garbage collection; the service assumes an operator-configured bucket lifecycle policy, which is not part of the shipped defaults. That means any delete the client misses is permanent, and the interpreter offers no way to recover the space. An internal reaper, or an authenticated admin delete that does not depend on the session key, would close it.

Worth considering alongside

Do not treat 404 as unconditionally terminal. It currently masks exactly this class of bug. A 404 from a route that does not exist and a 404 from an object that is already gone are indistinguishable to the caller, and the first should be loud.

Give the sweep a give-up or backoff. packages/api/src/files/sweep.ts has no retry cap, so a permanently-undeletable file is refetched and retried every hour for the life of the deployment. On our production instance that is ~500 futile DELETEs per day, growing with every file that fails.

PR #15316 is not related. That fix corrected the tenant namespace on these requests, and it is deployed here — but the requests 404 before authorisation is reached, so it could not have affected deletion in either direction. We initially assumed the two were connected; they are not.

Version Information

6. Environment

• LibreChat: fork of danny-avila/LibreChat, including PR 🎫 fix: Exclude System Tenant Sentinel from Code API JWT Claims #15316 (__SYSTEM__ tenant sentinel fix)
• Code Interpreter: LibreChat-AI/code-interpreter, main (the repo the README still calls ClickHouse/code-interpreter — the org was transferred)
• Auth: CODEAPI_AUTH_PROVIDER=librechat-jwt, EdDSA, single-tenant namespace configured
• Verified on two independent deployments, in two Azure tenants, with different topologies — a single VM with a local MinIO container, and a load-balanced two-node fleet with a shared MinIO node

Steps to Reproduce

Reproduction

Against a live deployment with a valid JWT, using deliberately non-existent identifiers so the only variable is the route:

# the path deleteCodeEnvFile uses
curl -s -o /dev/null -w '%{http_code}\n' -X DELETE \
  "http://<codeapi>:3112/v1/sessions/aaaaaaaaaaaaaaaaaaaaa/objects/bbbbbbbbbbbbbbbbbbbbb?kind=user&id=zz" \
  -H "Authorization: Bearer $TOKEN"
# -> 404

# the route the service actually exposes
curl -s -o /dev/null -w '%{http_code}\n' -X DELETE \
  "http://<codeapi>:3112/v1/files/aaaaaaaaaaaaaaaaaaaaa/bbbbbbbbbbbbbbbbbbbbb?kind=user&id=zz" \
  -H "Authorization: Bearer $TOKEN"
# -> 403

403 on the second call is the correct response — the route exists and sessionAuth rejects a session id that was never registered. 404 on the first means no handler is mounted for that method and path.

What browsers are you seeing the problem on?

No response

Relevant log output

Logs and outputs are provided in the description

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct