Skip to content

lifespan() hangs before yield — server never starts accepting connections (aiocron.crontab call never returns) #1235

Description

@khaagner

Summary

yente serve completes its startup sequence in the logs (index check, Index update complete., Released lock ...) but the ASGI server never actually starts accepting connections — no "Uvicorn running on ..." log line ever appears, and every request (including curl from inside the same container) gets connection refused/reset indefinitely.

Environment

  • Image: ghcr.io/opensanctions/yente:5.5.0 (also reproduced on 5.4.0)
  • Python 3.12 (per traceback paths: /opt/venv/lib/python3.12/site-packages/...)
  • Deployment: Docker Compose, single-node Elasticsearch 9.0.1 backend (discovery.type=single-node), YENTE_INDEX_URL pointing at it over the compose network
  • Manifest: a small custom dataset scope (sanctions, peps, debarment, crime, wanted) — not the default manifest, but I don't think that's relevant given where the hang is (see below)

Root cause (traced into source)

yente/app.py's lifespan():

@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
    await warm_up()
    log.info(
        "Setting up background refresh",
        crontab=settings.CRONTAB,
        auto_reindex=settings.AUTO_REINDEX,
    )
    settings.CRON = aiocron.crontab(settings.CRONTAB, func=refresh_catalog_cron_task)
    # Local handle keeps the cron alive for the lifetime of the lifespan context.
    _metrics_cron = aiocron.crontab("* * * * *", func=update_metrics_task)
    yield
    await close_provider()

"Setting up background refresh" is reliably the last log line ever emitted. yield — the point where control returns to uvicorn and the server starts accepting connections — is never reached. That places the hang inside one of the two aiocron.crontab(...) calls between the log line and yield.

Confirmed the second cron (update_metrics_task, "* * * * *") has no way to disable it via settings — setting YENTE_AUTO_REINDEX=false only skips the reindex-on-startup path and the first cron's underlying work; the app still hangs, just immediately after the same log line (since the second aiocron.crontab() call still runs unconditionally).

What I ruled out before concluding this is app-level, not environment-level

  • Elasticsearch dependency: reported healthy (Docker healthcheck), reachable from inside the yente container at the time of the hang.
  • Resource starvation: yente process was using ~0.15% CPU and ~360MiB of a 6.7GiB memory limit — essentially idle, not doing work or swapping.
  • Stuck container/volume state: reproduced identically after docker restart, a full docker compose up --force-recreate, and a complete wsl --shutdown (full Docker Desktop WSL2 VM reset — genuinely clean boot of the whole stack).
  • A boot race against Elasticsearch: one run did show yente's container starting ~11s before Elasticsearch's HTTP port was actually bound (despite depends_on: condition: service_healthy), so I specifically retested with Elasticsearch already confirmed healthy for 6+ minutes beforehand — still hung, with a completely clean log (no connection errors at all) at the exact same point.
  • Version-specific regression: reproduced identically on both 5.4.0 and 5.5.0.

Steps to reproduce

services:
  elastic:
    image: docker.elastic.co/elasticsearch/elasticsearch:9.0.1
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
    healthcheck:
      test: ["CMD-SHELL", "curl -sf http://localhost:9200/_cluster/health || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 10
      start_period: 20s

  opensanctions:
    image: ghcr.io/opensanctions/yente:5.5.0
    environment:
      - YENTE_INDEX_URL=http://elastic:9200
    ports:
      - "8080:8080"
    depends_on:
      elastic:
        condition: service_healthy

docker compose up -d, wait for indexing to complete in the logs, then curl http://localhost:8080/healthz — connection refused/reset, indefinitely.

Expected behavior

yield in lifespan() returns control, uvicorn logs "Uvicorn running on http://0.0.0.0:8080", and the server starts accepting connections.

Additional notes

aiocron is fairly lightly maintained and has had friction against changing asyncio event-loop APIs across Python versions historically — worth checking whether aiocron.crontab()'s internal asyncio.get_event_loop() (or similar) call is doing something that silently hangs under Python 3.12's asyncio runtime, as opposed to raising.

Happy to provide full container logs or test a patch if useful.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions