pdf-document-layout-analysis-async

Dependencies

• Docker Desktop 4.25.0 install link

Docker containers

A redis server is needed to use the service asynchronously. For that matter, it can be used the command make start-test that has a built-in redis server.

Containers with make start

Containers with make start-test

How to use it

1. Send PDF to extract

   curl -X POST -F 'file=@/PATH/TO/PDF/pdf_name.pdf' localhost:5051/async_extraction/[tenant_name]

2. Add extraction task

To add an extraction task, a message should be sent to a queue.

Python code:

queue = RedisSMQ(host=[redis host], port=[redis port], qname='segmentation_tasks', quiet=True)
message_json = '{"tenant": "tenant_name", "task": "segmentation", "params": {"filename": "pdf_file_name.pdf"}}'
message = queue.sendMessage(message_json).exceptions(False).execute()

3. Get paragraphs

When the segmentation task is done, a message is placed in the results queue:

queue = RedisSMQ(host=[redis host], port=[redis port], qname='segmentation_results', quiet=True)
results_message = queue.receiveMessage().exceptions(False).execute()

# The message.message contains the following information:
# {"tenant": "tenant_name", 
# "task": "pdf_name.pdf", 
# "success": true, 
# "error_message": "", 
# "data_url": "http://localhost:5051/get_paragraphs/[tenant_name]/[pdf_name]"
# "file_url": "http://localhost:5051/get_xml/[tenant_name]/[pdf_name]"
# }


curl -X GET http://localhost:5051/get_paragraphs/[tenant_name]/[pdf_name]
curl -X GET http://localhost:5051/get_xml/[tenant_name]/[pdf_name]

or in python

requests.get(results_message.data_url)
requests.get(results_message.file_url)

HTTP server

The container HTTP server is coded using Python 3.9 and uses the FastApi web framework.

If the service is running, the end point definitions can be founded in the following url:

http://localhost:5051/docs

The end points code can be founded inside the file app.py.

The errors are reported to the file docker_volume/service.log, if the configuration is not changed (see Get service logs)

Queue processor

The container Queue processor is coded using Python 3.9, and it is on charge of the communication with redis.

The code can be founded in the file QueueProcessor.py and it uses the library RedisSMQ to interact with the redis queues.

Service configuration

Some parameters could be configured using environment variables. If a configuration is not provided, the defaults values are used.

Default parameters:

REDIS_HOST=redis_paragraphs
REDIS_PORT=6379
POSTGRES_HOST=postgres_paragraphs
POSTGRES_PORT=28017
SERVICE_HOST=http://127.0.0.1
SERVICE_PORT=5051

Translation uses Ollama cloud. Set OLLAMA_API_KEY (create one at https://ollama.com/settings/keys). The base URL and model can be overridden:

OLLAMA_BASE_URL=https://ollama.com/v1
OLLAMA_API_KEY=
TRANSLATION_MODEL=deepseek-v4.1-flash:cloud
MAX_TRANSLATE_CONCURRENCY=3
MAX_TRANSLATE_RETRIES=3

MAX_TRANSLATE_CONCURRENCY bounds how many POST /translate requests reach Ollama at the same time. Excess requests wait for a slot instead of being rejected.

MAX_TRANSLATE_RETRIES is the number of extra attempts when an Ollama call fails with a network error, HTTP 429, or 5xx. Retries use exponential backoff (1s, 2s, 4s, … capped at 10s). Non-retryable failures (e.g. 4xx other than 429, malformed responses) fail immediately.

Materials (segmentation XML, paragraphs, OCR results) and uploaded PDFs are kept after being fetched and swept away after a retention window:

MATERIAL_RETENTION_HOURS=24
MATERIAL_SWEEP_INTERVAL_MINUTES=10

MATERIAL_RETENTION_HOURS is how many hours materials and uploaded PDFs are kept before a sweep deletes them; 0 (or less) disables sweeping entirely. MATERIAL_SWEEP_INTERVAL_MINUTES bounds how often the sweep runs: at most once per interval when a material is served, plus a background sweep in the API. Fetching a material does not extend its retention.

Supported translation languages

Translation requests identify languages with the Uwazi settings language keys, matched case-insensitively. Supported codes:

en es fr de it pt nl pl ru tr ja ko vi ar hi   zh-Hans zh-Hant   in

in is Indonesian. Requests with any other code (zh, cs, el, he, fa, ro, uk, …) are rejected: POST /translate answers 400 Language <code> not supported, and queue messages come back with success: false and an Unsupported language(s): <code> error message, so Uwazi never gets a translation made under the wrong language pair.

Deploy to Google Cloud Run

The service can be deployed to Google Cloud Run with GPU support. Make sure you are authenticated with your own Google account (gcloud auth login) and that billing is enabled on your project before running the command below.

gcloud run deploy pdf-document-layout-analysis \
  --image ghcr.io/huridocs/pdf-document-layout-analysis:0.0.37 \
  --region europe-west4 \
  --port 8080 \
  --command "gunicorn" \
  --args="-k,uvicorn.workers.UvicornWorker,--chdir,./src,drivers.rest.app:app,--bind,0.0.0.0:8080,--timeout,10000" \
  --gpu 1 --gpu-type nvidia-l4 \
  --cpu 4 --memory 16Gi \
  --no-cpu-throttling \
  --no-gpu-zonal-redundancy \
  --max-instances 1 \
  --concurrency 1 \
  --timeout 900 \
  --no-allow-unauthenticated

Notes:

• --no-allow-unauthenticated requires callers to authenticate with a Google account that has access to the service. Use gcloud run services add-iam-policy-binding to grant access.
• The service depends on a Redis server and a PostgreSQL database. Configure them via the environment variables described in Service configuration (e.g. with --set-env-vars).

Set up environment for development

It works with Python 3.12 [install] (https://runnable.com/docker/getting-started/)

make install_venv

Execute tests

make test