Dask Distributed: Basic remoting / serverless function example by amotl · Pull Request #1676 · crate/cratedb-examples

amotl · 2026-05-02T13:58:37Z

About

Demonstrate scheduling CTK I/O routines on Dask clusters, effectively including important details of remoting and serverless.

References

I/O: Add option to submit workloads to remote Dask clusters cratedb-toolkit#788

coderabbitai · 2026-05-02T13:58:45Z

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c50512e2-6647-4808-a188-8db81ed10116

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding a Dask Distributed framework example demonstrating remoting and serverless functions for CrateDB.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The pull request description accurately describes the changeset, explaining the intent to demonstrate scheduling CTK I/O routines on Dask clusters with details about remoting and serverless concepts.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch dask-distributed

Tip

💬 Introducing Slack Agent: Turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

Generate code and open pull requests
Plan features and break down work
Investigate incidents and troubleshoot customer tickets together
Automate recurring tasks and respond to alerts with triggers
Summarize progress and report instantly

Built for teams:

Shared memory across your entire org—no repeating context
Per-thread sandboxes to safely plan and execute work
Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get your free trial and get 200 agent minutes per Slack user (a $50 value).

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

amotl · 2026-05-02T16:06:12Z

+# Pre-flight requests to fix Dask accounting.
+docker compose run --rm --quiet-build --env="mode=ping" verify


Problem: When not invoking a request to this API first, the first task submitted is not recorded.
Action: Report this to Dask?

We've reported the observation to the upstream project.

High latency can cause get_task_stream() to miss tasks dask/distributed#9253

amotl · 2026-05-02T18:20:55Z

+  export-to-deltalake:
+    build:
+      context: .
+    command: >
+      ctk --debug save
+        "crate://crate:crate@cratedb/sys/summits"
+        "file+deltalake:///var/delta/summits?mode=overwrite"
+    environment:
+      DASK_SCHEDULER_ADDRESS: "tcp://dask-scheduler:8786"
+    profiles:
+      - tasks


While everything else is merely boilerplate, this is the actual Python workload that is diverted to the Dask compute cluster within this demonstration scenario. The task is submitted to the Dask scheduler, which will schedule it onto a worker.

Of course, that will not work magically, but relevant preparations aren't big either:

I/O: Add option to submit workloads to remote Dask clusters cratedb-toolkit#788

amotl · 2026-05-02T18:22:23Z

+  dask-scheduler:
+    build:
+      context: .
+    command: dask scheduler --dashboard
+    ports:
+      - "8786:8786"
+      - "8787:8787"
+    healthcheck:
+      test: ["CMD", "wget", "--quiet", "-O", "-", "http://localhost:8787/health"]
+      start_period: 3s
+      interval: 1.5s
+      retries: 30
+      timeout: 30s
+
+  dask-worker:
+    build:
+      context: .
+    command: dask worker --dashboard --dashboard-address ":8789"
+    environment:
+      DASK_SCHEDULER_ADDRESS: "tcp://dask-scheduler:8786"
+    depends_on:
+      dask-scheduler:
+        condition: service_healthy
+    healthcheck:
+      test: ["CMD", "wget", "--quiet", "-O", "-", "http://localhost:8789/health"]
+      start_period: 3s
+      interval: 1.5s
+      retries: 30
+      timeout: 30s


The Dask scheduler will receive tasks submitted by clients, and will schedule them onto Dask workers. That's it.

coderabbitai

Actionable comments posted: 1

♻️ Duplicate comments (1)

framework/dask-distributed/Dockerfile (1)
20-20: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Use COPY instead of ADD for requirements.txt.

Line 20 should use COPY; ADD is unnecessary here and triggers DL3020.
Proposed fix
-ADD requirements.txt /
+COPY requirements.txt /

🧹 Nitpick comments (2)

framework/dask-distributed/compose.yml (1)
55-56: ⚡ Quick win

Avoid hardcoding credentials in the command URI.

Embedding credentials in-line creates secret-scanner noise and encourages insecure copy/paste patterns. Prefer injecting the source DSN via environment variable.
Proposed refactor
     command: >
       ctk --debug save
-        "crate://crate:crate@cratedb/sys/summits"
+        "${CRATE_SOURCE_DSN:-crate://crate:crate@cratedb/sys/summits}"
         "file+deltalake:///var/delta/summits?mode=overwrite"
framework/dask-distributed/Dockerfile (1)

1-3: Pin image digests to ensure reproducible builds across CI runs and architectures.

Both docker.io/daskdev/dask:latest-py3.13 (line 3) and ghcr.io/astral-sh/uv:latest (line 14) use floating tags that can resolve to different images over time. Use the format IMAGE:TAG@sha256:MANIFEST_DIGEST instead—obtain the digest via docker manifest inspect IMAGE:TAG. This pins the exact multi-platform manifest, preventing unexpected behavior changes.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 2e216f55-c3ec-4623-a8ec-9642a12dc9d6

📥 Commits

Reviewing files that changed from the base of the PR and between ca2a929 and 3e42a75.

📒 Files selected for processing (7)

.github/dependabot.yml
.github/workflows/framework-dask-distributed.yml
framework/dask-distributed/Dockerfile
framework/dask-distributed/README.md
framework/dask-distributed/compose.yml
framework/dask-distributed/requirements.txt
framework/dask-distributed/test.sh

✅ Files skipped from review due to trivial changes (4)

framework/dask-distributed/requirements.txt
framework/dask-distributed/README.md
framework/dask-distributed/test.sh
.github/dependabot.yml

🚧 Files skipped from review as they are similar to previous changes (1)

.github/workflows/framework-dask-distributed.yml

amotl force-pushed the dask-distributed branch from 2d18292 to 3fd6c86 Compare May 2, 2026 14:40

amotl changed the title ~~Dask Distributed: Basic example~~ Dask Distributed: Basic remoting / serverless function example May 2, 2026

amotl force-pushed the dask-distributed branch 4 times, most recently from ab575d7 to 734bc4b Compare May 2, 2026 15:16

amotl mentioned this pull request May 2, 2026

container re-created when it shouldn't, if using docker-compose run docker/compose#13769

Closed

amotl force-pushed the dask-distributed branch 4 times, most recently from 6bcf8cc to ca2a929 Compare May 2, 2026 16:00

amotl commented May 2, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

amotl mentioned this pull request May 2, 2026

I/O: Add option to submit workloads to remote Dask clusters crate/cratedb-toolkit#788

Merged

amotl force-pushed the dask-distributed branch 3 times, most recently from e6556e0 to 3e42a75 Compare May 2, 2026 16:17

amotl mentioned this pull request May 2, 2026

High latency can cause get_task_stream() to miss tasks dask/distributed#9253

Closed

amotl marked this pull request as ready for review May 2, 2026 17:33

amotl commented May 2, 2026

View reviewed changes

coderabbitai Bot reviewed May 2, 2026

View reviewed changes

Comment thread framework/dask-distributed/compose.yml Outdated

amotl force-pushed the dask-distributed branch from 3e42a75 to 7d5edec Compare May 2, 2026 18:45

amotl force-pushed the dask-distributed branch from 7d5edec to 396d263 Compare May 14, 2026 09:44

Dask Distributed: Basic remoting / serverless function example

396d263

amotl merged commit e587604 into main May 14, 2026
2 checks passed

amotl deleted the dask-distributed branch May 14, 2026 09:47

Navigation Menu

Repository navigation

Dask Distributed: Basic remoting / serverless function example - #1676

Dask Distributed: Basic remoting / serverless function example#1676
amotl merged 1 commit into
mainfrom
dask-distributed

amotl commented May 2, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented May 2, 2026 •

edited

Loading

Review skipped

Uh oh!

amotl May 2, 2026

Uh oh!

amotl May 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

amotl May 2, 2026

Uh oh!

amotl May 2, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		# Pre-flight requests to fix Dask accounting.
		docker compose run --rm --quiet-build --env="mode=ping" verify

Conversation

amotl commented May 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

About

References

Uh oh!

coderabbitai Bot commented May 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

amotl May 2, 2026

Choose a reason for hiding this comment

Uh oh!

amotl May 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

amotl May 2, 2026

Choose a reason for hiding this comment

Uh oh!

amotl May 2, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

amotl commented May 2, 2026 •

edited

Loading

coderabbitai Bot commented May 2, 2026 •

edited

Loading

amotl May 2, 2026 •

edited

Loading