Skip to content

Commit 349725d

Browse files
authored
Fix/backend bugs and unused config (#1498)
1 parent fd8ff17 commit 349725d

129 files changed

Lines changed: 4059 additions & 2961 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎AGENTS.md‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,11 @@ Documentation is built with Sphinx from `docs/` (`pip3 install -e '.[docs]'`, th
4242
`make -C docs html SPHINXOPTS="-W --keep-going"`, see [docs/README.md](docs/README.md)). Pull
4343
requests that touch the docs must build without warnings: CI runs that same command.
4444

45+
Use the Python interpreter that has this checkout installed in editable mode (`pip show
46+
lithops` shows the *Editable project location*). Before reporting a change as done, run
47+
`ruff check .` and the localhost test suite. Run long jobs (the full test suite, Docker or
48+
runtime builds, docs builds) in the background.
49+
4550
## Repository map
4651

4752
| Path | Content |
@@ -75,6 +80,9 @@ requests that touch the docs must build without warnings: CI runs that same comm
7580
2. **Every backend looks the same to the core.** Backend-specific code stays in its backend
7681
package; the executor, invoker, job and worker code must not special-case a backend by
7782
name when an interface method or a config value can express it.
83+
Backends are self-contained: a new backend starts as a copy of a similar one, and logic
84+
may be duplicated between backends. Do not refactor backends into shared helpers or base
85+
classes to remove that duplication.
7886
3. **The worker must tolerate the environment it runs in.** Worker code runs in minimal
7987
runtimes (Lambda, containers, VMs, macOS/Windows localhost). Imports of optional
8088
dependencies stay lazy, and a failure in the user's function is reported in its status
@@ -114,8 +122,8 @@ requests that touch the docs must build without warnings: CI runs that same comm
114122
## Safety
115123

116124
- Do not run tests, examples or `lithops` CLI commands against real cloud backends (AWS, GCP,
117-
Azure, IBM Cloud, Aliyun, Oracle, Kubernetes clusters) and do not build or push runtime
118-
images without the maintainer's explicit approval: it creates billable resources and uses
119-
real credentials.
125+
Azure, IBM Cloud, Aliyun, Oracle, Kubernetes clusters), do not build or push runtime images,
126+
and do not publish content without the maintainer's explicit approval: it creates billable
127+
resources and uses real credentials.
120128
- Never commit credentials or configuration files (`~/.lithops/config`, `.lithops_config`).
121129
- Do not commit or push unless asked to.

‎CHANGELOG.md‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,15 +4,24 @@
44

55
### Changed
66
- [Core] A storage or monitoring service at a loopback address is refused for serverless workers, which cannot reach it; standalone and Singularity workers get a warning.
7-
- [Core] `RetryingFunctionExecutor` lists a bucket, prefix or directory before invoking, so that each object is retried on its own, and rejects `obj_chunk_size` and `obj_chunk_number`.
7+
- [Core] `wait()` timeouts no longer arm a SIGALRM: they are checked between polls, work from any thread and leave the caller's signal handler and alarm alone.
8+
- [Core] An `obj_chunk_size` no longer than `obj_newline` is raised to one byte more.
9+
- [Storage] Glob patterns in the input objects work on every storage backend: a key with `*` is a pattern, as it already was on AWS S3 and IBM COS, also from a Windows client.
10+
- [CLI] Times are shown in the local time zone without `pytz` and `tzlocal`, which are no longer used.
11+
- [Standalone] `lithops worker list` shows the last error of each worker and how many of its processes are busy.
12+
- [Knative] Updated to current Knative Serving (tested with 1.23): one container per call in flight with the current autoscaling annotations, `kubecfg_context`, `namespace` and `runtime_arch` keys, the Contour networking layer, cluster-local calls from inside the cluster, a deploy that updates an existing service and reports why its revision fails, and a default runtime that runs as a non-root user.
813

914
### Fixed
10-
- [Core] Chaining, cleanup, invocation errors, generator inputs, empty `fs`, `wait()` timeouts (threads, SIGALRM, `ANY_COMPLETED`), argument binding and `RetryingFunctionExecutor` retries and percentage waits.
15+
- [Core] Chaining, cleanup, invocation errors, generator inputs, empty `fs`, `wait()` with `ANY_COMPLETED`, argument binding, and `RetryingFunctionExecutor` retries and percentage waits; a failed object, or chunk of one, is retried on its own.
1116
- [Worker] Partitioned object streams (lost, duplicated or cut rows, rows longer than 128 KB, multi-byte `obj_newline`, `readline()`, iteration), module discovery of nested functions, a parameter named `future`, and `extra_env` restoring the previous environment.
12-
- [Localhost] Absolute interpreter paths are accepted as runtime; storage listing is a literal prefix match including dotfiles, and open-ended byte ranges are supported.
13-
- [Storage] `CloudFileProxy` path handling (`isfile`, `listdir`, `exists`, leading `/`), double `close()`, and `r+` mode writing back.
14-
- [Monitoring] Worker tokens of jobs dropped by `wait()`, replaced monitors and restarted invokers are handed back once, a held `__end__` is no longer replaced by an `__init__`, and the workers of a job that owes no token are forgotten instead of kept for the life of the executor.
17+
- [Localhost] An absolute interpreter path with any character, such as `@` or a space, is accepted as runtime; storage listing is a literal prefix match including dotfiles, and open-ended byte ranges are supported.
18+
- [Storage] `CloudFileProxy` path handling (`isfile`, `listdir`, `exists`, leading `/`), double `close()`, and `r+` mode writing back; Swift `generate_bucket_name`, Aliyun and Oracle listings past 1,000 objects, and open-ended byte ranges on GCS, Azure and Aliyun without modifying the caller's `extra_get_args`; glob patterns outside the bucket root on AWS S3 and IBM COS; Swift listings return `Key` and `Size` like the other backends and go past the first page.
19+
- [Serverless] Oracle Functions reports a missing `oracle_f` section, an Aliyun zip runtime builds without Docker, and Kubernetes no longer waits for ever on a runtime image it cannot pull or a master pod that does not start.
20+
- [Standalone] A failed `init()` on AWS EC2, GCP Compute Engine and IBM VPC now deletes every resource it created, including those whose tagging, attaching or rules failed; IBM VPC no longer uses the deprecated `code` of SDK errors; the `vm` backend loads its documented configuration, and Azure `request_spot_instances` creates Spot worker VMs (now `False` by default: B-series sizes, like the default worker, do not support Spot).
21+
- [Monitoring] Worker tokens of jobs dropped by `wait()`, replaced monitors and restarted invokers are handed back once, a held `__end__` is no longer replaced by an `__init__`, and the workers of a job that owes no token are forgotten instead of kept for as long as the job monitor runs; the shared RabbitMQ connection of a worker is closed at exit.
1522
- [Concurrent] `wait()` and `as_completed()` accept generators.
23+
- [Core] An executor that fails to be created no longer logs a misleading `job_monitor` error at exit.
24+
- [Knative] Calls longer than 5 minutes are no longer cut off by Knative, a load balancer ingress is reached at its own port and may be a hostname, `https` service URLs work, a failed call is reported instead of being invoked again for ever, and a service that never gets ready raises an error.
1625

1726
## [v3.8.0]
1827

‎CLAUDE.md‎

Lines changed: 0 additions & 18 deletions
This file was deleted.

‎CONTRIBUTING.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,9 @@ not exercised in CI, so mention in your pull request which backend you tested on
7676
- **Backends:** backend-specific code stays in its package (`lithops/serverless/backends/`,
7777
`lithops/standalone/backends/`, `lithops/storage/backends/`). Import the provider SDKs
7878
there, not in the core modules, so Lithops keeps working without the extras installed.
79+
Each backend is self-contained: to implement a new one, start from the code of a similar
80+
backend and adapt it. Duplicating logic between backends is fine; do not factor it out into
81+
shared modules or base classes, so that a change in one backend cannot break another.
7982
- **Configuration keys:** document new or changed keys in `config/config_template.yaml` and
8083
in the backend's page under `docs/source/compute_config/` or `docs/source/storage_config/`
8184
(general `lithops` keys in `docs/source/lithops_config_keys.csv`).

‎config/config_template.yaml‎

Lines changed: 15 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,7 @@
118118
#master_instance_type: t3.micro
119119
#worker_instance_type: t3.medium
120120
#delete_on_dismantle: True
121+
#worker_create_retries: 2 # Attempts to bring up each worker VM (create/reuse modes). Minimum 1
121122
#max_workers: 100
122123
#worker_processes: AUTO # Default: number of CPUs of the worker VM
123124
#ebs_volumes: # Optional list of EBS volume mappings
@@ -165,7 +166,6 @@
165166
#min_workers: 0
166167
#max_workers: 100
167168
#worker_processes: 1
168-
#invoke_pool_threads: 100
169169
#docker_server: pkg.dev # Marker for Artifact Registry image names
170170
#artifact_registry_repository: lithops
171171
#runtime_include_function: False
@@ -192,6 +192,8 @@
192192
#ssh_key_filename: ~/.ssh/id_rsa
193193
#request_spot_instances: False
194194
#delete_on_dismantle: True
195+
#worker_create_retries: 2 # Attempts to bring up each worker VM (create/reuse modes). Minimum 1
196+
#worker_public_ip: False # Also give worker VMs an external IP (default: Cloud NAT only)
195197
#max_workers: 100
196198
#worker_processes: AUTO # Default: number of CPUs of the worker VM
197199

@@ -245,8 +247,9 @@
245247
#ssh_username: ubuntu
246248
#ssh_password: <RANDOM_IF_NOT_PROVIDED>
247249
#ssh_key_filename: ~/.ssh/id_rsa
248-
#request_spot_instances: True
250+
#request_spot_instances: False # Spot worker VMs; B-series sizes do not support Spot
249251
#delete_on_dismantle: False # `True` is not yet supported on Azure
252+
#worker_create_retries: 2 # Attempts to bring up each worker VM (create/reuse modes). Minimum 1
250253
#max_workers: 100
251254
#worker_processes: AUTO
252255

@@ -319,6 +322,7 @@
319322
#master_profile_name: cx2-2x4
320323
#verify_resources: True
321324
#delete_on_dismantle: True
325+
#worker_create_retries: 2 # Attempts to bring up each worker VM (create/reuse modes). Minimum 1
322326
#max_workers: 100
323327
#worker_processes: AUTO
324328
#singlesocket: False
@@ -435,21 +439,21 @@
435439

436440
#knative:
437441
#kubecfg_path: <PATH_TO_KUBECONFIG>
438-
#networking_layer: kourier # One of: kourier, istio
439-
#ingress_endpoint: <INGRESS_URL>
442+
#kubecfg_context: <CONTEXT_NAME>
443+
#namespace: <NAMESPACE> # Defaults to the namespace of the context, or 'default'
444+
#networking_layer: kourier # One of: kourier, istio, contour
445+
#ingress_endpoint: <INGRESS_URL> # Found automatically when not set
440446
#docker_server: docker.io
441447
#docker_user: <REGISTRY_USERNAME>
442448
#docker_password: <REGISTRY_TOKEN>
443449
#docker_namespace: <REGISTRY_NAMESPACE>
444-
#git_url: <GIT_REPO_URL>
445-
#git_rev: <GIT_REVISION>
446450
#runtime: <RUNTIME_NAME>
447451
#runtime_cpu: 1 # vCPU
448452
#runtime_memory: 512 # MB
449-
#runtime_timeout: 600 # Seconds
453+
#runtime_timeout: 600 # Seconds; at most Knative's max-revision-timeout-seconds
454+
#runtime_arch: <amd64|arm64> # Defaults to cluster nodes; falls back to amd64
450455
#max_workers: 100
451456
#worker_processes: 1
452-
#invoke_pool_threads: 100
453457

454458
#openwhisk:
455459
#endpoint: <OW_ENDPOINT> # Mandatory
@@ -590,7 +594,7 @@
590594
#prometheus:
591595
#gateway: http://localhost:9091 # Pushgateway base URL. Default: http://localhost:9091
592596
#job: lithops # Pushgateway job label. Default: lithops
593-
#instance: <HOSTNAME> # Pushgateway instance label. Default: the hostname
597+
#instance: <HOSTNAME> # Pushgateway instance label. Default: $LITHOPS_TELEMETRY_INSTANCE, else the hostname
594598
#username: <USERNAME> # Basic auth user, when the gateway needs one
595599
#password: <PASSWORD> # Basic auth password, when the gateway needs one
596600
#timeout: 10 # Push timeout in seconds. Default: 10
@@ -603,6 +607,7 @@
603607
#endpoint: http://localhost:4318 # Collector endpoint. Default: http://localhost:4318
604608
#protocol: http/protobuf # One of: http/protobuf, grpc. Default: http/protobuf
605609
#service_name: lithops # OTel service.name. Default: lithops
606-
#instance: <HOSTNAME> # OTel host.name. Default: the hostname
610+
#instance: <HOSTNAME> # OTel service.instance.id and host.name. Default: $LITHOPS_TELEMETRY_INSTANCE, else the hostname
607611
#headers: {} # Extra headers, e.g. authentication
608612
#timeout: 10 # Export timeout in seconds. Default: 10
613+
#export_interval: 3600 # Seconds between the OTel SDK's own exports, a fallback to telemetry_interval. Default: 3600

‎docs/source/api_futures.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ Primary Executors
1818
* **RetryingFunctionExecutor** (``lithops.RetryingFunctionExecutor()``):
1919
A robust wrapper around ``FunctionExecutor`` that transparently handles retries on failed tasks.
2020
It supports the features of ``FunctionExecutor`` with added automatic retry logic, improving fault tolerance and reliability for unstable or transient-failure-prone environments.
21-
A failed call is retried with its own input: a bucket, prefix or directory passed to a function that processes objects is retried one object at a time, and splitting objects into chunks (``obj_chunk_size``, ``obj_chunk_number``) is not supported.
21+
A failed call is retried with its own input: a bucket, prefix or directory passed to a function that processes objects is retried one object, or one chunk of an object, at a time.
2222

2323
Secondary Executors
2424
-------------------

‎docs/source/cli.rst‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,9 @@ Open an ssh connection to the master VM (Only available for standalone backends)
111111
``lithops worker list``
112112
~~~~~~~~~~~~~~~~~~~~~~~
113113

114-
Lists the available workers in the master VM (Only available for standalone backends)
114+
Lists the available workers in the master VM (Only available for standalone backends). For each
115+
worker it shows its status, how many of its processes are running a task (*Busy*, for example
116+
``1/4``), the last error it reported, and the time left before it stops itself (*TTD*).
115117

116118
+------------------------+----------------------------------------------------------------+
117119
| Parameter | Description |

‎docs/source/compute_config/aws_ec2.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,7 @@ In summary, you can use one of the following settings:
123123
|aws_ec2 | master_instance_type | t2.micro | no | Profile name for the master VM |
124124
|aws_ec2 | worker_instance_type | t2.medium | no | Profile name for the worker VMs |
125125
|aws_ec2 | delete_on_dismantle | True | no | Delete the worker VMs when they are stopped. Master VM is never deleted when stopped |
126+
|aws_ec2 | worker_create_retries | 2 | no | Attempts to bring up each worker VM in create and reuse modes. A worker VM that is not reachable over SSH in time is deleted and created again until this many attempts have been made; then that worker fails and the job runs on the workers that came up. Minimum 1 |
126127
|aws_ec2 | max_workers | 100 | no | Max number of workers per `FunctionExecutor()`|
127128
|aws_ec2 | worker_processes | AUTO | no | Number of parallel Lithops processes in a worker. This is used to parallelize function activations within the worker. By default it detects the amount of CPUs in the `worker_instance_type` VM|
128129
|aws_ec2 | runtime | python3 | no | Runtime name to run the functions. Can be a container image name. If not set Lithops will use the default python3 interpreter of the VM |

‎docs/source/compute_config/azure_containers.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,8 @@ az provider register --namespace Microsoft.OperationalInsights --wait
5151
az containerapp env create --name lithops --resource-group LithopsResourceGroup --location westeurope
5252
```
5353

54+
Lithops deploys its container apps in the `region` of the configuration, so create the environment in that same region (or set `region` to the region of the environment). If Lithops cannot find the environment, the runtime deployment fails at once with an error naming the environment, resource group and region it looked for.
55+
5456
## Configuration
5557

5658
1. Access the [Storage Account](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts)

‎docs/source/compute_config/azure_vms.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,9 @@ Edit your Lithops config and add the relevant keys:
8989
|azure_vms | ssh_key_filename | ~/.ssh/id_rsa | no | Path to the ssh key file provided to access the VPC. It will use the default path if not provided |
9090
|azure_vms | master_instance_type | Standard_B1s | no | Profile name for the master VM |
9191
|azure_vms | worker_instance_type | Standard_B2s | no | Profile name for the worker VMs |
92+
|azure_vms | request_spot_instances | False | no | Create the worker VMs as Spot VMs. They are evicted when Azure needs the capacity back. B-series sizes, like the default `worker_instance_type`, do not support Spot |
9293
|azure_vms | delete_on_dismantle | False | no | Delete the worker VMs when they are stopped. Master VM is never deleted when stopped. `True` is NOT YET SUPPORTED |
94+
|azure_vms | worker_create_retries | 2 | no | Attempts to bring up each worker VM in create and reuse modes. A worker VM that is not reachable over SSH in time is deleted and created again until this many attempts have been made; then that worker fails and the job runs on the workers that came up. Minimum 1 |
9395
|azure_vms | max_workers | 100 | no | Max number of workers per `FunctionExecutor()`|
9496
|azure_vms | worker_processes | AUTO | no | Number of parallel Lithops processes in a worker. This is used to parallelize function activations within the worker. By default it detects the amount of CPUs in the `worker_instance_type` VM|
9597
|azure_vms | runtime | python3 | no | Runtime name to run the functions. Can be a container image name. If not set Lithops will use the default python3 interpreter of the VM |

0 commit comments

Comments
 (0)