Fix execution, storage, and backend reliability across Lithops

CHANGELOG.md

@@ -3,6 +3,9 @@
 ## [v3.8.1.dev0]
 
 ### Changed
+- [Worker] Start status is sent before launching the JobRunner process.
+- [Core] Dependency debug logs distinguish function modules from input-data modules.
+- [Azure Functions] Default runtime memory is now 512 MB.
 - [Core] A storage or monitoring service at a loopback address is refused for serverless workers, which cannot reach it; standalone and Singularity workers get a warning.
 - [Core] `wait()` timeouts no longer arm a SIGALRM: they are checked between polls, work from any thread and leave the caller's signal handler and alarm alone.
 - [Core] An `obj_chunk_size` no longer than `obj_newline` is raised to one byte more.
@@ -12,17 +15,39 @@
 - [Knative] Updated to current Knative Serving (tested with 1.23): one container per call in flight with the current autoscaling annotations, `kubecfg_context`, `namespace` and `runtime_arch` keys, the Contour networking layer, cluster-local calls from inside the cluster, a deploy that updates an existing service and reports why its revision fails, and a default runtime that runs as a non-root user.
 
 ### Fixed
+- [Code Engine] `clean --all` permanently deletes the project so its name can be reused.
+- [GCP Compute Engine] Cleanup tolerates VMs already deleted by another process.
+- [GCP] Compute Engine infers the project from credentials when omitted; Functions and Cloud Run honour explicit project IDs.
+- [GCP Functions] Cleaning runtimes works when service configuration is missing.
+- [Azure VMs] Fixed network request serialization with newer SDKs and enabled internet access for workers on newly created subnets.
+- [Runtime] Podman logins include the Docker Hub registry explicitly, so runtime builds can authenticate without a default registry in `registries.conf`.
+- [Core] `get_result()` with no futures returns `[]`. `call_async()` chains a list or tuple of futures as one call. `RetryingFunctionExecutor.wait()` shares one timeout across retry rounds and, when it expires, leaves the calls running instead of cancelling them.
 - [Core] Chaining, cleanup, invocation errors, generator inputs, empty `fs`, `wait()` with `ANY_COMPLETED`, argument binding, and `RetryingFunctionExecutor` retries and percentage waits; a failed object, or chunk of one, is retried on its own.
 - [Worker] Partitioned object streams (lost, duplicated or cut rows, rows longer than 128 KB, multi-byte `obj_newline`, `readline()`, iteration), module discovery of nested functions, a parameter named `future`, and `extra_env` restoring the previous environment.
 - [Localhost] An absolute interpreter path with any character, such as `@` or a space, is accepted as runtime; storage listing is a literal prefix match including dotfiles, and open-ended byte ranges are supported.
-- [Storage] `CloudFileProxy` path handling (`isfile`, `listdir`, `exists`, leading `/`), double `close()`, and `r+` mode writing back; Swift `generate_bucket_name`, Aliyun and Oracle listings past 1,000 objects, and open-ended byte ranges on GCS, Azure and Aliyun without modifying the caller's `extra_get_args`; glob patterns outside the bucket root on AWS S3 and IBM COS; Swift listings return `Key` and `Size` like the other backends and go past the first page.
-- [Serverless] Oracle Functions reports a missing `oracle_f` section, an Aliyun zip runtime builds without Docker, and Kubernetes no longer waits for ever on a runtime image it cannot pull or a master pod that does not start.
-- [Standalone] A failed `init()` on AWS EC2, GCP Compute Engine and IBM VPC now deletes every resource it created, including those whose tagging, attaching or rules failed; IBM VPC no longer uses the deprecated `code` of SDK errors; the `vm` backend loads its documented configuration, and Azure `request_spot_instances` creates Spot worker VMs (now `False` by default: B-series sizes, like the default worker, do not support Spot).
+- [Storage] `CloudFileProxy` path handling (`isfile`, `listdir`, `exists`, leading `/`), double `close()`, and `r+` mode writing back; Swift `generate_bucket_name`, Aliyun and Oracle listings past 1,000 objects, and open-ended byte ranges on GCS, Azure and Aliyun without modifying the caller's `extra_get_args`; glob patterns outside the bucket root on AWS S3 and IBM COS; Swift listings return `Key` and `Size` like the other backends and go past the first page. A missing bucket from an S3-compatible listing is `StorageNoSuchKeyError`. Swift uploads and non-404 reads raise, Swift has `head_bucket`, and an empty error page is not the end of a listing. Oracle, Azure and GCP no longer report an arbitrary failure as a missing key or bucket. `CloudFileProxy.exists()` ignores Lithops-internal keys. Swift logs in to current Keystone: a project is scoped by id or by name and domain, the domain defaults to `Default`, and the catalog URL is used as published, including `https`. A container that does not exist is created.
+- [Serverless] Oracle Functions reports a missing `oracle_f` section, an Aliyun zip runtime builds without Docker, and Kubernetes no longer waits for ever on a runtime image it cannot pull or a master pod that does not start. The Kubernetes registry secret uses the Docker config keys. A Knative, GCP Functions or Cloud Run deploy that cannot read runtime metadata deletes the service or function it created, and GCP Functions reads that metadata once. An unknown Knative `kubecfg_context` names the contexts that exist. Waiting for a GCP function or Cloud Run service to be deleted treats only HTTP 404 as gone, and a failed Cloud Run delete is reported.
+- [Standalone] A failed `init()` on AWS EC2, GCP Compute Engine and IBM VPC now deletes every resource it created, including those whose tagging, attaching or rules failed, and IBM VPC floating IPs allocated by that `init()`; IBM VPC no longer uses the deprecated `code` of SDK errors; the `vm` backend loads its documented configuration, and Azure `request_spot_instances` creates Spot worker VMs (now `False` by default: B-series sizes, like the default worker, do not support Spot). AWS EC2 raises a clear error when spot price history is empty.
 - [Monitoring] Worker tokens of jobs dropped by `wait()`, replaced monitors and restarted invokers are handed back once, a held `__end__` is no longer replaced by an `__init__`, and the workers of a job that owes no token are forgotten instead of kept for as long as the job monitor runs; the shared RabbitMQ connection of a worker is closed at exit.
 - [Concurrent] `wait()` and `as_completed()` accept generators.
 - [Core] An executor that fails to be created no longer logs a misleading `job_monitor` error at exit.
+- [Core] The FaaS invoker no longer runs up to two workers beyond `max_workers`. A `wait()` that times out drops the invocations still queued, the same as any other failed wait. `obj_chunk_number` splits an object into exactly that many chunks, or fewer non-empty ones when it is smaller. URL inputs are sized following redirects, an HTTP error raises, and only `Accept-Ranges: bytes` enables partitioning. A local input path that does not exist raises `FileNotFoundError`, and local inputs with no files raise like an empty bucket. A mode section with no body in the config file no longer crashes config loading.
+- [CLI] `lithops runtime update` stores `runtime_timeout` in the runtime metadata, which every later job needed.
+- [Standalone] Cleaning AWS EC2 workers reads every page of instances, so workers past the first 1,000 are stopped too. Cleaning IBM VPC workers, choosing its image and reusing a free floating IP read every page, which stops at 50, as do the VPC, subnet, gateway and SSH key lookups. Cleaning GCP Compute Engine workers reads every page of instances, which stops at 500. `get_result()` on one job no longer cancels the other jobs of the executor, and create mode stops its workers only once none of its jobs is left. Worker VMs whose setup fails, and the workers of a job whose master cannot be set up, are stopped instead of left running. Requests to the master from inside a worker send their body, and an error from the master raises. They wait up to 60 seconds for an answer instead of 1, and as long as the master needs to pull the runtime for its metadata; a POST no longer waits for ever. The `vm` backend runs jobs: it reads `ssh_username` and provides what the master and its budget keeper need. Azure VMs are deallocated when stopped, so they are no longer billed, the home directory follows `ssh_username`, and `delete_on_dismantle` defaults to `False` as documented.
+- [AWS Lambda] Invocations retry connection errors a few times, re-signing each attempt, instead of retrying any error for ever. `lithops runtime update` updates the code, configuration and tags of a deployed function, and the default runtime gets the `user_tags` too. `env_vars` and `user_tags` accept a `{NAME: VALUE}` mapping or a list of `{name, value}`. Layers with more than 50 versions are deleted. Runtimes and layers past the first page of the listing are found by `lithops runtime list` and `clean`, and deleting a runtime whose function is already gone works. A container runtime is looked up by its image tag, so it is found in a repository with more than 100 images.
+- [AWS Batch] Maps over more than 20 calls no longer fail on a fractional `max_workers`. Job definitions use the requested `runtime_memory`, checked against the Fargate combinations, and their names carry the requested version, so `lithops runtime update` and `delete` find them, also past the first page of job definitions. Deleting a runtime whose job definition is already gone works. Job queues and compute environments past the first page are found, and `lithops runtime update` registers a new job definition revision instead of keeping the CPU, roles and image from the first deploy.
+- [OpenWhisk] An invoke rejected with 401 or 404 raises instead of being reported as an activation id, which made `wait()` hang. A failure to read runtime metadata raises that error instead of a `TypeError`. Python 3.13 and 3.14 clients get a default runtime, built from the runtime Dockerfile with `--build-arg PYTHON_VERSION`. The client speaks to current OpenWhisk (tested with `openwhisk/standalone:nightly`): an `http://` endpoint is used as configured, actions and packages past the first page are listed, a package that already exists is reused, deleting a missing action works, and waiting for runtime metadata follows the activation instead of invoking again for ever. A non-blocking invocation opens its own connection and closes it, so the invoker threads do not queue on the requests connection pool. `lithops runtime build` builds for the machine it runs on; `--platform linux/amd64` selects amd64.
+- [Code Engine] Projects, job definitions and config maps past the first page are listed. `clear()` deletes every job run and config map of the executor, a leftover config map is replaced instead of failing every deploy, and waiting for a job run or a job definition deletion times out.
+- [Singularity] Runtime metadata that never arrives raises the intended timeout error.
+- [Kubernetes] Worker memory is compared in megabytes, so a `Mi` request is the same size as the memory it is checked against.
+- [Azure Functions] Fixed app lookup and CLI error reporting, invocation and metadata timeouts, metadata POST requests, per-memory deployments, and runtime lookup by name.
+- [Azure Container Apps] Added deployment progress logs and cleanup on interrupted creation. Fixed authentication error handling, runtime lookup by image, and numeric memory values.
+- [Aliyun FC] The synchronous metadata call waits for `runtime_timeout`. An asynchronous invoke only waits to be accepted. Retries while reading runtime metadata resend the payload instead of an empty body.
+- [Oracle Functions] Calls run on the function of their own runtime and memory, not on the function of the job started last. Functions and applications past the first page are found by `lithops runtime list`, `delete` and `clean`.
+- [GCP Functions] [GCP Cloud Run] An HTTPS invocation waits for `runtime_timeout` instead of giving up after 120 seconds, which is while the call is still running. Functions and services past the first page are found by `lithops runtime list` and `clean`.
 - [Knative] Calls longer than 5 minutes are no longer cut off by Knative, a load balancer ingress is reached at its own port and may be a hostname, `https` service URLs work, a failed call is reported instead of being invoked again for ever, and a service that never gets ready raises an error.
 
+
 ## [v3.8.0]
 
 ### Added

config/config_template.yaml

@@ -140,7 +140,7 @@
 # Google Cloud Platform – credentials shared by all gcp_* sections
 # =============================================================================
 #gcp:
-    #project_name: <GCP_PROJECT_ID>
+    #project_name: <GCP_PROJECT_ID>       # Optional; inferred from credentials when omitted
     #region: <REGION_NAME>                # e.g. us-east1
     #credentials_path: <ABSOLUTE_PATH_TO_SERVICE_ACCOUNT_JSON>
 
@@ -177,7 +177,7 @@
 
 # Google Compute Engine – standalone compute
 #gcp_compute_engine:
-    #project_name: <GCP_PROJECT_ID>       # Falls back to gcp.project_name
+    #project_name: <GCP_PROJECT_ID>       # Falls back to gcp.project_name, then credentials
     #zone: <ZONE_NAME>                    # e.g. us-east1-b. Mandatory
     #region: <REGION_NAME>                # Falls back to gcp.region; derived from zone if omitted
     #service_account: <SERVICE_ACCOUNT_EMAIL>
@@ -211,7 +211,7 @@
     #resource_group: <RESOURCE_GROUP_NAME># Falls back to azure.resource_group
     #region: <LOCATION>                   # Falls back to azure.region
     #runtime: <RUNTIME_NAME>
-    #runtime_memory: 1536                 # MB
+    #runtime_memory: 512                  # MB. Allowed: 512, 2048, 4096; CPU scales with memory
     #runtime_timeout: 300                 # Seconds
     #functions_version: 4
     #trigger: pub/sub                     # One of: pub/sub, https
@@ -532,13 +532,16 @@
     # Any additional key in this section is forwarded to redis.Redis(...)
 
 #swift:
-    #auth_url: <SWIFT_AUTH_URL>
+    #auth_url: <SWIFT_AUTH_URL>           # Keystone, with or without a trailing /v3
     #region: <SWIFT_REGION>
-    #user_id: <SWIFT_USER_ID>
-    #project_id: <SWIFT_PROJECT_ID>
+    #user_name: <USER_NAME>               # Name in user_domain_name. Or set user_id to the user id
+    #user_id: <USER_ID>
+    #project_name: <PROJECT_NAME>         # Name in project_domain_name. Or set project_id
+    #project_id: <PROJECT_ID>
     #password: <SWIFT_PASSWORD>
-    #user_domain_name: default
-    #project_domain_name: default
+    #user_domain_name: Default
+    #project_domain_name: Default
+    #endpoint: <STORAGE_URL>              # Optional. Replaces the URL from the Keystone catalog
     #storage_bucket: <CONTAINER_NAME>
 
 #infinispan:

docs/source/compute_config/aws_lambda.md

@@ -104,14 +104,38 @@ In summary, you can use one of the following settings:
 | aws_lambda | remote_invoker | False | no | Activate the remote invoker feature that uses one cloud function to spawn all the actual `map()` activations |
 | aws_lambda | architecture | x86_64 | no | Runtime architecture. One of **x86_64** or **arm64** |
 | aws_lambda | ephemeral_storage | 512 | no | Ephemeral storage (`/tmp`) size in MB (must be between 512 MB and 10240 MB) |
-| aws_lambda | user_tags | {} | no | List of {name: ..., value: ...} pairs for Lambda instance user tags |
-| aws_lambda | env_vars | {} | no | List of {name: ..., value: ...} pairs for Lambda instance environment variables |
+| aws_lambda | user_tags | {} | no | `{NAME: VALUE}` pairs for Lambda instance user tags (container runtimes only). A list of `{name: ..., value: ...}` pairs is also accepted |
+| aws_lambda | env_vars | {} | no | `{NAME: VALUE}` pairs for Lambda instance environment variables. A list of `{name: ..., value: ...}` pairs is also accepted |
 | aws_lambda | namespace |  | no | Virtual namespace. This can be useful to virtually group Lithops function workers. The functions deployed by lithops will be prefixed by this namespace. For example you can set it to differentiate between `prod`, `dev` and `stage` environments.  |
 | aws_lambda | runtime_include_function | False | no | If set to true, Lithops will automatically build a new runtime, including the function's code, instead of transferring it through the storage backend at invocation time. This is useful when the function's code size is large (on the order of tens of MB) and the code does not change frequently |
 
 
 ## Additional configuration
 
+### Environment variables and user tags
+To set environment variables or user tags on the Lithops lambdas, add them to the `aws_lambda` configuration section:
+
+```yaml
+aws_lambda:
+    execution_role: <EXECUTION_ROLE_ARN>
+    region: <REGION_NAME>
+    env_vars:
+        MY_VAR: my-value
+    user_tags:
+        project: my-project
+```
+
+The list form used in previous versions of this page is also accepted:
+
+```yaml
+aws_lambda:
+    env_vars:
+        - name: MY_VAR
+          value: my-value
+```
+
+User tags are set when a function is created. Run `lithops runtime update` after changing the environment variables so they are applied to the already deployed functions.
+
 ### VPC
 To connect the Lithops lambda to a VPC, add the following configuration to the `aws_lambda` configuration section:
 

docs/source/compute_config/azure_functions.md

@@ -87,7 +87,7 @@ az login
 |---|---|---|---|---|
 |azure_functions| resource_group | |no | Name of a resource group, for example: `LithopsResourceGroup`. Lithops will use the `resource_group` set under the `azure` section if it is not set here |
 |azure_functions| region |  |no | The Flex Consumption plan location for the runtime. Use `az functionapp list-flexconsumption-locations` to view the available locations. For example: `westeurope`, `westus2`, etc. Lithops will use the `region` set under the `azure` section if it is not set here|
-|azure_functions | runtime_memory | 2048 | no | Flex Consumption instance memory in MB. Supported values: `512`, `2048`, `4096`. Other values are mapped to the nearest supported size |
+|azure_functions | runtime_memory | 512 | no | Flex Consumption instance memory in MB. The default 512 MB receives approximately 0.25 CPU automatically. Supported values: `512`, `2048`, `4096`. Other values are mapped to the nearest supported size. Each instance memory size is deployed as a separate function app |
 |azure_functions | max_workers | 1000 | no | Max number of parallel workers. Although Azure limits the number of workers to 200, it is convenient to keep this value high|
 |azure_functions | worker_processes | 1 | no | Lithops-side parallelism setting. Not applied as an Azure Functions app setting on Flex Consumption |
 |azure_functions| runtime |  |no | Runtime name already deployed in the service|

docs/source/compute_config/azure_vms.md

@@ -206,7 +206,7 @@ lithops clean -b azure_vms -s azure_storage --all
 
 ## Architecture diagram
 
-Lithops creates a **Virtual Network** and **subnet** in your resource group (unless you provide existing ones). The master VM gets a **static public IP**; worker VMs use **private IPs only** and reach the internet through Azure's default subnet outbound routing.
+Lithops creates a **Virtual Network** and **subnet** in your resource group (unless you provide existing ones). The master VM gets a **static public IP**; worker VMs use **private IPs only**. Lithops explicitly enables default outbound access on subnets it creates so workers can download dependencies. User-provided subnets must supply outbound connectivity, such as a NAT gateway.
 
 ```mermaid
 flowchart TB

docs/source/compute_config/code_engine.md

@@ -128,7 +128,7 @@ Delete Lithops runtimes, job runs, and cached metadata:
 lithops clean -b code_engine -s ibm_cos
 ```
 
-To also delete the Code Engine project and local cache:
+To permanently delete the Code Engine project and local cache:
 
 ```bash
 lithops clean -b code_engine -s ibm_cos --all