A Kubernetes-native CLI plugin for managing Substrate Actor and Worker lifecycles.
There are two ways to run the tool, depending on whether you are developing locally or installing it permanently.
You can use go install to compile the tool and place the binary directly into your Go bin directory (which should be in your $PATH). Because the source folder is named kubectl-ate, Kubernetes will automatically recognize the resulting binary!
go install ./cmd/kubectl-ateYou can now run it seamlessly anywhere as a native Kubernetes command: kubectl ate <command>.
If you are testing changes to the codebase, you can bypass compilation and run the CLI directly from the source tree:
go run ./cmd/kubectl-ate <command>By default, kubectl-ate will automatically read your ~/.kube/config, discover the ate-api-server pods in your cluster, and establish a temporary background port-forward tunnel to execute gRPC calls securely.
If you prefer to route traffic directly (e.g., through a LoadBalancer or when running natively inside a cluster pod), simply provide the --endpoint flag to bypass the tunnel.
The CLI supports on-demand tracing using the --trace flag. When enabled, the CLI will generate a trace ID and signal to the server that it wants the request to be traced.
Prerequisites:
- The Google Cloud project must have the Cloud Trace API enabled. You can enable it using:
gcloud services enable cloudtrace.googleapis.com --project=PROJECT_ID- The GKE cluster must have Managed OpenTelemetry enabled. Clusters created by
setup-gcp create clusterorsetup-gcp bootstrapalways enable it. For a cluster that does not have it, enable it with:
gcloud beta container clusters update CLUSTER_NAME \
--project=PROJECT_ID \
--managed-otel-scope=COLLECTION_AND_INSTRUMENTATION_COMPONENTS \
--location=LOCATIONLocal (kind):
The kind overlay installed by hack/install-ate-kind.sh --deploy-ate-system already provisions an in-cluster OpenTelemetry Collector and a Jaeger all-in-one in the otel-system namespace. No additional setup is required.
Port-forward the Jaeger UI and invoke any command with --trace:
kubectl port-forward -n otel-system svc/jaeger 16686:16686 &
kubectl ate get actor my-counter-1 --trace
# open http://localhost:16686 and search for the most recent traceThese flags can be appended to any command:
| Flag | Short | Description | Default |
|---|---|---|---|
--kubeconfig |
Path to your kubeconfig file | ~/.kube/config |
|
--context |
Name of the kubeconfig context to use | current context | |
--endpoint |
Manual gRPC endpoint override (e.g., localhost:8080) |
||
--token-file |
Path to a bearer token for ate-api authentication, or - for stdin |
Kubernetes ServiceAccount token | |
--output |
-o |
Output format (table, json, yaml) |
table |
--trace |
Enable on-demand tracing for the request | false |
List and inspect the state of actors and workers across the cluster.
# List actors in one atespace; -a is shorthand for --atespace
kubectl ate get actors --atespace <atespace>
kubectl ate get actors -a <atespace>
# List actors across all atespaces
kubectl ate get actors -A
# Get a specific actor by name and output as raw YAML
kubectl ate get actor <actor-name> --atespace <atespace> -o yaml
# List all physical workers and see which actors are assigned to them
kubectl ate get workers
# Filter workers by Kubernetes namespace, assigned-actor atespace, or
# worker pool labels (same flags as `top workers`)
kubectl ate get workers -n <namespace>
kubectl ate get workers -a <atespace>
kubectl ate get workers -l <label-selector>Note:
get actorsrequires either--atespace <name>/-a <name>(one atespace) or-A/--all-atespaces(all atespaces) — there is no default atespace. Getting a single actor always requires--atespace/-a, since an actor is addressed by(atespace, name).-a(lower-case) scopes to one atespace;-A(upper-case) spans all.
Note: Actors, workers, and actor templates are not Kubernetes CRDs — they live in the Substrate control plane's PostgreSQL database, not
etcd.kubectl get actor,kubectl get worker, andkubectl get actortemplatewill not return anything; onlykubectl ate get …queries the control plane (see Actor Templates).kubectl get workerpooldoes work, because pools are CRDs.
| Column | Meaning |
|---|---|
ATESPACE |
The atespace the actor belongs to. Part of the actor's identity; folded into the storage key as actor:<atespace>:<name>. |
NAME |
The actor's name. User-provided for application actors; UUID for the golden actor that each template materialises during ResumeGoldenActor. |
TEMPLATE |
The ActorTemplate the actor was created from, displayed as <atespace>/<name>. |
STATE |
One of ACTOR_STATE_RESUMING, ACTOR_STATE_RUNNING, ACTOR_STATE_SUSPENDING, ACTOR_STATE_SUSPENDED. |
ATEOM POD |
The worker pod (namespace/name) currently hosting the actor. Empty while suspended. |
ATEOM IP |
The pod IP of that worker. Empty while suspended. |
VERSION |
Monotonic integer that increments on every state transition (resume / suspend / checkpoint). Useful for distinguishing snapshots. |
AGE |
Time elapsed since the actor was created. |
| Column | Meaning |
|---|---|
NAMESPACE |
The WorkerPool namespace. |
POOL |
The WorkerPool name. |
POD |
The worker pod name. |
STATUS |
FREE (idle, ready to receive an actor) or ASSIGNED (currently hosting an actor). |
ASSIGNED ACTOR |
If STATUS=ASSIGNED, the actor reference <namespace>/<template>/<actor-name>. |
An atespace is the isolation boundary an actor belongs to. It must exist before you can create actors in it.
# Create an atespace
kubectl ate create atespace <atespace>
# List all atespaces
kubectl ate get atespaces
# Get an atespace
kubectl ate get atespace <atespace>
# Delete an atespace (must be empty — fails if any actors remain)
kubectl ate delete atespace <atespace>Note:
create actor … -a <atespace>requires the atespace to already exist, otherwise it fails withFailedPrecondition.delete atespaceonly removes an empty atespace; delete its actors and snapshot tags first (cascade delete is not yet supported).
| Column | Meaning |
|---|---|
NAME |
The atespace name. Globally unique — atespaces are global-scoped. |
AGE |
Time elapsed since the atespace was created. |
An actor template describes what an actor runs: containers, volumes, snapshot policy, sandbox runtime, and worker selection. Templates live in an atespace and are immutable — there is no update; delete and recreate to change one.
# Create a template from a manifest (protojson-shaped ateapipb.ActorTemplate,
# a single YAML/JSON document; use -f - for stdin). The metadata's atespace
# must already exist.
kubectl ate create actor-template -f template.yaml
# List templates, or get one (also: -o yaml prints the re-applyable manifest).
kubectl ate get actor-templates -a <atespace>
kubectl ate get actor-template <name> -a <atespace> -o yaml
# Delete a template. This also deletes its golden actor and golden snapshot.
kubectl ate delete actor-template <name> -a <atespace>See
demos/counter/counter-template.yaml.tmpl
for a complete manifest example.
| Column | Meaning |
|---|---|
ATESPACE |
The atespace the template belongs to. |
NAME |
The template's name. |
SANDBOX CLASS |
The sandbox runtime family (SANDBOX_CLASS_GVISOR or SANDBOX_CLASS_MICROVM). |
GOLDEN SNAPSHOT |
The golden snapshot's name once it exists (actors can be created); empty while the golden build is still running. |
ERROR |
ERROR when the golden build failed; -o yaml shows the full message. |
AGE |
Time elapsed since the template was created. |
Manage the execution state of your workloads. (Note: Actors are identified by a user-provided name, which must be a valid DNS-1123 label)
# Create a new actor deriving from an ActorTemplate. The template name is
# resolved in the actor's atespace. -a/--atespace is required and the
# atespace must already exist (kubectl ate create atespace <atespace>).
kubectl ate create actor my-actor --template-ref=<template-name> -a <atespace>
# Resume an actor (assigns it to a free worker and restores its state)
kubectl ate resume actor my-actor -a <atespace>
# Suspend an actor (snapshots its state to storage and frees the worker)
kubectl ate suspend actor my-actor -a <atespace>
# Delete an actor (by default, requires the actor to be SUSPENDED or CRASHED).
kubectl ate delete actor my-actor -a <atespace>
# Delete an actor from any state (e.g. RUNNING, PAUSED), terminating workloads and detaching volumes.
kubectl ate delete actor my-actor -a <atespace> --any-stateSuspending an actor creates a durable snapshot. Tags give snapshots stable, Atespace-owned names; published tags may be used from other Atespaces.
# List snapshots, or resolve one canonical snapshot or tag.
kubectl ate get snapshots -a <atespace>
kubectl ate get snapshot <snapshot-name> -a <atespace>
kubectl ate get snapshot <tag-name> -a <atespace> --tag
# Tag a snapshot, then publish or unpublish the tag.
kubectl ate create snapshot-tag <tag-name> -a <atespace> --snapshot <snapshot-name>
kubectl ate update snapshot-tag <tag-name> -a <atespace> --scope published
kubectl ate update snapshot-tag <tag-name> -a <atespace> --scope atespace
# Create an actor from a tag and remove the tag when it is no longer needed.
kubectl ate create actor <actor-name> -a <atespace> --template-ref <template-name> --snapshot-tag <tag-atespace/tag-name>
kubectl ate delete snapshot-tag <tag-name> -a <atespace>kubectl ate logs requires a resource-type subcommand; running kubectl ate logs <actor-name> on its own prints help. The only supported resource type is actors:
# Print the logs an actor has produced on its current worker.
# -a/--atespace is required, since an actor is addressed by (atespace, name).
kubectl ate logs actors my-actor -a <atespace>
# Follow the logs with -f. The stream is aggregated across worker
# reassignments, so the same actor stays queryable as it teleports between pods.
kubectl ate logs actors my-actor -a <atespace> -f
# Show only one container's logs with -c/--container.
kubectl ate logs actors my-actor -a <atespace> -c my-containerLogs are streamable only while the actor is bound to a worker (i.e., ACTOR_STATE_RUNNING). For history across worker migrations, route through a centralized log backend (Cloud Logging, Loki, etc.); see docs/observability.md.
Commands for bootstrapping the Substrate control plane.
# Generate a new Actor ID CA pool and push it directly to a Kubernetes Secret
kubectl ate admin make-ca-pool \
--name actor-id-ca-pool \
--secret-namespace ate-system \
--ca-id "1"
# Generate a new JWT authority pool and push it to a Kubernetes Secret
kubectl ate admin make-jwt-pool \
--name actor-id-jwt-pool \
--secret-namespace ate-system \
--key-id "1"