Skip to content

sidecar-deployment.md

Latest commit

 

History

History
391 lines (313 loc) · 10.5 KB

sidecar-deployment.md

File metadata and controls

391 lines (313 loc) · 10.5 KB
Raw

Sidecar Deployment Guide

Deployment guide for running Talos CSR Signer as a sidecar in Kamaji TenantControlPlane with Talos workers.

Architecture

CSR Signer runs as a sidecar container in Kamaji TenantControlPlane, sharing the same LoadBalancer service on port 50001. Kamaji manages the control plane lifecycle.

Prerequisites

  • Kubernetes cluster with Kamaji installed
  • kubectl with cluster admin access
  • talosctl CLI
  • yq CLI
  • Talos worker nodes (bare metal, VMs, or cloud instances)

Step 1: Configure Environment

cp deploy/.env.example deploy/.env
vi deploy/.env
source deploy/.env

Example configuration:

export CLUSTER_NAME="my-cluster"
export NAMESPACE="default"
export CONTROL_PLANE_IP="10.10.10.100"
export KUBERNETES_VERSION="v1.33.0"
export TALOS_VERSION="v1.8.3"
export WORKER_IPS="10.10.10.201 10.10.10.202 10.10.10.203"

# Registry for CSR Signer image
export CSR_SIGNER_IMAGE="ghcr.io/clastix/talos-csr-signer"
export CSR_SIGNER_IMAGE_TAG="latest"

Step 2: Generate Talos Secrets

talosctl gen secrets -o secrets.yaml --force

Step 3: Prepare Credentials

Extract Talos credentials and create Kubernetes secret:

TALOS_CA_CRT=$(yq -r '.certs.os.crt' secrets.yaml | base64 -d)
# Talos uses "BEGIN ED25519 PRIVATE KEY" but cert-manager requires "BEGIN PRIVATE KEY" (RFC 7468)
TALOS_CA_KEY=$(yq -r '.certs.os.key' secrets.yaml | base64 -d | sed 's/ED25519 //g')
TALOS_TOKEN=$(yq -r '.trustdinfo.token' secrets.yaml)
TALOS_CLUSTER_ID=$(yq -r '.cluster.id' secrets.yaml)
TALOS_CLUSTER_SECRET=$(yq -r '.cluster.secret' secrets.yaml)

kubectl create secret generic ${CLUSTER_NAME}-talos-ca -n $NAMESPACE \
  --from-literal=tls.crt="$TALOS_CA_CRT" \
  --from-literal=tls.key="$TALOS_CA_KEY" \
  --from-literal=token="$TALOS_TOKEN"

Step 4: Generate the TLS Certificate with cert-manager

The gRPC Server uses a TLS certificate generated by cert-manager, a dependency already required by Kamaji.

Create the CA Issuer:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ${CLUSTER_NAME}-talos-ca
  namespace: $NAMESPACE
spec:
  ca:
    secretName: ${CLUSTER_NAME}-talos-ca
EOF

kubectl wait --for=condition=Ready issuer/${CLUSTER_NAME}-talos-ca -n $NAMESPACE --timeout=60s

Create the TLS Certificate for the gRPC server:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ${CLUSTER_NAME}-talos-tls-cert
  namespace: $NAMESPACE
spec:
  secretName: ${CLUSTER_NAME}-talos-tls-cert
  duration: 8760h # 1 year
  renewBefore: 720h # 30 days before
  isCA: false
  privateKey:
    algorithm: Ed25519
    size: 256
  usages:
    - digital signature
    - key encipherment
    - server auth
  ipAddresses:
    - 127.0.0.1
    - $CONTROL_PLANE_IP
  issuerRef:
    name: ${CLUSTER_NAME}-talos-ca
    kind: Issuer
    group: cert-manager.io
EOF

kubectl wait --for=condition=Ready certificate/${CLUSTER_NAME}-talos-tls-cert -n $NAMESPACE --timeout=60s

Step 5: Deploy Control Plane

Deploy TenantControlPlane with the CSR Signer sidecar:

kubectl apply -f - <<EOF
apiVersion: kamaji.clastix.io/v1alpha1
kind: TenantControlPlane
metadata:
  name: $CLUSTER_NAME
  namespace: $NAMESPACE
spec:
  dataStore: default
  controlPlane:
    deployment:
      replicas: 1
      additionalContainers:
        - name: talos-csr-signer
          image: ${CSR_SIGNER_IMAGE}:${CSR_SIGNER_IMAGE_TAG}
          imagePullPolicy: Always
          ports:
            - name: grpc
              containerPort: 50001
              protocol: TCP
          env:
            - name: TALOS_TOKEN
              valueFrom:
                secretKeyRef:
                  name: ${CLUSTER_NAME}-talos-ca
                  key: token
          volumeMounts:
            - name: talos-ca
              mountPath: /etc/talos-ca
              readOnly: true
            - name: tls-cert
              mountPath: /etc/talos-server-crt
              readOnly: true
      additionalVolumes:
        - name: talos-ca
          secret:
            secretName: ${CLUSTER_NAME}-talos-ca
        - name: tls-cert
          secret:
            secretName: ${CLUSTER_NAME}-talos-tls-cert
    service:
      serviceType: LoadBalancer
      additionalMetadata:
        annotations:
          metallb.io/loadBalancerIPs: $CONTROL_PLANE_IP
      additionalPorts:
        - name: talos-csr-signer
          port: 50001
          targetPort: 50001
          protocol: TCP
  networkProfile:
    address: $CONTROL_PLANE_IP
    port: 6443
  kubernetes:
    version: $KUBERNETES_VERSION
    kubelet:
      cgroupfs: systemd
  addons:
    coreDNS: {}
    kubeProxy: {}
    konnectivity: {}
EOF

kubectl wait --for=condition=Available deployment -l kamaji.clastix.io/name=$CLUSTER_NAME -n $NAMESPACE --timeout=60s
kubectl logs -n $NAMESPACE -l kamaji.clastix.io/name=$CLUSTER_NAME -c talos-csr-signer --tail=20

Step 6: Generate Talosconfig for Worker Management

Generate the talosconfig file to manage Talos worker nodes using talosctl CLI:

talosctl gen config $CLUSTER_NAME https://$CONTROL_PLANE_IP:6443 \
  --with-secrets secrets.yaml \
  --output-types talosconfig \
  --output talosconfig \
  --force

talosctl --talosconfig=talosconfig config endpoint $WORKER_IPS

Step 7: Create Worker Configuration

Extract Kubernetes credentials and create worker configuration:

kubectl get secret ${CLUSTER_NAME}-admin-kubeconfig -n $NAMESPACE \
  -o jsonpath='{.data.admin\.conf}' | base64 -d > ${NAMESPACE}-${CLUSTER_NAME}.kubeconfig

K8S_CA=$(kubectl --kubeconfig=${NAMESPACE}-${CLUSTER_NAME}.kubeconfig config view --raw \
  -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')

K8S_BOOTSTRAP_TOKEN=$(kubeadm --kubeconfig=${NAMESPACE}-${CLUSTER_NAME}.kubeconfig token create)

# Re-extract credentials from secrets.yaml (keep base64 encoded for worker.yaml)
TALOS_CA_CRT=$(yq -r '.certs.os.crt' secrets.yaml)
TALOS_TOKEN=$(yq -r '.trustdinfo.token' secrets.yaml)
TALOS_CLUSTER_ID=$(yq -r '.cluster.id' secrets.yaml)
TALOS_CLUSTER_SECRET=$(yq -r '.cluster.secret' secrets.yaml)

cat > worker.yaml <<EOF
version: v1alpha1
persist: true
machine:
  type: worker
  token: ${TALOS_TOKEN}
  ca:
    crt: ${TALOS_CA_CRT}
    key: ""
  kubelet:
    image: ghcr.io/siderolabs/kubelet:${KUBERNETES_VERSION}
    extraArgs:
      rotate-certificates: "true"
  install:
    disk: /dev/sda
    image: ghcr.io/siderolabs/installer:${TALOS_VERSION}
  features:
    rbac: true
    kubePrism:
      enabled: false
cluster:
  id: ${TALOS_CLUSTER_ID}
  secret: ${TALOS_CLUSTER_SECRET}
  controlPlane:
    endpoint: https://${CONTROL_PLANE_IP}:6443
  clusterName: ${CLUSTER_NAME}
  network:
    dnsDomain: cluster.local
    podSubnets:
      - 10.244.0.0/16
    serviceSubnets:
      - 10.96.0.0/12
  token: ${K8S_BOOTSTRAP_TOKEN}
  ca:
    crt: ${K8S_CA}
    key: ""
  discovery:
    enabled: true
    registries:
      kubernetes:
        disabled: true
      service:
        disabled: true
EOF

Step 8: Join Workers

Deploy workers and monitor joining:

for WORKER_IP in $WORKER_IPS; do
  talosctl apply-config --insecure --nodes $WORKER_IP --file worker.yaml
done

kubectl logs -n $NAMESPACE -l kamaji.clastix.io/name=$CLUSTER_NAME -c talos-csr-signer -f

Verify workers are joining:

kubectl --kubeconfig=${NAMESPACE}-${CLUSTER_NAME}.kubeconfig get nodes -o wide

Step 9: Deploy CNI

kubectl --kubeconfig=${NAMESPACE}-${CLUSTER_NAME}.kubeconfig apply -f \
  https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

kubectl --kubeconfig=${NAMESPACE}-${CLUSTER_NAME}.kubeconfig wait --for=condition=ready pod \
  -l app=flannel -n kube-flannel --timeout=120s

Step 10: Verify Cluster

Check Kubernetes nodes:

kubectl --kubeconfig=${NAMESPACE}-${CLUSTER_NAME}.kubeconfig get nodes -o wide
kubectl --kubeconfig=${NAMESPACE}-${CLUSTER_NAME}.kubeconfig get pods -A

Check CSR signing activity:

kubectl logs -n $NAMESPACE -l kamaji.clastix.io/name=$CLUSTER_NAME -c talos-csr-signer --tail=50

Verify Talos worker nodes:

FIRST_WORKER=$(echo $WORKER_IPS | awk '{print $1}')

talosctl --talosconfig=talosconfig -e $FIRST_WORKER -n $FIRST_WORKER version
talosctl --talosconfig=talosconfig -e $FIRST_WORKER -n $FIRST_WORKER get members
talosctl --talosconfig=talosconfig -e $FIRST_WORKER -n $FIRST_WORKER service kubelet status
talosctl --talosconfig=talosconfig -e $FIRST_WORKER -n $FIRST_WORKER dmesg | grep -i talos

Check all workers:

for WORKER_IP in $WORKER_IPS; do
  echo "=== Worker: $WORKER_IP ==="
  talosctl --talosconfig=talosconfig -e $WORKER_IP -n $WORKER_IP get machineconfig -o yaml | grep -A 2 kubelet
done

Note: Use -e <ip> -n <ip> for direct connection. Workers cannot forward Talos API requests in Kamaji deployments.

Troubleshooting

Workers Not Joining

kubectl logs -n $NAMESPACE -l kamaji.clastix.io/name=$CLUSTER_NAME -c talos-csr-signer

Common causes:

  • Token mismatch: Verify token in secret matches worker.yaml
  • Discovery disabled: Check cluster.discovery.enabled: true in worker.yaml
  • Port 50001 not accessible: Verify service has port 50001 exposed

CSR Signer Not Starting

kubectl describe pod -n $NAMESPACE -l kamaji.clastix.io/name=$CLUSTER_NAME

Common causes:

  • Secret not found: Verify secret exists in correct namespace
  • Invalid CA format: Ensure CA cert/key properly base64 decoded
  • Missing token: Check TALOS_TOKEN environment variable

Token Validation Failing

kubectl get secret ${CLUSTER_NAME}-talos-ca -n $NAMESPACE -o jsonpath='{.data.token}' | base64 -d
yq -r '.machine.token' worker.yaml

Tokens must match exactly.

Configuration Reference

Environment Variables

Variable Default Description
PORT 50001 gRPC server port
CA_CERT_PATH /etc/talos-ca/tls.crt Talos Machine CA certificate path
CA_KEY_PATH /etc/talos-ca/tls.key Talos Machine CA private key path
TLS_CERT_PATH /etc/talos-server-crt/tls.crt CSR gRPC server certificate path
TLS_KEY_PATH /etc/talos-server-crt/tls.key CSR gRPC server private key path
TALOS_TOKEN required Machine token for authentication

References