Skip to content

Latest commit

 

History

History
365 lines (299 loc) · 18.1 KB

09-kubernetes-sidecar.md

File metadata and controls

365 lines (299 loc) · 18.1 KB

BTrace in Kubernetes, the Sidecar Way

Package BTrace into a container the right way for three different jobs: an interactive troubleshooting image, a lean Kubernetes sidecar, and a minimal -javaagent layer for a distroless production image. You'll build each variant's Dockerfile pattern, attach exactly like Tutorial 1 — just across a container boundary — and see why the three official images are different sizes for different jobs.

Persona: a platform engineer who owns how Java workloads get packaged and deployed. Time: ~15 minutes.

This tutorial builds real Dockerfiles and a real Kubernetes manifest against your own Docker/ Kubernetes environment; every path, flag, and file layout below is copied from BTrace's shipped docker/ sources (not invented for this write-up). Build/registry/cluster output is inherently environment-specific, so "you should see" blocks are illustrative — image IDs, digests, and pod names will differ — the same way earlier tutorials mark timing numbers as illustrative.

What you'll need

  • Docker (or a compatible builder) locally, and — for the Kubernetes steps — a cluster with kubectl pointed at it
  • The demo app from this series (demo/DemoApp.java) — it's what you'll layer BTrace onto
  • Fifteen minutes and the muscle memory from Tutorial 1 (you'll reuse its exact oneliner, just from inside a container)

Step 1 — Pick your image variant

BTrace publishes three image variants, each for a different job (docker/README.md):

Variant Base image (from the Dockerfile) Ships Best for
ghcr.io/btraceio/btrace:3.0.0 bellsoft/liberica-openjdk-debian:21.0.13.0.3-cds Full toolchain, shell, samples Development, interactive debugging
ghcr.io/btraceio/btrace:3.0.0-alpine alpine:3.24 + openjdk21-jdk Full toolchain, smaller OS Kubernetes sidecars, resource-constrained environments
ghcr.io/btraceio/btrace:3.0.0-distroless gcr.io/distroless/java21-debian12 Runtime JARs only — no shell, no scripts Production apps using -javaagent

What just happened? Those base images come straight from docker/Dockerfile, docker/Dockerfile.alpine, and docker/Dockerfile.distroless. The first two COPY the entire built distribution (bin/, conf/, extensions/, libs/, samples/ — the build context's .dockerignore leaves out only the Windows .bat launchers and the license files) to /opt/btrace/ and put /opt/btrace/bin on PATH; the distroless variant copies only btrace/libs to /opt/btrace/libs — there's no bin/ and no entrypoint script, because (per the Dockerfile's own comment) "distroless has no shell". That one architectural difference is why Step 5 below looks nothing like Steps 2–4.

Step 2 — Layer BTrace onto your app with COPY --from

The common case: copy the BTrace distribution out of the official image and into your own, using a multi-stage build. This is docker/README.md's "Pattern 1: Multi-Stage Build", adapted to this series' demo app (full Dockerfile: demo/Dockerfile.k8s-sidecar-demo):

FROM ghcr.io/btraceio/btrace:3.0.0 AS btrace
FROM bellsoft/liberica-openjdk-debian:21.0.13.0.3-cds

WORKDIR /app
COPY DemoApp.java /app/

COPY --from=btrace /opt/btrace /opt/btrace
ENV BTRACE_HOME=/opt/btrace
ENV PATH="${PATH}:${BTRACE_HOME}/bin"

ENTRYPOINT ["java", "DemoApp.java"]

What just happened? COPY --from=btrace /opt/btrace /opt/btrace pulls the entire built tree out of the first stage — bin/, conf/, extensions/, libs/, samples/ — into your final image; only the official image's layers are discarded, not its files. BTRACE_HOME and PATH are set exactly the way docker/README.md's Quick Start does it. DemoApp.java runs via JDK 11+'s single-file source launch — no separate javac/jar step needed, same as running it on bare metal in Tutorial 1.

Step 3 — Build it and attach, same oneliner as Tutorial 1

From docs/tutorials/demo/:

docker build -f Dockerfile.k8s-sidecar-demo -t demo-app-with-btrace:local .
docker run -d --name demo-app demo-app-with-btrace:local

You should see (illustrative — your image ID and container ID will differ):

[+] Building 4.2s (10/10) FINISHED
 => exporting to image
 => => writing image sha256:1a2b3c4d5e6f...
 => => naming to docker.io/library/demo-app-with-btrace:local
7f3a9c21b8e4c6d5a0f1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3

Now find the PID and attach exactly like Tutorial 1 — just via docker exec instead of a local shell:

docker exec demo-app jps -m
docker exec demo-app btrace -n 'OrderService::processOrder @return { print method, time }' <PID>

jps -m prints 1 Main DemoApp.java: the app is a single-file source launch, so plain jps shows only the launcher's class name, Main (jps -l spells it out as jdk.compiler/com.sun.tools.javac.launcher.Main), and -m adds the DemoApp.java argument that identifies it. It's PID 1 because the JVM is the container's entrypoint process.

You should see the same shape of output as Tutorial 1, Step 3 (numbers will vary):

processOrder
execution time: 58 ms
processOrder
execution time: 79 ms
processOrder
execution time: 318 ms

What just happened? docker exec runs a new process inside the container's namespaces — it doesn't go through the image's ENTRYPOINT/docker-entrypoint.sh at all (that script only runs once, as PID 1, when the container starts). btrace on PATH is the same client binary from Step 2's COPY --from; it attaches to the DemoApp JVM inside the container the same way it attaches to any local JVM. Same bug, same fix — chargeCard's slow ~10% of calls, same as Tutorial 1 — just packaged differently.

Step 4 — The same app, as a Kubernetes sidecar

Rather than docker exec-ing into your app's own container, run BTrace in a separate container in the same pod and share its process namespace. This is docker/README.md's "Pattern 2: Kubernetes Sidecar" (full manifest: demo/k8s-sidecar-pod.yaml):

apiVersion: v1
kind: Pod
metadata:
  name: demo-app-with-btrace
spec:
  shareProcessNamespace: true # required - lets the sidecar see the app container's PIDs
  containers:
    - name: demo-app
      image: demo-app-with-btrace:local # <-- replace with your registry/tag
      imagePullPolicy: IfNotPresent

    - name: btrace-sidecar
      image: ghcr.io/btraceio/btrace:3.0.0-alpine
      command: ["/bin/sh", "-c", "while true; do sleep 30; done"]
      securityContext:
        capabilities:
          add: ["SYS_PTRACE"] # required for the sidecar to attach across containers
      volumeMounts:
        - name: btrace-scripts
          mountPath: /scripts

  volumes:
    - name: btrace-scripts
      configMap:
        name: btrace-scripts

Push your image to a registry kubectl can reach and update the image: field above. The pod mounts a ConfigMap named btrace-scripts at /scripts, so create it first — from Tutorial 2's demo/OrderTiming.java, which lands in the sidecar as /scripts/OrderTiming.java — then apply the pod and attach (still from docs/tutorials/demo/):

kubectl create configmap btrace-scripts --from-file=OrderTiming.java
kubectl apply -f k8s-sidecar-pod.yaml
kubectl exec demo-app-with-btrace -c btrace-sidecar -- \
  sh -c 'btrace $(pgrep -f "[D]emoApp") /scripts/OrderTiming.java'

(The [D]emoApp bracket trick keeps pgrep -f from matching the sh -c '...' wrapper itself, whose own command line contains the pattern; only the app container's java DemoApp.java matches.)

You should see (illustrative — depends on your cluster and registry), followed by the same per-worker lines as Tutorial 2, Step 4:

configmap/btrace-scripts created
pod/demo-app-with-btrace created
worker=order-worker-1 order #1 total=43ms at 1783778034521
worker=order-worker-0 order #1 total=57ms at 1783778034534
worker=order-worker-2 order #3 total=414ms at 1783778034964

What just happened? shareProcessNamespace: true is the one field that makes this pattern work at all — without it, the sidecar's pgrep/btrace would only ever see its own container's processes, never demo-app's JVM (docker/README.md's own Troubleshooting section calls this out: kubectl get pod myapp -o yaml | grep shareProcessNamespace is the first thing to check when attach fails). The sidecar uses the alpine variant, not the full or distroless one — it needs a shell and the full toolchain (btrace, jps) to run interactively via kubectl exec, and the alpine base is smaller than the Debian one.

The sidecar does not make the app image BTrace-free, though: the app container still needs BTrace installed at the same path (/opt/btrace). The client in the sidecar asks the target JVM to load the agent with the Attach API, passing the agent jar's path (/opt/btrace/libs/btrace.jar), and the target JVM opens that path in its own filesystem, not the sidecar's. That's why demo-app runs the demo-app-with-btrace image from Step 2; point the sidecar at an app image without /opt/btrace and the client fails with Agent JAR not found or no Agent-Class attribute / Failed to attach to PID 1, while the app container's own log shows Error opening zip file or JAR manifest missing: /opt/btrace/libs/btrace.jar. What the sidecar buys you is the client side — the shell, the launcher scripts, your probe scripts — kept out of the app container.

SYS_PTRACE is called out explicitly in docker/README.md's Best Practices as a required capability alongside shareProcessNamespace; the rest of a hardened securityContext (runAsNonRoot, readOnlyRootFilesystem, dropping ALL other capabilities) is standard Kubernetes practice, not BTrace-specific, so it's left out of the minimal manifest above — add it per your own cluster's policy.

Step 5 — Skip the sidecar: distroless production with -javaagent

For a permanently-running probe (rather than on-demand attach), skip the sidecar model entirely: copy just the BTrace runtime jar into a distroless image and load it as a -javaagent at JVM startup. This is docker/README.md's "Pattern 4: Distroless Production", made concrete (like the README's own copy, this assumes an earlier build stage that compiles myapp.jar — not shown, since it's your application's own build, not BTrace's).

The agent only loads precompiled probes: it skips a script= path ending in .java, and it checks the file for the class-file magic number, so a source file under any other name is skipped too (in both cases silently at the default log level). Nothing in the distroless image provides your probe either, so compile it first and put it in the build context, here with the full image's btracec (from docs/tutorials/demo/):

mkdir -p scripts
docker run --rm -v "$PWD":/work ghcr.io/btraceio/btrace:3.0.0 btracec -d scripts OrderTiming.java
# -> scripts/OrderTiming.class
# ... an earlier "build" stage that produces /app/target/myapp.jar goes here ...

FROM ghcr.io/btraceio/btrace:3.0.0-distroless AS btrace
FROM gcr.io/distroless/java21-debian12
WORKDIR /app

COPY --from=build /app/target/myapp.jar /app/
COPY scripts/ /scripts/
COPY --from=btrace /opt/btrace/libs /opt/btrace/libs

ENTRYPOINT ["java", \
  "-javaagent:/opt/btrace/libs/btrace.jar=script=/scripts/OrderTiming.class,stdout=true", \
  "-jar", "/app/myapp.jar"]

You should see in docker logs the agent starting, then the probe's lines interleaved with the app's own output (illustrative):

[main] INFO io.btrace.extension.impl.ExtensionLoaderImpl - Discovered 0 extension(s):
worker=order-worker-1 order #1 total=55ms at 1793530445096
worker=order-worker-0 order #1 total=89ms at 1793530445129

What just happened? Dockerfile.distroless only ever COPYs btrace/libs — there's no bin/, no entrypoint script, nothing to docker exec a shell into (distroless images ship no shell at all). That's the trade-off: you get the smallest, lowest-attack-surface image, but you give up on-demand btrace <PID> script.java attach entirely. The probe has to be decided at container build/deploy time, baked in via -javaagent, exactly like any other Java agent.

Three details follow from copying only libs/:

  • Where the output goes. Without stdout=true the agent writes the probe's output to a file in the JVM's working directory — here /app/OrderTiming.class-.default.<timestamp>.btrace — and logs Redirecting output to .... With no shell in the image, read it with docker cp <container>:/app/<file> .; stdout=true sends it to the container log instead, which is usually what you want in Kubernetes.
  • No extensions. There is no extensions/ directory, so the agent reports Discovered 0 extension(s). A probe that needs one (for example btrace-metrics) needs the fat agent from Step 6.
  • No conf/. The defaults apply; there's no /opt/btrace/conf/extensions.conf to edit.

Step 6 — Optional: bake in a fat agent instead of a bare JVM script

If your extension needs (say, btrace-metrics for percentile histograms, from Tutorial 4) are fixed at build time, you can go one step further than Step 5 and skip $BTRACE_HOME entirely with a fat agent JAR — the packaging BTrace builds for exactly this situation, covered in depth in Tutorial 8.

No official image contains one, so you can't COPY --from=btrace it: btrace-dist/build.gradle deliberately writes the fat agent to btrace-dist/build/fat-agent/ — outside the distribution tree the images copy, so it never rides along into the zip/tarball/deb/rpm — and buildDockerContext does not build it. Build it yourself and put it in your Docker build context:

# Option A: your own fat agent with exactly the extensions you chose (Tutorial 8, Steps 1-3)
cp ~/fat-agent-demo/build/libs/demo-btrace-agent.jar ./btrace-agent-fat.jar

# Option B: BTrace's release fat agent (all eight default extensions), from a source checkout.
# It's the same jar Maven Central serves as io.btrace:btrace:3.0.0.
./gradlew :btrace-dist:fatAgentJar
cp btrace-dist/build/fat-agent/btrace-agent-fat.jar ./btrace-agent-fat.jar

Then copy it from the build context rather than from an image:

FROM gcr.io/distroless/java21-debian12

# Copy only the fat agent (no BTRACE_HOME, no /opt/btrace/libs needed)
COPY btrace-agent-fat.jar /opt/btrace/

and point Step 5's -javaagent: at /opt/btrace/btrace-agent-fat.jar instead of /opt/btrace/libs/btrace.jar. Remember from Tutorial 8 that a privileged embedded extension still needs allowExtensions=<id> (or allowPrivileged=true) among the agent arguments.

Troubleshooting

  • Cannot attach to process / Can not attach to PID / BTrace server not available at port 2020 — in plain Docker, a separate BTrace container has to join the target container's PID namespace (to see and attach to the JVM) and its network namespace (after loading the agent, the client connects to it on localhost:2020, which is only the target's localhost if the network namespace is shared). From docs/tutorials/demo/, against Step 3's container:

    docker run --rm --pid=container:demo-app --network=container:demo-app -v "$PWD":/scripts \
      ghcr.io/btraceio/btrace:3.0.0 btrace 1 /scripts/OrderTiming.java

    With --pid alone the agent loads but the client stops with BTrace server not available at port 2020. The target image must still contain BTrace at /opt/btrace (see Step 4). In Kubernetes, containers of one pod already share the network namespace; confirm shareProcessNamespace: true actually applied: kubectl get pod <pod> -o yaml | grep shareProcessNamespace.

  • BTRACE_HOME not found — verify the COPY --from layer landed where you expect: docker exec <container> sh -c 'ls -la $BTRACE_HOME' (quoted, so $BTRACE_HOME expands inside the container rather than in your host shell) and docker exec <container> env | grep BTRACE.

  • Java tools not available (jps/javac missing) — you copied a JRE-only base image instead of a full JDK; the sidecar/app image needs the JDK. On Java 9+ bases you may also need --add-exports jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED on JAVA_OPTS for PID discovery tools to work.

  • The sidecar can't see the app's process at all — Pod Security Policies/admission controllers can block ptrace-family syscalls even with shareProcessNamespace: true; add the SYS_PTRACE capability (Step 4) and check your cluster's PSP/OPA policy. Note GKE Autopilot in particular may restrict SYS_PTRACE outright — use standard GKE if you hit this.

  • You tried to docker exec sh into the distroless image — that's expected to fail: the distroless variant ships no shell by design (Step 1). Use it only for the baked-in -javaagent pattern (Step 5/6), not for interactive attach.

Clean up

docker rm -f demo-app
docker rmi demo-app-with-btrace:local
kubectl delete pod demo-app-with-btrace
kubectl delete configmap btrace-scripts

Go deeper