Skip to content

Latest commit

 

History

History
1160 lines (913 loc) · 34.4 KB

Troubleshooting.md

File metadata and controls

1160 lines (913 loc) · 34.4 KB

BTrace Troubleshooting Guide

This guide helps you diagnose and resolve common BTrace issues.

Note: Examples use btrace.jar -- the single masked JAR (BTrace 3.0+). If using a legacy multi-JAR distribution, replace btrace.jar with btrace-agent.jar (and add -Xbootclasspath/a:btrace-boot.jar where needed).

Table of Contents

  1. JVM Attachment Issues
  2. Script Compilation Errors
  3. Script Verification Errors
  4. No Output from Scripts
  5. Class/Method Not Found
  6. Performance Issues
  7. Agent Loading Problems
  8. Compatibility Issues
  9. Debugging BTrace Scripts
  10. Known Limitations

JVM Attachment Issues

Start with the read-only diagnostic before trying to load the agent:

btrace doctor <pid>
# Machine-readable automation output:
btrace doctor <pid> --json

Doctor reports whether the Attach API is available and the target is accessible, but reports dynamic-agent loading permission as not_tested: determining that permission would require an agent-load attempt. If a normal BTrace attach is explicitly rejected because dynamic loading is disabled, BTrace prints two next-deployment choices instead of an unconditional stack trace: relaunch with -XX:+EnableDynamicAgentLoading, or configure reviewed prepared mode with -javaagent:/path/to/btrace.jar=port=0.

Unable to attach to target VM

Error message: the client prints Failed to attach to PID <pid>; run it with -v to see the underlying exception:

com.sun.tools.attach.AttachNotSupportedException: Unable to attach to target VM

Causes and solutions:

  1. Permission mismatch

    • Cause: BTrace and target JVM run as different users
    • Solution: Run BTrace as the same user as the target JVM
    # Check process owner
    ps aux | grep <PID>
    
    # Run as correct user
    sudo -u <username> btrace <PID> script.java
  2. Attach mechanism disabled or restricted

    For JDK 20 and older:

    • Cause: Target JVM started with -XX:+DisableAttachMechanism
    • Solution: Remove this flag and restart the application

    For JDK 21+ (JEP 451):

    • Cause: Dynamic agent loading triggers warnings (still works but shows warning)
    • Warning message: WARNING: A Java agent has been loaded dynamically
    • Solution: Add -XX:+EnableDynamicAgentLoading to target JVM startup to suppress warnings:
      java -XX:+EnableDynamicAgentLoading -jar your-application.jar
    • Note: In a future JDK release, dynamic agent loading will be disabled by default. The -XX:+EnableDynamicAgentLoading flag will be required to use BTrace's attach mode.
    • Alternative: Use BTrace in agent mode (attach at startup) instead of dynamic attach:
      java -javaagent:/path/to/btrace.jar=script=YourScript.class -jar your-application.jar
  3. JRE instead of JDK

    • Cause: JRE doesn't include attach API tools
    • Solution: Install JDK and ensure JAVA_HOME points to it
    # Verify JDK installation
    which javac
    echo $JAVA_HOME
  4. Process in container/namespace

    • Cause: Target process in Docker/different namespace
    • Solution: Run BTrace in same container/namespace
    # For Docker
    docker exec -it <container> btrace <PID> script.java
  5. SELinux/AppArmor restrictions

    • Cause: Security policies prevent ptrace
    • Solution: Adjust security policies or disable temporarily
    # Check SELinux
    getenforce
    
    # Temporarily disable (not recommended for production)
    sudo setenforce 0

Connection refused

Error message:

java.net.ConnectException: Connection refused

Solutions:

  • Check if BTrace client port (default 2020) is available
  • Specify different port: btrace -p 3030 <PID> script.java
  • Check firewall rules

Script Compilation Errors

Compilation failed

Error message:

error: cannot find symbol

Common causes:

  1. An explicit BTraceUtils import suppresses the flat-DSL auto-import

    BTrace 3.0 injects import static io.btrace.BTrace.* and import io.btrace.core.annotations.* into every script, so bare println(...), @OnMethod and the other flat-DSL helpers compile without any import. If your script still carries import static io.btrace.core.BTraceUtils.* (the 2.x style), the compiler treats that as "the script manages its own DSL import" and does not inject io.btrace.BTrace.*; anything that exists only in the flat DSL then fails with cannot find symbol.

    // Fails for flat-DSL helpers: the BTraceUtils import disables the auto-import
    import static io.btrace.core.BTraceUtils.*;
    
    @BTrace
    public class MyTrace {
        @OnMethod(...)
        public static void handler() {
            println(str(monotonic()));  // cannot find symbol: monotonic() is flat-DSL only
        }
    }
    
    // Works: no imports at all (or `import static io.btrace.BTrace.*;` explicitly)
    @BTrace
    public class MyTrace {
        @OnMethod(...)
        public static void handler() {
            println(str(monotonic()));  // OK
        }
    }
  2. Wrong package for annotations

    // Old (pre-2.0)
    import com.sun.btrace.annotations.*;
    
    // Old (2.x)
    import org.openjdk.btrace.core.annotations.*;
    
    // New (3.0+)
    import io.btrace.core.annotations.*;
  3. Classpath issues

    • Solution: Add required JARs to classpath
    btrace -cp /path/to/lib.jar <PID> script.java

Script Verification Errors

BTrace script verification failed

Error message: the compiler names each violation, then the client gives up:

error: /path/to/MyTrace.java:12:for loops are not allowed [null]
BTrace compilation failed

BTrace enforces safety restrictions. Common violations:

1. Creating new threads

Wrong:

@OnMethod(...)
public static void handler() {
    new Thread(() -> {}).start();  // ERROR: Cannot create threads
}

Right:

@OnMethod(...)
public static void handler() {
    // Use @OnTimer instead
}

@OnTimer(1000)
public static void periodic() {
    // Executes every second
}

2. Using synchronization

Wrong:

private static Object lock = new Object();

@OnMethod(...)
public static void handler() {
    synchronized (lock) {  // ERROR: Cannot use synchronization
        // ...
    }
}

Right:

import static io.btrace.core.BTraceUtils.Atomic.*;
import java.util.concurrent.atomic.AtomicInteger;

private static AtomicInteger counter = newAtomicInteger(0);

@OnMethod(...)
public static void handler() {
    incrementAndGet(counter);  // Thread-safe without synchronization
}

3. Loops

Wrong:

@OnMethod(...)
public static void handler() {
    // ERROR: All loops forbidden in safe mode
    while (condition) { }
    for (int i = 0; i < 100; i++) { }
    do { } while (condition);
    for (String s : collection) { }
}

Right:

@OnMethod(...)
public static void handler() {
    // Use BTraceUtils methods instead
    // For iteration, use trusted mode (@BTrace(trusted = true) plus the -u flag) if absolutely necessary
}

Note: ALL loop constructs (for, while, do-while, enhanced for) are forbidden in safe mode. Use trusted mode only in controlled environments if loops are required: the script must be declared @BTrace(trusted = true) and deployed with -u; -u alone does not relax the compile-time checks.

4. File I/O operations

Wrong:

@OnMethod(...)
public static void handler() {
    FileWriter fw = new FileWriter("out.txt");  // ERROR: No file I/O
}

Right:

@OnMethod(...)
public static void handler() {
    println("Output");  // Use BTraceUtils output functions
}

// Or run in trusted mode: declare the script @BTrace(trusted = true) and deploy it with
// btrace -u <PID> script.java

5. Calling external methods

Wrong:

@OnMethod(...)
public static void handler() {
    MyOtherClass.someMethod();  // ERROR: Cannot call external class methods
    System.out.println("test");  // ERROR: Cannot call System methods
}

Right:

@BTrace
public class MyTrace {
    @OnMethod(...)
    public static void handler() {
        // Call helper methods within same BTrace class - OK
        myCustomMethod();

        // Use BTraceUtils methods
        println(str(timestamp()));
    }

    private static void myCustomMethod() {
        // Helper methods in same class are allowed
        println("Helper method called");
    }
}

Note: You CAN call private static methods within the same BTrace class. You CANNOT call methods from external classes (except BTraceUtils).

6. Catching exceptions

Wrong:

@OnMethod(...)
public static void handler() {
    try {
        // ...
    } catch (Exception e) {  // ERROR: Cannot catch exceptions
        // ...
    }
}

Right:

// BTrace handles exceptions automatically
@OnMethod(...)
public static void handler() {
    // Any exception here is caught and logged by BTrace
}

// Or use @OnError to track exceptions
@OnMethod(..., location = @Location(Kind.ERROR))
public static void onError(Throwable t) {
    println("Exception: " + str(t));
}

No Output from Scripts

Script attaches but produces no output

Diagnostic checklist:

  1. Verify method is being called

    // Add entry point logging
    @OnMethod(clazz = "com.example.MyClass", method = "<init>")
    public static void onInit() {
        println("Constructor called - script is working!");
    }
  2. Check class and method names

    # Class names are case-sensitive and must be fully qualified
    
    # Wrong
    @OnMethod(clazz = "MyClass", method = "process")
    
    # Right
    @OnMethod(clazz = "com.example.pkg.MyClass", method = "process")
  3. Test with wildcard patterns

    // Too specific - might not match
    @OnMethod(clazz = "com.example.MyClass", method = "processData")
    
    // Broader - helps identify the issue
    @OnMethod(clazz = "/com\\.example\\..*/", method = "/.*/")
    public static void catchAll() {
        println("Matched something!");
    }
  4. Verify regex escaping

    // Wrong - dots match any character
    @OnMethod(clazz = "/com.example..*/")
    
    // Right - dots are escaped
    @OnMethod(clazz = "/com\\.example\\..*/")
  5. Check for inner classes

    // Inner classes use $ separator
    @OnMethod(clazz = "com.example.Outer$Inner", method = "method")
  6. Enable verbose output

    btrace -v <PID> script.java

    -v turns on the client's debug log. It also puts the agent in debug mode, but only when this attach is the one that loads the agent into the target; the agent's log (for example Attempting to retransform class: ...) is written to the target JVM's output, not to the client.

  7. Verify instrumention occurred

    # Dump instrumented classes
    btrace -d /tmp/btrace-dump <PID> script.java
    
    # Check if classes were modified
    ls -la /tmp/btrace-dump/

    The instrumented target classes (<Class>.class next to <Class>_orig.class) are dumped only when this attach is the one that loads the agent; against an agent that is already loaded, -d dumps the probe's own classes only.

Class/Method Not Found

No methods matched

Symptom: BTrace prints no message when a probe matches no class or method: the client reports Successfully started BTrace probe: ... and the handler never fires.

Solutions:

  1. Use jcmd to find exact class names

    # List loaded classes
    jcmd <PID> VM.classes | grep -i myclass
    
    # Find specific methods
    jcmd <PID> VM.class_hierarchy | grep -A 5 MyClass
  2. Verify BTrace is retransforming loaded classes

    • BTrace scans ALL already-loaded classes when attaching
    • It retransforms matching classes via Instrumentation.retransformClasses()
    • It also listens for newly loaded classes
    • If classes aren't being instrumented, check if they are modifiable:
    # Some classes cannot be retransformed (e.g., native methods, JVM internals)
    # Use -v on the attach that loads the agent to see, in the target JVM's output,
    # which classes are being retransformed
    btrace -v <PID> script.java
  3. Check class loader hierarchy

    // Match classes loaded by any classloader
    @OnMethod(clazz = "+com.example.MyClass", method = "method")
    //         ^ plus sign matches subclasses and all classloaders
  4. Verify method signature for overloaded methods

    // Multiple methods with same name - need signature
    @OnMethod(clazz = "com.example.MyClass",
             method = "process",
             type = "void (java.lang.String, int)")

Performance Issues

BTrace causes significant slowdown

Symptoms:

  • Application becomes unresponsive
  • High CPU usage
  • Increased latency

Solutions:

  1. Use sampling

    @Sampled(kind = Sampled.Sampler.Adaptive)
    @OnMethod(...)
    public static void handler() {
        // Only executes on sampled invocations
    }
  2. Add level filtering

    @OnMethod(clazz = "...", enableAt = @Level(">=2"))
    public static void heavyHandler() {
        // Only active when level >= 2
    }
    
    // Start at a given level with a script argument: btrace <PID> MyTrace.java level=0
    // Change it later from a handler, e.g. @OnEvent("quiet") -> BTraceUtils.setInstrumentationLevel(0),
    // triggered from the client console (Ctrl-C, "send event"). There is no console "level" command.
  3. Avoid tracing high-frequency methods

    // BAD - called millions of times per second
    @OnMethod(clazz = "java.lang.String", method = "charAt")
    
    // GOOD - application-specific methods
    @OnMethod(clazz = "com.example.Service", method = "handleRequest")
  4. Minimize instrumentation scope

    // Too broad - matches thousands of methods
    @OnMethod(clazz = "/java\\..*/", method = "/.*/")
    
    // Focused - matches specific target
    @OnMethod(clazz = "com.example.Service", method = "slowMethod")
  5. Reduce output volume

    // Bad - prints every invocation
    @OnMethod(...)
    public static void handler() {
        println("Called");
    }
    
    // Better - periodic summary
    // (needs `import static io.btrace.core.BTraceUtils.*;` and
    // `import java.util.concurrent.atomic.AtomicInteger;`)
    private static AtomicInteger count = newAtomicInteger(0);
    
    @OnMethod(...)
    public static void handler() {
        incrementAndGet(count);
    }
    
    @OnTimer(5000)
    public static void report() {
        println("Calls in last 5s: " + str(get(count)));
        set(count, 0);
    }
  6. Record distributions with the metrics extension instead of logging every call: inject MetricsService from the bundled btrace-metrics extension and report histogram snapshots on a timer (see Getting Started).

Probe removed: a budget was exceeded

Error message: the client prints one line and exits with code 5:

BTrace: probe HotPathProbe removed: 23478274 hits/s exceeded probeMaxRate=10000
BTrace: probe HotPathProbe removed: lifetime exceeded probeTtl=600s

HotPathProbe is the probe class as you declared it, with its package if it has one. Read exit code 5 together with this line: a probe can also end itself with exit(5). A client that has stopped reading, or whose exit takes too long, is disconnected instead, possibly without this line or exit code; the line logged in the target JVM is then the record.

Cause: the operator started the target JVM with probeMaxRate or probeTtl (launch-time agent argument or io.btrace.* system property, see Probe budgets) and the probe went over it. The agent stopped the probe's handlers and removed its instrumentation. The check runs once per second, so a rate breach is acted on about two seconds after deploy.

Solutions:

  • Narrow the probe: a more specific clazz/method, @Sampled, or a handler that runs less often.
  • Ask the operator to raise the limit; it is fixed at JVM launch and a client cannot change it.
  • A detached probe has no client to tell, so its removal is recorded only in the target JVM's log, as a WARN line from the BTrace Budget Exit thread:
    [BTrace Budget Exit] WARN io.btrace.agent.Client - BTrace: probe HotPathProbe removed: 23478274 hits/s exceeded probeMaxRate=10000
    
    Afterwards the probe no longer appears in the probe list.
  • A probe loaded from script= or scriptdir=, or bundled into the agent, is limited only when the agent was loaded by dynamic attach. It has no client either: its removal is logged in the target JVM the same way and written to the probe's output.

Trusted mode is not available under a probe budget

Error message: a deploy with -u prints one line and exits with code 1:

BTrace: trusted mode is not available: this JVM has a probe budget set by its operator

Cause: the target JVM has a probe budget (probeMaxRate or probeTtl, see Probe budgets). A trusted probe is not verified and could defeat the budget, so the agent refuses it. Nothing was instrumented and the target keeps running.

Solutions:

  • Deploy without -u. Most probes do not need trusted mode.
  • If the probe really needs trusted mode, ask the operator. Only trusted=true in the target's launch-time -javaagent arguments enables it while a budget is set. An attach-time trusted=true is ignored, so a script= probe that needs trusted mode fails verification.
  • A bundled probe that needs trusted mode is then skipped, and agent startup continues, with one line in the target JVM: bundled probe '<name>' did not pass verification and was not loaded: trusted mode is not available: this JVM has a probe budget set by its operator.

Agent Loading Problems

Agent JAR not found

Error message:

Error opening zip file or JAR manifest missing : /path/to/btrace.jar

Solutions:

# Verify JAR exists
ls -la /path/to/btrace.jar

# Use absolute path
java -javaagent:/absolute/path/to/btrace.jar=script=Script.class MyApp

# Or set BTRACE_HOME
export BTRACE_HOME=/path/to/btrace
java -javaagent:$BTRACE_HOME/libs/btrace.jar=script=Script.class MyApp

Script class not found in agent mode

Symptom: no error is printed. A script= entry that the agent cannot read as a compiled probe (a path that does not exist, or a .java source) is skipped silently and the application starts untraced; with debug=true the agent logs initial script is ... and nothing more for it.

Solutions:

# Specify absolute path to the compiled script; a relative path is resolved against the
# application's working directory
java -javaagent:btrace.jar=script=/absolute/path/MyTrace.class MyApp

Invalid agent option: the command server was not started

Error message:

[BTrace Agent] invalid agent option bindAddress=0.0.0.0: BTrace 3.0 binds loopback addresses only (127.0.0.1 or ::1); the command server was not started

(or invalid agent option port=70000: expected a port number between 0 and 65535; ...)

Cause: the port= or bindAddress= agent argument has a value the agent cannot use. The agent names the option and value on one line, skips the command server, and lets the application and any script= startup probes run; btrace <pid> later reports no endpoint (btrace doctor <pid> exits 2). An invalid io.btrace.probeTtl/io.btrace.probeMaxRate system property is handled the same way, except that when the agent is loaded by dynamic attach it loads no probe itself (script=, scriptdir=, bundled probes) and the line ends with attach-time probes were not loaded.

Solutions:

  • bindAddress must be a loopback address (127.0.0.1 or ::1); remote prepared mode is not supported in 3.0.0
  • port must be 0–65535; use port=0 to let the OS pick a free port that 3.0 clients discover through the Attach API

A fixed port that is already in use is a different failure: binding throws a plain java.net.BindException and agent initialization aborts with [BTrace Agent] FATAL: Initialization failed: java.net.BindException ...; the JVM then exits before the application starts (FATAL ERROR in native method: processing of -javaagent failed). Pick another port or use port=0.

Compatibility Issues

JDK 25+ Thread Method Exclusions

BTrace automatically excludes certain java.lang.Thread methods from instrumentation on JDK 25+ to prevent infinite recursion and interference with JVM internals:

  • ThreadLocal accessors — threadLocals(), setThreadLocals(), inheritableThreadLocals(), setInheritableThreadLocals(), terminatingThreadLocals(), setTerminatingThreadLocals(): JDK 25 changed direct field access to accessor methods. Instrumenting these causes infinite recursion when BTrace runtime uses ThreadLocal.
  • Interrupt methods — getAndClearInterrupt(), setInterrupt(), clearInterrupt(): Used internally by Thread.interrupted(), which BTrace's own runtime calls.
  • Exception dispatch — dispatchUncaughtException(), getUncaughtExceptionHandler(): Instrumenting these interferes with thread exception handling.

These exclusions are enforced in ClassFilter.SENSITIVE_METHODS and cannot be overridden by user scripts. The methods are skipped silently, so a probe on one of them never firing is expected behavior.

Java Version Mismatch

Error message:

UnsupportedClassVersionError

Solutions:

  • Ensure BTrace supports your Java version. BTrace 3.0 runs on Java 8–27+. Running BTrace against a JVM older than Java 17 is deprecated: it continues to work throughout 3.x but emits a deprecation warning. Support for Java < 17 will be removed in the next major release (4.0).
  • Compile scripts with target version matching JVM
  • Check JAVA_HOME points to correct version

Module System Issues (Java 9+)

Error message:

IllegalAccessError: class X cannot access class Y

Solutions:

# Add the required --add-opens flags to the target JVM startup
# (the btrace launcher takes no JVM flags)
java --add-opens java.base/java.lang=ALL-UNNAMED \
     --add-opens java.base/jdk.internal.misc=ALL-UNNAMED \
     MyApp

Old Script Syntax

Error: Scripts written for older BTrace versions fail to compile

Solution: Update imports to the current package namespace:

// BTrace 1.x
import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;

// BTrace 2.x
import org.openjdk.btrace.core.annotations.*;
import static org.openjdk.btrace.core.BTraceUtils.*;

// BTrace 3.0+
import io.btrace.core.annotations.*;
import static io.btrace.core.BTraceUtils.*;

Note that in BTrace 3.0 most imports are unnecessary: the compiler auto-imports the annotations and the flat DSL (import static io.btrace.BTrace.*) into every script. Also, compiled 2.x probe .class files that reference org.openjdk.btrace are automatically migrated to the io.btrace namespace by the agent at load time — no recompilation needed.

Debugging BTrace Scripts

Enable verbose logging

btrace -v <PID> script.java

Dump instrumented classes

btrace -d /tmp/btrace-classes <PID> script.java

Then examine with:

javap -c /tmp/btrace-classes/com/example/MyClass.class

Print diagnostic info

@OnMethod(clazz = "com.example.MyClass", method = "myMethod")
public static void handler(@ProbeClassName String clazz,
                          @ProbeMethodName String method) {
    println("Class: " + clazz);
    println("Method: " + method);
    println("Thread: " + threadName(currentThread()));
    println("Stack:");
    printStack();  // Print the current stack
}

Test incrementally

Start simple and add complexity:

// Step 1: Verify attachment
@OnTimer(1000)
public static void heartbeat() {
    println("BTrace alive");
}

// Step 2: Verify class matching
@OnMethod(clazz = "com.example.MyClass", method = "/.*/")
public static void anyMethod() {
    println("Matched");
}

// Step 3: Add specific logic
@OnMethod(clazz = "com.example.MyClass", method = "specificMethod")
public static void specific(String arg) {
    println("Arg: " + arg);
}

Kubernetes and Cloud Deployments

Finding Process ID in Pods

Problem: Need to find Java process ID inside a pod

Solutions:

# List Java processes in pod
kubectl exec <pod-name> -- jps

# Or use ps
kubectl exec <pod-name> -- ps aux | grep java

# For multi-container pods, specify container
kubectl exec <pod-name> -c <container-name> -- jps

Multi-Container Pod Considerations

Problem: Cannot attach to process in different container

Solution: Kubernetes pods with multiple containers have separate process namespaces by default.

# Enable process namespace sharing
apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  shareProcessNamespace: true  # Required for cross-container attachment
  containers:
  - name: app
    image: myapp:latest
  - name: btrace-sidecar
    image: ghcr.io/btraceio/btrace:3.0.0-alpine

Note: With shareProcessNamespace: true, all containers can see each other's processes.

Sidecar attach times out when the app's working directory is read-only

Problem: attaching from a sidecar fails with Unable to open socket file /proc/1/root/tmp/.java_pid1: target process 1 doesn't respond, but only when the application's working directory is not writable (for example a read-only volume used as workingDir).

Cause: the attaching JVM first creates a trigger file, .attach_pid<pid>, in the target's working directory. When that fails, a JDK 11 or 17 client (11.0.32 and 17.0.21 tested) falls back to the target's /tmp only if the target's PID differs between the two containers. With a shared PID namespace it does not, so the client creates the trigger in its own /tmp, the target never sees it, and the attach times out. JDK 21 and later (21.0.13 and 25.0.5 tested) fall back to the target's /tmp.

Solution: run the client on JDK 21 or later. On an older client, create the trigger in the target's /tmp before attaching (<pid> as seen from the sidecar, <nspid> the target's PID in its own namespace, the last value of the NSpid line in /proc/<pid>/status):

touch /proc/<pid>/root/tmp/.attach_pid<nspid>
btrace <pid> script.java

Either way the target's /tmp must be writable; with readOnlyRootFilesystem: true, mount an emptyDir at /tmp in the application container.

Port Conflicts in Pods

Problem: BTrace fails with port already in use

Error:

java.net.BindException: Address already in use

Cause: BTrace uses port 2020 by default; multiple BTrace instances or port conflicts

Solutions:

# Use different port
kubectl exec <pod> -- btrace -p 3030 <PID> script.java

# Check port usage
kubectl exec <pod> -- netstat -an | grep 2020

Security Policies and RBAC

Problem: Pod Security Policy blocks BTrace attachment

Error:

Operation not permitted

Common causes:

  1. Pod Security Standards (PSP replacement in K8s 1.25+):
apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  securityContext:
    # May need to adjust based on policy
    runAsNonRoot: true
    capabilities:
      add:
      - SYS_PTRACE  # Required for process attachment
  1. SELinux/AppArmor:
# Check if SELinux is blocking
kubectl exec <pod> -- getenforce

# Check AppArmor profile
kubectl exec <pod> -- cat /proc/self/attr/current

Solutions:

  • Add SYS_PTRACE capability if allowed by security policy
  • Run as same user as target JVM
  • Adjust security context constraints (OpenShift)

JDK Not Available in Container

Problem: BTrace requires JDK but only JRE in container

Error:

Unable to locate tools.jar. Please, make sure JAVA_HOME points to a valid JDK installation

Solutions:

Option 1: Use JDK base image

# Instead of JRE
FROM eclipse-temurin:21-jre
# Use JDK
FROM bellsoft/liberica-openjdk-debian:21-cds

Option 2: Install JDK in running pod (temporary)

kubectl exec <pod> -- apt-get update && apt-get install -y openjdk-21-jdk

Option 3: Sidecar with JDK

spec:
  shareProcessNamespace: true
  containers:
  - name: app
    image: myapp-jre:latest  # Can use JRE
  - name: btrace
    image: bellsoft/liberica-openjdk-debian:21-cds     # Sidecar has JDK

Service Mesh Compatibility

Istio/Linkerd:

BTrace works with service meshes but some considerations apply:

Traffic Interception:

  • Service mesh sidecars don't interfere with BTrace (different layer)
  • BTrace communication port (2020) not intercepted by mesh

mTLS:

  • BTrace uses local socket communication (not HTTP)
  • mTLS between services doesn't affect BTrace

Resource Limits:

# BTrace overhead may trigger limits
resources:
  requests:
    cpu: "100m"
    memory: "128Mi"
  limits:
    cpu: "500m"      # Increase if BTrace causes throttling
    memory: "512Mi"   # Increase if needed

ConfigMap for BTrace Scripts

Pattern: Store scripts in ConfigMap for easy distribution

apiVersion: v1
kind: ConfigMap
metadata:
  name: btrace-scripts
data:
  TraceMethod.java: |
    import io.btrace.core.annotations.*;
    import static io.btrace.core.BTraceUtils.*;

    @BTrace
    // Classname 'TraceMethod' must correspond to the file name 'TraceMethod.java'
    public class TraceMethod {
        @OnMethod(clazz = "com.example.Service", method = "process")
        public static void onProcess() {
            println("Method called");
        }
    }
---
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    image: myapp:latest
    volumeMounts:
    - name: scripts
      mountPath: /btrace-scripts
  volumes:
  - name: scripts
    configMap:
      name: btrace-scripts

Usage:

kubectl exec <pod> -- btrace <PID> /btrace-scripts/TraceMethod.java

Batch Tracing Across Pods

Pattern: Trace all pods in a deployment

#!/bin/bash
# trace-deployment.sh

DEPLOYMENT=$1
SCRIPT=$2

# Get all pod names for deployment
PODS=$(kubectl get pods -l app=$DEPLOYMENT -o jsonpath='{.items[*].metadata.name}')

for POD in $PODS; do
  echo "Attaching to $POD..."
  kubectl exec $POD -- btrace 1 $SCRIPT &
done

wait
echo "All traces complete"

Usage:

./trace-deployment.sh myapp /tmp/trace.java

Persistent Tracing with StatefulSets

Issue: BTrace doesn't persist across pod restarts

Solutions:

  1. Store output in persistent volume:
kubectl exec <pod> -- btrace -o /data/trace.log <PID> script.java
  1. Stream to external system:
// Use the bundled btrace-statsd extension (import io.btrace.statsd.Statsd;
// the agent must allow it: allowExtensions=btrace-statsd)
@Injected
private static Statsd statsd;

@OnMethod(...)
public static void handler() {
    statsd.increment("metric.name");
}
  1. Agent mode in pod spec:
spec:
  containers:
  - name: app
    image: myapp:latest
    env:
    - name: JAVA_TOOL_OPTIONS
      value: "-javaagent:/opt/btrace/libs/btrace.jar=script=/scripts/trace.class"

Cloud-Specific Issues

AWS EKS:

  • No specific issues; standard K8s patterns apply
  • Use same region for kubectl access

Google GKE:

  • GKE Autopilot: May restrict SYS_PTRACE capability
  • Use standard GKE for full BTrace support

Azure AKS:

  • No specific issues; standard K8s patterns apply
  • Ensure node has sufficient resources

OpenShift:

  • Security Context Constraints (SCC) may be stricter
  • May need to add SCC for SYS_PTRACE:
oc adm policy add-scc-to-user anyuid -z default

For more K8s deployment patterns, see Getting Started: K8s and FAQ: Microservices.

Known Limitations

Cannot instrument

Native methods:

  • Methods implemented in native code cannot be instrumented (JVM limitation)

Sensitive classes (to avoid infinite recursion):

  • Core classes BTrace depends on: java.lang.Object, String, ThreadLocal, Integer, Number
  • Package prefixes: java.lang.instrument.*, java.lang.invoke.*, java.lang.ref.*
  • Concurrency classes: everything under java.util.concurrent.* (locks, atomics, executors, ...)
  • JDK internals: jdk.internal.*, sun.invoke.*, sun.reflect.*, java.lang.classfile.*
  • BTrace itself: io.btrace.*

Classes annotated with @BTrace (BTrace scripts themselves)

Note: Classes already transformed by other agents CAN be retransformed. Boot classpath and JDK classes CAN be instrumented (except the sensitive classes listed above).

Cannot trace

  • Very early VM initialization (use agent mode instead of attach mode)
  • Class loading before agent initialization
  • JVM shutdown sequence (partially)

Platform-specific issues

Windows:

  • Attach API may require admin privileges
  • Path separators: use \\ or /

macOS:

  • SIP (System Integrity Protection) may block attachment
  • Disable for debugging: csrutil disable (not recommended for production)

Linux:

  • Some distros have ptrace restrictions in /proc/sys/kernel/yama/ptrace_scope
  • Set to 0 to allow: echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

Getting Help

If issues persist:

  1. Check BTrace version

    btrace --version
  2. Search existing issues

  3. Provide details when asking for help:

    • BTrace version
    • Java version (java -version)
    • OS and version
    • Complete error message
    • Minimal reproducing script
    • Target application details
  4. Community channels:

See Also