Start a JVM in prepared mode, prove it is ready without loading anything, attach with no flags, and prove that nothing is left behind when you leave. The target app reports its own throughput, so the cost of the probe, and its disappearance, show up as a number you can watch.
Recorded from demos/07-production-safe-attach/run.sh; the MP4 is next to the GIF.
Persona: an SRE or on-call engineer who has been burned by agents that cannot be removed. Time: ~10 minutes.
- JDK 11 or newer with
JAVA_HOMEset (bin/btracerefuses to start without it) - BTrace 3.0 installed, with
BTRACE_HOMEpointing at it andbin/btraceon your PATH (installation options) - The two files from demos/07-production-safe-attach: HotLoopApp.java (the target) and HotPathProbe.java (the probe). Run the commands below from that directory.
- Three terminal windows
The target is a single thread calling one cheap method, Work.step, in a tight loop. It prints its
own throughput every two seconds. Start it in terminal 1 with the BTrace agent loaded at launch in
prepared mode (port=0):
java -javaagent:$BTRACE_HOME/libs/btrace.jar=port=0 HotLoopApp.javaYou should see, after a couple of dozen [main] INFO lines in which the agent reports
initializing its extension system and loading the bundled extensions (left out here; absolute
numbers depend on your machine, so yours will differ):
[demo] hot loop running - stop with Ctrl+C
[demo] 60.6 M calls/s
[demo] 65.7 M calls/s
Note the baseline: that is what "healthy" looks like. Get the PID in terminal 2 (jps -m, look for
Main HotLoopApp.java); you need it for every command below.
What just happened? The agent started, but it instrumented nothing. Prepared mode means the agent only opens a control endpoint: it binds a loopback address (never a wildcard or remote one), generates an owner-protected token file, and publishes the actual address, port, and token-file path through JVM properties.
port=0asks the OS for a free port, so there is nothing to collide with. The token is never a process argument or system property.
Before you attach anything, check readiness. In terminal 2:
btrace doctor <PID>You should see:
BTrace doctor for PID 89156
Status: READY (exit 0)
Target JDK: 17.0.12 (Oracle Corporation)
Attach API: available
Target attach: accessible
Dynamic agent loading: not tested (doctor never loads an agent)
BTrace endpoint: ready at 127.0.0.1:58360
Authentication: required; credential file readable
Detail: A BTrace control endpoint is ready.
Observation: doctor did not inspect the target VM flag state.
Actions: read target system properties and detached; no agent load or BTrace command connection was attempted.
(On some JDKs a Java HotSpot ... warning: Option AllowRedefinitionToAddDeleteMethods was deprecated line precedes the report. It comes from the JVM, not from BTrace.) The command exits
with code 0.
Line by line:
Status: READY (exit 0)- the exit codes are stable and scriptable:0ready,2PREPARATION_REQUIRED,3INACCESSIBLE,4unexpected failure or invalid invocation.btrace doctor <PID> --jsongives the same answer for automation.Target JDKandAttach API/Target attach- the JVM can be reached through the Attach API.Dynamic agent loading: not tested- doctor never loads an agent, so it cannot tell you whether one would be allowed. That is deliberate.BTrace endpoint: ready at 127.0.0.1:58360- the loopback address and the port the OS picked.Authentication: required; credential file readable- the token file exists and your user can read it.Actions:- the receipt. It states exactly what doctor did: read system properties and detach.
What just happened? Doctor read the JVM's published properties through the Attach API and detached. No agent was loaded, no class was touched, and the BTrace command socket was never opened. It is safe to run against anything.
For contrast, start the same app in terminal 3 without the agent:
java HotLoopApp.javaFind its PID and ask the doctor:
btrace doctor <PLAIN_PID>You should see:
BTrace doctor for PID 89358
Status: PREPARATION_REQUIRED (exit 2)
Target JDK: 17.0.12 (Oracle Corporation)
Attach API: available
Target attach: accessible
Dynamic agent loading: not tested (doctor never loads an agent)
BTrace endpoint: not ready
Detail: No BTrace control endpoint is published.
Observation: doctor did not inspect the target VM flag state.
Remediation:
- Relaunch with -XX:+EnableDynamicAgentLoading before using dynamic attach.
- Relaunch with -javaagent:/path/to/btrace.jar=port=0 for reviewed prepared mode.
Actions: read target system properties and detached; no agent load or BTrace command connection was attempted.
The exit code is 2.
What just happened? Doctor found no published endpoint and told you your two options for the next deployment: allow dynamic agent loading, or launch in prepared mode as you did in Step 1. Production is where you want to have chosen prepared mode ahead of time. Stop this plain JVM now (
Ctrl+Cin terminal 3); you do not need it again.
In terminal 2, attach the probe to the prepared JVM from Step 1. No -p, no token:
btrace <PID> HotPathProbe.javaYou should see:
[main] INFO io.btrace.client.Main - Attaching BTrace to PID: 89156
[main] INFO io.btrace.client.Client - Successfully started BTrace probe: HotPathProbe.java
[probe] attached
The [probe] attached line repeats every two seconds. Now look at terminal 1:
[demo] 6.3 M calls/s
[demo] 7.2 M calls/s
[demo] 7.1 M calls/s
The throughput fell by roughly an order of magnitude, from the baseline you noted in Step 1.
What just happened? The client found the port and the token on its own: it asked the target JVM through the Attach API for the published address, port, and token-file path, then used the token. Nothing was typed, and nothing secret appeared on a command line. The slowdown is real but is a worst case on purpose. HotPathProbe.java has an empty handler on a method that is called tens of millions of times per second, so every bit of the cost is call-site dispatch, with no probe logic at all. A typical production method is called orders of magnitude less often.
Press Ctrl+C in terminal 2. The client opens a small menu; choose 1. exit. Then watch
terminal 1:
[demo] 7.1 M calls/s
[demo] 29.5 M calls/s
[demo] 51.7 M calls/s
Within a few seconds the app is back near the baseline of Step 1 (here 51.7 against 65.7 M calls/s; a busy machine will not match the baseline exactly).
What just happened? Two things, and the app's own number shows both. When the client exits, the agent restores the original bytecode of the classes it instrumented (a retransform), so the probe's code is gone from
Work.step. And the probe handlers are dispatched throughinvokedynamiccall sites that become no-ops once the client is gone, so even a window in which the old bytecode is still running leaves no lingering overhead.
So far you removed the probe yourself. The operator of the JVM can also set a limit that the agent enforces on its own. Stop the app in terminal 1 and restart it with a rate ceiling of 10,000 probe hits per second:
java -javaagent:$BTRACE_HOME/libs/btrace.jar=port=0,probeMaxRate=10000 HotLoopApp.javaAttach the same probe from terminal 2 and check the exit code:
btrace '~HotLoopApp' HotPathProbe.java; echo "exit code: $?"You should see (the hits/s figure is what the agent measured, so yours will differ):
[main] INFO io.btrace.client.Main - Attaching BTrace to PID: 70839
[main] INFO io.btrace.client.Client - Successfully started BTrace probe: HotPathProbe.java
BTrace: probe HotPathProbe removed: 23478274 hits/s exceeded probeMaxRate=10000
[BTRACE] terminal cleanup: runtime=cleaned,mbean=absent
exit code: 5
The client returned on its own, about four seconds after it started. Terminal 1 shows the app's throughput (these figures come from a separate run on another machine than the numbers in the earlier steps, with a baseline near 360 M calls/s):
[demo] 352.6 M calls/s
[demo] 107.5 M calls/s
[BTrace Budget Exit] WARN io.btrace.agent.Client - BTrace: probe HotPathProbe removed: 23478274 hits/s exceeded probeMaxRate=10000
[demo] 263.1 M calls/s
[demo] 352.1 M calls/s
[demo] 352.0 M calls/s
What just happened? The agent counts executions of the handlers a probe attaches to application code. Once a second it compared the count with
probeMaxRate, saw far more hits than allowed, switched the probe's handlers off, removed its instrumentation, sent the client the one line above, and logged the same line in the target JVM, which is the only record when the probe is detached. The dip in the two throughput windows around the removal is the time between attach and the governor's first check.probeTtl=<seconds>works the same way for a probe's lifetime. Exit code5tells scripts that this was a budget removal, not a clean exit. The limits are set by whoever launches the JVM; a client cannot raise them. See Probe budgets.
Stop the app in terminal 1 with Ctrl+C (or kill <PID>). Generated token files are removed during
orderly shutdown of the JVM, and the plain JVM from Step 3 was already stopped. There is nothing
else to undo.
btrace doctorexits3(INACCESSIBLE) - doctor could not reach the target through the Attach API. The usual cause is that BTrace and the target JVM run as different OS users; runbtraceas the target's user (see Unable to attach to target VM).- The app exits at startup, or
port=fails with aBindException- you used a fixedport=NNNNthat is already in use. That aborts agent initialization, unlike an invalid option value, which only skips the command server. Preferport=0whenever the port is not under your control. - A 2.x client cannot connect - a 3.0 prepared endpoint requires authentication before every command, and a 2.x client does not know how to authenticate, so the connection fails. Use a 3.0 client. (Dynamic attach remains compatible with 2.x clients.)
btrace <PID>reports no endpoint but you started the agent - checkbtrace doctor <PID>; an invalidport=orbindAddress=value makes the agent skip the command server and print one line naming the option (details).
- Prepared mode,
authTokenFile=,bindAddress=andbtrace doctor: Getting Started - Attach failures and their fixes: Troubleshooting: JVM Attachment Issues
- The runnable version of this tutorial: demos/07-production-safe-attach
- Same prepared-mode ideas across a container boundary: BTrace in Kubernetes, the Sidecar Way
