Compact terminal dashboards for monitoring ComfyUI workloads on Ubuntu/Linux with either AMD ROCm or NVIDIA CUDA GPUs.
BF95 VRAMWATCH provides live GPU and system-memory telemetry to help distinguish normal rendering spikes from sustained memory pressure, swap thrashing, and gradual ComfyUI memory accumulation.
Both the AMD ROCm and NVIDIA CUDA versions have been successfully tested on physical hardware.
| Script | Platform | GPU telemetry |
|---|---|---|
BF95-AMD-VRAMWATCH.sh |
AMD ROCm | amd-smi |
BF95-CUDA-VRAMWATCH.sh |
NVIDIA CUDA | nvidia-smi |
- GPU VRAM usage and free VRAM
- System RAM, available RAM, and swap usage
- ComfyUI process RSS, anonymous memory, swap, and total footprint
- Peak RSS, kernel
VmHWM, lowest available RAM, and process uptime - Short- and long-term RSS trends to expose gradual memory accumulation
- Major page-fault rate, swap-in/out activity, and Linux PSI memory pressure
- GPU utilization and memory-controller activity
- GPU temperatures, cooling, power draw, and clock speeds
- Rolling 60-second memory guard with percentage-based warning thresholds
- Colour terminal dashboard with ASCII and no-colour modes
- No sudo requirement
- No continuous kernel-log scanning
- No Python packages or additional libraries required
- Ubuntu or another Linux distribution with
/proc - Bash
- Standard command-line tools including
awk,grep,getconf, andps - ComfyUI launched as a Python
main.pyprocess
- A supported AMD GPU
- A working ROCm installation
amd-smiavailable inPATH
- A supported NVIDIA GPU
- A working proprietary NVIDIA driver
nvidia-smiavailable inPATH
Some NVIDIA telemetry, such as VRAM temperature, depends on the GPU model and driver. Unsupported values are displayed as N/A rather than causing the dashboard to fail.
Clone or download the repository, then make the appropriate script executable:
chmod +x BF95-AMD-VRAMWATCH.sh
chmod +x BF95-CUDA-VRAMWATCH.sh./BF95-AMD-VRAMWATCH.sh./BF95-CUDA-VRAMWATCH.sh-g, --gpu ID GPU index to watch (default: 0)
-i, --interval SEC Refresh interval in seconds (default: 2)
-w, --width COLS Meter width, 10-60 (default: 32)
-1, --once Print one snapshot and exit
--no-color Disable ANSI colours
--ascii Use ASCII bar characters
-h, --help Show help
Examples:
./BF95-AMD-VRAMWATCH.sh --gpu 0 --interval 1
./BF95-CUDA-VRAMWATCH.sh --once
NO_COLOR=1 ./BF95-AMD-VRAMWATCH.shBy default, both scripts look for a process matching:
[p]ython.*main.py
The detection pattern can be overridden when needed:
VRAMWATCH_COMFY_PATTERN='python.*main.py' ./BF95-AMD-VRAMWATCH.shThe same environment variable works with the CUDA version.
The memory guard uses a rolling 60-second average so that normal short-lived rendering spikes do not immediately trigger a restart warning.
Default thresholds scale with installed system RAM:
- Caution: available RAM below 20%, ComfyUI RSS above 75%, or ComfyUI swap at or above 2 GiB
- Critical: available RAM below 10% or ComfyUI RSS above 85%
The thresholds and RSS trend windows can be adjusted with environment variables. Run either script with --help to see the supported variables.
The dashboard includes several Linux and GPU-monitoring terms that may be unfamiliar. The following section explains what each reading means and which ones deserve the most attention.
VRAM means Video Random Access Memory. It is the dedicated memory installed on the graphics card.
ComfyUI uses VRAM to hold models, tensors, latent images, and other data required during generation.
High VRAM usage is often normal. Large diffusion models may use nearly all available VRAM while generating an image.
Watch for:
- A generation failing with an out-of-memory error
- Repeated model unloading and reloading
- Severe slowdowns caused by data being moved between VRAM and system RAM
A red or nearly full VRAM bar does not automatically mean something is wrong.
RAM is the computer's main system memory.
The System RAM bar shows how much total memory Linux is currently using. The accompanying Available system RAM value is especially important because it estimates how much memory remains available for new work without relying heavily on swap.
Watch for:
- Available RAM remaining below 20% for an extended period
- Available RAM continuing to fall after each completed generation
- Available RAM approaching zero
Swap is disk space that Linux uses as an overflow area when physical RAM becomes crowded.
Swap is much slower than RAM. A small amount of swap usage is not necessarily a problem, especially if it remains stable. Active swap movement during generation can cause major slowdowns.
Watch for:
- ComfyUI swap usage that keeps increasing
- Sustained Swap In or Swap Out activity
- Generations becoming progressively slower
- The desktop or SSH session becoming sluggish
RSS means Resident Set Size. It measures how much physical system RAM is currently occupied by the ComfyUI process.
This is one of the most useful readings for identifying gradual memory accumulation.
Watch for:
- RSS increasing across several completed generations
- RSS failing to fall after a workflow finishes
- RSS approaching most of the computer's installed RAM
Normal workflows may cause RSS to rise and fall as models and working data are loaded and released.
Anonymous RSS, shown as Anon RSS, is memory allocated directly by the ComfyUI Python process rather than memory backed by ordinary files on disk.
This commonly includes tensors, model working memory, Python objects, and other runtime allocations.
A high value is not automatically abnormal, but continuous growth may indicate memory being retained between jobs.
The total footprint combines ComfyUI's resident RAM and its process swap usage.
It provides a broader estimate of how much system-memory capacity is associated with ComfyUI, including memory that Linux has moved out to swap.
Peak RSS is the highest ComfyUI RSS value observed since BF95 VRAMWATCH was started.
This helps reveal brief memory spikes that may have disappeared before you looked at the dashboard.
VmHWM means Virtual Memory High Water Mark.
Despite the name, this value represents the highest resident-memory usage recorded by the Linux kernel for the ComfyUI process since that process started.
Unlike the dashboard's Peak RSS reading, Kernel VmHWM may include peaks that occurred before BF95 VRAMWATCH was launched.
Low Avail is the lowest amount of available system RAM observed since BF95 VRAMWATCH was started.
This is useful for determining how close the system came to exhausting its memory during a generation.
RSS growth compares the current ComfyUI RSS with its value when BF95 VRAMWATCH began monitoring the process.
A positive value is normal while a model or workflow is loading. Continued growth across many completed jobs is more significant.
The short and long RSS trends show how ComfyUI memory usage has changed over recent time windows.
Interpretation:
- A positive trend during generation may be normal
- A trend that returns toward zero after completion usually indicates memory reuse
- A consistently positive long-term trend may indicate gradual memory accumulation
- A strongly negative trend usually means a model or large working allocation was released
A page fault occurs when a process requests a memory page that is not immediately available in physical RAM.
BF95 VRAMWATCH focuses on major page faults. A major fault may require Linux to retrieve data from disk or swap, which is much slower than accessing RAM.
Watch for:
- Sustained major faults while ComfyUI is sampling
- Major faults combined with active swap input
- Slow generations while available RAM is very low
An occasional burst during a model transition is not necessarily a problem.
PSI means Pressure Stall Information. It is a Linux measurement of how much time processes spend delayed because a resource is under pressure.
The dashboard displays memory PSI:
- PSI some: at least one process was delayed by memory pressure
- PSI full: all runnable work was stalled by memory pressure at the same time
Brief low readings are generally harmless. Sustained or rising readings suggest that insufficient RAM or active swapping is affecting performance.
GPU utilization shows how busy the GPU's processing cores are.
During sampling, a reading near 100% usually means the GPU is being used efficiently. Low utilization may be normal during model loading, VAE decoding, file saving, or CPU-dependent stages.
Memory-controller activity indicates how heavily the GPU is accessing its VRAM.
It does not need to match GPU utilization. Some operations are limited primarily by computation, while others depend more heavily on memory bandwidth.
These readings show the GPU's operating conditions:
- GPU temperature: primary GPU core temperature
- Hotspot: hottest reported area of an AMD GPU, when available
- VRAM temperature: graphics-memory temperature, when supported
- Fan: cooling-fan activity
- Power: current GPU power draw
- GFX Clock: current graphics-core clock speed
- Performance state: NVIDIA power and performance state, such as
P0
Temperatures and fan behaviour vary between GPU models. Consult the GPU manufacturer's specifications when evaluating safe operating limits.
For most ComfyUI users, the most important readings are:
- Available system RAM
- ComfyUI RSS and its long-term trend
- ComfyUI swap usage
- Swap In activity
- Major page faults
- PSI memory pressure
- The rolling Memory Guard status
A typical healthy generation may show:
- Very high VRAM use
- High GPU utilization
- A temporary increase in RSS
- Little or no active swap input
- Available RAM recovering when the job finishes
- RSS trends stabilizing or falling after completion
Possible signs that a ComfyUI restart may be useful include:
- Available RAM remains critically low
- ComfyUI RSS rises after every completed job
- Process swap continues increasing
- Swap In remains active
- Major page faults continue during sampling
- PSI pressure remains elevated
- Each new generation becomes slower than the previous one
- The Memory Guard remains in caution or critical status
The dashboard is intended as a diagnostic aid rather than an automatic failure detector. Consider several readings together rather than treating any single red bar as proof of a problem.
The AMD version uses the current amd-smi interface rather than the deprecated rocm-smi command.
The CUDA version uses nvidia-smi for GPU telemetry and process-specific CUDA VRAM reporting.
It has been successfully tested on physical NVIDIA CUDA hardware. Hardware- or driver-specific telemetry that is unavailable is shown as N/A.
BF95 VRAMWATCH is released under the MIT License. See the included LICENSE file.