BSimVis

BSimVis is a tool to analyze similarities across a collection of binaries, based on Ghidra analyzers and the BSim (Behavioral Similarity) plugin. It provides an API and Web interface to upload large quantities of decompiled binaries and BSim feature vectors to a Kvrocks database for similarity analysis, function diffing, and family clustering.

BSimVis uses a custom database because Ghidra's BSim databases don't store decompiled code and other metadata. This alternative BSim database and API provide filtering and visualization of this additional data across multiple binaries at once. It doesn't aim to replace Ghidra's BSim plugin, but to enable more advanced analysis and visualization of the similarities on a large scale (family clustering, etc.).

Features

Analysis

Upload decompiled functions and BSim vectors from Ghidra
Similarity search with score filtering across multiple binaries
Function diffing based on BSim features
BSim feature correlation with decompiled C tokens / Pcode blocks
Call graph navigation (callers and callees)

Clustering

HDBSCAN-based binary family clustering and interactive file dendrogram visualization
Cluster search view with dendrogram and packing diagram
Stability and parent cluster filtering

Cross-Collection Pools

Combine multiple collections into a pool for similarity search and clustering across their union
Search endpoints accept a pool parameter; per-pool build/cluster jobs with sync-status tracking

Search & Filtering

Full text search on files and features with sorting, filtering, and pagination
Search history and caching

Web Interface

Similarity graph
Dynamic window management for multiple code previews
Tag management for files, functions, and similarities
Quick preview tooltips for clusters and diffs
Table selection and copy across all views

Analyst Notes & AI Insights

Analyst notes system for files and functions
LLM assistant for file and function summaries (local or remote via Ollama)

API

REST API with Swagger documentation
Upload API: processor/compiler config, profiling, batch metadata, and similarity params

Screenshots

Web UI Similarity Search Graph view

Web UI Diffing

Web UI Cluster Dendrogram

Call graph

Requirements

Java 21+ (Ghidra 12 requirement) — install.sh drops a portable Temurin JDK 21 into bin/ and sets JAVA_HOME in .env if the system Java is missing or older
Ghidra and pyghidra install
Redis and Kvrocks databases
Function ID databases in the Ghidra install (see below) — without them, function library tags stay empty
Ollama (optional) for LLM analyst insights and function summaries. Can be run locally (default port 11434) or remotely (configurable via ollama_url in bsimvis_config.toml).

Installation

Copy the example configuration files and customize them:

cp .env.example .env
cp bsimvis_config.toml.example bsimvis_config.toml

Run the install script to set up portable Redis, Kvrocks, and optionally Ghidra:

./install.sh

Milvus support is optional and can be enabled via the .env file (ENABLE_MILVUS=true).

YARA rulesets

The yara analysis module runs against data/yara_rules/. No third-party rule is kept in git — only the two in-house rules under house/. The other ~587 files come from ReversingLabs (MIT), elastic/protections-artifacts (Elastic License 2.0) and Neo23x0/signature-base (DRL 1.1), fetched at pinned commits by install.sh. Skip them with SKIP_YARA=1 ./install.sh, or fetch by hand later:

./scripts/fetch_yara_rules.sh

The Elastic rules carry the only non-OSI licence in the tree: self-hosting is fine, but it forbids offering a substantial set of their functionality as a hosted multi-tenant service. SKIP_ELASTIC=1 ./scripts/fetch_yara_rules.sh leaves them out, at the cost of roughly half the ELF/botnet detections (see doc/yara-botnet-coverage.md). A thin or missing ruleset is not fatal — it means fewer tags, not failed jobs.

Pulling this change deletes the previously committed rule files from your working tree. Re-run the fetch script afterwards, or the module scans with the two house/ rules alone.

Function ID databases

Ghidra's Function ID (FID) analyzer is what produces the lib:<library>:<version>:<name> tags BSimVis puts on functions: standard library code identified by exact and operand-masked instruction hashes. Stock Ghidra ships FID databases for Visual Studio runtimes only, so on ELF samples nothing gets identified until you add databases to $GHIDRA_HOME/Ghidra/Features/FunctionID/data/.

install.sh fetches the open-source threatrack/ghidra-fidb-repo (MIT — 67 MB download, 215 MB unpacked, 47 databases: glibc, uClibc/OpenWrt, gcc, OpenSSL, Qt5, SDL, libsodium, CentOS 6/7, across x86 32/64, ARM, AARCH64, MIPS, PowerPC, SPARC, SuperH, m68k). Skip it with SKIP_FIDB=1 ./install.sh.

To install them by hand, into your own Ghidra:

curl -LO https://github.com/threatrack/ghidra-fidb-repo/releases/download/20200530/ghidra-fidb-repo_20200530.zip
unzip ghidra-fidb-repo_20200530.zip -d "$GHIDRA_HOME/Ghidra/Features/FunctionID/data/"

Those are the pre-unpacked .fidbf files, so no Ghidra rebuild is needed. Restart the workers afterwards — Ghidra reads the data directory at startup.

Coverage is per-architecture and per-toolchain. No database exists for PowerPC e500, MIPS64r6, RISC-V or LoongArch, and a statically linked binary built with a different compiler or -O level than the one the FIDB was built from misses every hash even where a database exists. Build your own with Ghidra's Tools -> Function ID -> Populate Function ID Database when a library matters and nothing matches it.

Running

Use the launch script to start all services in screen sessions:

./launch.sh

Use --clear to kill stale sessions before restarting:

./launch.sh --clear

launch_tmux.sh is the tmux equivalent, and additionally caps the worker count by host RAM and runs each worker under a memory-limited systemd scope.

Services are configured via .env (see .env.example). Key variables:

Variable	Default	Description
`APP_HOST`	`0.0.0.0`	Bind address for the API server
`APP_PORT`	`5000`	API / Web UI port
`REDIS_HOST`	`localhost`	Redis host
`REDIS_PORT`	`6379`	Redis job queue port
`KVROCKS_HOST`	`localhost`	Kvrocks host
`KVROCKS_PORT`	`6666`	Kvrocks database port
`WORKERS_COUNT`	`5`	Number of background workers
`DATA_BASE_DIR`	`./data`	Storage path for all service data
`ENABLE_MILVUS`	`false`	Enable optional Milvus vector DB
`MILVUS_HOST`	`localhost`	Milvus host (when enabled)
`MILVUS_PORT`	`19530`	Milvus gRPC port
`MILVUS_HEALTH_PORT`	`9091`	Milvus health-check port
`MINIO_PORT`	`9000`	MinIO data port (Milvus storage)
`MINIO_CONSOLE_PORT`	`9001`	MinIO console port
`ETCD_PORT`	`2379`	Etcd port (Milvus metadata)
`GHIDRA_INSTALL_DIR`	(auto)	Path to Ghidra install (set by `install.sh` if needed)
`WORKER_MEMORY_MAX`	`3G`	Per-worker memory cap (`launch_tmux.sh` only)

Configuration — `bsimvis_config.toml`

Copy bsimvis_config.toml.example and edit to taste. Key sections:

Analysis profiles ([profiles.fast] / [profiles.full] / [profiles.no-analysis]): sets of Ghidra analyzer toggles. Pass profile=fast (default) or profile=full on upload.

Similarity ([similarity]):

Key	Default	Description
`algo`	`unweighted_cosine`	Default similarity algorithm
`top_k`	`1000`	Max neighbours per function
`min_score`	`0.9`	Minimum similarity score
`min_features`	`10`	BSim floor — functions below it are matched by exact FunctionID hash

Clustering ([clustering]):

Key	Default	Description
`engine`	`threshold_uf`	Function cluster engine: `threshold_uf`, `hierarchical_uf`, or `hdbscan`
`bin_engine`	`threshold_uf`	Binary cluster engine (same choices)
`uf_threshold`	`0.98`	Similarity edge threshold for `threshold_uf` / `hierarchical_uf`
`bin_uf_threshold`	`0.1`	Same, for binary clustering
`cohesion_cut`	`0.9`	Minimum cohesion for a cluster to be kept
`idle_debounce_seconds`	`30`	Quiet period after uploads before auto-cluster fires
`lane_stale_seconds`	`1800`	Stale lane timeout (self-heals a crashed worker)

Analysis modules ([analysis_modules]): enabled = [] lists modules that run on every upload by default. Choices: FunctionID, capa, yara, rulezet. Override per-request with --enable/--disable (CLI) or enable=/disable= (API).

LLM ([llm]): ollama_url, model, prompt.

Ghidra ([ghidra]): max_heap_mb (per-worker JVM heap cap), max_ram_percent (CLI upload only), jvm_args (GC tuning).

Rulezet ([rulezet]): mirror of rulezet.org YARA rules. Set url, api_key (optional — enables single-call sync and tag recovery), allow_licenses, baseline_dirs (clean binaries for auto-quarantine gating), drop (tag glob blacklist), [rulezet.tags] (source-namespace → BSimVis-namespace routing). Synced with bsimvis rulezet sync.

Search ([search]): max_filter_buckets — cap on index buckets one wildcard filter may resolve to (default 200000).

Storage ([storage]): upload_dir — where uploaded binaries are written before Ghidra analysis (default data/uploads).

Test script

uv run test_api_endpoints.py

Benchmark

# Ingest + similarity pipeline over the JSON fixtures in data/bench/
uv run bsimvis-bench --clear

# Benchmark the pool paths, save metrics, compare against a baseline
uv run bsimvis-bench --bench-pools --save data/bench_results/run.json
uv run bsimvis-bench --compare data/bench_results/run.json

API

Full endpoint reference in doc/api_documentation.md; curl examples in doc/api_examples.md. Interactive Swagger UI is at /api/ when running.

Binary upload

One call is the whole story: analysis, indexing, and per-file similarity all run as part of this one job, and the collection's function/binary clusters rebuild automatically a short while after uploads to it go quiet (clustering.idle_debounce_seconds in bsimvis_config.toml, default 30s) -- see Batch upload below for why you don't need to call anything else, even for many files at once.

# upload to /api/file/upload
curl -X POST --data-binary "@/path/to/file" \
  "http://localhost:5000/api/file/upload?collection=main&file_name=my_binary&profile=fast"

# Response
{
    "status": "processing",
    "file_md5": "b7680c697c69aff3cd8f44fffcb7d683",
    "pipeline_id": "b2e1a4c0-...",
    "message": "Binary uploaded. Analysis started."
}

Add &priority=high to jump this file's analysis ahead of other pending jobs on the shared worker pool -- useful when another collection has a long-running job tying up workers and you need one file processed fast regardless. It does not preempt a job already running, only reorders what an idle worker picks up next.

Supported upload formats

/api/file/upload takes the raw bytes of whatever you have. Containers are unpacked server-side and every binary inside is analyzed as its own file, tagged with the format it came from and carrying the container's md5 in parent_md5.

Upload	What gets analyzed	Tag
Raw executable (ELF, PE, Mach-O, ...)	The file itself	—
`.zip`, `.tar`, `.tar.gz`, `.tar.bz2`, `.tar.xz`	Every member, one file each	`container:archive`
Encrypted zip (ZipCrypto or AES)	Same, password defaults to `infected`	`container:archive`
`.apk`, `.aab`	Only `.dex`, `.so` and `.jar` members — resources and assets are skipped	`container:apk`
Fat/universal Mach-O	One file per architecture slice, named `<file>:x86_64`, `<file>:arm64`, ...	`container:macho-fat`
UPX-packed executable	Both the packed file and its unpacked child, so packed-vs-unpacked is a normal binary diff	`packer:upx`
`.gpr.zip` (Ghidra project)	Imported whole, never unpacked — see below	—

Notes:

Archive password: --archive-password on the CLI, archive_password= on the API. Default is infected, the usual convention for shipped malware samples.
Unpacking UPX needs the upx binary, installed into bin/ by install.sh. Without it a packed sample is still analyzed, just packed.
Containers are followed 2 levels deep, capped at 200 children.
--no-unpack (unpack=false) analyzes the upload exactly as-is.
Unpacked something with your own tooling? Declare the lineage yourself with --parent-md5 / --parent-name (parent_md5= / parent_file_name=).

Adding another format means one entry in HANDLERS in bsimvis/app/services/unpack_service.py.

Ghidra project upload

Files in ghidra projects won´t get reanalyzed to not overwrite analyst work, meaning if no analyzers were ran in this project, no functions will be found

# Zip your .gpr and .rep project
gpr_name="my_project" # .gpr
zip -r "$gpr_name.gpr.zip" "$gpr_name.gpr" "$gpr_name.rep"

# upload to /api/file/upload
curl -X POST --data-binary @$gpr_name.gpr.zip \
  "http://localhost:5000/api/file/upload?file_name=$gpr_name&collection=main&profile=fast"

Batch upload (optional)

Just upload each file with plain POST /api/file/upload calls, in a loop or in parallel -- no batch_uuid, no finalize call needed. Every collection has its own lane: all the files you upload analyze fully in parallel across workers, and once uploads to that collection go quiet, the lane automatically groups everything uploaded in that window, clears old cluster/bin_sim results, and rebuilds function clusters, binary similarity, and binary clusters -- once, covering the whole batch:

for f in /path/to/*.bin; do
  curl -s -X POST --data-binary "@$f" \
    "http://localhost:5000/api/file/upload?collection=main&file_name=$(basename $f)"
done
# each upload analyzes immediately; clustering fires on its own ~30s after the last one

POST /api/cluster/rebuild_all and POST /api/file/upload/batch_finalize (explicit pipeline_ids list, batch_uuid grouping) still work for forcing a rebuild right now instead of waiting, or for scripts that already manage their own batching -- they queue behind whatever's currently active for that collection instead of racing it (a collection's lane only ever runs one clear/rebuild at a time; two overlapping requests used to corrupt each other's results). Add "priority": "high" to jump an explicit request ahead of other rebuilds already queued for that same collection. See doc/api_documentation.md for batch_finalize's full parameters.

Follow pipeline progress

curl -s "http://localhost:5000/api/jobs/pipe_f4f87081-ab7d-4077"

# Wait for completed status 
{
...
    "id": "pipe_f4f87081-ab7d-4077",
    "progress": "100",
    "status": "completed",
...
}

Build function clusters and Binary similarities

Runs automatically after uploads to a collection go quiet (see Batch upload above). Call this directly to force it right now instead of waiting -- it queues behind whatever's currently active for that collection rather than running concurrently with it:

curl -X POST -H "Content-Type: application/json" \
 -d '{"collection": "main", "algo": "unweighted_cosine"}' \
 http://localhost:5000/api/cluster/rebuild_all

# Response -- pipeline_id is pollable via GET /api/jobs/{pipeline_id} either way,
# whether it started immediately or is waiting behind another active rebuild
{
    "job_id": "b2e1a4c0-...",
    "pipeline_id": "b2e1a4c0-...",
    "status": "queued"
}

CLI tool

Upload BSIM data

Assuming you have the API running, upload binary or ghidra project (files in ghidra projects won´t get reanalyzed to not overwrite analyst work, meaning if no analyzers were ran in this project, no functions will be found)

uv run bsimvis upload <target1> <target2> ... <targetN> -c <collection_name>

Key upload flags:

Flag	Description
`--batch-split N`	Upload in independent batches of N files; each part is finalized and clustered before the next starts — results appear incrementally instead of at the end
`--enable MODULE`	Enable an analysis module for this run (repeatable): `FunctionID`, `capa`, `yara`, `rulezet`
`--disable MODULE`	Disable an analysis module for this run (repeatable)
`--local-analysis`	Run Ghidra analysis locally instead of on the server (requires Ghidra + pyghidra installed locally)
`--no-unpack` / `unpack=false`	Analyze the upload exactly as-is without unpacking archives or UPX
`--metadata FILE`	Enrich uploads with a metadata CSV (pipe-delimited, matched by filename)
`--archive-password PASSWORD`	Password for encrypted zip archives (default: `infected`)
`--profile fast\|full`	Ghidra analysis profile (default: `fast`)
`--save-json PATH`	Save analyzed JSON to disk (in addition to or instead of uploading)

See bsimvis_config.toml for analysis module defaults ([analysis_modules].enabled).

Job management

# List all jobs
uv run bsimvis job list

# View logs of a specific job
uv run bsimvis job status <job_id>

# Performance stats for a job/pipeline
uv run bsimvis job perf <job_id>

# Cancel a job
uv run bsimvis job cancel <job_id>

Worker management

# Start workers
uv run bsimvis worker start --count 5

Building clusters

Clustering fires automatically after uploads to a collection go quiet (see clustering.idle_debounce_seconds). To trigger it manually:

uv run bsimvis cluster build -c <collection_name>

Binary similarity management

# Build binary similarities
uv run bsimvis binsim build -c <collection_name>

# Get binary similarity diff between two files
uv run bsimvis binsim diff --md5-a <md5_a> --md5-b <md5_b> -c <collection_name>

Collection management

# Wipe and delete a collection completely
uv run bsimvis collection delete -c <collection_name>

# Clean up temporary raw/JSON upload keys in a collection
uv run bsimvis collection clean -c <collection_name>

Metadata propagation

# Propagate metadata from a pipe-delimited CSV file
uv run bsimvis metadata propagate -m <metadata_csv_path> -c <collection_name>

Rulezet YARA mirror

Mirror and gate rules from rulezet.org into data/rulezet/ — a build artifact, never in git, separate from the data/yara_rules/ set above. Configure the [rulezet] section in bsimvis_config.toml first.

# Fetch, tag, compile and auto-quarantine the mirror
uv run bsimvis rulezet sync

# Force a full re-fetch (ignores last sync date)
uv run bsimvis rulezet sync --full

# Recover curated MISP-galaxy tags (requires api_key in config)
uv run bsimvis rulezet index-tags mitre-attack-pattern

# Show quarantined rules (false-positive gated)
uv run bsimvis rulezet quarantine

# Release a specific rule from quarantine
uv run bsimvis rulezet quarantine --release <uuid>

Full CLI reference

usage: bsimvis [-h] [-H HOST] {features,index,sim,cluster,binsim,job,worker,upload,collection,metadata,rulezet} ...

Unified BSimVis CLI

positional arguments:
  {features,index,sim,cluster,binsim,job,worker,upload,collection,metadata,rulezet}
    features            BSim Feature management (Indexing)
    index               Index health and statistics
    sim                 Similarity management
    cluster             Unsupervised clustering management
    binsim              Binary-level similarity management
    job                 Job & Pipeline management
    worker              Worker management
    upload              Upload binaries to redis/kvrocks
    collection          Collection management
    metadata            Metadata management and propagation
    rulezet             Mirror YARA rules from rulezet.org

options:
  -h, --help            show this help message and exit
  -H, --host HOST       API host:port (default: localhost:5000, or from .env, or from bsimvis_config.toml)

Name		Name	Last commit message	Last commit date
bsimvis		bsimvis
data		data
doc		doc
img		img
kvrocks_data		kvrocks_data
scripts		scripts
.env.example		.env.example
.gitignore		.gitignore
.python-version		.python-version
AGENTS.md		AGENTS.md
LICENSE		LICENSE
README.md		README.md
app.py		app.py
bsimvis_config.toml.example		bsimvis_config.toml.example
docker-compose.yml		docker-compose.yml
install.sh		install.sh
kvrocks.conf		kvrocks.conf
launch.sh		launch.sh
launch_tmux.sh		launch_tmux.sh
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt
uv.lock		uv.lock

Uh oh!

Latest commit

History

Folders and files

Repository files navigation