LLAMA.CPP setup summary on Jetson Orin NX 8 GB

1. Initial setup

1. Cloned repo and prepared environment

   git clone https://github.com/ggml-org/llama.cpp
   cd llama.cpp

2. Build with CMake + CUDA support

   cmake -B build -DGGML_CUDA=ON
   cmake --build build -j$(nproc)

   • Solved original Makefile replaced error by using CMake.
   • Required CMake ≥ 3.18 (upgrade system CMake if needed).
   • Successfully compiled llama-cli and llama-server.

3. Downloaded model

   • Mistral 7B Instruction Q4_K_M GGUF format:

     ~/work/models/mistral-7b-instruct-v0.2.Q4_K_M.gguf
     

   • Copied to local SSD instead of external USB for faster load.

---

2. First test

./build/bin/llama-cli -m ~/work/models/mistral-7b-instruct-v0.2.Q4_K_M.gguf \
  -ngl 20 -c 2048 -p "Explain edge AI in one paragraph"

• Problem: Very slow, model load ~10 min, then 1 token/sec or less.
• Analysis:
  • GPU barely used (GR3D_FREQ 0–20%)
  • CPU heavily used, 2 cores offline
  • Swap heavily used (~4.4 GB)
  • RAM almost full (~6.7/7.3 GB)
• Root cause: Memory pressure + CPU cores offline + GPU not fully utilized.

---

3. Optimizations applied

A. Enable all CPU cores

sudo nvpmodel -m 0
sudo jetson_clocks

• Orin NX has 6 cores; lower power modes only activate 4.
• Enabling all cores improved prefill and generation speed slightly.

B. Move model to fast local storage

• Copied from external drive to internal SSD (~/work/models).
• Reduces disk I/O and swap usage.

C. Reduce memory footprint

• Original context: -c 2048 → lowered to -c 512
• Reduced KV cache memory → fewer pages to swap.

D. Adjust GPU layer offload

• Optimized -ngl from 20 → 28
• Maximizes number of layers on GPU while staying within VRAM (~3.6 GB used out of 7.3 GB).

E. Limit CPU threads for smoother system performance

-t 4

• Leaves other cores for system tasks (DeepStream, OS).

F. Optional tricks

• --mlock → prevents memory paging
• --no-mmap → reduces swap jitter
• Running console-only (no browser/IDE) → frees RAM

---

4. Results after optimization

Prompt: 5.7 tok/s
Generation: 2.4 tok/s

• Model load: 1–2 minutes (from internal SSD)
• GPU utilization: ~30–50%
• Swap usage: minimal (~0–200 MB)
• CPU fully online
• RAM usage: ~6.6 GB
• Context size: 512 tokens
• GPU layers: 28

Significant improvement compared to initial setup (previously under 0.5 tok/s and heavy swap).

---

5. Server setup and GPU parameters

After building llama.cpp, you can run a LAN-accessible inference server:

./llama-server \
  -m ~/work/models/mistral-7b-instruct-v0.2.Q4_K_M.gguf \
  -ngl 28 \
  -c 512 \
  -t 4 \
  --host 0.0.0.0 \
  --port 8080

Key server arguments

Argument	Recommended setting	Purpose
-m	Path to .gguf model	Loads the selected model
-ngl	28	Number of layers to offload to GPU (GPU-heavy)
-c	512	Context window (tokens)
-t	4	CPU threads for non-GPU tasks (adjust based on enabled CPU cores)
--host	0.0.0.0	Bind server to all network interfaces for LAN access
--port	8080	Server port for HTTP requests

Pushing GPU a bit further

To maximize GPU usage on Orin NX 8 GB and increase tokens/sec:

./llama-server \
  -m ~/work/models/mistral-7b-instruct-v0.2.Q4_K_M.gguf \
  -ngl 30 \
  -c 512 \
  -t 6 \
  --host 0.0.0.0 \
  --port 8080 \
  --cache-ram 512

Changes:

• -ngl 30 — more layers on GPU (risk: may exceed VRAM; reduce if loading fails).
• -t 6 — more CPU threads for off-GPU computation and I/O.
• --cache-ram 512 — reserve extra host RAM for prompt caching to reduce GPU stalls.

Tips

• Close other apps (browser, IDE) to free system RAM and swap.
• Keep GPU VRAM headroom ≥1 GB to avoid crashes.
• Monitor jtop or tegrastats — lower -ngl if CUDA memory spikes or the server fails to load.
• Context (-c) can be temporarily reduced to 256–384 tokens if you need faster throughput.

With these settings, expect slightly higher token/sec (~2.5–3 t/s) while leaving room for other CUDA tasks (e.g. DeepStream pipelines).

---

6. Observations / constraints

• Orin NX 8 GB is borderline for real-time 7B LLMs.
• DeepStream + 7B LLM together is challenging due to VRAM limits.
• Token/sec depends heavily on:
  • -ngl (GPU layers)
  • -c (context size)
  • number of CPU threads
  • system load / swap

---

7. Potential improvements and tricks

Optimization	Description	Impact
Enable all CPU cores	nvpmodel -m 0 + jetson_clocks	Faster CPU prefill / better generation speed
Use internal SSD	Move model from USB → internal	Reduces load time, swap pressure
Reduce context	-c 512	Avoid swap, maintain throughput
Increase GPU layers	-ngl 28–30	Push layers to GPU → faster generation
Limit CPU threads	-t 4	Leaves cores free for OS / other pipelines
MLock / no-mmap	--mlock --no-mmap	Reduce page faults → smoother token generation
Console-only session	Close browser/IDE	Frees RAM for model

---

8. Performance expectation (Orin NX 8 GB)

Model	Context	GPU layers	Tokens/sec
Mistral 7B Q4	512	28	2–4 t/s
3–4B models	512	full	5–12 t/s
1–2B models	512	full	10–25 t/s

Note: Always budget ~10–20% VRAM and RAM headroom for stability.

---

Conclusion

You now have a working 7B LLM setup on Jetson Orin NX. The system can generate tokens reasonably fast (~2.4 t/s) and stays stable if:

• GPU layers are tuned
• Context is moderate
• CPU cores are fully online
• Swap is minimized