Hailo-Ollama for Apple Silicon macOS

English | 简体中文

Run Hailo-10H LLMs from standard Ollama and OpenAI clients on Apple Silicon macOS.

This project ports Hailo-Ollama v5.3.0 to macOS and connects it to the hailort-macos-usb userspace USB runtime. Inference executes on the ASUS UGen300 Hailo-10H; the Mac hosts the HTTP server, token streaming and USB transport.

Important

This is a community project, not an official Hailo or ASUS macOS release. It has been validated on one Mac Studio M2 Ultra with the official Qwen3-1.7B HAILO10H HEF.

What works

Capability	Status
Native Apple Silicon build	✅
Ollama /api/chat and /api/generate	✅
Ollama streaming and non-streaming responses	✅
OpenAI /v1/models	✅
OpenAI /v1/chat/completions	✅
OpenAI SSE streaming with data: [DONE]	✅
OpenAI Python SDK	✅
Qwen3-1.7B	✅
Authentication and TLS	❌ Reverse proxy required
Full OpenAI API	❌ Chat compatibility subset only
Arbitrary GGUF models and LoRA	❌ HEF models only
Multiple simultaneous LLM/VLM instances	❌ Exclusive device KV cache

Quick start

The default scripts expect these repositories to be siblings:

workspace/
  hailort-macos-usb/
  hailo-ollama-macos/

1. Build and boot HailoRT

cd workspace
git clone https://github.com/relaxcloud-cn/hailort-macos-usb.git
git clone https://github.com/relaxcloud-cn/hailo-ollama-macos.git

cd hailort-macos-usb
brew install cmake libusb pkg-config qemu openssl@3
scripts/macos/build_hailort_usb.sh
scripts/macos/load_ugen300_runtime.sh /absolute/path/to/hailo-loader.qcow2
scripts/macos/doctor.sh

Preparing the loader VM requires the official ASUS package. Follow the loader VM guide.

2. Build Hailo-Ollama

cd ../hailo-ollama-macos
scripts/macos/build_hailo_ollama.sh

The installed binary uses @loader_path/../lib and carries its matching libhailort.5.3.0.dylib under build-macos/install/lib. It does not embed a developer's home-directory path.

3. Register a supported HEF

scripts/macos/link_local_model.sh \
  qwen3 \
  ../hailort-macos-usb/models/Qwen3-1.7B-Instruct.hef

The script verifies the HEF against the official manifest SHA-256 and creates a symbolic link in the ignored runtime store. It does not duplicate the model file.

HEFs and model weights are not included in this repository. Obtain them from an official Hailo distribution channel and use them under their own terms.

4. Start and test

scripts/macos/run_hailo_ollama.sh

In another terminal:

scripts/macos/smoke_test.sh qwen3

The smoke test checks version discovery, model discovery, a non-streaming completion and an OpenAI SSE streaming completion.

OpenAI client example

The macOS launcher listens on 127.0.0.1:8000 by default.

export OPENAI_BASE_URL=http://127.0.0.1:8000/v1
export OPENAI_API_KEY=local-only

from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="qwen3",
    messages=[{"role": "user", "content": "Explain least privilege briefly."}],
    max_tokens=64,
    stream=False,
)
print(response.choices[0].message.content)

Streaming uses the normal OpenAI SDK interface:

stream = client.chat.completions.create(
    model="qwen3",
    messages=[{"role": "user", "content": "Explain zero trust briefly."}],
    max_tokens=64,
    stream=True,
)

for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

OPENAI_API_KEY only satisfies SDK-side validation. The server does not authenticate it.

Ollama client example

curl --silent http://127.0.0.1:8000/api/chat \
  -H 'Content-Type: application/json' \
  -d '{
    "model":"qwen3",
    "messages":[{"role":"user","content":"Explain least privilege briefly."}],
    "options":{"num_predict":64},
    "stream":false
  }'

API compatibility

Endpoint	Compatibility
GET /api/version	Ollama
GET /api/ps	Ollama
GET /api/tags	Ollama
POST /api/pull	Hailo model library
POST /api/show	Ollama
DELETE /api/delete	Ollama
POST /api/generate	Ollama
POST /api/chat	Ollama
GET /hailo/v1/list	Hailo model discovery
GET /v1/models	OpenAI
POST /v1/chat/completions	OpenAI JSON and SSE

This is not a complete OpenAI server. Embeddings, files, batches, assistants, audio, images and other unrelated endpoints are not implemented.

Architecture

OpenAI SDK / Ollama client / Open WebUI
  └─ Hailo-Ollama HTTP server on macOS
      └─ HailoRT GenAI C++ API
          └─ Hailo HRPC over libusb
              └─ ASUS UGen300 / Hailo-10H

USB reconnect behavior

UGen300 returns to loader mode after power loss. After unplugging or resetting it:

1. stop Hailo-Ollama;
2. reload the UGen300 runtime using hailort-macos-usb;
3. restart Hailo-Ollama.

An old HTTP process can stay reachable while its VDevice is invalid. Generation then returns HTTP 500 with HAILO_DEVICE_NOT_CONNECTED; restarting the process is required.

Security boundary

Hailo-Ollama has no built-in authentication, TLS, tenant isolation or request-level authorization. The launcher therefore defaults to loopback. Do not set OLLAMA_HOST=0.0.0.0:8000 on an untrusted network unless a secured reverse proxy provides those controls.

The server executes one request while one additional request waits. Only one LLM or VLM can own the device KV cache at a time.

Changes from upstream v5.3.0

• Apple Silicon build and runtime wrappers.
• Manifest-verified local HEF registration.
• Correct handling of untagged manifests such as qwen3/manifest.json.
• OpenAI SSE streaming for /v1/chat/completions.
• OpenAI model discovery through /v1/models.
• Correct OpenAI timestamps and preservation of the last streamed token.
• Self-contained macOS @loader_path layout for libhailort.

Documentation

• Complete macOS guide
• Supported model catalog
• Companion HailoRT USB port
• Contributing
• Security policy

Upstream and license

This repository preserves upstream history and currently branches from hailo-ai/hailo_model_zoo_genai v5.3.0. The official repository is configured locally as the upstream remote.

The source remains available under the MIT license. Third-party notices are in LICENSE-3RD-PARTY.md. Firmware, HEFs, model weights and vendor packages are excluded and retain their own license terms.