Skip to content

Commit b269536

Browse files
bilgeyucelbilgeyucel
authored
Add Kolibri integration page (#640)
1 parent ae1bb60 commit b269536

2 files changed

Lines changed: 213 additions & 0 deletions

File tree

‎integrations/kolibri.md‎

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
---
2+
layout: integration
3+
name: Kolibri
4+
description: Use Aleph Alpha Kolibri open-weight German–English models with Haystack for generation, reasoning, and agents
5+
authors:
6+
- name: deepset
7+
socials:
8+
github: deepset-ai
9+
twitter: deepset_ai
10+
linkedin: https://www.linkedin.com/company/deepset-ai/
11+
pypi: https://pypi.org/project/vllm-haystack/
12+
repo: https://huggingface.co/Aleph-Alpha/Kolibri-1
13+
type: Model Provider
14+
report_issue: https://github.com/deepset-ai/haystack-core-integrations/issues
15+
logo: /logos/kolibri.svg
16+
version: Haystack 2.0
17+
toc: true
18+
---
19+
20+
### **Table of Contents**
21+
22+
- [Overview](#overview)
23+
- [Setup](#setup)
24+
- [Usage](#usage)
25+
- [Generative chat (German)](#generative-chat-german)
26+
- [Reasoning mode](#reasoning-mode)
27+
- [Agentic tool calling](#agentic-tool-calling)
28+
29+
## Overview
30+
31+
**[Kolibri](https://aleph-alpha.com/en/kolibri/)** is Aleph Alpha’s open-weight model for German and English. Serve it with vLLM, then use it in Haystack for chat, reasoning, and agents.
32+
33+
Key capabilities:
34+
35+
<div class="styled-table">
36+
37+
| Capability | Details |
38+
| --- | --- |
39+
| **Generative** | Chat completion for drafting, Q&A, coding, and long-document work |
40+
| **German & English** | Native bilingual model; German is a first-class language (≈24% of pre-training mix) |
41+
| **Reasoning** | Controllable thinking via `reasoning_effort` (`none` / `low` / `medium` / `high`) |
42+
| **Agentic** | Hermes-style tool calling with vLLM’s `kolibri1` parser |
43+
| **Long context** | Native 262,144 tokens; validated up to 1,048,576 (recommend ≤256k for complex tasks) |
44+
45+
</div>
46+
47+
## Setup
48+
49+
Kolibri needs Aleph Alpha’s vLLM plugin (`aleph-alpha-inference`). Stock vLLM alone cannot load `Kolibri1ForCausalLM` or the `kolibri1` parsers.
50+
51+
**Option A — pip (pins a supported vLLM, currently 0.29):**
52+
53+
```bash
54+
pip install 'aleph-alpha-inference>=1'
55+
```
56+
57+
**Option B — container:**
58+
59+
```bash
60+
docker pull ghcr.io/aleph-alpha/aleph-alpha-inference
61+
```
62+
63+
Serve with reasoning and tool calling enabled:
64+
65+
```bash
66+
vllm serve Aleph-Alpha/Kolibri-1 \
67+
--kv-cache-dtype fp8 \
68+
--reasoning-parser kolibri1 \
69+
--tool-call-parser kolibri1 \
70+
--enable-auto-tool-choice
71+
```
72+
73+
For BF16 weights, serve `Aleph-Alpha/Kolibri-1-BF16` and omit `--kv-cache-dtype fp8`. For contexts beyond 262,144 tokens, add `--max-model-len 1048576 --hf-overrides '{"max_position_embeddings": 1048576}'`.
74+
75+
Recommended sampling defaults from the model card: `temperature=1.0`, `top_p=0.97`, `top_k=128`.
76+
77+
Install the Haystack client for the OpenAI-compatible vLLM server:
78+
79+
```bash
80+
pip install vllm-haystack
81+
```
82+
83+
See also the [vLLM](/integrations/vllm) integration.
84+
85+
## Usage
86+
87+
Once Kolibri is serving (default `http://localhost:8000/v1`), use [`VLLMChatGenerator`](https://docs.haystack.deepset.ai/docs/vllmchatgenerator) for generation, reasoning, and Haystack [`Agent`](https://docs.haystack.deepset.ai/docs/agent) agent loops.
88+
89+
### Generative chat (German)
90+
91+
Kolibri is specialized for German. This example asks for a short explanation entirely in German:
92+
93+
```python
94+
from haystack.dataclasses import ChatMessage
95+
from haystack_integrations.components.generators.vllm import VLLMChatGenerator
96+
97+
chat = VLLMChatGenerator(
98+
model="Aleph-Alpha/Kolibri-1",
99+
api_base_url="http://localhost:8000/v1",
100+
generation_kwargs={
101+
"temperature": 1.0,
102+
"top_p": 0.97,
103+
"extra_body": {
104+
"top_k": 128,
105+
"chat_template_kwargs": {
106+
"enable_thinking": False,
107+
"reasoning_effort": "none",
108+
},
109+
},
110+
},
111+
)
112+
113+
result = chat.run(
114+
messages=[
115+
ChatMessage.from_user(
116+
"Erkläre kurz, was ein Mixture-of-Experts-Modell ist und warum es für deutsche Verwaltungstexte hilfreich sein kann."
117+
)
118+
]
119+
)
120+
print(result["replies"][0].text)
121+
```
122+
123+
### Reasoning mode
124+
125+
Thinking is on by default when the server is started with `--reasoning-parser kolibri1`. Control effort per request with `chat_template_kwargs`. When thinking is enabled, Haystack exposes the reasoning trace on the reply’s `.reasoning` field:
126+
127+
```python
128+
from haystack.dataclasses import ChatMessage
129+
from haystack_integrations.components.generators.vllm import VLLMChatGenerator
130+
131+
chat = VLLMChatGenerator(
132+
model="Aleph-Alpha/Kolibri-1",
133+
api_base_url="http://localhost:8000/v1",
134+
generation_kwargs={
135+
"temperature": 1.0,
136+
"top_p": 0.97,
137+
"extra_body": {
138+
"top_k": 128,
139+
"chat_template_kwargs": {
140+
"enable_thinking": True,
141+
"reasoning_effort": "high", # none | low | medium | high
142+
},
143+
},
144+
},
145+
)
146+
147+
result = chat.run(
148+
messages=[
149+
ChatMessage.from_user(
150+
"Ein Amt hat 3 Anträge: A braucht 2 Tage, B 5 Tage, C 1 Tag. "
151+
"In welcher Reihenfolge minimiert man die mittlere Wartezeit? Begründe."
152+
)
153+
]
154+
)
155+
reply = result["replies"][0]
156+
print(reply.reasoning) # thinking trace (when enabled)
157+
print(reply.text) # final answer
158+
```
159+
160+
Set `reasoning_effort` to `"none"` or `enable_thinking` to `False` for a direct reply without a thinking block.
161+
162+
### Agentic tool calling
163+
164+
Serve with `--tool-call-parser kolibri1 --enable-auto-tool-choice`, then plug Kolibri into a Haystack `Agent`. Tool calling can be combined with reasoning. This agent looks up a mock product price in German:
165+
166+
```python
167+
from typing import Annotated
168+
169+
from haystack.components.agents import Agent
170+
from haystack.dataclasses import ChatMessage
171+
from haystack.tools import tool
172+
from haystack_integrations.components.generators.vllm import VLLMChatGenerator
173+
174+
175+
@tool
176+
def preis_abrufen(produkt: Annotated[str, "Produktname für die Preissuche"]) -> str:
177+
"""Sucht einen Beispielpreis für ein Produkt."""
178+
preise = {"laptop": "999 €", "tastatur": "99 €", "monitor": "349 €"}
179+
return preise.get(produkt.lower(), "Unbekanntes Produkt")
180+
181+
182+
agent = Agent(
183+
chat_generator=VLLMChatGenerator(
184+
model="Aleph-Alpha/Kolibri-1",
185+
api_base_url="http://localhost:8000/v1",
186+
generation_kwargs={
187+
"temperature": 1.0,
188+
"top_p": 0.97,
189+
"extra_body": {
190+
"top_k": 128,
191+
"chat_template_kwargs": {
192+
"enable_thinking": True,
193+
"reasoning_effort": "medium",
194+
},
195+
},
196+
},
197+
),
198+
tools=[preis_abrufen],
199+
system_prompt=(
200+
"Du hilfst Nutzern auf Deutsch und rufst die bereitgestellten Tools auf, wenn sie relevant sind. Antworte klar und knapp."
201+
),
202+
)
203+
204+
result = agent.run(messages=[ChatMessage.from_user("Was kostet ein Laptop?")])
205+
print(result["last_message"].text)
206+
```
207+
208+
For English-only assistants, the same pattern works — keep tools and prompts in English. For RAG pipelines, use Kolibri as the generator after retrieval the same way you would any other chat model behind vLLM.

‎logos/kolibri.svg‎

Lines changed: 5 additions & 0 deletions
Loading

0 commit comments

Comments
 (0)