Skip to content

Commit 2743dad

Browse files
Add AISIX AI Gateway integration page (#603)
* Add AISIX AI Gateway integration page Signed-off-by: Yilia Lin <Yilialin@apache.org> * Update integrations/aisix.md --------- Signed-off-by: Yilia Lin <Yilialin@apache.org> Co-authored-by: Kacper Łukawski <kacperlukawski@users.noreply.github.com>
1 parent 04badb2 commit 2743dad

1 file changed

Lines changed: 190 additions & 0 deletions

File tree

‎integrations/aisix.md‎

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
---
2+
layout: integration
3+
name: AISIX AI Gateway
4+
description: Route Haystack model calls through the open-source AISIX AI Gateway using OpenAI-compatible endpoints.
5+
authors:
6+
- name: Yilia Lin
7+
socials:
8+
github: Yilialinn
9+
repo: https://github.com/api7/aisix
10+
type: Custom Component
11+
report_issue: https://github.com/api7/aisix/issues
12+
version: Haystack 3.1
13+
toc: true
14+
---
15+
16+
### **Table of Contents**
17+
18+
- [Overview](#overview)
19+
- [Installation](#installation)
20+
- [Configure AISIX](#configure-aisix)
21+
- [Use AISIX in a RAG Pipeline](#use-aisix-in-a-rag-pipeline)
22+
- [Retry Ownership](#retry-ownership)
23+
- [License](#license)
24+
25+
## Overview
26+
27+
[AISIX AI Gateway](https://github.com/api7/aisix) is an Apache-2.0-licensed open-source AI gateway maintained by API7.ai. It is an independent project and is not an Apache Software Foundation project.
28+
29+
AISIX exposes OpenAI-compatible endpoints, so Haystack applications can use the built-in [`OpenAIChatGenerator`](https://docs.haystack.deepset.ai/docs/openaichatgenerator) without installing an AISIX-specific component. The Haystack application authenticates with a gateway caller key and selects a caller-facing model alias. AISIX keeps the upstream provider credential and resolves that alias to a direct model or routing group.
30+
31+
The examples below were validated with Haystack 3.1.1 and AISIX 1.2.0.
32+
33+
## Installation
34+
35+
Install Haystack:
36+
37+
```bash
38+
pip install haystack-ai==3.1.1
39+
```
40+
41+
Deploy AISIX by following the [open-source AISIX gateway quickstart](https://docs.api7.ai/ai-gateway/getting-started/gateway-quickstart), using the `ghcr.io/api7/aisix:1.2.0` image tag to reproduce this example. No additional Haystack integration package is required.
42+
43+
## Configure AISIX
44+
45+
The following AISIX resources define two direct upstream models and expose them to Haystack as one `rag-model` [failover alias](https://docs.api7.ai/ai-gateway/routing/routing-and-failover). Replace the provider endpoint and model identifiers with values available in your environment.
46+
47+
```yaml
48+
_format_version: "1"
49+
50+
provider_keys:
51+
- display_name: haystack-provider
52+
provider: openai
53+
adapter: openai
54+
api_key: ${UPSTREAM_API_KEY}
55+
api_base: ${UPSTREAM_API_BASE}
56+
57+
models:
58+
- display_name: rag-primary
59+
provider: openai
60+
model_name: ${UPSTREAM_PRIMARY_MODEL}
61+
provider_key: haystack-provider
62+
- display_name: rag-backup
63+
provider: openai
64+
model_name: ${UPSTREAM_BACKUP_MODEL}
65+
provider_key: haystack-provider
66+
- display_name: rag-model
67+
routing:
68+
strategy: failover
69+
retries: 0
70+
max_fallbacks: 1
71+
targets:
72+
- model: rag-primary
73+
- model: rag-backup
74+
75+
api_keys:
76+
- display_name: haystack-caller
77+
key_env: HAYSTACK_CALLER_KEY
78+
allowed_models:
79+
- rag-model
80+
```
81+
82+
Set the referenced environment variables in the AISIX process and validate the resources before starting or reloading the gateway:
83+
84+
```bash
85+
aisix validate --resources /etc/aisix/resources.yaml
86+
```
87+
88+
The Haystack process needs only the gateway URL and caller key:
89+
90+
```bash
91+
export AISIX_BASE_URL="http://localhost:3000"
92+
export HAYSTACK_CALLER_KEY="replace-with-a-gateway-caller-key"
93+
```
94+
95+
Do not expose `UPSTREAM_API_KEY` to the Haystack application.
96+
97+
## Use AISIX in a RAG Pipeline
98+
99+
This example uses in-memory BM25 retrieval so that the only external request is the generation call through AISIX.
100+
101+
```python
102+
import os
103+
104+
from haystack import Document, Pipeline
105+
from haystack.components.builders import ChatPromptBuilder
106+
from haystack.components.generators.chat import OpenAIChatGenerator
107+
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
108+
from haystack.dataclasses import ChatMessage
109+
from haystack.document_stores.in_memory import InMemoryDocumentStore
110+
from haystack.utils import Secret
111+
112+
document_store = InMemoryDocumentStore()
113+
document_store.write_documents(
114+
[
115+
Document(
116+
content=(
117+
"If the checkout service returns HTTP 503, fail over to the "
118+
"secondary region and page the on-call engineer."
119+
)
120+
),
121+
Document(
122+
content=(
123+
"If checkout latency exceeds two seconds, inspect the payment "
124+
"provider dashboard before scaling the application."
125+
)
126+
),
127+
]
128+
)
129+
130+
template = [
131+
ChatMessage.from_system(
132+
"Answer only from the supplied runbook documents."
133+
),
134+
ChatMessage.from_user(
135+
"Runbook documents:\n"
136+
"{% for document in documents %}"
137+
"- {{ document.content }}\n"
138+
"{% endfor %}"
139+
"Question: {{ question }}"
140+
),
141+
]
142+
143+
pipeline = Pipeline()
144+
pipeline.add_component(
145+
"retriever",
146+
InMemoryBM25Retriever(document_store=document_store, top_k=1),
147+
)
148+
pipeline.add_component(
149+
"prompt_builder",
150+
ChatPromptBuilder(
151+
template=template,
152+
required_variables={"documents", "question"},
153+
),
154+
)
155+
pipeline.add_component(
156+
"llm",
157+
OpenAIChatGenerator(
158+
api_key=Secret.from_env_var("HAYSTACK_CALLER_KEY"),
159+
model="rag-model",
160+
api_base_url=f"{os.environ['AISIX_BASE_URL'].rstrip('/')}/v1",
161+
max_retries=0,
162+
generation_kwargs={"temperature": 0},
163+
),
164+
)
165+
166+
pipeline.connect("retriever.documents", "prompt_builder.documents")
167+
pipeline.connect("prompt_builder.prompt", "llm.messages")
168+
169+
question = "What should we do when checkout returns HTTP 503?"
170+
result = pipeline.run(
171+
{
172+
"retriever": {"query": question},
173+
"prompt_builder": {"question": question},
174+
}
175+
)
176+
177+
print(result["llm"]["replies"][0].text)
178+
```
179+
180+
A response grounded in the retrieved runbook should instruct the operator to fail over to the secondary region and page the on-call engineer.
181+
182+
## Retry Ownership
183+
184+
The example sets `max_retries=0` on `OpenAIChatGenerator` because the displayed AISIX routing model owns failover. This makes one Haystack generation call correspond to one gateway request while AISIX tries the configured targets within that request.
185+
186+
You can keep client-side retries for a different failure policy, but size the Haystack/OpenAI client retry budget and the AISIX retry/failover budget together. Otherwise, one application-level call can create more upstream attempts than expected.
187+
188+
## License
189+
190+
AISIX is distributed under the [Apache License 2.0](https://github.com/api7/aisix/blob/main/LICENSE).

0 commit comments

Comments
 (0)