|
| 1 | +--- |
| 2 | +layout: integration |
| 3 | +name: AISIX AI Gateway |
| 4 | +description: Route Haystack model calls through the open-source AISIX AI Gateway using OpenAI-compatible endpoints. |
| 5 | +authors: |
| 6 | + - name: Yilia Lin |
| 7 | + socials: |
| 8 | + github: Yilialinn |
| 9 | +repo: https://github.com/api7/aisix |
| 10 | +type: Custom Component |
| 11 | +report_issue: https://github.com/api7/aisix/issues |
| 12 | +version: Haystack 3.1 |
| 13 | +toc: true |
| 14 | +--- |
| 15 | + |
| 16 | +### **Table of Contents** |
| 17 | + |
| 18 | +- [Overview](#overview) |
| 19 | +- [Installation](#installation) |
| 20 | +- [Configure AISIX](#configure-aisix) |
| 21 | +- [Use AISIX in a RAG Pipeline](#use-aisix-in-a-rag-pipeline) |
| 22 | +- [Retry Ownership](#retry-ownership) |
| 23 | +- [License](#license) |
| 24 | + |
| 25 | +## Overview |
| 26 | + |
| 27 | +[AISIX AI Gateway](https://github.com/api7/aisix) is an Apache-2.0-licensed open-source AI gateway maintained by API7.ai. It is an independent project and is not an Apache Software Foundation project. |
| 28 | + |
| 29 | +AISIX exposes OpenAI-compatible endpoints, so Haystack applications can use the built-in [`OpenAIChatGenerator`](https://docs.haystack.deepset.ai/docs/openaichatgenerator) without installing an AISIX-specific component. The Haystack application authenticates with a gateway caller key and selects a caller-facing model alias. AISIX keeps the upstream provider credential and resolves that alias to a direct model or routing group. |
| 30 | + |
| 31 | +The examples below were validated with Haystack 3.1.1 and AISIX 1.2.0. |
| 32 | + |
| 33 | +## Installation |
| 34 | + |
| 35 | +Install Haystack: |
| 36 | + |
| 37 | +```bash |
| 38 | +pip install haystack-ai==3.1.1 |
| 39 | +``` |
| 40 | + |
| 41 | +Deploy AISIX by following the [open-source AISIX gateway quickstart](https://docs.api7.ai/ai-gateway/getting-started/gateway-quickstart), using the `ghcr.io/api7/aisix:1.2.0` image tag to reproduce this example. No additional Haystack integration package is required. |
| 42 | + |
| 43 | +## Configure AISIX |
| 44 | + |
| 45 | +The following AISIX resources define two direct upstream models and expose them to Haystack as one `rag-model` [failover alias](https://docs.api7.ai/ai-gateway/routing/routing-and-failover). Replace the provider endpoint and model identifiers with values available in your environment. |
| 46 | + |
| 47 | +```yaml |
| 48 | +_format_version: "1" |
| 49 | + |
| 50 | +provider_keys: |
| 51 | + - display_name: haystack-provider |
| 52 | + provider: openai |
| 53 | + adapter: openai |
| 54 | + api_key: ${UPSTREAM_API_KEY} |
| 55 | + api_base: ${UPSTREAM_API_BASE} |
| 56 | + |
| 57 | +models: |
| 58 | + - display_name: rag-primary |
| 59 | + provider: openai |
| 60 | + model_name: ${UPSTREAM_PRIMARY_MODEL} |
| 61 | + provider_key: haystack-provider |
| 62 | + - display_name: rag-backup |
| 63 | + provider: openai |
| 64 | + model_name: ${UPSTREAM_BACKUP_MODEL} |
| 65 | + provider_key: haystack-provider |
| 66 | + - display_name: rag-model |
| 67 | + routing: |
| 68 | + strategy: failover |
| 69 | + retries: 0 |
| 70 | + max_fallbacks: 1 |
| 71 | + targets: |
| 72 | + - model: rag-primary |
| 73 | + - model: rag-backup |
| 74 | + |
| 75 | +api_keys: |
| 76 | + - display_name: haystack-caller |
| 77 | + key_env: HAYSTACK_CALLER_KEY |
| 78 | + allowed_models: |
| 79 | + - rag-model |
| 80 | +``` |
| 81 | +
|
| 82 | +Set the referenced environment variables in the AISIX process and validate the resources before starting or reloading the gateway: |
| 83 | +
|
| 84 | +```bash |
| 85 | +aisix validate --resources /etc/aisix/resources.yaml |
| 86 | +``` |
| 87 | + |
| 88 | +The Haystack process needs only the gateway URL and caller key: |
| 89 | + |
| 90 | +```bash |
| 91 | +export AISIX_BASE_URL="http://localhost:3000" |
| 92 | +export HAYSTACK_CALLER_KEY="replace-with-a-gateway-caller-key" |
| 93 | +``` |
| 94 | + |
| 95 | +Do not expose `UPSTREAM_API_KEY` to the Haystack application. |
| 96 | + |
| 97 | +## Use AISIX in a RAG Pipeline |
| 98 | + |
| 99 | +This example uses in-memory BM25 retrieval so that the only external request is the generation call through AISIX. |
| 100 | + |
| 101 | +```python |
| 102 | +import os |
| 103 | + |
| 104 | +from haystack import Document, Pipeline |
| 105 | +from haystack.components.builders import ChatPromptBuilder |
| 106 | +from haystack.components.generators.chat import OpenAIChatGenerator |
| 107 | +from haystack.components.retrievers.in_memory import InMemoryBM25Retriever |
| 108 | +from haystack.dataclasses import ChatMessage |
| 109 | +from haystack.document_stores.in_memory import InMemoryDocumentStore |
| 110 | +from haystack.utils import Secret |
| 111 | + |
| 112 | +document_store = InMemoryDocumentStore() |
| 113 | +document_store.write_documents( |
| 114 | + [ |
| 115 | + Document( |
| 116 | + content=( |
| 117 | + "If the checkout service returns HTTP 503, fail over to the " |
| 118 | + "secondary region and page the on-call engineer." |
| 119 | + ) |
| 120 | + ), |
| 121 | + Document( |
| 122 | + content=( |
| 123 | + "If checkout latency exceeds two seconds, inspect the payment " |
| 124 | + "provider dashboard before scaling the application." |
| 125 | + ) |
| 126 | + ), |
| 127 | + ] |
| 128 | +) |
| 129 | + |
| 130 | +template = [ |
| 131 | + ChatMessage.from_system( |
| 132 | + "Answer only from the supplied runbook documents." |
| 133 | + ), |
| 134 | + ChatMessage.from_user( |
| 135 | + "Runbook documents:\n" |
| 136 | + "{% for document in documents %}" |
| 137 | + "- {{ document.content }}\n" |
| 138 | + "{% endfor %}" |
| 139 | + "Question: {{ question }}" |
| 140 | + ), |
| 141 | +] |
| 142 | + |
| 143 | +pipeline = Pipeline() |
| 144 | +pipeline.add_component( |
| 145 | + "retriever", |
| 146 | + InMemoryBM25Retriever(document_store=document_store, top_k=1), |
| 147 | +) |
| 148 | +pipeline.add_component( |
| 149 | + "prompt_builder", |
| 150 | + ChatPromptBuilder( |
| 151 | + template=template, |
| 152 | + required_variables={"documents", "question"}, |
| 153 | + ), |
| 154 | +) |
| 155 | +pipeline.add_component( |
| 156 | + "llm", |
| 157 | + OpenAIChatGenerator( |
| 158 | + api_key=Secret.from_env_var("HAYSTACK_CALLER_KEY"), |
| 159 | + model="rag-model", |
| 160 | + api_base_url=f"{os.environ['AISIX_BASE_URL'].rstrip('/')}/v1", |
| 161 | + max_retries=0, |
| 162 | + generation_kwargs={"temperature": 0}, |
| 163 | + ), |
| 164 | +) |
| 165 | + |
| 166 | +pipeline.connect("retriever.documents", "prompt_builder.documents") |
| 167 | +pipeline.connect("prompt_builder.prompt", "llm.messages") |
| 168 | + |
| 169 | +question = "What should we do when checkout returns HTTP 503?" |
| 170 | +result = pipeline.run( |
| 171 | + { |
| 172 | + "retriever": {"query": question}, |
| 173 | + "prompt_builder": {"question": question}, |
| 174 | + } |
| 175 | +) |
| 176 | + |
| 177 | +print(result["llm"]["replies"][0].text) |
| 178 | +``` |
| 179 | + |
| 180 | +A response grounded in the retrieved runbook should instruct the operator to fail over to the secondary region and page the on-call engineer. |
| 181 | + |
| 182 | +## Retry Ownership |
| 183 | + |
| 184 | +The example sets `max_retries=0` on `OpenAIChatGenerator` because the displayed AISIX routing model owns failover. This makes one Haystack generation call correspond to one gateway request while AISIX tries the configured targets within that request. |
| 185 | + |
| 186 | +You can keep client-side retries for a different failure policy, but size the Haystack/OpenAI client retry budget and the AISIX retry/failover budget together. Otherwise, one application-level call can create more upstream attempts than expected. |
| 187 | + |
| 188 | +## License |
| 189 | + |
| 190 | +AISIX is distributed under the [Apache License 2.0](https://github.com/api7/aisix/blob/main/LICENSE). |
0 commit comments