Repository navigation
feat: add native Azure OpenAI batch backend - #727
Merged
madiator merged 1 commit intoJul 13, 2026
Merged
Conversation
Adds a native `backend="azure"` batch request processor, since Azure OpenAI was previously only reachable via the litellm backend, which doesn't support batch mode. Batch was already supported natively for OpenAI, Anthropic, Gemini, and Mistral, but not Azure. Azure's OpenAI-compatible "v1" API surface (https://<resource>.openai.azure.com/openai/v1/) lets AzureBatchRequestProcessor subclass OpenAIBatchRequestProcessor directly and reuse its file/batch lifecycle handling almost unmodified, rather than reimplementing it. It accounts for two documented differences from OpenAI's own API: the batches.create() `endpoint` argument omits the "/v1" prefix, and the request body's "model" field must be the Azure deployment name rather than curator's `model` config (which stays the underlying model name for litellm-based cost lookups). Also adds a `_AzureCostProcessor` (mirroring the existing KlusterAI/ InferenceNet pattern) to prefer litellm's `azure/`-prefixed pricing table, and a small refactor extracting OpenAIBatchRequestProcessor's hardcoded batch endpoint string into an overridable class attribute so Azure doesn't need to duplicate create_batch(). Not verified against a live Azure resource (no Azure credentials available in this environment) - the design follows Microsoft's current official documentation, and the integration test is left @pytest.mark.skip()-decorated pending a run against a real resource. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
madiator
approved these changes
Jul 13, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #538 — Add Native Azure OpenAI Backend with Batch API Support.
Azure OpenAI was previously only reachable via the litellm backend, which doesn't support batch mode in Curator. Native batch support already existed for OpenAI, Anthropic, Gemini, and Mistral, but not Azure — so Azure customers had no working batch path at all.
This adds a native
backend="azure"batch processor:Design
Azure now exposes an OpenAI-SDK-compatible "v1" API surface (
https://<resource>.openai.azure.com/openai/v1/), confirmed against Microsoft's current official docs. This meansAzureBatchRequestProcessorcan subclassOpenAIBatchRequestProcessordirectly and inherit its file-upload/batch-lifecycle handling almost unmodified, rather than reimplementing ~400 lines for a parallel client. It only needs to account for two documented differences from OpenAI's own API:client.batches.create()'sendpointargument omits the/v1prefix ("/chat/completions", not"/v1/chat/completions"), even though the batch file's per-line"url"keeps it."model"field must be the Azure deployment name, not curator'smodelconfig — Azure deployment names are user-chosen and don't map 1:1 to a model name, somodelstays the underlying model (e.g."gpt-4o") for litellm-based cost lookups, and a newazure_deploymentconfig field carries the deployment name separately.Changes
config.py: newazure_deploymentfield onBatchRequestProcessorConfig+BatchBackendParams.openai_batch_request_processor.py: extracted the hardcoded batch endpoint string into an overridable_BATCH_ENDPOINTclass attribute (1-line diff tocreate_batch), so Azure doesn't need to duplicate that method.azure_batch_request_processor.py(new): the processor itself — validatesbase_url/azure_deploymentare set, overrides the endpoint string, substitutes the deployment name into the request body, and prefers Azure-specific cost lookups.cost.py: new_AzureCostProcessor, mirroring the existing_KlusterAICostProcessor/_InferenceNetCostProcessorpattern, preferring litellm'sazure/-prefixed pricing table._factory.py: registersbackend="azure"as an explicit opt-in (not auto-detected from the model name — Azure deployment names are arbitrary and aren't a reliable signal, same reasoning as whyklusterai/inference.netaren't auto-detected either).Caveat
I don't have a real Azure OpenAI resource/credentials available to test against, so this hasn't been verified end-to-end against live Azure infrastructure — the design follows Microsoft's current official documentation rather than a live run. The integration test (
tests/integrations/test_azure_batch.py) is left@pytest.mark.skip()-decorated, mirroring how the existing Mistral batch integration test is handled, pending a run against a real resource by a maintainer or someone with Azure access.Test plan
tests/unittests/test_batch.py(no network required) covering: missingbase_url/azure_deploymentvalidation errors, the deployment-name-vs-model substitution in the request body, the/chat/completionsvs/v1/chat/completionsendpoint divergence, and both branches of the cost processor (Azure-specific pricing found / not found)main)_RequestProcessorFactorypath, including error branches (missingAZURE_OPENAI_API_KEY, online-mode rejection)ruff check/ruff formatclean