Skip to content

feat: add native Azure OpenAI batch backend - #727

Merged
madiator merged 1 commit into
bespokelabsai:mainfrom
EpicSRN021:feat/azure-batch-backend-538
Jul 13, 2026
Merged

madiator merged 1 commit into
bespokelabsai:mainfrom
EpicSRN021:feat/azure-batch-backend-538

Conversation

@EpicSRN021

Copy link
Copy Markdown
Contributor

Summary

Closes #538 — Add Native Azure OpenAI Backend with Batch API Support.

Azure OpenAI was previously only reachable via the litellm backend, which doesn't support batch mode in Curator. Native batch support already existed for OpenAI, Anthropic, Gemini, and Mistral, but not Azure — so Azure customers had no working batch path at all.

This adds a native backend="azure" batch processor:

curator.LLM(
    model_name="gpt-4o",
    backend="azure",
    batch=True,
    backend_params={
        "base_url": "https://<resource>.openai.azure.com/openai/v1/",
        "azure_deployment": "<your-batch-deployment-name>",
    },
)

Design

Azure now exposes an OpenAI-SDK-compatible "v1" API surface (https://<resource>.openai.azure.com/openai/v1/), confirmed against Microsoft's current official docs. This means AzureBatchRequestProcessor can subclass OpenAIBatchRequestProcessor directly and inherit its file-upload/batch-lifecycle handling almost unmodified, rather than reimplementing ~400 lines for a parallel client. It only needs to account for two documented differences from OpenAI's own API:

  1. client.batches.create()'s endpoint argument omits the /v1 prefix ("/chat/completions", not "/v1/chat/completions"), even though the batch file's per-line "url" keeps it.
  2. The request body's "model" field must be the Azure deployment name, not curator's model config — Azure deployment names are user-chosen and don't map 1:1 to a model name, so model stays the underlying model (e.g. "gpt-4o") for litellm-based cost lookups, and a new azure_deployment config field carries the deployment name separately.

Changes

  • config.py: new azure_deployment field on BatchRequestProcessorConfig + BatchBackendParams.
  • openai_batch_request_processor.py: extracted the hardcoded batch endpoint string into an overridable _BATCH_ENDPOINT class attribute (1-line diff to create_batch), so Azure doesn't need to duplicate that method.
  • azure_batch_request_processor.py (new): the processor itself — validates base_url/azure_deployment are set, overrides the endpoint string, substitutes the deployment name into the request body, and prefers Azure-specific cost lookups.
  • cost.py: new _AzureCostProcessor, mirroring the existing _KlusterAICostProcessor/_InferenceNetCostProcessor pattern, preferring litellm's azure/-prefixed pricing table.
  • _factory.py: registers backend="azure" as an explicit opt-in (not auto-detected from the model name — Azure deployment names are arbitrary and aren't a reliable signal, same reasoning as why klusterai/inference.net aren't auto-detected either).

Caveat

I don't have a real Azure OpenAI resource/credentials available to test against, so this hasn't been verified end-to-end against live Azure infrastructure — the design follows Microsoft's current official documentation rather than a live run. The integration test (tests/integrations/test_azure_batch.py) is left @pytest.mark.skip()-decorated, mirroring how the existing Mistral batch integration test is handled, pending a run against a real resource by a maintainer or someone with Azure access.

Test plan

  • 6 new unit tests in tests/unittests/test_batch.py (no network required) covering: missing base_url/azure_deployment validation errors, the deployment-name-vs-model substitution in the request body, the /chat/completions vs /v1/chat/completions endpoint divergence, and both branches of the cost processor (Azure-specific pricing found / not found)
  • Full existing unit test suite run — no regressions (pre-existing failures needing live network/API keys are identical on main)
  • Manually constructed the processor end-to-end and through the full _RequestProcessorFactory path, including error branches (missing AZURE_OPENAI_API_KEY, online-mode rejection)
  • ruff check / ruff format clean
  • Not yet verified against a live Azure OpenAI resource (see caveat above)

Adds a native `backend="azure"` batch request processor, since Azure OpenAI
was previously only reachable via the litellm backend, which doesn't support
batch mode. Batch was already supported natively for OpenAI, Anthropic,
Gemini, and Mistral, but not Azure.

Azure's OpenAI-compatible "v1" API surface (https://<resource>.openai.azure.com/openai/v1/)
lets AzureBatchRequestProcessor subclass OpenAIBatchRequestProcessor directly and
reuse its file/batch lifecycle handling almost unmodified, rather than
reimplementing it. It accounts for two documented differences from OpenAI's
own API: the batches.create() `endpoint` argument omits the "/v1" prefix, and
the request body's "model" field must be the Azure deployment name rather
than curator's `model` config (which stays the underlying model name for
litellm-based cost lookups).

Also adds a `_AzureCostProcessor` (mirroring the existing KlusterAI/
InferenceNet pattern) to prefer litellm's `azure/`-prefixed pricing table,
and a small refactor extracting OpenAIBatchRequestProcessor's hardcoded batch
endpoint string into an overridable class attribute so Azure doesn't need to
duplicate create_batch().

Not verified against a live Azure resource (no Azure credentials available
in this environment) - the design follows Microsoft's current official
documentation, and the integration test is left @pytest.mark.skip()-decorated
pending a run against a real resource.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@madiator
madiator merged commit 08e42e7 into bespokelabsai:main Jul 13, 2026
1 check failed
@madiator madiator mentioned this pull request Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Native Azure OpenAI Backend with Batch API Support

2 participants