Skip to content
Help guide the future of agentgateway — Take the survey

For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Azure

Verified : Automated tests passed for: Configuration files: The configuration files on this page were checked against their schema. End-to-end run: Every step ran in a fresh test environment and the results were checked.
Page as Markdown

Route agentgateway LLM traffic to models hosted on Microsoft Azure AI.

Configure Microsoft Azure AI as an LLM provider in agentgateway.

Authentication

Before you can use Azure as an LLM provider, you must authenticate by using one of the standard Azure authentication methods. In standalone mode, this authentication is configured with llm.models[] fields (for example, params.apiKey or auth.azure). In routing-based configurations, use policies.backendAuth.azure.

Important

Azure CLI authentication requires az or azd to be installed and signed in. Agentgateway calls the CLI when it needs a token. It does not open an interactive flow or run az login or azd auth login for you. Agentgateway does not bundle either command. Mounting a credential directory such as ~/.azure makes cached login state available inside the container, but it does not install the CLI. Use Azure CLI authentication only when running agentgateway directly on your local machine. If agentgateway runs in a container, use an API key, client secret, managed identity, or workload identity.

Configuration

Azure supports two endpoint types:

  • Azure AI Foundry (foundry): Connect to Azure AI Foundry project endpoints at {resourceName}-resource.services.ai.azure.com.
  • Azure OpenAI (openAI): Connect directly to Azure OpenAI Service deployments at {resourceName}.openai.azure.com.

You can add the model in the UI or in your configuration file.

To add the model in the UI, follow these steps. The steps create the same models as the Config file tab.

  1. Optional: To authenticate with an API key, set your API key in the AZURE_API_KEY environment variable. The UI checks that the variable exists when you save the model.

    export AZURE_API_KEY='<your-api-key>'
  2. Create a configuration file with an empty llm section, and start agentgateway in the same terminal. Without an llm section, the UI does not show the LLM > Models page.

    cat > config.yaml <<'EOF'
    # yaml-language-server: $schema=https://agentgateway.dev/schema/config
    llm:
      models: []
    EOF
    agentgateway -f config.yaml
  3. Open LLM > Models, such as http://localhost:15000/ui/llm/models, and click Add model.

  4. From the Provider dropdown, select Azure.

  5. In the Incoming model match field, replace the default azure/* with the model name to match, such as * or gpt-4.1.

  6. For the Azure credentials, select one of the following options.

    • Default: Uses implicit Entra ID authentication.
    • Managed: Uses a managed identity. Optionally, enter the client ID of a user-assigned identity.
    • API key: Select Env var, and enter AZURE_API_KEY.
  7. Enter the Azure resource name, such as your-resource-name.

  8. For the Azure resource type, select Foundry or OpenAI. For Foundry, also enter the Azure project name, such as your-project-name.

  9. Optional: Enter an Azure API version. Leave the field empty to use the default.

    Add model drawer with Azure selected as the provider, Default as the Azure credentials, your-resource-name as the Azure resource name, Foundry as the Azure resource type, and your-project-name as the Azure project name
    Add model drawer with Azure selected as the provider, Default as the Azure credentials, your-resource-name as the Azure resource name, Foundry as the Azure resource type, and your-project-name as the Azure project name
  10. Click Save model.

The API key option saves the key in params.apiKey. The Foundry (API key) example in the Config file tab uses an auth.key block instead, which sends the key in the api-key header. To write that block, click the Raw icon (<>) in the drawer header, or use the Config file tab.

Whether the UI can save depends on your storage mode. For more information, see Save changes from the UI.

Advanced configuration

For advanced Azure AI scenarios, use the traditional listener/route configuration format. The following tabs show examples for different authentication methods.

Azure AI Foundry with implicit auth: Use DefaultAzureCredential to automatically detect credentials from the environment (Azure CLI, managed identity, workload identity, or environment variables).

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
gateways:
  default:
    port: 3000
routes:
- matches:
  - path:
      pathPrefix: /azure
  backends:
  - ai:
      name: azure
      provider:
        azure:
          resourceName: "your-resource-name"
          projectName: "your-project-name"
          resourceType: foundry
          model: gpt-4.1

Review the following example configuration.

SettingDescription
ai.nameThe name of the LLM provider for this AI backend.
ai.provider.azure.resourceNameThe Azure resource name used to construct the endpoint hostname.
ai.provider.azure.resourceTypeThe endpoint type: foundry for Azure AI Foundry, or openAI for Azure OpenAI Service.
ai.provider.azure.projectNameThe Foundry project name. Required for foundry type.
ai.provider.azure.modelOptionally set the model to use for requests. If set, any models in the request are overwritten. If not set, the request must include the model to use.
backendAuth.azure.implicitUse implicit authentication via DefaultAzureCredential, which automatically detects credentials from the environment.

Use Claude models on Azure AI Foundry

Azure AI Foundry hosts Anthropic Claude models at native Anthropic endpoints. When you set azureResourceType: foundry and a model name that starts with claude-, agentgateway automatically routes requests to the Anthropic-native path (/anthropic/v1/messages) instead of the OpenAI-compatible path, and injects the required anthropic-version header. No extra configuration is needed beyond specifying a Claude model name.

Note

For more information about Claude models on Azure AI Foundry, see the Microsoft documentation.

Review the following example configuration.
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
gateways:
  default:
    port: 3000
routes:
- name: azure
  matches:
  - path:
      pathPrefix: /azure-anthropic #prefix example
  backends:
  - ai:
      name: azure
      provider:
        azure:
          resourceName: your-foundry-resource
          projectName: your-project-name
          resourceType: foundry
          model: claude-sonnet-4-6
    policies:
      backendAuth:
        key:
          value: your-api-key
Review the following table to understand this configuration.
SettingDescription
nameThe exact Claude model name to match in incoming requests, such as claude-3-5-haiku-20241022. Use * to match any model name.
providerSet to azure for Azure AI Foundry.
backendAuth.key.valueThe Azure AI Foundry API key. You can reference environment variables using the $VAR_NAME syntax. The key is automatically sent in the Authorization header. Other auth method can be applied Backend authentication
params.azureResourceNameThe Azure AI Foundry resource name used to construct the endpoint hostname.
params.azureResourceTypeSet to foundry to use Azure AI Foundry endpoints.
params.azureProjectNameThe Foundry project name.

After running agentgateway with this configuration, send a request to verify:

curl -X POST http://localhost:4000/azure-anthropic \
  -H "Content-Type: application/json" \
  -d '{
    "max_tokens": 256,
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Use Decisions on Azure

Azure models can answer classification questions about an input, such as which of several choices fits best or how likely a statement is true. agentgateway accepts two client request formats for these models:

  • Decisions API (/v1/decisions): The OpenAI-compatible format, with predicate, choice, and score question types. For another example of this API, see the Jev guardrail guide.
  • SystemOne (/v1/systemone): The TypeSafe format, with corresponding noul, choice, and score question types.

Choose the Azure endpoint that hosts your model. Agentgateway converts requests and responses to the upstream endpoint’s native format.

  • Azure OpenAI: Deploy a model that supports the Azure OpenAI Decisions API, such as gpt-6-luna. Azure OpenAI receives both client formats at /openai/v1/decisions: Decisions requests are sent unchanged, and SystemOne requests are converted to Decisions.
  • Azure AI Foundry: Deploy Microsoft-Decision-1 in your Foundry resource. Foundry receives both client formats at https://{resourceName}.cognitiveservices.azure.com/providers/microsoft/v1/systemone: SystemOne requests are sent unchanged, and Decisions requests are converted to SystemOne.

Before you begin, sign in with the Azure CLI or the Azure Developer CLI as an identity that has inference access to the resource.

  1. Save the following configuration as config-decisions.yaml. Choose the tab for the endpoint that hosts your model, and replace the resource and deployment names with your own values.

    Review the following example configuration.
    # yaml-language-server: $schema=https://agentgateway.dev/schema/config
    llm:
      models:
      - auth:
          azure:
            developerImplicit: {}
        name: azure-decisions
        params:
          azureResourceName: my-openai-resource
          azureResourceType: openAI
          model: my-decision-deployment
        provider: azure
      port: 3000
    Review the following table to understand this configuration.
    SettingDescription
    nameThe model name that clients send in the model field. This name is not the Azure deployment name. A request with any other model name returns a 404 error.
    providerSet to azure for Azure.
    auth.azure.developerImplicitAuthenticate with the credentials from your Azure CLI or Azure Developer CLI sign-in.
    params.azureResourceNameThe Azure resource name used to construct the endpoint hostname.
    params.azureResourceTypeSet to openAI for Azure OpenAI or foundry for Azure AI Foundry.
    params.modelThe name of your Azure deployment. agentgateway sends this name upstream and replaces the model value in the client request.

    Both request paths are served from the same model. You do not need to configure a route for each path.

  2. Validate the configuration, then start agentgateway.

    agentgateway --validate-only -f config-decisions.yaml
    agentgateway -f config-decisions.yaml
  3. In another terminal, send a predicate question through the Decisions API.

    curl --fail-with-body --max-time 120 http://localhost:3000/v1/decisions \
      -H 'Content-Type: application/json' \
      -d '{
        "input": "I was charged twice. Can I get a refund?",
        "model": "azure-decisions",
        "questions": [{
          "instructions": "Is the customer asking for a refund?",
          "name": "refund_requested",
          "type": "predicate"
        }]
      }'

    The response includes the probability that the customer requested a refund and the request’s token usage. For more information about the request format, see Decisions.

  4. Send the same question in the SystemOne client format. SystemOne takes the input in state, the questions as a map keyed by name, and returns an answers map.

    curl --fail-with-body --max-time 120 http://localhost:3000/v1/systemone \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "azure-decisions",
        "questions": {
          "refund_requested": {
            "instructions": "Is the customer asking for a refund?",
            "type": "noul"
          }
        },
        "state": "I was charged twice. Can I get a refund?"
      }'

    Note

    Prompt guards are disabled for requests in the SystemOne format.

  5. When you finish, stop agentgateway with Ctrl+C.

Was this page helpful?
Agentgateway assistant

Ask me anything about agentgateway configuration, features, or usage.

Note: AI-generated content might contain errors; please verify and test all returned information.

Tip: one topic per conversation gives the best results. Use the + button in the chat header to start a new conversation.

Switching topics? Starting a new conversation improves accuracy.
↑↓ navigate ↵ select esc dismiss

What could be improved?

Your feedback helps us improve assistant answers and identify docs gaps we should fix.

Need more help? Join us on Discord: https://discord.gg/y9efgEmppm

Want to use your own agent? Add the Solo MCP server to query our docs directly. Get started here: https://search.solo.io/.