Enterprise chatbot projects rarely die because the model can't answer questions - they die in procurement review, when security asks "who can see what, and can you prove it," and nobody has an answer. Azure AI Foundry exists precisely for that gap: a single control plane over model deployment, orchestration, safety filtering, evaluation, and identity that satisfies both the engineering team and the governance team. Here's how a typical enterprise chatbot build looks in Foundry.
Projects and Hubs: The Organizing Structure
Azure AI Foundry organizes work into hubs (the shared infrastructure boundary - networking, compute quota, connected resources) and projects underneath a hub (the actual workspace where a team builds a specific application). A hub might belong to a whole department; a project maps to one chatbot, one evaluation pipeline, one deployment.
az ml workspace create \
--kind hub \
--name enterprise-ai-hub \
--resource-group rg-ai-platform \
--location eastus2
az ml workspace create \
--kind project \
--name support-chatbot-project \
--hub-id /subscriptions/***/resourceGroups/rg-ai-platform/providers/Microsoft.MachineLearningServices/workspaces/enterprise-ai-hub
This separation matters operationally - security reviews the hub once (network isolation, private endpoints, managed identity configuration) and individual project teams don't need to re-litigate infrastructure decisions every time they ship a new bot.
Model Catalog and Deployment
Foundry's model catalog spans Azure OpenAI models (GPT-4.1, GPT-4.1-mini, o3-mini) alongside Microsoft's own Phi family and third-party models like Llama and Mistral, all deployable behind the same endpoint and authentication pattern.
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
project = AIProjectClient.from_connection_string(
credential=DefaultAzureCredential(),
conn_str="eastus2.api.azureml.ms;***;rg-ai-platform;support-chatbot-project",
)
chat_client = project.inference.get_chat_completions_client()
response = chat_client.complete(
model="gpt-4.1-mini",
messages=[
{"role": "system", "content": "You are a support assistant for Contoso Billing."},
{"role": "user", "content": "Why was I charged twice this month?"},
],
temperature=0.2,
)
Picking gpt-4.1-mini over full gpt-4.1 for a support bot is usually the right call - the latency and cost savings compound quickly at enterprise ticket volume, and grounding via retrieval closes most of the quality gap.
Orchestration with Prompt Flow
Prompt flow is Foundry's visual/code-hybrid orchestration layer - it chains retrieval, model calls, and post-processing into a DAG you can version, test, and deploy as a single managed endpoint.
# flow.dag.yaml
inputs:
question:
type: string
nodes:
- name: retrieve_context
type: python
source:
type: code
path: retrieve.py
inputs:
query: ${inputs.question}
- name: generate_answer
type: llm
source:
type: code
path: generate.jinja2
inputs:
deployment_name: gpt-4.1-mini
context: ${retrieve_context.output}
question: ${inputs.question}
outputs:
answer:
type: string
reference: ${generate_answer.output}
Each node is independently testable, and prompt flow tracks the full execution trace per request - which retrieval hits fed which generation call - so debugging a wrong answer means opening one trace, not grepping through application logs.
Content Safety as a Gate, Not an Afterthought
Azure AI Content Safety filters attach directly to a deployment and run before the request reaches the model and after the response leaves it. Configure severity thresholds per harm category (hate, violence, self-harm, sexual) and per direction (prompt vs. completion).
from azure.ai.contentsafety import ContentSafetyClient
from azure.ai.contentsafety.models import AnalyzeTextOptions
safety_client = ContentSafetyClient(endpoint, credential)
result = safety_client.analyze_text(
AnalyzeTextOptions(text=user_message)
)
if any(category.severity >= 4 for category in result.categories_analysis):
return "I can't help with that request. Let me connect you with a specialist."
We wire this as a hard gate in the orchestration graph, not a best-effort check in the UI layer - anything that reaches the model has already passed the filter.
Evaluation: Proving the Bot Is Actually Good
Foundry's evaluation SDK runs built-in evaluators - groundedness, relevance, coherence, fluency - plus custom ones, against a labeled test set, and the results become part of your CI gate before a new prompt or model version ships.
from azure.ai.evaluation import evaluate, GroundednessEvaluator, RelevanceEvaluator
results = evaluate(
data="eval_dataset.jsonl",
evaluators={
"groundedness": GroundednessEvaluator(model_config=judge_model_config),
"relevance": RelevanceEvaluator(model_config=judge_model_config),
},
evaluator_config={
"groundedness": {"column_mapping": {"context": "${data.context}"}},
},
)
assert results["metrics"]["groundedness.gpt_groundedness"] >= 0.85
We block a deployment if groundedness drops below the threshold established in the last sign-off review - it's the one metric compliance actually asks about by name.
Enterprise Auth and RBAC
Every Foundry project sits behind Azure AD, and access to a deployed endpoint, the underlying storage, and the evaluation results are all governed by standard Azure RBAC roles (Azure AI Developer, Azure AI Inference Deployment Operator, etc.) rather than a separate API-key system to manage. Managed identities let the chatbot's backend call the model endpoint without a secret ever touching application code.
- Data plane isolation - private endpoints on the hub keep model traffic off the public internet
- Least privilege by project - a support-bot team gets access to their project only, not the whole hub
- Audit trail - every inference call is logged with the calling identity, satisfying the "who queried what" question that inevitably comes up in a security review
Building the chatbot itself is the easy 20%. Azure AI Foundry earns its keep in the other 80% - deployment governance, safety enforcement, and evaluation discipline - which is exactly the part that gets a project killed in review if you skip it.