How Do You Red Team AI Systems and LLM Applications in the Enterprise?

AI red teaming — the structured adversarial testing of large language models and the agentic systems built around them — has crossed from research curiosity to enterprise mandate in 2026. IBM reports that 79% of enterprises are deploying AI agents, yet 97% lack proper security controls to validate whether those systems behave safely under adversarial conditions. Regulatory frameworks including the EU AI Act, NIST AI RMF, and ISO/IEC 42001 now treat adversarial evaluation as a required engineering practice for high-risk AI deployments, not an optional audit exercise.

The challenge is that most security teams know how to red-team infrastructure but not AI. Traditional penetration testing methodologies do not transfer directly: you are not looking for unpatched services or injection points in SQL queries. You are probing probabilistic systems for behavioral failures — cases where the model generates harmful outputs, leaks sensitive data, ignores safety guardrails, or is manipulated through its own inputs to take actions its designers never intended. A system can pass every unit test, every static analysis scan, and still fail catastrophically under a single adversarial interaction.

This guide covers the complete AI red teaming methodology for enterprise teams: when to run an engagement, what attack categories to cover, which tools to use, and how to integrate adversarial testing into your MLOps pipeline. For the runtime controls that prevent attacks once they are found, see our guide on securing AI agents at the infrastructure level.

What Is AI Red Teaming and How Is It Different from Traditional Red Teaming?

Traditional red teaming targets infrastructure: finding misconfigured servers, weak credentials, unpatched software, and access-control flaws. The adversary's goal is unauthorized access — they move through the network, escalate privileges, and exfiltrate data. AI red teaming targets behavior, not access. The adversary's goal is to manipulate an AI system into violating its own constraints — generating content it should refuse, revealing information it should protect, or taking actions it should not take under any conditions.

NIST's AI Risk Management Framework defines AI red teaming as 'a structured testing exercise that enables organizations to discover risks from adversarial attacks on AI systems.' MITRE ATLAS provides the attack taxonomy — AI-specific techniques mapped to the MITRE ATT&CK structure — that gives red teams a consistent vocabulary for cataloguing findings. In March 2026, a NIST-organized competition with Gray Swan and the UK AI Security Institute found successful attacks against all 13 frontier models tested in agentic scenarios, across more than 250,000 attack attempts. Every system had at least one exploitable failure.

When Must You Red Team an AI System?

Red teaming is not a one-time event. The attack surface of an AI system changes with every model update, prompt revision, RAG source expansion, and new tool connection. A system that passed adversarial testing in January may have new failure modes by April if its underlying model was updated or its context window expanded. The following triggers require a red team engagement — treating any one of them as optional creates a blind spot that attackers will find first.

• →Before first production deployment: any LLM application or AI agent processing real user data or triggering real actions requires adversarial validation before it handles live traffic. This is the minimum bar — not a best practice.
• →After model upgrades: swapping the underlying model, even a minor version, can shift behavioral boundaries unexpectedly. A prompt injection defense tuned for one model checkpoint may not hold for its successor.
• →After significant prompt changes: system prompt revisions alter the model's behavioral contract. Any change that modifies safety instructions, persona definition, or tool access rules triggers a regression test against the prior attack suite.
• →After RAG source expansions: adding data sources to a retrieval-augmented system expands the indirect prompt injection surface. New document collections should be evaluated for embedded adversarial payloads before being connected to a production agent.
• →When adding tool calls to an existing LLM application: giving the model the ability to take actions fundamentally changes the attack surface. A system that previously only generated text can now modify databases, send messages, or execute code. The red team scope grows proportionally.
• →When handling regulated data: HIPAA, GDPR, and the EU AI Act's high-risk classification explicitly require documented adversarial robustness testing. Compliance is the floor, not the ceiling.

The Five-Phase AI Red Teaming Methodology

A structured methodology prevents the most common failure mode of disorganized probing: it produces a list of jailbreaks but no actionable risk picture. The approach used by Microsoft's AI Red Team (AIRT) and NIST's adversarial evaluation center follows five phases that convert adversarial findings into prioritized engineering work.

• →Phase 1 — Scope and threat model: define the system under test (the model, deployment context, and tool access), identify the harm categories in scope (PII leakage, harmful content generation, tool misuse, IP extraction, regulatory violation), and establish success criteria. A customer service chatbot and an autonomous DevOps agent have entirely different threat models. A generic scope guarantees incomplete coverage.
• →Phase 2 — Threat taxonomy mapping: map harm categories to a structured classification. MITRE ATLAS provides AI-specific attack techniques (AML.T0051 for prompt injection, AML.T0054 for jailbreaking). The OWASP Top 10 for LLM Applications provides the vulnerability classification framework. Systematic mapping ensures coverage is comprehensive, not ad hoc.
• →Phase 3 — Attack execution: run both manual and automated probing campaigns in parallel. Manual red teamers find novel attack patterns that automation misses; automated tooling verifies those patterns at scale and catches regressions. Crescendo attacks — multi-turn conversations that gradually escalate toward harmful territory — are a human specialty that most automated tools cannot replicate effectively.
• →Phase 4 — Severity scoring and triage: classify each finding by exploitability, potential harm, and deployment context. A jailbreak requiring 40 turns in a sandboxed system is lower severity than a single-turn prompt injection that exfiltrates PII in a customer-facing application. Document each finding with a reproducible test case and a CVSS-equivalent severity rating.
• →Phase 5 — Remediation and re-test: implement mitigations — input handling, output filtering, prompt hardening, tool access restriction — then re-run original attack cases plus new variants. A mitigation that blocks the exact attack but fails against a paraphrase is incomplete and counts as an open finding.

What Attack Categories to Cover in Every Engagement

Coverage is the primary quality bar for a red team engagement. An exercise that tests only jailbreaks misses the categories most relevant to enterprise production systems. These constitute the minimum baseline for any LLM application processing sensitive data or triggering consequential actions — and they map directly to the vulnerabilities documented in the OWASP Top 10 for LLM Applications.

• →Direct prompt injection: instructions in user input that attempt to override the system prompt, change the model's persona, or extract the system prompt contents. Every application with a system prompt is exposed to this attack class.
• →Indirect prompt injection: malicious instructions hidden in content the model processes from external sources — documents, emails, web pages, database records, tool outputs. This is the highest-severity category for agentic systems because the injection arrives through data channels, not user input, and is far harder to detect and filter.
• →Safety guardrail bypass (jailbreaking): structured attempts to elicit content the model is trained to refuse — harmful instructions, CBRN-adjacent material, illegal activity guidance. Tests the robustness of alignment under sustained adversarial pressure.
• →Sensitive data extraction: probing whether the model reveals PII from its context window, exposes system prompt contents, or surfaces memorized training data in response to crafted queries. Relevant to any system processing confidential customer or employee information.
• →Tool and function-call abuse: for agentic systems, testing whether adversarial prompts can cause the model to misuse connected tools — deleting records it should not delete, sending messages to unauthorized recipients, or escalating its own permissions through a chain of legitimate-looking calls.
• →Memory and context poisoning: in systems with persistent memory or session context, testing whether injected content from one turn influences behavior in later turns in ways that persist across sessions or contaminate shared memory stores.
• →Model output integrity: testing for cases where the model generates plausible but false citations, fabricated compliance statuses, or invented data that a downstream system or human could act on with harmful consequences.

The Enterprise Toolchain: PyRIT, Garak, and Promptfoo

Three tools constitute the enterprise standard for AI red teaming automation in 2026. They are complementary, not competing: each has a distinct philosophy and covers a different part of the attack surface. Using any one alone leaves gaps that the others fill.

• →PyRIT (Microsoft Azure AI Red Teaming Tool): the dominant enterprise tool, tightly integrated with Azure AI Foundry. PyRIT's signature capability is multi-turn conversational attack simulation — crescendo attacks that gradually escalate toward harmful territory across several exchanges, adapting strategy based on each response. An AI Red Teaming Agent released in April 2025 supports automated attack generation, response scoring via a judge LLM, and structured reporting. PyRIT is the right primary tool for agentic systems where single-turn testing misses the realistic attack sequence.
• →Garak (NVIDIA): the LLM equivalent of Nessus — a comprehensive scanner with 37+ probe modules covering hallucination, toxicity, jailbreaking, prompt injection, and data leakage across most major model APIs. Garak's philosophy is breadth over depth: run it first to find where the surface is weak, then send findings to targeted manual investigation. It is the fastest way to establish a baseline coverage map of a new LLM deployment.
• →Promptfoo: covers 50+ vulnerability types with native CI/CD integration via GitHub Actions and GitLab CI. Promptfoo is the right tool for shifting red teaming left — running automated adversarial checks as a quality gate in the model deployment pipeline, blocking promotions when new vulnerabilities are detected. It supports custom assertion writing for domain-specific risk definitions that generic probes do not cover.
• →Manual quarterly engagements: automation finds known patterns at scale; humans find novel patterns automation has never seen. A 1–2 day engagement with internal security engineers or a specialist vendor targets the tail of the attack distribution — complex multi-step jailbreaks, domain-specific attacks, and creative attack sequences that canned probe libraries do not yet contain.

Why Agentic Systems Have a Fundamentally Different Attack Surface

Standard LLM red teaming covers approximately 30% of the attack surface of an equivalent agentic system, according to adversarial testing research from the March 2026 NIST competition. An LLM that only generates text can hallucinate, leak data, and produce harmful content — but it cannot modify a database, send an email, or execute code. An agent with those tools converts a successful prompt injection into a consequential real-world action.

The attack surface compounds with every tool an agent can call. An agent with access to an email tool, a CRM read API, a calendar write API, and a code execution sandbox has four separate tool-abuse vectors on top of every standard LLM vulnerability. Each tool description in the system prompt is itself an injection surface — a compromised or malicious tool description can redirect the model's behavior without touching the user-facing input. This is the threat category that MCP tool poisoning prevention addresses; the red team must cover it as a distinct attack class with its own probe suite.

For agentic systems, red teaming must also cover workflow-level attacks: multi-step attack sequences where the payload is distributed across several turns and multiple tool calls, state persistence attacks where an injection in one session affects behavior in a later session through shared memory, and action-chain amplification where a small prompt deviation causes a disproportionate consequence through cascading tool calls.

Integrating AI Red Teaming into CI/CD and LLMOps

Adversarial testing that only happens quarterly misses the regression window between engagements. Model updates, prompt changes, and RAG source additions happen continuously in a mature LLM deployment — the red team methodology must match the deployment cadence. Shifting left means that adversarial testing gates deployment, not audits it after the fact.

• →Pre-merge gate: run Promptfoo's automated attack suite on every pull request that modifies system prompts, model configuration, or tool definitions. A PR that introduces a regression in prompt injection resistance fails the gate and cannot merge to the deployment branch.
• →Model registry evaluation: before a new model version is promoted from staging to production in your LLM gateway, run Garak's probe suite against it and compare results against the current production baseline. Block promotion if new vulnerability categories appear that did not exist in the incumbent model.
• →RAG change validation: when new document collections are added to a retrieval-augmented pipeline, scan them automatically for indirect prompt injection payloads — adversarial instructions embedded in documents that could redirect the model when retrieved. The volume of enterprise document collections makes manual review impractical; automated scanning is the only viable approach.
• →Quarterly manual engagement: automated tools catch known patterns; quarterly engagements with a dedicated red team target the novel attack surface that automation has not yet learned to probe. Findings from each manual engagement become new probes in the automated suite, continuously raising the baseline.
• →Incident-triggered re-test: any production anomaly resembling adversarial manipulation — unusual tool call sequences, model behavior inconsistent with the system prompt, data access patterns outside normal usage — triggers an immediate targeted red team review of that specific attack vector.

Frequently Asked Questions

What is the difference between AI red teaming and penetration testing?

Penetration testing targets infrastructure: unauthorized access through misconfigured systems, unpatched vulnerabilities, and weak credentials. AI red teaming targets behavior: whether an AI system generates harmful outputs, leaks data it should protect, or takes actions it should not take under adversarial conditions. The methodologies, tools, and skill sets are distinct — most engagements require both, not one or the other.

How often should an enterprise red team its AI systems?

At minimum: before every production deployment, after every model upgrade, and after significant prompt or RAG source changes. In addition, run a quarterly manual engagement and integrate automated adversarial checks into your CI/CD pipeline as a continuous gate. High-risk AI systems under the EU AI Act require documented adversarial testing as a condition of deployment — that requirement applies continuously, not only at initial launch.

Is automated red teaming with tools like Garak or PyRIT sufficient on its own?

No. Automated tools find known attack patterns at scale and provide consistent regression coverage — they are essential infrastructure. But they cannot find novel jailbreaks, domain-specific attacks, or complex multi-turn attack sequences that a skilled human red teamer constructs for a specific deployment context. The correct model is automation for continuous coverage plus manual engagements for discovering the attacks automation has not yet learned.

What qualifications does an AI red team need?

A combination of security and AI expertise that is genuinely rare. AI red teamers need to understand LLM behavior — prompt engineering, context management, model fine-tuning dynamics — alongside security adversarial techniques including social engineering, jailbreak construction, and injection payload design. Most enterprise teams start with existing security engineers, provide AI-specific training, and supplement with specialist vendors for quarterly manual engagements.

Does AI red teaming satisfy EU AI Act compliance requirements?

Adversarial robustness testing is a required element of the EU AI Act conformity assessment for high-risk AI systems, with the August 2026 compliance deadline now active. Red teaming provides the documented evidence of systematic adversarial evaluation that conformity assessments require. However, it must be paired with risk management documentation, human oversight controls, data governance, and post-market monitoring obligations — red teaming alone is necessary but not sufficient for full compliance.

How Belsoft Helps Harden AI Systems Against Adversarial Attack

Belsoft's security and scalability engineering practice designs, tests, and hardens AI applications for enterprise production. We run structured AI red team engagements against agentic and LLM systems — covering the full attack taxonomy from direct prompt injection to multi-step tool-call abuse — and integrate continuous adversarial testing into your MLOps pipeline so regressions are caught before they reach users. We build the mitigations too: input handling, output validation, guardrail architecture, and least-privilege tool access that contains the blast radius when an attack succeeds. If you are deploying AI agents into a security-sensitive environment and need to know where your exposure is, book a technical scoping call to get started.

“Knowing your AI system is secure is not a feeling — it is a test result. Run the attacks before your adversaries do.”