2 November 2026 · 10 min read · By Michael van Mameren, Lead Penetration Tester (OSCP)
What is AI red teaming? How adversarial testing of AI systems works, what it covers and which standards now ask for it
AI red teaming is the structured, adversarial testing of an AI system, its model, its data, its tools and the application around it, to find how it can be made to misbehave or be abused before a real attacker does. The term covers two different practices, and buyers who do not separate them end up with the wrong test. This guide explains both, what an AI red team actually tests, how an exercise runs, how it differs from an AI penetration test and a classic red team, and which US and EU frameworks now expect it.
What AI red teaming is
AI red teaming is adversarial testing of an AI system by people who think and act like an attacker, with the explicit goal of making the system fail in ways its owners did not intend. The object under test is not only the model. In a deployed system it includes the system prompt and guardrails, the retrieval pipeline that feeds documents into the context, the tools and connectors the model can call, the identity and permissions it runs under, the application that renders its output, and the logging that is supposed to catch misuse. A red team that only sends clever prompts to a chat window is testing one layer of a six-layer system.
The term was borrowed from cybersecurity, where a red team emulates a real adversary against an organisation. In AI it has acquired two meanings. Model developers use it for expert panels probing a new model for harmful outputs before release; security teams use it for objective-driven attacks on deployed AI applications. For an enterprise running a customer-facing assistant, an internal copilot over company data or an agent that acts in business systems, the relevant question is rarely 'can this model be made to say something offensive'. It is 'can a customer, an employee, a supplier document or a web page make this system leak data, take an action it should not, or give a user something they are not entitled to'. That is a security question, answered by security red teaming rather than by a safety evaluation of the base model.
The two meanings: safety red teaming of models vs security red teaming of AI systems
Both are called AI red teaming. They share a word and little else. Most enterprise buyers need the right-hand column.
| Aspect | Safety red teaming (of a model) | Security red teaming (of an AI system) |
|---|---|---|
| Primary question | Can the model be induced to produce harmful, biased or policy-violating content? | Can an adversary make the deployed system leak data, take unauthorised actions or be abused against the business? |
| Object under test | The foundation or fine-tuned model and its alignment training | Model, system prompt, guardrails, retrieval, tools, identities, application, logging |
| Who performs it | Model developers, domain experts, trust-and-safety and policy teams | Offensive security testers with application, cloud and identity experience plus LLM-specific skills |
| Output | Harm rates, refusal behaviour, capability evaluations, model cards | Reproducible findings with impact, attack narratives, remediation, regression test set |
| Driven by | Provider obligations: EU AI Act Article 55 for systemic-risk models | Deployer obligations: EU AI Act Articles 9 and 15, SOC 2, ISO 27001, ISO/IEC 42001, customer due diligence |
What an AI red team actually tests
Described at concept level. The classes map to the OWASP Top 10 for LLM Applications 2025 and to MITRE ATLAS techniques, which is how a good report labels them.
Prompt injection, direct and indirect
Direct injection is a user telling the model to ignore its rules. Indirect injection is the enterprise problem: instructions hidden in a web page, an email, a PDF in the knowledge base or a support ticket that the model reads as data and follows as a command (OWASP LLM01). The red team maps every channel through which untrusted text reaches the model.
Guardrail bypass and jailbreaks
Can the system be moved outside its intended persona, scope or acceptable-use policy? For a bank's assistant that means giving advice it is not licensed to give; for an HR copilot, discussing other employees. The test measures whether guardrails hold under persistent, creative pressure, not whether a filter exists.
Sensitive information and system prompt disclosure
Does the system reveal personal data, other tenants' documents, credentials in configuration, or the system prompt itself (OWASP LLM02 and LLM07)? Retrieval is tested for authorisation: does the model only see documents the current user may see, or does it see everything and rely on instructions to stay quiet?
Data exfiltration through tools and output channels
When a model can browse, send email, call APIs or render rich content, injected instructions can turn those capabilities into exfiltration paths the user never sees. The red team chains injection with every outbound capability, which is where most high-severity findings sit.
Retrieval and data poisoning
Who can write to the knowledge base, vector store, shared drive or ticket queue the assistant reads from? Poisoning one document can change answers for every user or plant a persistent injection (OWASP LLM04 and LLM08). The test covers ingestion paths, embedding integrity and detectability afterwards.
Excessive agency and agent misuse
Agents that create tickets, change records, move money or run code are tested for the confused-deputy problem: acting with their own broad permissions for a user who should have narrow ones (OWASP LLM06). Approval gates, tool allow-lists, rate limits and the blast radius of a compromised agent identity are in scope.
Model theft, supply chain and infrastructure
Fine-tuned weights, embeddings and prompts are assets. The red team checks model registries, artefact storage, serialised model formats that can carry code, third-party models and plugins from public hubs (OWASP LLM03), and the cloud and identity layer that hosts inference.
Improper output handling and unbounded consumption
Model output passed to a browser, shell, database or downstream API without validation revives classic injection classes with the model as the vector (OWASP LLM05). Unbounded consumption (LLM10) covers denial-of-wallet, runaway agent loops and resource exhaustion through crafted inputs.
How an AI red team exercise runs
The same discipline as any offensive engagement, with two additions: a model-specific threat model and a regression harness at the end.
1. Scoping and threat modelling
Which systems, models, providers, data classes, tools and user populations; where the trust boundaries lie (user to model, model to retrieval, model to tools, tenant to tenant). Rules of engagement cover production versus staging, test accounts, data handling, the provider's acceptable-use terms, and what success looks like for the attacker: read another customer's record, trigger a payment, exfiltrate a document.
2. Architecture and configuration review
Before attacking, the team reads the system: system prompts, guardrail configuration, retrieval authorisation logic, tool definitions and permissions, the identity the agent runs under, logging. This is grey-box by design; black-box AI red teaming wastes days rediscovering what a diagram would show and still misses the authorisation flaws.
3. Model-layer adversarial testing
Manual, creative pressure on the model's behaviour within the deployed context: injection through every input channel, persona and scope escape, disclosure attempts, multi-turn and multilingual variants, multimodal inputs where the system accepts images or files. Automated attack suites add breadth; the chained, context-specific findings come from people.
4. Integration and application testing
The surrounding system gets a proper application and API penetration test: authentication, authorisation on retrieval and tool endpoints, input and output validation, rate limiting, secrets, cloud configuration. Most AI incidents to date were enabled by ordinary application weaknesses with a model in the middle.
5. Objective-driven attack chains
The pieces are combined into end-to-end scenarios against the agreed objectives: a poisoned document that makes the assistant exfiltrate a record through a tool; an injected email that makes an agent approve a request. Each chain is documented with evidence and business impact, not a severity score alone.
6. Reporting and remediation workshop
Findings are mapped to the OWASP Top 10 for LLM Applications 2025, MITRE ATLAS and the client's obligations (EU AI Act Articles 9 and 15, NIST AI RMF, ISO/IEC 42001). Remediation is architectural as often as it is a filter change: least-privilege tool scopes, retrieval authorisation, output encoding, approval gates, monitoring for injection patterns.
7. Retest and regression harness
Every successful attack becomes a reproducible test case. After remediation the team retests and hands the case set over so it can run against every model upgrade, prompt change and new tool. AI systems change weekly; a one-off report ages faster than any other security deliverable.
AI red teaming vs an AI pentest vs a classic red team
Three engagements that are regularly confused in procurement. The right one depends on what you need to prove and to whom.
| Aspect | AI red teaming | AI penetration test | Classic red team |
|---|---|---|---|
| Goal | Find how the AI system can be abused against defined objectives | Cover the AI application, model integration and infrastructure for known weakness classes | Test the organisation's detection and response against a realistic adversary |
| Scope | One or more AI systems, their data, tools and users; open-ended within rules of engagement | Defined AI application, APIs, retrieval, cloud components; coverage-driven | People, physical, network, cloud, identity; AI may be one path among many |
| Approach | Adversarial, creative, objective-based; manual with automated breadth | Methodical, checklist-backed (OWASP LLM Top 10, ATLAS, WSTG, API Top 10) | Stealthy, intelligence-led, long duration, blue team usually unaware |
| Typical findings | Chained injection-to-exfiltration, agent abuse, guardrail failure under pressure, poisoning paths | Authorisation gaps, output handling, secrets, misconfiguration, missing rate limits | Initial access via phishing or exposed services, lateral movement, detection gaps |
| Duration and cadence | Two to four weeks; repeat on major model, prompt or tool changes | One to three weeks; per release or annually | Four to twelve weeks; annually or per regulatory cycle |
| Evidence | Attack narratives, impact demonstrations, regression test set | Findings list with CVSS and remediation, attestation for auditors and customers | Timeline, detection and response metrics, strategic recommendations |
| Best when | You run agents or assistants over sensitive data and need to know what an adversary can do | You need auditable coverage before launch or for SOC 2, ISO 27001 or customer due diligence | You need to test the security organisation, not one system |
Standards and regulation that ask for AI red teaming
The OWASP Top 10 for LLM Applications 2025 is the common vocabulary: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) is the ATT&CK-style knowledge base of tactics and techniques against machine-learning systems, the right reference for describing attack chains and building detections afterwards. NIST AI 100-2 E2025, the adversarial machine learning taxonomy published in March 2025, supplies the formal terminology for evasion, poisoning, privacy and misuse attacks.
In the United States, the NIST AI Risk Management Framework 1.0 (January 2023) sets out the Govern, Map, Measure and Manage functions, and its Generative AI Profile, NIST AI 600-1 (July 2024), lists AI red teaming among the suggested actions for measuring risks such as information security, data privacy and harmful content. Federal policy has shifted between administrations: the October 2023 executive order that required developers of the largest models to report red-team results was rescinded in January 2025, while the Office of Management and Budget memoranda of April 2025 still require agencies to test high-impact AI before use. CISA and NSA published guidance on deploying AI systems securely in April 2024 that recommends adversarial testing of deployed systems. Sector regulators read existing obligations (GLBA Safeguards, HIPAA Security Rule, NYDFS Part 500, SEC disclosure duties) as covering AI systems that process regulated data, which makes a documented adversarial test the practical way to show diligence.
In the European Union, Regulation (EU) 2024/1689, the AI Act, makes adversarial testing explicit at two levels. Article 55 requires providers of general-purpose AI models with systemic risk to perform model evaluation including documented adversarial testing; those obligations have applied since 2 August 2025. For high-risk AI systems, Article 9 requires a risk-management system with testing against defined metrics, and Article 15 requires accuracy and cybersecurity, naming resilience against data poisoning, model poisoning, adversarial examples, model evasion and confidentiality attacks; those obligations are scheduled to phase in from August 2026, subject to the amendment process in Brussels. ISO/IEC 42001:2023, the AI management system standard, expects evidence of testing and is becoming the certification enterprise customers ask for. For organisations under NIS2 Article 21(2)(f) or DORA Article 25, an AI system behind an essential or critical function falls inside the existing testing programme like any other system.
What a good AI red team deliverable contains
If the proposal does not promise these, you are buying a prompt-hacking session, not an assessment.
- A threat model of the AI system: trust boundaries, data classes, tools, identities and the agreed attacker objectives
- Findings with reproducible evidence, business impact and a severity that reflects what the attacker achieved
- Mapping of every finding to the OWASP Top 10 for LLM Applications 2025 and to MITRE ATLAS techniques
- Attack narratives for each successful chain, from entry point to objective, readable by engineers and the risk committee
- Guardrail and retrieval-authorisation assessment: what held, what failed, and where the design relies on instructions instead of access control
- Architectural remediation, not only filter tuning: least-privilege tool scopes, approval gates, output encoding, ingestion controls, monitoring
- A regression test set of the successful attacks your team can run after every model, prompt or tool change, plus a retest of remediated findings
When to commission an AI red team, and how HackersHub approaches it
Commission one before a customer-facing assistant goes live, before an agent is given write access to any business system, after a change of model or provider, after an incident, and annually for anything that touches regulated data. The trigger that matters most is the step from answering questions to taking actions: once an AI system can send, change, approve or pay, the consequences of a successful injection move from embarrassing to material.
HackersHub approaches AI red teaming as offensive security first. Our testers hold at least OSCP, with OSWE and CREST-certified testers on the team, and the AI work builds on 10+ years of application, cloud and identity testing: the retrieval pipeline is an authorisation problem, the agent is a confused-deputy problem, the output channel is an injection problem, and the model is the new component that makes all three reachable through natural language. Engagements are scoped individually, follow a documented methodology aligned with OWASP, MITRE ATLAS and NIST SP 800-115, and end with a report that an engineer, an auditor and a board member can each use. Delivery is remote from Amsterdam, Netherlands, with on-site work where an engagement needs it.
Frequently asked questions
Related reading
- AI penetration testing by HackersHub
- Red teaming services by HackersHub
- Penetration testing services by HackersHub
- What is red teaming? Adversary simulation explained
- Red team vs penetration testing: which one do you need?
- Purple teaming: turning offensive findings into detections
- API penetration testing
- Cloud penetration testing
Deploying assistants or agents over sensitive data?
We scope AI red teaming and AI penetration tests against your systems and objectives, test the model, the data, the tools and the application around them, and deliver findings your engineers, auditors and board can act on. Talk to an expert.