AI Red Teaming: How to Test AI System Security

AI red teaming is testing the security of an AI system by playing the attacker, to find weaknesses before real attackers do. The group that plays the attacker is called a red team. NIST (the US National Institute of Standards and Technology) defines it as a structured testing effort, often using adversarial methods, to find flaws and vulnerabilities in an AI system, including unforeseen or undesirable behaviors and risks from misuse of the system. This AI red teaming how-to explains the scope, the steps, the tools, and how to read the results.

How AI red teaming differs from ordinary security testing

OWASP (the open software security community) published the GenAI Red Teaming Guide, version 1.0, in January 2025. It defines GenAI red teaming as a structured methodology that combines human expertise with automation and AI tools, to uncover gaps in safety (of the users), security (of the system operator), trust, and performance in systems that include generative AI (AI that creates new content, such as text or images). CISA (the US cybersecurity agency) wrote in a November 2024 blog post that AI red teaming is the third-party safety and security evaluation of AI systems, and a subset of TEVV (testing, evaluation, verification, and validation), which software has used for a long time. CISA therefore advises building on existing software practice rather than creating a separate process.

OWASP lists several differences from traditional red teaming. The main ones are:

• A wider scope: besides technical flaws, you must test what the model produces, such as harmful content, bias, or false statements the AI makes with confidence (hallucination), and look from both the attacker's side and the side of the affected user
• Unpredictable output: a model can answer the same question differently, so testing must be statistical, meaning repeated trials and a look at proportions, not a single pass or fail
• Harder criteria for success: telling normal model error apart from the effect of an attack is difficult. OWASP asks how you know, when a model is more than 90% accurate, whether a wrong answer comes from normal error or from an attacker
• A multidisciplinary team: responsible AI experts look at bias and harmful content, while cybersecurity experts look at backdoors, bypassed controls, and real-world implementation

What to test: four areas of an AI system

A working AI is more than a model. It is the model, what wraps around it, the surrounding systems, and the people who use it. So OWASP splits testing into four areas:

• Model evaluation: test the weaknesses of the model itself, such as resistance to inputs designed to deceive it, generation of harmful content, bias, and leakage of data the model learned from
• Implementation: test what wraps around the model, namely the system prompt (the hidden starting instructions the developer gives the model), guardrails (rules and filters that control the AI's answers), and RAG (a setup where the AI searches your documents to build an answer), to see whether they can be tricked into letting something through
• System: test APIs, data storage, the supply chain of models and libraries, and connections to other systems, including cases where the AI is given too much permission
• Runtime: test the interaction between people and the AI, and between agents (AI that works through tasks on its own) and other systems, including over-reliance on the AI and social engineering of people

OWASP gives the Microsoft Copilot exploits shown at Black Hat USA 2024 (a security conference) as an example. They largely did not target model weaknesses. They used loosely set search permissions in the surrounding system to expose sensitive data. The lesson is to test the whole system, not only the model. If the system has an agent that calls tools and acts on its own, test its permissions and approvals as well, see Agentic AI Security: Controlling AI That Acts on Its Own.

How to run AI red teaming, step by step

OWASP has a Quick Start Guide with initial steps, and an activity view of 11 evaluation steps that runs from scoping to retesting after fixes. The steps below are drawn from these two parts, and the ordering is our own. The steps are:

• Set objectives and scope by risk: start with AI systems that are critical to the business or use sensitive data, and state clearly which system is tested, for what, and what is out of scope. OWASP observes that internal-facing systems are often less risky than externally exposed ones
• Assemble a multidisciplinary team: AI engineers, security specialists, and, where possible, specialists in ethics or compliance
• Do threat modeling: a systematic walk through the system to see how an attacker could get in. OWASP uses the four questions from the Threat Modeling Manifesto: what are we working on, what can go wrong, what are we going to do about it, and did we do a good enough job
• Agree the rules of engagement and get authorization: the rules of engagement say where testing happens, what may be used, what is forbidden, and who decides when something goes wrong
• Test all four areas with both people and tools: use automated tools for volume, and people to review and to think up scenarios that tools do not
• Report with evidence and severity: record the test case, the evidence, the impact, and the recommended fix, and keep a path to senior management for serious findings
• Fix, then retest: OWASP says red teaming is not a one-time event, so retest after fixes and repeat periodically across the system's life cycle

Use reference frameworks to plan the tests

You do not need to invent every test yourself. The frameworks below each answer a different question:

• OWASP Top 10 for LLM Applications 2025: the ten risks for applications that use an LLM (large language model, the engine behind chatbots), used as a checklist of what to test, such as prompt injection (planting instructions that make a model act against its setup)
• MITRE ATLAS (a public knowledge base of attack methods against AI systems): it collects the tactics, techniques, and procedures of attackers who target AI systems, laid out much like MITRE ATT&CK (a knowledge base of attacker techniques against ordinary IT systems). The data at version 2026.09 has 16 tactics, along with mitigations and case studies. For example, AML.T0051 LLM Prompt Injection splits into Direct, Indirect, and Triggered. Use it to turn attacker scenarios into test cases and to cite technique IDs in reports
• NIST AI 100-2 E2025: a taxonomy (a way of sorting into categories) of attacks and mitigations, final in March 2025, covering both predictive AI and generative AI. Use it as a shared vocabulary for naming types of attack
• NIST AI 600-1: the Generative AI Profile (July 2024), which describes AI red-teaming and the forms a testing team can take. OWASP uses it as one of three main references for testing scope

OWASP also advises testing by the model's life cycle stage: acquisition, experimentation and training, and serving. It takes its life cycle vocabulary from ISO/IEC 5338:2023. We found no ISO standard specific to AI red teaming in this check. For the seven layers of an AI system to protect, and for governance, see AI Security and AI Governance: Protect Every AI Component.

Who tests, and how to use automated tools

NIST AI 600-1 says the quality of red-teaming results relates to the background and expertise of the team itself. Teams that are diverse and understand the deployment context help find flaws in different situations. NIST lists four forms of team:

• General public: people expected to use the system, who bring their own experience to try to elicit harmful behavior. NIST says this form can be more effective with large groups of testers
• Expert: specialists in a domain, such as medicine, biotech, or cybersecurity
• Combination: using both groups, for example experts modifying or verifying the test prompts that general users wrote
• Human and AI: generative AI helping or leading the testing. NIST says it can be more cost effective than human testers alone, and that each form may suit different kinds of harm

Automated tools let you try attacks quickly and repeat them. OWASP lists open source tools in its appendix, such as Microsoft's PyRIT, garak, and promptfoo, which are only examples, not a ranking. NIST AI 100-2 describes one kind of automated testing as using an attacker model, a target model, and a judge model that rates whether an answer is harmful. OWASP warns that people must review tool results, because they can judge wrongly in both directions, and that passing most automated tests does not mean a system is secure, while failing them does not mean it is insecure, because the test sets may be tuned for a different kind of model.

How to read and measure the results

Because model answers are not fixed, a single test result says little. OWASP offers starting metrics to adapt to each use case. Examples are:

• Attack Success Rate (ASR): the share of adversarial inputs that actually cause the undesired behavior
• Detection Rate: the share of adversarial inputs that defensive mechanisms correctly detect and flag
• Severity of findings: OWASP suggests four levels, Critical, High, Medium, and Low, with each organization setting its own criteria
• Operating metrics: vulnerability discovery rate, time to detection, coverage of testing, false positive rate, and effectiveness of remediation

NIST AI 100-2 warns that defenses need to be evaluated against strong adaptive attacks (attacks where the attacker knows the defense and changes the method to match), and admits that designing mitigations is an ad hoc and fallible process. OWASP says no AI model is ever truly "done" or "secure". OWASP also says the red team finds the problems while closing the gaps usually falls to the blue team (the defenders) or the purple team (which coordinates attackers and defenders), so every finding should have an owner who takes it away to fix. Read the results as follows:

• Passing automated tests does not mean secure: it is evidence only for the items tested
• Retest after any change: OWASP gives the example that after RAG was added, a model became more factual but less emotionally intelligent, with no change to the model itself
• Try many times and adapt: one failed attack does not prove the defense holds

Keeping the test itself safe

Simulating attacks can cause real harm if there are no rules. OWASP lists topics the rules of engagement should cover:

• Environment and tools: where testing happens, which tools and techniques may be used, and the standards for documentation and communication
• Safety controls: how data is handled, model access controls, output monitoring, incident response, and the ability to roll back
• Ethical boundaries: sensitive topics, content restrictions, privacy, and legal requirements
• Emergency and escalation procedures: if something serious is found, whom to tell and what to do at once
• Permission: the necessary permissions from stakeholders must be in place before starting

General safety rules for simulation, such as isolating the lab from production and getting written authorization, are covered in Network and Cyber Security Simulators: Types and Uses. For the human side, such as over-reliance on AI and deception, see Security Awareness and Phishing Simulation That Works.

Common mistakes

• Testing only the model: ignoring guardrails, surrounding systems, and permissions, even though the Copilot case OWASP cites came from loosely set search permissions, not a weakness in the model
• Testing once and stopping: the model, the starting instructions, and the data sources can change, so old results do not last
• Trusting automated tool results without review: tools can judge wrongly in both directions
• Starting without scope or permission: with no rules, nobody knows when to stop
• A team of only security people or only developers: missing people who look at harmful content, bias, and the context of use
• No owner for the findings: a report nobody takes away to fix does not make the system safer

Start today

• Build an inventory of AI systems: include those you built and the AI features inside off-the-shelf software, and record what data and tools each one can reach
• Pick the one riskiest system: for example one connected to sensitive data or able to take actions
• Answer the four threat modeling questions: what are we working on, what can go wrong, what are we going to do about it, and did we do a good enough job
• Write a one-page scope and rules: state what is tested, what must not be touched, who gives permission, and whom to tell when something serious is found
• Try one small test case in a lab: hide a deceptive instruction in a document the system will read, and see whether the system follows it
• Set starting metrics and book a retest: begin with ASR and detection rate, and set a date to retest after the fixes

To see where the rest of your AI controls stand, try the self-assessment for a prioritized view of next steps.

For how red team, pen test and vulnerability assessment differ for conventional systems, see Pen Test vs Vulnerability Assessment vs Red Team.

References and further reading

• OWASP GenAI Red Teaming Guide: version 1.0, January 2025, with the four testing areas, steps, metrics, and a tool list
• OWASP: Announcing the Gen AI Red Teaming Guide: the launch announcement of January 22, 2025
• OWASP Top 10 for LLM Applications 2025: the ten risks used as a testing checklist
• MITRE ATLAS: a knowledge base of attacker techniques against AI systems, with case studies
• MITRE ATLAS data (GitHub): the machine-readable ATLAS data, used to check the version and the number of tactics
• NIST AI 100-2 E2025: Adversarial Machine Learning: the taxonomy of attacks and mitigations, final in March 2025, including a definition of red teaming
• NIST AI 600-1: Generative AI Profile: the generative AI risk profile of July 2024, with a section on AI red-teaming
• CISA: AI Red Teaming: Applying Software TEVV for AI Evaluations: the blog post of November 26, 2024