Vulnerability assessment, penetration testing (pen test) and red team are three kinds of security testing that people often confuse. A vulnerability assessment looks for known weaknesses, mostly with automated tools. A pen test is when authorized experts try to break into systems to show what can really be reached. A red team is an authorized team that acts like a real attacker to see whether the defenders detect and handle the attack.

Why the three terms get confused

Each provider uses different names and works in a different way. CREST (a body that sets standards for security testing providers) says in its buyer's guide that methodology varies between providers, so the service name in a quote does not tell you everything. Always read the scope, the test method and the deliverables.

NIST SP 800-115, NIST's technical guide to security testing and assessment, also warns that no single technique gives a complete picture of a system's security. Organizations should combine several techniques.

Vulnerability assessment: finding known weaknesses

According to CREST, a vulnerability assessment uses automated techniques to map, scan and identify weaknesses in an environment. The result shows how exposed the environment is to known, published vulnerabilities. It is often repeated regularly, for example daily, weekly or monthly, and has a broad scope covering networks, servers, applications and wireless networks.

NIST SP 800-115 explains that a vulnerability scanner works by identifying which operating system and software a host runs and matching that against the scanner's database of known vulnerabilities. It can therefore find outdated software versions, missing patches and misconfigurations. The limit is that a vulnerability assessment shows that a weakness exists, but does not confirm that an attacker could actually break in through it.

Pen test: trying to break in to prove it can be done

CREST describes a pen test as a mix of automated and manual techniques to find and exploit known and unknown vulnerabilities, including policy and control gaps, which makes it more rigorous and intrusive than a vulnerability assessment. It is done less often, usually when something triggers it, such as before a new system is released, to meet a regulation, or once a year. The scope is usually narrower and more focused, such as web applications, external or internal infrastructure, cloud, mobile and IoT devices, and sometimes physical testing or social engineering (tricking staff).

NIST SP 800-115 gives an example of splitting a pen test into four phases, and states that other ways of grouping the work are also acceptable. NIST's example is:

  • Planning: set the rules, finalize and document management approval, and set the test goals; no actual testing happens in this phase
  • Discovery: gather information and scan to find targets, then analyze vulnerabilities using vulnerability databases and the testers' own knowledge
  • Attack: try to exploit the vulnerabilities found to verify that they work; if an attack succeeds, identify the safeguards that would reduce that exposure
  • Reporting: summarize what was found and the recommendations
INFOGRAPHIC
The four pen test phases (NIST example)
1
Planning
rules, approval, goals
2
Discovery
gather, scan, analyze vulnerabilities
3
Attack
try exploits to verify they work
4
Reporting
findings and recommendations

NIST notes that manual vulnerability analysis can find new or obscure vulnerabilities that a scanner may miss, but is much slower than a scanner. NIST also provides a template for the rules of engagement (ROE), agreed before testing starts, which gives the test team the authority to carry out defined activities. The NCSC (the UK's national cyber security center) warns that testing can occasionally have unexpected effects on systems, so agree in advance how scope changes will be handled.

Red team: testing whether defenders see and stop an attacker

NIST cites a definition from CNSSI 4009-2022: a red team is a group formally authorized to imitate how a potential adversary would attack. Its purpose is to improve security by showing how serious successful attacks could be and which defensive measures hold up in real operations. This also helps the defenders (the blue team). CREST gives a close meaning: simulating a real threat actor to achieve a specific objective while testing detection and response.

Our own view is that the key difference is what is being tested. A pen test asks whether this system can be broken into. A red team asks whether our monitoring and response team can see and stop an attacker. MITRE ATT&CK (a knowledge base of the ways attackers have really operated) can also help emulate specific threats; see MITRE ATT&CK for Executives: How to Read and Use It.

Comparing the three by the question each answers

In our own view, each one answers a different question:

  • Vulnerability assessment: which known vulnerabilities do we have, and where are they
  • Pen test: can those vulnerabilities really be used to break in, and how far can an attacker get
  • Red team: can our monitoring and response team see and stop an attacker who has a defined goal
INFOGRAPHIC
The question each one answers (our view)
01
Vulnerability assessment
what flaws exist
02
Pen test
can it be broken into
03
Red team
do defenders see and stop it

From the CREST guide: a vulnerability assessment is frequent, broad and largely automated, while a pen test is less frequent, narrower and a hybrid of automated and manual work. The CREST guide also separates red teaming from penetration testing, but we did not read its detail on red team frequency or scope, so we do not summarize it here.

Which to choose first

The content of this section is our own view, not a requirement of the sources. The order below suits many organizations:

  • No asset inventory yet, or no regular scanning: start with a vulnerability assessment and build a fix cycle around it; see A Vulnerability Management Program That Works
  • A new system about to go live, or a regulation that requires periodic checks: run a pen test on that system, since CREST says pen tests are usually triggered by releases, regulation or an annual cycle
  • Web applications: use the OWASP Web Security Testing Guide (WSTG, OWASP's guide to testing web security) as a reference for what testers should check
  • A monitoring team that works in practice and has already passed a pen test: consider a red team to test detection and response; see AI Red Teaming: How to Test AI System Security if the system under test is AI

The NCSC advises that a pen test should not be your main way of finding vulnerabilities; it is a periodic external check of how well you manage them. Tests may be a year or more apart, so internal processes must catch problems in between.

How to buy without getting burned

The NCSC says test quality depends heavily on the testers' skills, so choose testers who are qualified and experienced, and flag unusual systems such as mainframes or bespoke hardware during the bidding. It advises scoping together with the risk owners, the technical staff who know the system and a representative of the testing team, and sharing your latest vulnerability assessment so the testers can check its accuracy.

Our own view is that before signing, you should ask the following:

  • Scope: which systems are in scope, which are out, and at which times testing is forbidden
  • Method: the share of automated tooling versus manual testing by the testers, and which methodology or standard is used
  • Rules and contact: a signed ROE document, and a technical contact who can be reached at short notice during testing, which the NCSC advises arranging
  • Report and retest: the report format you will get, and whether a retest after fixes is included

When the report arrives, the NCSC advises feeding the results into your own vulnerability management process and making your own risk decisions, because testers may rate severity differently from you. Give special attention to vulnerabilities you did not already know about, and do not accept the proposed fix automatically, since removing unneeded software, adding compensating controls or increasing monitoring may suit better.

Common mistakes

  • Paying pen test prices for scan results: if the quote does not say whether testers try the vulnerabilities by hand, the work may be only a vulnerability assessment
  • Treating one pen test a year as the whole security program: the NCSC says a pen test is a periodic check, not the main way of finding vulnerabilities
  • Fixing only the reported items and stopping there: our own view is to retest and to work out how to detect the same kind of problem in future, which matches what the NCSC advises considering
  • Starting with a red team before the basics are ready: if there is no working monitoring or many known vulnerabilities are still open, our own view is that the result will tell you what you already know
  • Starting a test without approved rules: NIST says management approval must be finalized and documented in the planning phase, and the ROE gives the test team authority for the defined activities

Start today

  • Write down the question you need answered: decide whether you want to know the vulnerabilities you have, the chance of being broken into, or your readiness to detect
  • Check whether regular scanning exists: if not, start with a vulnerability assessment
  • Pick the first system to pen test: a new system, an internet-facing one, or one that a regulation names
  • Draft the scope and rules: use the NCSC topics and NIST's ROE template as a starting point, then have the system owner approve them

To see where the organization stands overall, try the self-assessment. For choosing attack-simulation tools, see the Network and Cyber Security Simulator guide.

Scope and accuracy

The information here comes from NIST SP 800-115 (September 2008), which is final, and the NIST page we read on 4 November 2026 shows no newer revision, plus the NCSC guidance page, the CREST buyer's guide page, the NIST red team glossary page and the OWASP WSTG project page, all read the same day; OWASP states that the latest stable version is v4.2 and that v5.0 is in development. We read NIST SP 800-115 from the PDF by extracting the text of the pen test sections. The NIST content is old, so use it together with newer guidance. The choice of order, the pre-purchase questions and the explanation of what each test examines are our own view.

References and further reading