AI Penetration Testing: What It Can and Cannot Do

TL;DR: AI penetration testing is good at breadth: enumerating thousands of hosts, trying known weaknesses in parallel and running the same checks every week. It still misses business logic, the long chains that turn a small foothold into real access, and the context of what matters to your business. Buy it with a human in charge of the test.

AI penetration testing is showing up in sales pitches, insurance conversations and security questionnaires, usually with a promise that the robot found what the humans missed. Some of that is true. Some of it is a vulnerability scan with a new label. Here is how to tell real AI penetration testing from a relabeled scan, from someone who runs both kinds of testing.

What does "AI penetration testing" actually mean?

The phrase covers two different jobs.

Testing with AI. A penetration test is a controlled attack on your own systems, done with permission. The UK's National Cyber Security Centre describes it as "a method for gaining assurance in the security of an IT system" by "using the same tools and techniques as an adversary might." (NCSC, penetration testing guidance) AI penetration testing hands some of that work to software agents: they map what you expose, pick what to try next, run the tools and write up what they found.

Testing your AI. If your business added a chatbot to its website or an AI assistant connected to your inbox, that system needs testing too. CISA calls this AI red teaming and defines it as "the third-party safety and security evaluation of AI systems," part of the broader testing and evaluation of software. (CISA, November 2024)

Most of this post is about the first meaning, because that is what is being sold to small businesses right now. The second gets its own section below.

What does AI penetration testing find well?

Start with the good news. The clearest public test so far is a Stanford and Carnegie Mellon study published at ICLR 2026. Researchers put ten professional penetration testers and several AI agents on a live university network of about 8,000 hosts across 12 subnets. Their own agent, ARTEMIS, "placed second overall, discovering 9 valid vulnerabilities with an 82% valid submission rate and outperforming 9 of 10 human participants." (Lin et al., Comparing AI Agents to Cybersecurity Professionals, 2026)

The authors named where agents had the edge: "systematic enumeration, parallel exploitation, and cost." Some agent setups cost about $18 an hour to run, against $60 an hour for a professional, a figure based on salary rather than what a firm bills. Federal data is in the same range: the median information security analyst earned $129,180 in May 2025. (BLS Occupational Outlook Handbook)

DARPA ran a two-year competition on a related job: finding and fixing flaws in open source code. In the August 2025 final, the teams' automated systems "discovered 54 unique synthetic vulnerabilities in the Final Competition's 63 challenges" and patched 43 of them. They also found 18 real vulnerabilities nobody had planted, and patches were submitted in an average of 45 minutes. (DARPA, August 2025)

Put together, here is where I would trust AI penetration testing to carry real weight:

• Breadth. Every public address, every subdomain, every open port, checked the same way each time. NIST's testing guide says automated checks "can be done very quickly and provide consistent, repeatable results," while a person checking thousands of settings by hand is "tedious and error-prone." (NIST SP 800-115)
• Parallel follow-up. The Stanford agent probed several targets at once. The researchers watched one human note a vulnerable server and never return to it. Agents do not tire of the boring targets.
• Old and odd systems. In the same study, an aging server with an outdated encryption setting would not load in modern browsers. No human found its vulnerability; the agent did, because it worked from the command line.
• Repetition. Rerunning the same checks after every change, which slips when it depends on someone remembering.

That is real value. It is also the part of a test that was already the most mechanical.

What does AI penetration testing miss?

The same research shows the gaps. Three matter most for a small business.

Business logic

Business logic flaws are weaknesses in how your application is supposed to work, not in the software underneath it. OWASP's testing guide uses a simple example: a login that expects steps 1, 2 and 3 in order. What happens if a user goes from step 1 straight to step 3? OWASP is blunt about who finds these: "This type of vulnerability cannot be detected by a vulnerability scanner and relies upon the skills and creativity of the penetration tester." It adds that "Automation of business logic abuse cases is not possible." (OWASP Web Security Testing Guide 4.2)

That guide predates today's agents, which reason better than the scanners it had in mind. The core problem for AI penetration testing has not changed. To spot a logic flaw, the tester has to know what the business intended. A property manager in Worcester with a tenant portal knows that one tenant must never see another tenant's lease. An AI penetration testing tool sees two documents that both load.

Same for a refund requested twice from two browser tabs, or a staff role that cannot open payroll but can export the report that contains it.

Chained access

In my experience, the serious findings are rarely one dramatic flaw. They are small steps strung together: a leaked username, a reused password, a forgotten admin page, one more permission than anyone meant to grant.

AI penetration testing agents can chain some of these steps. They still lose the thread on long chains. The PentestGPT study, published at USENIX Security 2024, found models were good at "specific sub-tasks" such as "using testing tools, interpreting outputs, and proposing subsequent actions," but "encounter difficulties maintaining a whole context of the overall testing scenario." (Deng et al., USENIX Security 2024)

The Stanford study saw the same habit from the other side. Top human testers were "more likely to pivot or deepen their foothold after finding a vulnerability," while the agent "tends to submit findings immediately." On one device, it reported two minor configuration issues and moved on, missing a remote code execution flaw that 80% of the human participants found. (Lin et al., 2026)

A minor finding and a critical one can sit on the same box. The difference is whether someone keeps pulling.

Context and judgment

Findings from AI penetration testing need checking. The Stanford agent "is also more prone to false positives than humans." In one case it reported a successful login with default credentials because the server answered "200 OK," when those responses were really redirects back to the login page. (Lin et al., 2026) NIST made the same point about scanners years ago: they "can have a high false positive error rate," and "an individual with expertise in networking and OS security should interpret the results." (NIST SP 800-115)

Then there is the judgment no tool owns. Which finding would stop your billing on a Friday? Is this test safe to run against the system your front desk uses all day? Those calls come from knowing the business, and they belong to a named person.

Is pentesting being replaced by AI?

Not on the evidence so far. The best agent in the Stanford study beat most professionals on a large network and still missed a flaw most humans caught. DARPA presented its competition results as a step forward for cyber defense.

The labor forecasters are not betting on replacement either. BLS projects employment of information security analysts to "grow 21 percent from 2025 to 2035." (BLS)

My read: AI penetration testing is making the mechanical half of testing cheaper and faster. The thinking half still finds the things that hurt.

How should you read an AI penetration testing offer?

Ask any AI penetration testing provider these questions, ours included, and listen for plain answers.

1. Who is the human on this test, by name? Someone should own the scope, review every finding before you see it, and be reachable while testing runs.
2. What is automated, and what is done by hand? A good answer names specific work, like business logic and access between user roles, that a person does directly.
3. What did you agree not to touch? Get scope and rules of engagement in writing, signed by the owner of the systems, before anything starts.
4. How do you stop it? The Stanford researchers had a team member watch "each session in real time with the ability to immediately terminate any out-of-scope or unsafe activity," with three separate ways to cut it off. (Lin et al., 2026) Ask for the same kind of answer.
5. Where does my data go? If an AI model reads your responses, logs or code, ask which provider runs it, whether your data is kept or used for training, and who can see the transcripts.
6. How do you confirm a finding is real? Every finding in the report should come with steps a person reproduced, not just a tool's say-so.
7. Is a retest included? A finding is closed when the same attack stops working, not when someone says it was fixed.

Two answers would make me walk away: "fully autonomous, no human needed" and "we will find everything." Neither is true of any test. NCSC puts it well: a test "can only validate that your organisation's IT systems are not vulnerable to known issues." (NCSC)

Amelia, my wife and Kief Studio's co-founder and CEO, wrote a good guide on how to evaluate AI tools without getting sold (opens in a new tab). The same questions work on security vendors.

If your business uses AI, how do you test the AI itself?

This is the second meaning of AI penetration testing, and more businesses here need it as they add AI tools. In the Census Bureau's latest survey, published October 22, 2026, roughly one in four Massachusetts businesses said they used AI in a business function in the past two weeks. (U.S. Census Bureau, Business Trends and Outlook Survey)

OWASP ranks prompt injection first among the risks for these systems and describes it as user prompts that "alter the LLM's behavior or output in unintended ways." OWASP also says "it is unclear if there are fool-proof methods of prevention," and its recommended mitigations include regular penetration testing and breach simulations. (OWASP LLM01:2025) The NCSC went further in December 2025, warning these attacks "may never be totally mitigated" the way older injection attacks were, and that the work should turn to "reducing the risk and impact." (NCSC, December 2025)

That changes what a test of an AI feature looks for. The question is less "can someone trick the chatbot?" and more "what can the chatbot reach when it is tricked?" A useful test checks:

• which systems and records the AI can read, and whether it needs all of them,
• which actions it can take on its own, such as sending email, issuing refunds or changing bookings,
• whether text from a customer, a document or a web page can change those actions, and
• whether a person approves anything that moves money or reaches a customer.

We wrote a plain explainer on prompt injection attacks if you want the background first.

How do humans and automation fit together in AI penetration testing?

The split that works: automation running all the time, and people going deep on a schedule and after big changes.

NCSC compares a penetration test to a financial audit: a way to gain assurance that your vulnerability management works, "not as a primary method for identifying vulnerabilities." (NCSC) Let automation find and close the routine issues every week, so paid human hours go to what only people find.

In practice, for a small business, that looks like this:

1. Every week, automated. Scan what you expose, track what changed, and flag known weaknesses. A tool can do this without getting bored. I wrote about replacing our own scanning stack with one tool (opens in a new tab).
2. Every week, a person reads it. Someone throws out the false alarms, ranks what is left by your real exposure, and makes sure fixes happen. That is vulnerability management, and in my experience it is where most risk gets removed.
3. Once a year and after big changes, a human-led test. AI penetration testing widens the coverage. People handle business logic, follow footholds as far as they go, and decide what matters.
4. After every fix, a retest. Automated where it can be, confirmed by a person where it counts.

If you want to see the hands-on side, I keep a pentester's cheat sheet (opens in a new tab) that walks from reconnaissance through exploitation to the write-up. Most of those commands are exactly what agents now run well. The scoping, the judgment and the decision about what matters are not on it, and that is the point.

What does this look like for a Central Massachusetts business?

A small business has an advantage here. Fewer systems means AI penetration testing covers most of the ground, and the human hours can go to the few workflows that touch money and customer data. For example:

• A dental office in Shrewsbury might automate external scanning and spend the human time on its patient portal: can one patient reach another's records or appointments?
• An accounting firm in Westborough using an AI assistant on client documents might test what that assistant can read and send before testing anything else.