XBOW automated AI hacker matches 20-year veteran pentester in 28 minutes

An AI-powered automated security testing system has seemingly matched the performance of leading cybersecurity experts in a recent experiment after solving the same number of penetration-testing “benchmarks” in just 1.1% of the time it took its human counterparts. 

On Aug. 5, XBOW founder and CEO Oege de Moor published the results of an experiment in which XBOW AI’s penetration testing capabilities were measured against professional human penetration testers, or “pentesters” for short.

A penetration test is an authorized, simulated cyberattack on a computer system performed to evaluate the security of that system.

XBOW created 104 novel benchmarks — a term for realistic security scenarios — covering various vulnerabilities, designed to be unsolvable through web searches.

A total of five professional human pentesters from leading cybersecurity firms were given 40 hours to solve them.

The leading pentester, named "Principal 1" solved the same amount as the AI system, however, XBOW completed the tasks in 28 minutes, compared to the 40 hours it took the Principal 1, to complete.

“I just learned that XBOW got as many solves as I did. I am shocked. I expected it would not be able to solve some of the challenges I tackled at all, ” said principal pentester Federico Muttis.

A pentester is a cybersecurity professional who specializes in testing the security of computer systems, networks, and web applications.

Pentesters are typically white hat or “ethical” hackers who use the same tools and techniques that malicious hackers might use, but for defensive purposes.

The advantage the AI has over its human counterparts is that it can run continuously during software development, unlike infrequent human pentesting.

de Moor explained that that the approach “ensures that vulnerabilities are identified and addressed while the system is still under development, well before bad actors have a chance to exploit them.”

Several security experts say advancements in AI-powered security testing could be a major benefit the crypto industry which has already been plagued by over $1.4 billion in hacks so far this year.

CertiK Chief Security Officer Kang Li told Cointelegraph Magazine that crypto exchanges, wallets, and blockchain platforms could benefit from continuous AI-driven security testing specifically when it comes to the auditing of smart contracts and other crypto security systems.