In the past few years, automation in many spheres of cybersecurity has increased dramatically, but pen testing has remained stubbornly immune to this. While crowdsourced security has evolved as an alternative to pen testing in the past ten years, it’s not based on automation but simply throwing more humans at a problem (and in the process, creating its own set of weaknesses). Recently though, automated pen testing tools have now surfaced to a point where they are usable to automate pen testing under certain conditions. This begs the question, are human pen testers heading for redundancy? Can we replace them with these tools?
To answer this question, we need to understand how they work, and crucially, what they don’t do. While I’ve spent a great deal of the past year testing these tools and comparing them in like-for-like tests against a human pen tester, the big caveat here is that these automation tools are improving at a phenomenal rate, so depending on when you read this, it may already be out of date.
First of all, the ‘delivery’ of the pen test is done by either an agent or a VM, which effectively simulates the pen tester’s laptop and/or attack proxy plugging into your network. So far, so normal. The pen testing bot will then perform reconnaissance on its environment by performing identical scans to what a human would do – so where you often have human pen testers perform a vulnerability scan with their tool of choice or just a ports and services sweep with nmap or masscan. Once they’ve established where they sit within the environment they will filter through what they’ve found and this is where their similarities to vulnerability scanners end.
Vulnerability scanners will simply list a series of vulnerabilities and potential vulnerabilities that have been found with no context as to their exploitability and will simply regurgitate CVE references and CVSS scores. They will sometimes paste ‘proof’ that the system is vulnerable but don’t cater well to false positives. The automated pen testing tools will then choose out of this list of targets the ‘best’ system to take over, making decisions based on ease of exploit, noise and other such factors.
So for example, if it was presented with an windows machine which was vulnerable to eternalblue it may favor this over brute forcing an open SSH port that authenticates with a password as it’s a known quantity and much faster/easier exploit.
Once it gains a foothold, it will propagate itself through the network, mimicking the way a pen tester or attacker would do it, but the only difference being it actually installs a version of its own agent on the exploited machine and continues its pivot from there (there are variations in how different vendors do this). It then starts the process again from scratch, but this time will also make sure it forensically investigates the machine it has landed on to give it more ammunition to continue it’s journey through your network.
This is where it will dump password hashes if possible or look for hardcoded credentials or SSH keys. It will then add this to its repertoire for the next round of its expansion. So while previously it may have just repeated the scan/exploit/pivot this time it will try a pass the hash attack, or try connecting to an SSH port using the key it just pilfered. Then, it pivots again from here and so on and so forth.
If you notice a lot of similarities between how a human pen tester behaves then you’re absolutely right – a lot of this is exactly how pen testers (and to a less extent) attackers behave. The toolsets are similar and the techniques and vectors used to pivot are identical in many ways. So what’s different? Well first of all, the act of automation gives a few advantages over the ageing pen testing methodology (and equally chaotic crowdsourced methodology).
The speed of the test and reporting is many magnitudes faster, and the reports are actually surprisingly readable (after verifying with some QSA’s, they will also pass the various PCI-DSS pen testing requirements). No more waiting days or weeks for a report that has been drafted by humans hands and gone through a few rounds of QA before being delivered into your hands.
This is one of the primary weaknesses of human pen tests, since the adoption of continuous delivery has caused many pen test reports to become out of date as soon as they are delivered since the environment on which the test was completed has been updated multiple times since, and therefore, had potential vulnerabilities and misconfigurations introduced that weren’t present at the time of the pen test. This is why traditional pen testing is more akin to a snapshot of your security posture at a particular point in time.
Automated pen testing tools get around this limitation by being able to run tests daily, or twice daily, or on every change, and have a report delivered almost instantly. This means you can potentially pen test your environment daily and detect changes in configuration on an exploitability level on a daily basis too rather than relying on a report delivered weeks later.
The second advantage is the entry point. While with a human pen test you may typically give them a specific entry point into your network, with an automated pen test you can run the same pen test multiple times from different entry points to uncover vulnerable vectors within your network and monitor various impact scenarios depending on the entry point.
While this is theoretically possible with a human it would require a huge budgetary investment due to having to pay each time for a different test.
So what are the downsides to all this? Well first off, automated pen testing tools don’t understand web applications – at all. While they will detect something like a web server at the ports/services level they won’t understand that you have an IDOR vulnerability in your internal API or a SSRF in an internal web page that you can use to pivot further. This is because the web stack today is complex, and to be fair even specialist scanners (like Web Application Scanners), have a hard time detecting vulnerabilities that aren’t low hanging fruit (such as XSS or SQLi).
This leads to a secondary weakness in automated pen testing tools in that you can only use them ‘inside’ the network. As most exposed company infrastructure will be web based, and automated pen testing tools don’t understand these, you’ll still need to stick to a good ol’ fashioned human pen tester for testing from the outside.
To conclude, the technology shows a lot of promise, but it’s early days and while they aren’t ready to make human pen testers redundant just yet, they do have a role in meeting today’s offensive security challenges that can’t be met without automation.