For several years now, crowdsourced security offerings have been growing in popularity to the point where some companies dispense with pen tests entirely. Are we ready to throw away pen testing as a methodology?
What is crowdsourced security?
Fundamentally it’s about bringing scale to the human aspect of pen testing. While a single pen tester will have one skillset, one methodology and one way of looking at things, a crowd simply scales on those strengths to cover the weaknesses that have crept into pen testing over the years. Now instead of one pen tester, you have dozens, and in some rare cases, hundreds. Instead of your pen test lasting five days, now it lasts weeks, or forever.
Penetration testing – the challenges
Various weaknesses have crept into pen testing over the years, with many factors outside the control of the security industry, while others are of our own making:
Development cycles and continuous delivery - While pen testing is traditionally an annual activity for many companies, this no longer keeps up with the speed of deployment. Weekly, daily and sometimes continuous delivery models introduce changes to software. While a pen test can only give a snapshot of your security posture at a particular point in time any update to software potentially introduces new functionality and therefore new vulnerabilities, effectively invalidating your pen test findings that are barely a week old.
The obvious counter to this is to have more frequent pen tests but then you’ll eventually come up against budgetary limitations, not to mention if your pen test turns up no vulnerabilities, you still pay out.
Pen testers work against a time-limit - Pen testers don’t enjoy the luxury of time. On an engagement that has been billed for five days, one of those days is already lost to ‘report writing’, while the first day of the engagement is typically spent using automated tools to assist in reconnaissance of the target and map out the attack surface. The remaining three days are spent manually trying to exploit vulnerabilities.
If a particularly interesting error message is being examined that may eventually lead to a vulnerability, it will be left behind after a while simply because there is a finite amount of time limit that can be spent on any discovery.
Skillset / Testing approach - We all know the technology stack has become more complex and varied and changes quickly. While one pen tester may feel comfortable testing PHP with a SQL backend and a front end written in Angular.js, they may not feel as comfortable when encountering Ruby, or understanding the foibles of EmberJS.
Add to this stack a content delivery network and cloud-hosted configurations, and you can understand why an individual may not see all the flaws in one go. Companies are well aware of this issue which has led to the ‘cycling’ of pen testing companies, which means the next time you have a pen test, it will be a different set of eyes who may unearth different things.
Interestingly this really is tackling a problem, but on a much smaller scale than crowdsourced pen tests. The problem with cycling pen testing companies is that there is a depressingly small amount of pen testers in any given geographical location. I’ve even had the experience of cycling pen testing companies to fall upon the very same pen tester who tested the same site the previous year.
Pen tester syndrome - This is the act of making things worse than they appear, and is also why no one in the history of security has ever seen an empty pen test report. It is a neurotic obsession of the industry to have ‘something’ written on a report, even when nothing is wrong.
This is why you will see pen testing reports filled with all sorts of junk hardening measures that have no discernible value and aren’t immediately exploitable. A missing X-XSS protection header anyone?
This issue is really one of our own making as an industry and as security professionals. Pen testing companies have become complacent and put ‘vulnerabilities’ on a report which are anything but, only for the fear of a competitive pen testing company writing a ‘better’ report with ‘more’ vulnerabilities which really is a race to the bottom when comparing value between two pen testing companies.
Likewise recipients of pen testing reports don’t challenge or ask for proof of concept and have taken reports at face value – a lack of offensive security knowledge and misinterpreting likelihood of certain attacks is at fault here.
Business model - Pen testers are expensive to maintain and hard to recruit. They get poached easily and there aren’t that many around. Also four to six week lead times are pretty common if you want to plan a pen test, and this can stretch further if your pen testing requirements becomes more ‘niche’ – for example you have only mobile app that you want tested or want some reverse engineering on ‘a thing’.
So these are the issues and challenges facing pen testing today. Look out for the second part of this article later this week where we will look at how crowdsourced security tackles these issues, and if the approach offers any advantages to pen testing today.