Following the weaknesses identified in pen testing in the first part of this article published earlier this week, we now need to consider if crowdsourced security actually tackles them effectively.
Speed of development & time limited tests - Many crowdsourced programs are effectively ‘open-ended’ with no time limit, effectively translating to a ‘constant pen test’ if the incentives for researchers are right. This maps better to current threats since attackers are not constrained by time limitations either.
An individual vs the crowd - This is by far the greatest advantage. Crowdsourced engagements have been successful in dredging up critical vulnerabilities in the most used sites today that previously had relied only on pen testing. The reasons are clear – the more people you have looking at something, the more vulnerabilities you are likely to find.
Because of the wide mix of technologies in use today, the crowd acts as a big equaliser in this field, ensuring that you will eventually get someone looking at your site that has experience in a specific flaw that would have been missed by someone else.
Pen tester syndrome - Crowdsourced security programs rely on proof of concept to show impact, which effectively eliminates ‘pen tester syndrome’. If you feel a certain missing HTTP header is a vulnerability, then you better have a good proof of concept as if not your report will be dismissed as ‘spam’ and if you log enough of them, some crowdsourced programs won’t even let you submit vulnerabilities anymore. This provides a clear advantage to companies too who can now focus on fixing real threats and not chasing phantom risk.
Business model - Leveraging a crowd helps the scarcity that is present when looking at the offensive security skillset as you are scaling across borders to find individuals who would never work for the pen testing companies you traditionally rely upon. Crowdsourced outfits also don’t have to pay salaries, or pensions, or invest in training or send anyone to conferences. There is an ethical quandary to this but more on that later.
The Downsides of crowdsourced security
For now, external assets are easier to test with a crowd - When you want to do an ‘internal’ pen test, nothing beats calling up a local company and have a pen tester physically turn up at your premises. To have someone from the outside test inside your network you’d have to create a VPN access or some kind of authenticated proxy access, and then if you didn’t want them annoying end-users you’d probably have to re-create a test environment for them too. It’s worth adding the skillset for testing inside a network is not as common within ‘the crowd’.
Controlling the crowd and the performance hit - Having personally experienced this many times, when a new target is opened up on a crowdsourced engagement, it sometimes crumbles under the performance hit of several dozen enthusiastic researchers. While there are ways to limit the number of people on an engagement, this can catch the unwary off-guard.
If incentives are poor, the crowd will stay away - Crowdsourced researchers are paid per vulnerability. They aren’t paid if they don’t find anything, so understandably, if your site is offering small rewards, or worse, no rewards at all, don’t expect a flock of highly experienced bug hunters to descend upon your site.
There is also an ethical element to this. If a researcher spends 20-30 hours on your site and finds nothing, should he be rewarded? Arguments you may hear when discussing the so-called ‘gig’ economy may surface here.
This being said, crowdsourced companies have started rolling out programs recently that actually pay researchers for time spent, even if no vulnerabilities are discovered.
Budget - Although crowd-sourced outfits will argue against it, for now pen testing is just more cost-effective, since you’re paying a flat day rate, and only five days at that on average. If a pen tester turns up 100 vulnerabilities in that time frame, then you still pay the same amount. The same can’t be said for a crowdsourced program.
Finally, some myths
- Crowdsourced reports won’t count as pen testing for PCI-DSS. This is simply not true, a report can be produced which now aggregates the vulnerabilities found into a simple report that reads very much like a pen test report.
- Researchers earn megabucks with very little effort. Not true. While some bugs will earn big payouts for relatively little investment, many researchers will often regale you with tales of turning up empty handed after days of searching or worse: finding a bug that was already found by someone else before you (a duplicate).
- There are hundreds and thousands of researchers in ‘the crowd’. Again, an exaggeration but is pushed heavily in some marketing. The most active researchers on all platforms amount to a few dozen. When you hear of a certain site having 100,000 researchers, this is simply misdirection since many of those are just people who have registered for an account and done absolutely nothing. To give an example, I remember signing up to a crowdsourced platform in 2014 and leaving my account inactive for a few years. I hadn’t logged a single vulnerability but when I logged back in I was ranked 5341. For a platform that advertised over 100,000 researchers this means I was in the top five percent of researchers without having logged a single vulnerability!
So can I replace my entire pen testing program with a crowdsourced program? I’d say if your budget can handle it, and you’ve already gone through a few cycles of pen testing, you can actually dispense with pen testing entirely and switch to a crowdsourced approach, especially if you just want to test your external assets.