Many organizations may find they’re better off hiring pen testers and in-house security researchers directly than running bug bounty programs, according to new MIT research.
The New Solutions for Cybersecurity paper features a surprising analysis of bug bounty programs in the chapter, Fixing a Hole: The Labor Market for Bugs.
It studied 61 HackerOne bounty programs over 23 months — including those run for Twitter, Coinbase, Square and other big names — and one Facebook program over 45 months.
It claimed that, contrary to industry hype, organizations running these programs don’t benefit from a large pool of white hats probing their products. Instead, an elite few produce the biggest volume and highest quality of bug reports across multiple products, earning the biggest slice of available rewards.
It’s also claimed that even these elite “top 1%” ethical hackers can’t make a decent wage by Western standards.
The top seven participants in the Facebook program studied made just $34,255 per year from an average of 0.87 bugs per month, while from the HackerOne dataset it was estimated that they made just $16,544 from 1.17 bugs per month.
There are, of course, exceptions: last week we reported that one company has upped its maximum payout for iOS zero-day exploits to $2m. However, it appears that these programs offer more of a salary top-up to Western researchers than a main source of income.
Security research firm Trail of Bits claimed the findings proved that firms should reconsider their security strategies by hiring “boffins” directly as consultants instead of running bug bounty programs.
“The authors of Fixing a Hole argue that bug bounties should be designed to incentivize the elite. They say that making bounties invite-only lowers the operational cost of managing a tsunami of trivial, non-issue, and duplicate bugs. Only 4-5% of bugs from Google, Facebook, and GitHub’s public-facing bounty programs were eligible for payment,” it argued in a blog post.
“According to the authors, a small number of bounty hunters are indispensable and hold significant power to shape the market for bug bounty programs. Based on this, hiring security consultants under terms and conditions that can be controlled seems more practical.”
That view is unsurprisingly not shared by HackerOne CEO, Marten Mickos, who said the MIT study is not representative.
“If it is based on HackerOne data, it is only based only on a fragment of it. The hacker community is indeed power-law distributed,” he added in comments sent to Infosecurity.
“The top performers are orders of magnitude more productive than newcomers. The beauty is that many newcomers rise very quickly in the ranks. Within this merit-based system, there is unlimited opportunity for one with skill and will."
Report co-author and CEO of Luta Security, Katie Moussouris, doubled down on the findings, claiming that independent researchers are “better off pen testing or living the good life of in-house research staff.”
“Orgs can't #bugbounty their way to secure, same as they can't pen test their way to secure,” she tweeted. “The myth of ‘many eyes’ is convenient, but untrue as proven in both open source & bounties. Skilled bug bounty hunters rarely make a good living by Western standards.”