Security experts are urging users to change all of their online passwords after a problem at content delivery network Cloudflare exposed customer data from countless clients including Uber, Fitbit and OK Cupid.
The source of the problem – which was discovered accidentally by Google Project Zero bod, Tavis Ormandy – was a memory leak caused by a broken HTML parser chain.
However, it was compounded by the fact that leaked data was then cached by search engines.
The leaked data included “private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data,” Cloudflare CTO, John Graham-Cumming explained in a lengthy blog post.
“We quickly identified the problem and turned off three minor Cloudflare features (email obfuscation, Server-side Excludes and Automatic HTTPS Rewrites) that were all using the same HTML parser chain that was causing the leakage. At that point it was no longer possible for memory to be returned in an HTTP response,” he added.
Although Graham-Cumming claimed the bug was fixed globally in under seven hours, it may have been leaking highly sensitive data for months.
“The greatest period of impact was from February 13 and February 18 with around one in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that’s about 0.00003% of requests),” he added.
In fact, given the extent of the info cached by search engines, Cloudflare clients will now be under pressure to inform their own customers of the extent of the privacy snafu.
“The examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I've informed Cloudflare what I'm working on,” said Ormandy.
“I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.”
Although he praised Cloudflare for its response to the issue, it’s also true the firm’s bug bounty offers little in the way of rewards for white hat researchers – free t-shirts, rather than money.
Former Google click fraud boss and current Shape Security CTO, Shuman Ghosemajumder, argued that it is “one of the widest exposures of confidential and sensitive consumer data ever observed.”
“This incident has many people suggesting that everyone in the world should change all of their passwords immediately,” he said.
“The total exposure is likely not that large – i.e., not all of your passwords have been compromised – but the problem is that almost any one of your passwords on over four million websites could have been compromised, so the safest course of action is to act as though all of your passwords were compromised.”
Kaushik Narayan, CTO at Skyhigh Networks, analyzed over 30 million enterprise users worldwide and found 99.7% of companies have at least one employee that used a Cloudbleed vulnerable cloud application.
“This means hackers could have stolen user passwords for these cloud applications – and may even have access to session keys exposed, while a session is live. But this user-data also revealed another surprise – out of 128 enterprise-ready applications that could have been compromised, only four were vulnerable,” he added.
“Cloudbleed is the latest in a string of vulnerabilities that should be of concern to enterprise IT security and a reminder us of the problems caused by user password reuse across corporate services and personal web sites and cloud services.”