Over 4% of Googlebots are not Mountain View crawlers but malicious imposters masquerading as Google software to commit DDoS attacks, scraping, spamming and other malicious activity, according to Incapsula.
The security firm analysed 400 million search engine visits to 10,000 sites, resulting in over 2.19 billion page crawls over a 30 day period, to compile its findings. This amounted to 50 million “Googlebot imposter” visits.
“The actual ‘type’ of these impostors may vary, but all of them should be deemed suspicious by default, due to their attempt to assume a false identity,” product evangelist Igal Zeifman said in a blog post.
Of the 50 million fake Googlebot sessions spotted by Incapsula, the majority (65.7%) were actually classified as “suspicious” rather than “downright malicious”.
A third (34.3%) were branded explicitly suspicious with the most popular use for the bots in Layer 7 DDoS
attacks (23.5%). In fact, fake Googlebots are the third most commonly used bots in DDoS attacks, Zeifman said.
“Website operators who use such [rate-limiting] solutions [rather than case-by-case traffic inspection] are unable to identify real Googlebots from fakes. As a result, when the attack bells go off, they are presented with a harsh ‘all or nothing’ dilemma: to block all Googlebot agents and risk loss of traffic, or to allow all Googlebots in and suffer downtime,” he added
“With DDoS events that can last for days, weeks or even months, each of these alternatives is extremely damaging for the target, making the attack a success – either way.”
IP, ASN verification and other heuristics tools and techniques can help organizations spot and block these Google imposters, although they require “excessive” processing power, warned Zeifman.
Scraping (5.3%), spamming (3.8%) and hacking (1.7%) were the next most popular uses for malicious imposter Googlebots.
The botnets of compromised computers that power these fake Google crawlers are mostly located in the United States (25%), followed by China (16%), Turkey (15%), Brazil (13%) and India (8%), according to Incapsula.
The appearance of Brazil in that list is still open to debate.
“These numbers may have something to do with the myriad internet devices brought into the country by 1 million tourists, some of whom probably should pay more attention to what they download,” said Zeifman.