Google, Facebook, Yahoo and other web players have joined forces to create a new blacklist designed to clamp down on click fraud.
The Trustworthy Accountability Group (TAG) pilot program will identify bots run in data centers that manage to avoid detection by the IAB/ABC International Spiders & Bots List, according to Google product manager, Vegard Johnsen.
“Well-behaved bots announce that they’re bots as they surf the web by including a bot identifier in their declared User-Agent strings,” he wrote in a blog post. “The bots filtered by this new blacklist are different. They masquerade as human visitors by using User-Agent strings that are indistinguishable from those of typical web browsers.”
To give an idea of the blacklist’s effectiveness, Johnsen claimed it managed to filter out 8.9% of all clicks during May on ad management platform DoubleClick Campaign Manager – preventing a potentially large distortion in click metrics.
He outlined two common software tools used to generate false clicks: UrlSpirit and HitLeap.
Google estimates that there were 6,500 data center installations of the former running at the end of May, generating a combined monthly rate of 500 million ad requests.
The other tool, HitLeap, had 4,800 installations running in VMs in data centers in mid-June at an average monthly rate of at least one billion fraudulent ad requests.
“Not only were these publishers collectively responsible for billions of automated ad requests, but their websites were also often extremely deceptive,” explained Johnsen.
“For example, of the top 10 webpages visited by HitLeap bots in June, nine of these included hidden ad slots –meaning that not only was the traffic fake, but the ads couldn’t have been seen even if they had been legitimate human visitors.”
It’s not all deliberately malicious traffic, however, as some “advertising competitive intelligence” companies use automated bots to analyze ad creatives, which sometimes requires them to click through to landing pages.
“While the aim of this company is not to cause advertisers to pay for fake traffic, the company’s scrapers do waste advertiser spend,” concluded Johnsen.
“Google has always invested to prevent this and other types of invalid traffic from entering our ad platforms. By contributing our data-center blacklist to TAG, we hope to help others in the industry protect themselves.”
Google’s motivation in corralling the industry to act on click fraud is not surprising given that it still makes a huge amount of money from advertising. However, advertisers are less willing to pay as much for those clicks as they were a few years ago as competition heats up.
The others mentioned as members of this TAG effort are Rubicon Project, TubeMogul, MediaMath, Quantcast and Dstillery.