The scourge of the bots shows no sign of waning: A fresh study shows that upward of 60 percent of an organization’s Web traffic may be generated by programs that operate as an agent for a user or another program or simulate human activity.
The finding has prompted Akamai Technologies to debut the Akamai Bot Manager, aimed at helping users better identify and understand what types of bot traffic are hitting their sites.
Nearly all online businesses can be impacted by various types of bot traffic. This traffic may include scrapers that grab content or price information, automated “clicks” that fraudulently increase ad revenues and transactional bots that can be used to purchase limited availability goods and services, making them unavailable to legitimate customers.
Further, there are situations where the impact of bot activity on the business may be beneficial, while the impact on site performance is not. As such, organizations require a way not only to identify the type of bot activity they are experiencing, but also to provide a variety of techniques to most effectively respond to different types of bot traffic beyond simple blocking. In the case of malicious bots, simply blocking them alerts the bot operator that protections are in place and triggers the bot to evolve in order to better evade detection.
Instead, Akamai Bot Manager uses a variety of management techniques—slowing or delaying bot traffic, serving alternate content, redirecting to an alternate origin, or identifying bot traffic and allowing customers to take independent action.
“Bot activity is in many ways a ‘cost of doing business’ when you sell online, and up until now, there has not been a good way to achieve the visibility into bot traffic necessary to make truly informed decisions,” said William Avellan, IT director at internet retailer U.S Auto Parts Network. “With Bot Manager, we have the information we need to solve all of the bot problems we’ve been facing including content theft, price scrapers, and even identifying the IP transit providers hosting these bots.”
Bot Manager also contains a pre-defined directory of more than 1,300 pre-defined bot signatures in 15 different categories of legitimate web and business services, making it easier to rapidly identify commonly seen bot traffic. And, companies can create custom bot signatures and categories reflecting the impacts that new and/or unique bots to their sites have on their business and IT infrastructure.
Detection features include the automatic identification of clients that have engaged in web scraping behavior against other Akamai customers; customers can then apply a unique management policy to each custom or pre-defined category.
“The web is full of bots and until now, companies had two choices, block them or suffer in silence. Unfortunately, neither choice was ideal,” said Stuart Scholly, senior vice president and general manager, Cloud Security Solutions, Akamai. “With Bot Manager, we’re changing the game when it comes to bots. We’re giving our customers the power and flexibility to put a true bot management strategy in place that best fits their business goals and objectives.”
Photo © Ken Wolter/Shutterstock.com