This is the stark result of an analysis of 80 new viruses found and tested by Imperva. The company used the TOR network to scour the dark net to find the latest malware, and then developed an automated testing and monitoring process to test the samples using the VirusTotal website. VirusTotal allows people to submit samples for testing against more than 40 of the current free and paid-for anti-virus products, and produces a report on the results.
The results obtained by Imperva are not encouraging. “The average detection rate of a newly created virus is 0%,” and “Typically, it takes four weeks for just 25% of antivirus vendors to detect a new virus.”
Finding the viruses was not easy. Imperva hid its intentions behind the obfuscation of TOR and visited dubious websites, known infected servers and Russian forums. The last was a rich source: “The availability of malicious code and viruses in these forums was extremely high. Any kid could build a virus by themselves or download one ready-made.”
Having obtained the new samples Imperva set about testing the ability of AV products to detect them. It chose to use VirusTotal, and wrote a script to automate its submissions. The results were collected and collated, and the process was repeated weekly for six weeks. When VirusTotal detects a new virus it informs the AV industry. It can be assumed, then, that each AV vendor learned about the new virus shortly after the first Imperva test run. By repeating its tests each week for six weeks, Imperva was able to monitor how quickly the vendors incorporated detection capabilities into their products.
On the surface, the report is a damning indictment of anti-virus capabilities. Imperva doesn’t go so far as to suggest that AV should be abandoned, but does recommend additional and alternative defenses. “Despite the inadequacy of anti-virus solutions,” it says, “Imperva does not recommend completely eliminating it from an effective security posture. Instead, security teams should focus on detecting abnormal behavior such as unusually fast access speeds or large volume of downloads and adjust its security spend on modern solutions to address today’s threats.”
However, it has to be said that the real value of VirusTotal is in allowing users to check whether a suspect file is actually malware – it was designed to check malware, not to check AV products. David Emm of Kaspersky Lab pointed to two particular problems. Firstly, he told Infosecurity, “The scan engines used on VirusTotal aren’t necessarily the latest versions provided by vendors. VirusTotal is available to *anyone*, including malware writers. No vendor wants to wittingly assist malware writers to test their creations.” Secondly he pointed out that the site only shows signature detection, and not its heuristic behavioral detection capabilities. “Today,” he said, “the best anti-malware products use a wide array of technologies to protect their customers. A vendor may be able to keep their customer safe even without the need for a specific signature.”
David Harley, a senior research fellow at ESET, was equally concerned. “VirusTotal was never intended as a test of scanner performance, and isn't suitable for the job,” he told Infosecurity, adding, “there is more than a whiff of marketing about this exercise.” Harley pointed to a paper he co-authored with VirusTotal’s Julio Canto. It points out that, “VirusTotal uses a group of very heterogeneous engines. AV products may implement roughly equivalent functionality in enormously different ways, and VT doesn’t exercise all the layers of functionality that may be present in a modern security product.”
The paper concludes that VirusTotal “can be used for useful research or can be misused for purposes for which it was never intended, and the reader must have a minimum of knowledge and understanding to interpret the results correctly. With tools that are less impartial in origin, and/or less comprehensively documented, the risk of misunderstanding and misuse is even greater.”
|
This story has just got a new lease of life following articles in the New York Times and the Register. Both articles discuss the Imperva report without criticism. However, the lack of any reply from the AV industry in the articles has prompted an angry response from the AV companies. The Anti-Malware Testing blog, for example, comments, “While Imperva’s recent quasi-test ‘proving’ that anti-malware products ‘are rubbish‘ has been thoroughly debunked by my colleague David Harley...” and ESET’s Righard Zwienenberg and Trend’s Rik Ferguson et al... Infosecurity’s position remains as stated in our report: VirusTotal is an excellent service for users who wish to check whether a suspect file might be malware; it is not, was never designed to be, and should not be used as a reliable test of anti-malware products.
|