“Bit9,” says the report, “commissioned Tolly to evaluate leading endpoint security solutions to compare the effectiveness of traditional anti-virus products and the most recent application control approach against malware and zero-day threats.” The intent is to see how the products compare in defending against contemporary ‘advanced’ threats. Five attacks were tested. Bit9 blocked all five attacks; Symantec Endpoint Protection 12.1 blocked three; and McAfee Endpoint Protection Suite blocked one.
“This side by side test provides buyers with the confidence that Bit9 delivers on their promise of protecting enterprises’ valuable Intellectual Property from malware attacks,” said Kevin Tolly, founder of the Tolly Group. “Bit9 Parity Suite stopped all five exploits that compromised laptops and servers in these tests.” The test does indeed confirm Bit9’s efficiency against these attacks, but should not be used as condemnation of McAfee and Symantec products. The report carries a rider from Symantec: “Symantec advocates a layered approach to endpoint security... Despite our request to The Tolly Group to test the proper products for protecting Web-facing servers, Symantec Critical System Protection AND Symantec Endpoint Protection. The Tolly Group proceeded with the test using ONLY Symantec Endpoint Protection.”
Anti-malware testing is notoriously difficult. Indeed, the anti-malware industry has established its own organization (the Anti-Malware Testing Standards Organization – AMTSO) in an attempt to define a valid, consistent, and fair methodology for product testing and comparison. The Tolly Group is not currently listed as a member of AMTSO. It is important, therefore, to see this evaluation in context.
Independent security researcher David Harley told Infosecurity that he had some concerns. “I wouldn’t personally trust a test that made a broad statement about the overall effectiveness of the products on the basis of a handful of simulated attacks in an artificial environment. To do so,” he continued, “I’d have to assume that the choice of competitive products was appropriate (which Symantec doesn’t seem to think was the case), that the product configuration was appropriate (which McAfee doesn’t seem to think was the case), and that the sketchily described methodology was appropriate.”
Harley’s concern is that “an accurate test would have to play to strengths and weaknesses of the sponsored product and any other products included for comparison, with all appropriate functionality enabled and tested, and at least attempting to utilize all likely attack types and vectors. Otherwise,” he told us, “there’s a danger that the test could be dismissed as an ‘apples and oranges’ test.”
None of this detracts from Bit9’s ability to defend against this particular selection of attacks. It is the negative implication leveled against Symantec and McAfee that should be viewed with caution.
09 April 2012
Its not recommended to conclude that Bit9 is doing better that their competitors just by testing 5 Exploits in the test. So this report doesn't provide any useful information. This report may be misleading for users.
Note: The majority of comments posted are created by members of the
public. The views expressed are theirs and unless specifically stated are not those
Elsevier Ltd. We are not responsible for any content posted by members of the public
or content of any third party sites that are accessible through this site. Any links
to third party websites from this website do not amount to any endorsement of that
site by the Elsevier Ltd and any use of that site by you is at your own risk. For
further information, please refer to our Terms & Conditions.
Comment on this article
You must be registered and logged in to leave a comment
about this article.