Related Links

Top 5 Stories


Performance comparison between Bit9, Symantec and McAfee

14 March 2012

The Tolly Group has published a new report: 'Comparison of Bit9 Advanced Threat Solution versus McAfee Endpoint Protection Suite and Symantec Endpoint Protection 12.1'. But are they apples and oranges?

Bit9,” says the report, “commissioned Tolly to evaluate leading endpoint security solutions to compare the effectiveness of traditional anti-virus products and the most recent application control approach against malware and zero-day threats.” The intent is to see how the products compare in defending against contemporary ‘advanced’ threats. Five attacks were tested. Bit9 blocked all five attacks; Symantec Endpoint Protection 12.1 blocked three; and McAfee Endpoint Protection Suite blocked one.

“This side by side test provides buyers with the confidence that Bit9 delivers on their promise of protecting enterprises’ valuable Intellectual Property from malware attacks,” said Kevin Tolly, founder of the Tolly Group. “Bit9 Parity Suite stopped all five exploits that compromised laptops and servers in these tests.” The test does indeed confirm Bit9’s efficiency against these attacks, but should not be used as condemnation of McAfee and Symantec products. The report carries a rider from Symantec:  “Symantec advocates a layered approach to endpoint security... Despite our request to The Tolly Group to test the proper products for protecting Web-facing servers, Symantec Critical System Protection AND Symantec Endpoint Protection. The Tolly Group proceeded with the test using ONLY Symantec Endpoint Protection.”

Anti-malware testing is notoriously difficult. Indeed, the anti-malware industry has established its own organization (the Anti-Malware Testing Standards Organization – AMTSO) in an attempt to define a valid, consistent, and fair methodology for product testing and comparison. The Tolly Group is not currently listed as a member of AMTSO. It is important, therefore, to see this evaluation in context.

Independent security researcher David Harley told Infosecurity that he had some concerns. “I wouldn’t personally trust a test that made a broad statement about the overall effectiveness of the products on the basis of a handful of simulated attacks in an artificial environment. To do so,” he continued, “I’d have to assume that the choice of competitive products was appropriate (which Symantec doesn’t seem to think was the case), that the product configuration was appropriate (which McAfee doesn’t seem to think was the case), and that the sketchily described methodology was appropriate.”

Harley’s concern is that “an accurate test would have to play to strengths and weaknesses of the sponsored product and any other products included for comparison, with all appropriate functionality enabled and tested, and at least attempting to utilize all likely attack types and vectors. Otherwise,” he told us, “there’s a danger that the test could be dismissed as an ‘apples and oranges’ test.”

None of this detracts from Bit9’s ability to defend against this particular selection of attacks. It is the negative implication leveled against Symantec and McAfee that should be viewed with caution.

This article is featured in:
Internet and Network Security  •  Malware and Hardware Security



vishalsk1 says:

09 April 2012
Its not recommended to conclude that Bit9 is doing better that their competitors just by testing 5 Exploits in the test. So this report doesn't provide any useful information. This report may be misleading for users.

Note: The majority of comments posted are created by members of the public. The views expressed are theirs and unless specifically stated are not those Elsevier Ltd. We are not responsible for any content posted by members of the public or content of any third party sites that are accessible through this site. Any links to third party websites from this website do not amount to any endorsement of that site by the Elsevier Ltd and any use of that site by you is at your own risk. For further information, please refer to our Terms & Conditions.

Comment on this article

You must be registered and logged in to leave a comment about this article.

We use cookies to operate this website and to improve its usability. Full details of what cookies are, why we use them and how you can manage them can be found by reading our Privacy & Cookies page. Please note that by using this site you are consenting to the use of cookies. ×