“Bit9,” says the report, “commissioned Tolly to evaluate leading endpoint security solutions to compare the effectiveness of traditional anti-virus products and the most recent application control approach against malware and zero-day threats.” The intent is to see how the products compare in defending against contemporary ‘advanced’ threats. Five attacks were tested. Bit9 blocked all five attacks; Symantec Endpoint Protection 12.1 blocked three; and McAfee Endpoint Protection Suite blocked one.
“This side by side test provides buyers with the confidence that Bit9 delivers on their promise of protecting enterprises’ valuable Intellectual Property from malware attacks,” said Kevin Tolly, founder of the Tolly Group. “Bit9 Parity Suite stopped all five exploits that compromised laptops and servers in these tests.” The test does indeed confirm Bit9’s efficiency against these attacks, but should not be used as condemnation of McAfee and Symantec products. The report carries a rider from Symantec: “Symantec advocates a layered approach to endpoint security... Despite our request to The Tolly Group to test the proper products for protecting Web-facing servers, Symantec Critical System Protection AND Symantec Endpoint Protection. The Tolly Group proceeded with the test using ONLY Symantec Endpoint Protection.”
Anti-malware testing is notoriously difficult. Indeed, the anti-malware industry has established its own organization (the Anti-Malware Testing Standards Organization – AMTSO) in an attempt to define a valid, consistent, and fair methodology for product testing and comparison. The Tolly Group is not currently listed as a member of AMTSO. It is important, therefore, to see this evaluation in context.
Independent security researcher David Harley told Infosecurity that he had some concerns. “I wouldn’t personally trust a test that made a broad statement about the overall effectiveness of the products on the basis of a handful of simulated attacks in an artificial environment. To do so,” he continued, “I’d have to assume that the choice of competitive products was appropriate (which Symantec doesn’t seem to think was the case), that the product configuration was appropriate (which McAfee doesn’t seem to think was the case), and that the sketchily described methodology was appropriate.”
Harley’s concern is that “an accurate test would have to play to strengths and weaknesses of the sponsored product and any other products included for comparison, with all appropriate functionality enabled and tested, and at least attempting to utilize all likely attack types and vectors. Otherwise,” he told us, “there’s a danger that the test could be dismissed as an ‘apples and oranges’ test.”
None of this detracts from Bit9’s ability to defend against this particular selection of attacks. It is the negative implication leveled against Symantec and McAfee that should be viewed with caution.