Knowing what to buy and how well a technology works can be quite the dilemma: do you go by the analyst report, the company’s marketing or presentation, the recommendation from a colleague, or maybe even the test report?
For the latter, NSS Labs has been testing products for over 10 years and Infosecurity met with founder Vikram Phatak. He explained that the company's model is focused around “testing the world’s security products.” He said that every time NSS Labs has reviewed a product, it has taken no payment for it, and the creators of the ones that do well are welcome to buy the rights to use the report.
So how does NSS Labs choose what to test? Phatak said that after deciding what product area they want to test, they will look at what the analysts are saying and see what the client is asking for.
“When somebody goes to lengths to avoid testing, it’s probably an indication that there is a problem,” he claimed, saying that usually people want to put a product into a group test, and “show the world that it is good.”
He explained that in one case, the incumbent vendor that the client was using didn’t want to be part of the test, and the client chose another product. “Sometimes you want a hammer, sometimes you want a screwdriver, different companies have different needs,” he said.
Phatak explained that tests are carried out in several ways. These include performance testing in a test environment, and security testing using exploits, evasions and malware using a “library of attacks.” He also said that it places real machines on the internet, which run custom software to engage with thousands of websites, and it can track performance against real attacks. This is enabled by a team who are monitoring attacks and building custom software reflective of what clients typically encounter, while a threat research team monitor for nation state style malware.
However, Phatak was keen to stress the role that the testers play in the security lifecycle, saying “we’re not the adversary, we help their [vendor] customers recognize the value that they provide” as buyers find it hard to determine what is better about one product over another, “so we provide ground truth so people can make decisions based on facts.”
Phatak praised those companies that he said were doing well, saying that they care and “want their customers protected and are less worried about doing well in any individual test for marketing reasons than they are about making sure clients are protected.”
Asked if he is able to spot a product from afar, and know what is going to be a game changing piece of technology, Phatak said that NSS Labs can see potential, as “the technology will tell us,” but also when it is working with a company, the approach taken is often a very strong indicator. “This means that if they are open to the fact that they are not perfect, then they can start to fix things,” he said. “If they put their head in the sand, it’s not going to help much.”
Phatak said that there are companies who get test results, and then spend to improve as when the test result is not good, sales decline, and then if there is an improvement in engineering and the next test is better, sales go can up.
In the case of breaking news, Phatak highlighted a major attack such as Sasser, saying that there were companies claiming that they could protect you from the exploits.
“There are three possibilities here: they are flat out lying, they can solve it, but it is only with a specific version that they have seen once, and if it is polymorphic they cannot prevent it, while the third is those who do it right - who look at the malware and variants that an attacker can use.” He said that the third is far too rare, and often they see the first two. “This is where we come in, we see this stuff and we can tell the executives who tell their teams not to do shortcuts.”
Phatak said that in some other test environments, a test is done with a “known” malware variant, and the technology will pass the test. “When we test it the way a bad guy would, you don’t get a clean bill of health.”
He said that a big motivator is because clients are enterprise businesses, which tends to motivate people to get things fixed when they have “money on the line, but our job is to help these folks as we have leverage across the entire industry.”
Concluding, Infosecurity asked Phatak about what the future of testing looks like, and he said that there is a lot more to be done, especially considering cloud, DevOps and IoT. “With the move to DevOps and the cloud, how can we give the tools to these folks so that they can do the testing and then ship us the information, and we can help them with the analysis to know what it means.”