ICSA Labs, which put security products through a battery of tests before accrediting them as appropriate for business use, found that some products fared better than others. For example, 100% of web application firewalls that are submitted for security certification eventually make the grade. However, only 29% of network intrusion prevention systems ever attain security certification.
George Japak, managing director at ICSA Labs, explained that IPS is one of the newer security certification programs operated by the labs, and the technology is more recent, and more complex in nature. When the program began, the majority of IPS vendors participated, but many dropped out because they couldn't meet the security testing requirements.
"Typically they left with the promise of returning when their product was better prepared to pass the criteria", said the report. "Unfortunately for users, these products continue to be sold (sans ICSA Labs certification) in the interim."
Documentation also causes problems for many vendors, Japak warned. "Documentation is one of the last things that a company developer works on getting to an appropriate level. Time and again, we find the information in the documentation unhelpful and/or misleading", he said. "In some cases we get back in touch with the vendor because we think they sent us the wrong documentation, but it turns out that's all they had."
Unfortunately for product vendors, the security certification process does not stop once a product has been accredited. Because of changing threats and alterations in the product itself, products must be recertified. 43% of network IPSs that make it through the accreditation process and get security certification subsequently lose that certification at a later stage.
Common violation types include revisions and patching (21%), and logging (57%), the report added.
"They may break other functionality because of poor QA. It may be that when a fix is pushed forward, they forgot to include previous fixes", Japak said of software patching. "In some cases vendors play musical chairs when it comes to development groups."
Although network IPS products were the worst of the bunch, overall, 18% of all security products on average never attain certification, the report concluded.