“Testing the Testers”: Certification and Cloud Computing

Written by

By the (ISC)² U.S. Government Advisory Board Executive Writers Bureau (EWB)


Cloud computing is becoming ubiquitous throughout the federal government, and while the adoption of this technology may be more widespread in the private sector, the rigor and transparency provided by the FISMA certification process brings a unique challenge to this revolutionary technology. Federal agencies have responded to OMB pressure to utilize cloud technologies with a rate of adoption that has outstripped existing architectural, security, and operational standards and policies, and this gap represents an opportunity to reevaluate the security testing and certification practices that have developed in the confines of independently managed networks.

The National Institute of Standards and Technology (NIST) recently released a Cloud Computing Standards Roadmap, which noted a fundamental difference between the authorization decision that is made in an agency-owned or agency-managed environment and the decision that is made in a cloud environment. In the cloud environment, consumers are renting services in three models: Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and Software as a Service (SaaS). In each of these models, there is less direct visibility into the computing resources that provide the service than is typical in a standard enterprise environment, and there are new challenges to existing standards for managing and certifying the service.

SaaS applications, for example, are mostly consumed using a Web browser or other Web service using other application clients. Portability and interoperability, two key requirements for successful cloud implementation, require data format standards and interfaces if services are to migrate freely and support compliance requirements at the same time. This is especially important for email messaging systems, which have been the cloud ‘service-of-choice’ for federal agencies and have the most stringent eDiscovery and records management compliance requirements.

PaaS functional interfaces encompass the runtime environment and offer standard-based APIs, but there are significant issues regarding data format for backup and migration of application workload that need standardization if the portability promise of cloud computing is to be met.

Similarly, standard data formats and interfaces are needed to describe service-level agreement (SLA) and quality of service (QoS) in the degree necessary to allow authorizing authorities to make comparisons between the traditional and cloud environments.

Security and privacy pose perhaps the greatest challenge, especially in the out-sourced or off-site environment that is typical of the commercially provided cloud offerings. Identity and access management challenges are radically different in the cloud environment, where the implementation of these capabilities may span different network and administration domains. Single sign-on interfaces and protocols are another area where NIST notes a standards weakness.

The auditing and compliance needs of FISMA-certified systems are not going to be exclusive to the federal domain for long. Industry systems are increasingly interconnected, and the demand for a standardized approach to assuring the security state of those systems is only going to grow.

NIST is anticipating one aspect of this challenge with the publication of a draft inter-agency advisory report (NISTIR 7328), which addresses the requirements for security assessment providers – specifically providers who are offering assessment as a service. The document points out that “Security assessments are not about checklists, simple pass-fail results, or generating paperwork to pass inspections or audits – rather, they are the last line of defense in knowing the strengths and weaknesses of an organization’s information system…” Certainly, the quality of the assessment service is critical to the authorization decision and the ability for systems to share data and computing resources, whether in the federal environment or the private sector.

Key to the assessment process is the test procedure itself. In all too many cases, testing is, in fact, the mere execution of a checklist with a pass or fail result. This approach is inadequate at best in a cloud environment, and in the intermediate term – defined by the lack of stable standards and policies – it cannot support a valid authorization decision. The interesting question is, who determines what constitutes an effective test plan?  

Software development methodologies have associated testing procedures that have a long-standing history in practice. It might be useful to evaluate these processes to determine what might constitute a standard for testing and the development of test plans in the dynamic world of cloud computing. In the NIST Risk Management Framework, testing and the development of test plans are a critical component of the continuous authorization process. The shift to a cloud paradigm might be the opportunity to standardize the test and assessment process as well as other promising service offerings that this technology can provide.

What’s hot on Infosecurity Magazine?