11 September 2007

Disaster recovery six years after 11 September

John-Paul Kamath, Computer Weekly

The 11 September terrorist attacks, six years ago today, brought the realities of IT disaster recovery sharply into focus. More than half of all small to medium-sized enterprises affected by 9/11 did not trade again.

Although some businesses have learned from their mistakes and refined their recovery plans, others have a long way to go.

One of the lessons of 9/11 is the need for companies to have a back-up datacentre located away from their primary datacentre.

Bill Crichton, consultancy manager of business continuity and recovery services at Hewlett-Packard, was in New York at the time of the attacks. He warned that many firms still wrongly assume that simply having two datacentres (main and back-up) will protect them in a disaster.

But when most of Manhattan closed down, organisations that had sited their main and back-up datacentres in the same area found they could not resume business.

"Unfortunately, there are still many businesses that have two data­centres within a central location, where a terrorist attack can render both sites unusable," Crichton said.

Merrill Lynch is one company that decentralised its core IT systems after the attacks. The financial services firm lost two datacentres on 11 September. It has now moved its primary datacentre to Staten Island, where it runs on a separate electrical grid to mitigate against the loss of power in one area. The New York site functions as a back-up.

Morgan Stanley is another financial firm that has separated its trading and back-up facilities, which were within close proximity and dependent on the same transport and power infrastructure prior to 9/11.

Include people and processes

A second lesson from 9/11 is that organisations need to think of the company as a whole, including people and processes, as well as IT.

"Prior to 9/11, many firms did not really take account of staff in their plans. It is all very well having remote datacentres, but if you have no staff, or absent key staff as a result of an incident, this can bring an organisation to its knees," said Ron Miller, managing consultant at SunGard Availability Services.

Morgan Stanley conducts major tests every year. Over the Easter weekend, the firm takes advantage of the powering down of all its main datacentres in Canary Wharf to run a full simulation of a total loss of data at all offices with 200 people.

The firm also conducts business process testing twice a year, in which it sends a team to the recovery site to execute trades. "It is very important in terms of familiarising staff with everything from getting to the site to knowing how to use it," said Richard Deighton, continuity manager EMEA business at Morgan Stanley.

The latest Department of Trade & Industry survey on disaster recovery, published last year, found that of the 60% of UK firms that had a disaster recovery plan, less than 50% had conducted live tests involving staff in the past year. The danger, say experts, is that many companies base their plans on misconceptions and false assumptions.

Jon France, business continuity manager at business information provider LexisNexis, which conducts a series of live disaster recovery simulations throughout the year, said that full scenario testing can be expensive and time-consuming, which is why some companies are reluctant to test regularly.

The frequency of a company's simulation should reflect the rate of change within the business, he said. "When people change jobs, or strategic direction at the company changes, or when significant equipment refreshes occur, our recovery plans are tested against this criteria," said France.

Professional services firm KPMG said business continuity is a human resources issue as well as an IT issue. But it has taken events such as 9/11 and Hurricane Katrina to move firms' attention to the human factors.

Bob Piggott, head of group crisis management at HSBC, said the finance sector has learnt crucial lessons from 9/11. Keeping staff informed about what is going on is vital.

"In the UK, all our staff have a telephone number they can ring to receive an updated status message in the event of an incident," he said.

HSBC has direct communications links to Transport for London and the Metropolitan Police. Providing this service can help reassure staff, who might receive confusing reports from the media during a disaster.

Organisations have done well on the technology side of things, said Lyndon Bird, technical services director at the Business Continuity Institute. "But without managing people, the most automated operation in the world will still fail, so having the right people doing the right things in the right places at the right time is absolutely vital, and more work still needs to be done," he said.

This article first appeared on the web-site of Computer Weekly, at http://www.computerweekly.com/Articles/2007/09/11/226632/disaster-planning-and-business-continuity-after-911.htm. © Reed Business Information 2007.

Flood-zone security suppliers confident of business continuity (23 July 2007)

Staying power (business continuity feature, July/August 2006)

News index