"In the event of a crisis, can we rely on our backup?" Following the tragic events of September 11, that is a question that many information technology executives have been asking themselves. With millions of dollars at stake in volatile financial markets, few can afford the risk of being unsure.
In the past few months, much has been said about developing new disaster recovery sites that can emulate primary site trading desks and operations, or purchasing more hardware to expand capacity at backup data centers. The idea behind this is that by spending more money on hardware and fixed locations, firms will be better prepared for any future disturbances to operations.
This line of thinking, however, is a step back to the days when companies focused primarily on protecting their data and less on the ergonomics of accessing the data. Most firmsâ disaster recovery programs failed because they didnât provide a workable environment for their trading operations.
Crisis Adds to Crisis
On September 11, the financial services industry learned that most disaster recovery plans were grossly insufficient. Many firms found themselves having to quickly vacate their premises and relocate to their disaster recovery site (DRS), where â they assumed â duplicate systems were installed and configured for immediate use.
Yet, managers quickly discovered that the DRS was inadequately prepared because:
-not enough hardware was available;
-the software, if loaded at all, was outdated;
-database backups were sorely inadequate - the data was, in some cases, weeks old and, at best, was missing intra-day transactions;
-in the very few instances where data, software and equipment operated together, the systems worked in isolation. In essence, the users did not have the network of systems it required to trade effectively.
Most importantly, though, firms realized that they had underestimated the number of people required to run their operations â even in a scaled down mode. Most trading units had less than half the desks that they needed to resume operations, and people werenât situated in the best places for efficient workflow.
Instead of recovery in the event of a crisis, firmsâ DRSs offered an alternative set of problems that compounded existing difficulties. Most of the trading systems took days to restore, and some even took weeks â which translated into a substantial loss of business that many firms could not endure.
How Could This Have Happened?
Some of the reasons for deficiencies in DRSs are easy to identify. First, with the declining economic growth of the last year, most firms have reined in costs; and, until recently, backup sites have typically been relegated to a low priority status.
Second, teams set up for managing the DRS are often inadequate. Disaster recovery has always been an after-thought, and system administrators, who were often responsible for implementing and monitoring the DRS, were not able to become intimately familiar with the nuances of the systems at the site. Furthermore, software engineers didnât foresee the need to develop systems that were configurable enough to be run from multiple sites. Keeping track of all the system dependencies and replicating them at the DRS can be a daunting and time-consuming task.
Finally, there was not enough attention paid to workflow: Who would sit where? How would different units, such as sales and traders, interact with each other?
Moving forward, financial services firms need to create solutions that address both database backups and workflow management.
The first step in creating an effective disaster recovery program is to develop backup systems that ensure data reliability. Post trade data needs to be updated throughout the day, ideally in real-time, and trade confirmation at the backup data center needs to be a condition for clearing the trade. The business needs to stop, or at least slow down, if the secondary site is not operational.
Also, firms need to ask themselves if one backup site is enough? Is New Jersey, for example, safe enough to host a backup site for a firm based in New York? A better alternative to local backup sites can be found in the approach that many Internet companies have adopted of having redundant data centers throughout the country. There is no reason that a well-managed backup site canât be hosted in Texas or Colorado, or both.
Finally, itâs impossible to develop a DRS that offers the perfect allocation of space for various workgroups. By recognizing the need for mobility and flexibility, firms can develop systems and environments that enable more efficient workflow. Users need to be able to arrive at the DRS and set themselves up without major help from system administrators and the hardware must be interchangeable.
To create such an environment, firms should use Web-based software solutions. Because they require minimum installations and can be accessed from any desk with an Internet connection, Web-based solutions are an ideal tool for operations at a DRS.
Most financial systems today, however, are not Web-based. And, indeed, primary trading systems need not be. Going forward, however, management should mandate that all systems also support Web-based interfaces. These interfaces need only be scaled down versions that allow business to continue in case of a disaster.
Firms could arm users with pre-configured laptops that grant them network access and offer a scaled down Web-enabled trading application, which would allow traders, salespeople and support staff to quickly resume operations at a DRS, hotel room, or even from their homes on a dial-up Internet connection.
Different firms will allocate resources differently in the months to come. But the firms with the vision to set up multiple database backup sites and supplement their systems with web-based interfaces will be the ones that are best prepared for any future disasters.