Core vendors and credit unions among the many ramping up backup capabilities post 9/11

<p>COLUMBIA, S. C. – Last Sept. 11, USERS’s senior managers were gathered with many of their clients for an educational conference in Scottsdale, Ariz. Backup systems and operational continuity plans, of course, were something they all had in place, but no one could have expected the test they were about to undergo. “We shared the experience together,” said Larry Tankeloff, senior vice president of operations for the Valley Forge, Pa., core processor for more than 350 credit unions. “We immediately set in motion a contingency plan in the event some of our clients had to evacuate and declare disasters,” Tankeloff said. “We escalated the situation to a `year-end support’ status, meaning that we had the staff available around the clock ready to handle any issue. We also beefed up our FTP site to handle the surge in media transmissions we expected to result from the disruption of normal mail and overnight delivery services,” he said. The events of Sept. 11 seemed surreal. Like science fiction. A bad movie. But it was real, and in the aftermath of the attacks, credit unions, like every other transaction-critical industry in America, have spent the past year working with their technology providers to assess how they can keep the doors open when the unthinkable happens. Adding to that uncertainty is the pressure that comes with increasing consumer expectation for all the time, anywhere access to their accounts, to their money. The essence of doing so is communications. Electronic connection over telephone lines, computer lines, optic fiber, through the airwaves, whatever it takes. Credit unions and their key partners, the core processors, have long been developing an extensive network of backup data centers, offsite tape storage, and contingency plans of all shapes and sizes. And now they have to make sure the connections to those sites are there, so they can keep it all “live.” It’s perhaps easy to forget now, but it wasn’t 9/11 that first represented a big step forward in that kind of thinking for many. “It was Y2K that really put business contingency planning at the top of the priority list for many financial institutions,” observed Annette Zimmerman at Mountain America Credit Union in Utah. “During that time period, plans were developed. That was the difficult part,” said the $1 billion credit union’s senior vice president and CIO. “Once a good plan is developed, regular reviews, updates and tests will ensure its viability.” But the attacks of Sept. 11 gave that new urgency, and added a new sense of scale, according to the analysts at IDC who recently published a post-9/11 version of a report titled “A Guide to Business Continuity Planning: Protection and Recovery Services for Your Communications Infrastructure.” “The attacks were eerily reminiscent of the final scene in the hit movie Fight Club in which the main character sought to `erase the debt record’ by blowing up all the offices of the credit card companies,” the IDC analysts wrote. “This, of course, would not work, given that most important information is backed up in ultra-secure, remote facilities. “Given current events, however, such facilities are bound to become even more important, and interfacing with them during both disaster and non-disaster periods is even more important.” That’s because the consuming public, including credit union members, expect service 24/7, 365 days a year and from anywhere. “In earlier years, databases and systems could be down for as long as 48 to 72 hours. In today’s environment, being down as little as 30 minutes can be disastrous, as several prominent online firms can attest,” the IDC analysts wrote. “In this new context, acceptable downtimes are measured in seconds and maximum acceptable data loss is zero,” they wrote. Even the definition of business recovery services has changed. It used to mean being able to recover quickly. Now it means never going down at all. Preventing that means continuous assessment of all the disparate and connected networks of software, hardware, people and wires that a credit union and its technology supplier rely on to keep the business going through such problems as the routine power outage or the unthinkable that hit the United States last September. “Modern continuity services combine straightforward planning for normal problems with a need to prepare for the real potential of worst-case scenarios,” the IDC report said. It said developing a business continuity plan should include these three key steps: The assessment stage. This involves determining the cost of interruptions to a business and levels of service availability that would meet business objectives. The planning and design stage. Here, you map out the enterprise environments needed to support those business availability requirements. That would include things such as examining network architectures to ensure there is adequate bandwidth for expected transaction loads. The remote management stage. Many credit unions and their core processors have become familiar with this one after Sept. 11 as they took a look at the range of capabilities the vendors maintain at remote processing sites. This stage involves an ongoing set of services designed to “proactively minimize downtime,” the IDC analysts said. Included is remote monitoring to anticipate and prevent technological problems. Predictive trend analysis can come into play here, using computer-powered thinking to anticipate transaction loads and take steps to accommodate that vital heartbeat of member activity. Credit unions are responding to such advice, according to those who help them do that. “Overall, we’re seeing increased activity related to business recovery planning and testing,” said Chris McWilliams, manager of business recovery services at USERS. “Credit unions are setting up more meetings to discuss the topic with senior management. They’re asking more questions. They’re testing more frequently, they’re testing more than just the core system, and their boards are getting involved,” McWilliams said. “They also seem to be more diligent about the elecommunications piece of the puzzle – making sure there is enough bandwidth to support multiple access points and multiple delivery channels,” the USERS manager said. Other vendors are reporting the same interest and activity. “We see many more customers duplicating parts of their system. For instance, one of our customers, the Credit Union of Texas, even has everything doubled,” said Stephanie Shah, director of product marketing and management for Harland Financial Solutions. “We’re seeing much more testing with our Disaster Recovery Centers. We’re seeing more investing spare solutions offsite, and more frequent updating of credit unions’ business recovery plans,” said Shah, whose company serves more than 420 credit unions through its ULTRADATA system. The disastrous terrorist attacks changed the breadth of the planning, too. “After 9/11, there were new considerations,” Shah said. “What if there were no airplanes to deliver data or staff to an offsite location? What if it were days before mail was delivered? “Today, we have more of our customers taking advantage of our system’s ability to provide journaling not only to their processing system, but also to another system at an entirely different location,” she said. Answering these “what if” questions has become an integral part of the daily business activities of many credit unions in many ways, including expanding the horizons of possibilities good and bad. “The events of 9/11 changed the way we view the potential scope of a disaster,” said McWilliams at USERS. “Before, we tended to focus on a disaster being very localized or maybe regional, as in an earthquake or flood,” the business recovery services manager said. “September 11 forced us to see the very real possibility of a disaster being much more widespread, involving many credit unions all over the country. “As a result, we’ve taken additional steps to assure we have the capacity to handle multiple disasters at the same time. Having three identical data centers (in Pennsylvania, Washington state and Hawaii) is a major asset in that regard,” McWilliams said. McWilliams’ comments allude to a new economic reality. All this preparation is itself driving a new market. IDC predicts worldwide spending on recovery services, including consulting and operation services, to grow from $2.3 billion in 2000 to $6.3 billion in 2004. “This spending is driven by the purchase of recovery services that are designed to support availability of all aspects of IT infrastructure, including all hardware and software components, as well as business processes,” the IDC report said. Adding to that is the reality that increasingly sophisticated customer touch points, such as voice-over IP systems and call centers, are making the backup and recovery job that much more complicated, and expensive. Zimmerman sees that. “Our business recovery plan includes a greater number of critical systems than ever before,” the Mountain America CIO said. “Integrating multiple systems is one our greatest challenges,” she said. “Equally challenging is hiring and retaining staff who can manage multiple operating systems and a very complex network. “Many applications are sold to us to fill a certain niche or as an `independent application.’ In reality, these applications must be integrated into the infrastructure in order for us to reap the most benefit from our investment in them.” Zimmerman is not alone in that assessment. “Comprehensive disaster recovery services don’t necessarily come cheap,” the IDC analysts wrote. “Providers in this market will be charged with helping clients find the right level of service, as well as balancing expenses and needs. In the current economic climate, the issue of cost is paramount because companies need to assess their IT budgets and telecom assets more closely.” The bottom line won’t change, though. And, in the members’ eyes, it’s not about the money their financial institution spends. It’s about their own. “Members have become accustomed to and now expect to have access to their accounts 24/7. Downtime, even in the middle of the night, is no longer acceptable,” says Zimmerman at Mountain America. She also sees the challenge as continuous. “Business contingency planning is a journey. Our organization is continually changing, expanding our delivery systems to better serve our membership, which directly impacts contingency plans,” Zimmerman said. Tankeloff would agree. “USERS doesn’t see the challenges slowing down anytime soon,” the senior vice president of operations said. “No one can say what will happen next or how and where it will surface. “The next event is likely to present itself very differently, just as 9/11 did.” Indeed, the IDC researchers said, those responsible for business continuity can probably expect to have to deal with more routine, but nonetheless serious and disruptive, challenges to their daily operations. “The vast majority of service interruptions are caused by a few standard reasons: online security breaches (i.e., hacking and cracking), viruses, product bugs and the like. This will continue to be the case,” the analysts concluded. -</p> <p>[email protected]</p>