Credit unions examine data safety, operations recovery

WASHINGTON- As much as they did for anyone, the Sept. 11 terrorist attacks prompted Pentagon Federal Credit Union to re-think its crucial data backup operations. So, it turned half a continent away, to Omaha, Neb., where a new dedicated hot site has been added to the big CU’s existing call center. Credit unions across the land have reassessed how they store data (and ensure business operations) in the wake of the attacks, and took measures ranging from simply assuring themselves with audits and tests that they’re satisfied with what they have to wholesale changes, such as Pentagon’s abandoning its reliance on a third-party backup site. “This plan assumed that most of the technical staff would survive the disaster and could travel to the commercial recovery site in another city,” Frank Leser, senior vice president and CIO at $5 billion Pentagon FCU, says of assumptions that went out the window on Sept. 11. And while none of its 30 staffers were injured at the big CU’s operations inside the Pentagon (nor at its headquarters in suburban Virginia), the attacks also forced the big CU to dismiss the notion that it could always count on being able to physically transport tapes to its backup hot site or that public utility infrastructure would be available to do it electronically. The result of throwing out all those assumptions was the decision to move the backup operations inside and far away, Leser says. The Omaha center is fully equipped to run Pentagon FCU’s core systems and maintain services to its 480,000 members, its CIO says, and other changes are continuing. “Since it’s already on our internal data communications network, service could be quickly restored in the event of an emergency in the metropolitan Washington area,” Leser says. “IT staff positions were transferred to Omaha to provide the needed technical skills, and the final step is the protection of member data. “Pentagon Federal now is vaulting the back-up data tapes in Omaha and will soon begin electronic vaulting of completed transactions in real time.” All the way across the country, the attacks reverberated in real time to organizations like Kinecta FCU, which found that not only were several of its member service centers around the Los Angeles airport suddenly shut down, but it couldn’t send its recovery team to Philadelphia, where its backup systems are located. “Luckily, we didn’t have to,” says Rudy Pereira, senior vice president and CIO for the $2.8 billion, 230,000-member organization based in Manhattan Beach, Calif. But preparations have been made to deal with such a possibility, he adds. “For several months after Sept. 11, our emergency operations committee met on a weekly basis and discussed disaster recovery and data backup alternatives,” he says. “We decided not make an immediate change, but are exploring a local recovery solution in addition to an out-of-state solution.” Kinecta has already made some changes, however. Pereira says business resumption and crisis communication plans were strengthened, including an emergency 800 number for employees (on a laminated card they all got) and information areas on the CU’s intranet. EARTHQUAKES AND HURRICANES Terrorist attacks weren’t on the minds of the bosses at Wescom Credit Union when they first opted for a disaster-recovery hot site about 60 miles away from its Pasadena, Calif., headquarters. “We live in earthquake country, and our hot site was already in the works when 9/11 happened,” says Lesley Chinen, spokeswoman for $2.2 billion Wescom, a particularly self-reliant outfit that now offers processing and backup services to other credit unions. “The events last year confirmed the need for our hot site, and after 9/11 we made a thorough review of what is included in our emergency plans,” Chinen says. Hurricanes, meanwhile, had been on the minds of planners at MacDill FCU on Florida’s Gulf Coast long before terrorists eyed the nation’s military and financial citadels. “Since we’re in a hurricane zone, our procedures already cover evacuation scenarios and the possibility of losing access to our facilities for an extended period of time,” says Brad Sears, senior vice president and CIO at the $900 million, 132,000-member CU in Tampa. Data and systems backup plans are intended, in fact, to address “most scenarios, regardless of any specific threat,” Sears says, adding, “If anything, we have become more aware of physical security. “People or objects that appear out of place are scrutinized a little more than before. We’ve also increased our surveillance capabilities at some of our offices.” Meanwhile, human actions of a far less drastic nature have prompted Campus Federal Credit Union to make sure its data backup systems were adequate all along, according to staffers at the $200 million, 34,000-member organization in Baton Rouge, La. “Because of users who delete their files, we have had numerous opportunities to test our backup tapes,” says Juan Batista, Campus Federal’s network manager. “We use BackupExec from Veritas on all our Windows NT/200 servers, and have a tape library from TreeFrog that includes two drives and stores up to 15 tapes,” he says. “The library automatically changes tapes as needed. We perform full backups every night. We also perform full backups at the end of each month. These are reasons why we felt we did not have to change anything,” Batista says, adding that tapes, of course, are stored offsite. Same with RTP Federal Credit Union, which also cites size and resources in its considerations. “Prior to 9/11, we already had a well-established disaster recovery plan in place, and we’re careful to back up all systems daily and take the backups offsite,” says Doug Wilkerson, president of the $60 million, 13,000-member CU in Durham, N.C. “Since 9/11, we’ve heard of larger businesses creating a secondary data center to be ready to take over if something happens to the primary center, but that takes resources that we have not felt we have available at the present time,” Wilkerson says. DOCUMENTING, Y2K AND THE HUMAN FACTOR You try to cover all the bases, but there’s only so much you can anticipate. That’s one of the lessons of Sept. 11. At Motorola Employees Credit Union-West, “our backup strategy is designed from a recovery and business continuity approach, providing procedures to follow instead of attempting to plan for every possible disaster scenario,” says Tom Gessel, vice president of information technology at the $525 million CU in Phoenix, Ariz. MECU-West has system backups in place, of course, and they were examined thoroughly in light of Sept. 11, Gessel says. “We found our processes to be prudent and reasonable but realized some of these practices needed to be documented more thoroughly,” he says. “As such, related procedures and work instructions have been updated to reflect the backup process.” Data-processing recovery planning is nothing new for most credit unions, and certainly not at the nation’s largest, where experience has caused it to expand its thinking beyond simply storing the numbers. “We had been rehearsing data processing recovery twice annually for over a decade. About a year before 9/11, building on our Y2K experience, we revisited our recovery strategy and changed our focus from computer systems recovery to business process recovery,” says Ardin Goss, CIO at the $14 billion, 2.1 million-member Navy Federal Credit Union in suburban Vienna, Va. “We did a risk analysis and categorized each of our business functions as either mission critical, essential or non-essential and then designed recovery strategies for each category,” Goss says. The outcome of that process? “There were two elements of our planning that we found needed more thought and better contingency planning: preparation for total unavailability of our campus and better planning for the human aftermath of a catastrophe,” Goss says. The terrorist attacks added to that process at Navy FCU, its CIO says. For instance, a more robust communications network has been installed that allows the credit union to continue operating “as a subset of our branches,” Goss says. -