Municipal CU opens high-tech back-up facility; CU leaders say criticism of disaster recovery plan too harsh

NEW YORK – From a technology standpoint, was Municipal CU’s disaster recovery plan woefully inadequate to deal with a disaster? Was their IT department asleep at the switch? Heck no, says CEO Bill Porter who says he’s tired of uninformed people criticizing the CU’s preparedness. At press time as the CU is populating the data for its new revamped disaster recovery facility in New Jersey, Porter says he has a bad taste in his mouth from some of the criticism from the industry. “Some of the things I’m hearing really bother me. For some people out there in the movement there’s some misconceptions of what we had and what we didn’t have, and what we truly had to go through to get back into real-time mode,” said Porter. Porter said if this was a normal disaster, if there is such a thing, the CU would have been in great shape, but no one could have planned for a national shutdown of the airlines. The CU did have a hot site back-up established with Sungard Recovery Services, a well-known disaster recovery company. When the CU contacted Sungard to declare a disaster it was informed that Sungard’s Philadelphia facility, where the CU had chosen to recover, was oversubscribed because of the number of companies affected by 9/11. Sungard offered facilities in Atlanta and Chicago as alternatives, which the CU couldn’t utilize because of the grounded planes. “Our hot site in Pennsylvania was designed so all we had to do was switch our processing over to it. But they were at capacity,” said Porter. For any disaster recovery the most critical components are the back-up tapes, a system to run them on, and communication lines. On Sept. 12. some MCU staffers were able to re-enter the CU’s headquarters on 22 Cortland St., which once overlooked World Trade Center Square, to retrieve some critical tapes and disk drives. The tapes were delivered to the home of the CU’s Chief Technology Officer, Barry Grant, so he could start the restoration process at AFTECH’s facility in Malvern, Pa. Grant said the tech team had to start as if the CU’s entire headquarters had been destroyed. “Based on what we had to deal with, and what we had to rebuild, very few management teams would have been able to do what we did. We had full connectivity to our branches by that following Monday. My staff and I worked 20 hour days to get this done,” said Grant. The CU decided to use its Elmont Branch as a recovery site for the computer systems. It contacted Compaq (now Hewlett-Packard) to deliver replacement equipment as dictated by the RecoverAll Contract the CU had in place. The RecoverAll Contract was an element of disaster recovery that was also unreported and many people didn’t know the CU had, said Grant. On Sept. 12 it implemented a plan to get all of its branches live by Sept. 17. The CU had a number of obstacles. The Verizon network it used for high-speed communication lines was in shambles. It had to move to microwave services from AT&T to get communication up and running. Tech issue after tech issue kept coming up that forced the CU to change plans. For example, the Elmont branch it had started using as the recovery site could not be used for the microwave connectivity because there was no sight line to a microwave tower. The CU found a more favorable environment at its Battery Park location. It then installed 130 new PCs at its new back office location, and had microwave connectivity that could support a T-3 line. These are examples of the extent the CU went through to get back up said Grant. Grant admits that the CU’s disaster recovery plan wasn’t flawless, but was adequate for a typical disaster. “I’m not going to say the disaster recovery plan was prefect. I knew there were some weaknesses. We were working on things to make it better. There’s only so much that can be done in a certain period of time,” said Grant. Time ironically became a big issue in this story. Something most people don’t know said Grant is the CU had planned to change its ATM protocol on Sept. 22, 2001, just nine days after 9/11. It was changing its connectivity method from the slower BISYNC to the faster TCP/IP. That conversion could have helped significantly with the ATM withdrawal problem. Going from BISYNC to TCP/IP isn’t done overnight, said Grant. The CU’s core processor, AFTECH, had never supported a TCP/IP connection so the CU contracted with ACI Worldwide to provide hardware and software to drive its own ATMs with a TCP/IP connection to NYCE. Contrary to popular belief, the CU was able to get a real-time connection with NYCE despite all its problems. However this did not re-establish communications to MCU’s proprietary ATMs. Prior to Sept. 11, the CU’s proprietary ATMs supported 23% of all the CU’s members ATM transactions. This additional transaction volume moved to NYCE ATMs. That extra volume along with the intermittency of a dial-up network connection caused transactions to time out. Once transactions timed out, the NYCE network became overwhelmed causing NYCE to generate reversal transactions as it started to `stand in.’ It had to stand in on about 50% of transactions said Grant. “NYCE has certain `stand in’ standards. If they send a transmission to a financial institution and don’t get a response within 10 seconds, they are going to reject the transaction,” said Kam Wong, CFO and Chief of Staff for Municipal. The CU made the now famous decision of telling NYCE if it didn’t get a response from the CU in that 10-second timeframe, to give the money to the members anyway. By Oct. 1, AT&T was able to implement the microwave solution to re-connect the CU’s branches and its proprietary ATMs. That alleviated some of the pressure on the NYCE network, but it was still standing in 50% of the time in peak hours. Municipal did take a proactive technology step once it saw that some members were abusing the ATMs. It started creating files to send to NYCE to flag those accounts that were seriously overdrawn as “hot cards”. That meant that NYCE would not authorize any transactions for those cards. It’s a decision Porter said he would do again if the same situation came up. “The idea that we we’re going to have $15 million in losses going against the bond fund was crazy. There was no fraud here. We knew we couldn’t claim anything,” said Porter. He said management and the board knew they were taking a risk, but that it was worth it under the circumstances. Of the $15 million overdrawn, the CU has already recovered $9 million and expects to recover $12 million by the end of the year. Things have changed however on the disaster recovery side. The credit union has built a complete redundancy system in New Jersey that CU officials say would keep the CU down for maybe an hour and a half if the same scenario arose. “There’s a redundancy host system. A redundancy system for processing ATM transactions. We have a complete other set of equipment,” said Grant. Grant said the new set-up forms a communications triangle, so if any one leg of the triangle is broken communication can still occur. [email protected]