The median $1 billion financial institution pays 0.07% of its assets, or $700,000, on debit processing, Cornerstone said. However, 25% of similar-size institutions actually pay 0.047% for the same services. And institutions in the top quartile pay 0.221% of assets, or $2.2 million per year, it said.

"The network fees charged by Visa and MasterCard for signature debit, for instance, as well as those of PIN networks for PIN debit, are published on rate cards for the whole world to see," the study said. "When it comes to debit pricing, however, the pricing landscape lies in the hands of an opaque oligopoly. Consolidation has brought the bulk of the business into the hands of a few vendors. Those vendors don't publish rate cards, and will insist during contract negotiations that they cannot share pricing information."

That can hit the bottom line hard.

"For the typical $1 billion-asset institution earning a 1% return on assets, the difference between paying at the 25th percentile and 75th percentile – roughly $1.7 million – is equivalent to about 17% of its annual profit. Thought of another way, a bank with a net interest margin of 3% would need to make about $56 million more in loans to achieve the same bottom-line results," the study said.

A difference of five cents per transaction amounts to more than $1.5 million over the course of a five-year contract, Roth noted.

Financial institutions are making money issuing debit cards and are focusing on areas other than payments, and vendors know that, so they use it to their advantage, Cornerstone said.

"They will shift the focus of contract negotiations, offering discounts on core processing in exchange for greater debit processing margins," the report said. "This sounds like a good tradeoff to bankers, brought up in a world where core solutions – the technology that runs the bank and traditionally is the single biggest IT expense – have always been a focal point. For many banks nowadays, however, it's not a good deal at all."

Read more: Learn five ways to spend less on debit card processing …

Banks that saved the most during their negotiations adhered to five best practices, according to the study.

1. They don't auto-renew their contracts. "It's the easy thing to do, and something the vendor encourages for obvious reasons. But it's rarely good for the customer…The market is competitive and there's nothing like a hungry rival to encourage some give on the pricing," the study said.

2. They bundle core and debit processing businesses. "Bringing two large pieces of business to the table instead of one, making them co-terminus – even if the contracts wind up with different vendors – and bidding them out can enhance a bank's negotiating leverage and result in better pricing on both debit and core processing," it said.

3. Their senior managers handle the negotiations. "Making such contracts a priority helps ensure things are done right. It also sends a signal to the vendor that the bank is serious about finding the right pricing," Cornerstone said.

4. They get help negotiating via an advisor. "Bankers shouldn't necessarily strive for the absolute lowest possible price; negotiations shouldn't be cutthroat, but they should be fair," the study noted. "Knowing what other banks are paying for similar services – what the vendors have accepted as fair in other negotiations – allows for a fully informed discussion and decision-making process."

5. They have a technology strategy. "Every successful negotiation outcome started with a sound technology plan," the study said. "In some instances it was to save money with current vendors to reinvest in new technology. In some, it was to improve integration. In others it was to consolidate vendors and drive down costs."