The ruling issued Friday by the U.S. Appeals Court for the District of Columbia overturned a lower court's decision in July that favored the merchants and was a setback for banks.
In the July ruling, a federal judge struck down the Fed's cap on so-called "swipe fees," saying the Fed didn't have the authority to set the limit the way it did in 2011, improperly including data that made the cap too high.
The retail groups had sued the Fed over its setting the cap at an average of about 24 cents per debit-card transaction.
Congress mandated a ceiling on debit-card swipe fees as part of the 2010 financial regulatory overhaul. Prior to the cap, fees averaged 44 cents per swipe. The Fed had initially proposed a 12-cent fee limit, and the retailers argued that the Fed buckled under pressure from bank lobbyists when it doubled that level.
The retailers had argued that the Fed deviated from the 2010 law's intent by factoring banks' expenses into the cap that the law didn't allow.
The three-judge panel of the appeals court said that in making that argument, "far from summiting the steep hill, the merchants have barely left basecamp." The judges said they decided to defer to the Fed's "reasonable interpretation" of the law and to reject the retailers' challenge.
The banking industry swiftly welcomed the ruling. "Reasonable minds have prevailed," Richard Hunt, president of the Consumer Bankers Association, said in a statement.
Hunt said the group still believes the mandate for a fee cap is flawed, but also acknowledges that further changes to the fees "would only pile on the negative consequences for consumers."
"Make no doubt about it - consumers must come first in this process, not the bottom-line of retailers," Hunt said.
Spokesmen for the National Retail Federation had no immediate comment.
The Fed's cap is the first limit on debit card fees. Before it took effect in October 2011, banks had negotiated such fees with merchants. A big chain like Starbucks would likely get a better rate than a local coffee shop because it handles more customers. The fees were typically based on a percentage of the purchase price.
In addition to the National Retail Federation, the suit against the Fed was brought by the National Association of Convenience Stores; the National Restaurant Association; the Food Marketing Institute; Boscov's Department Store, a chain of 40 stores based in Reading, Pa.; and Miller Oil Co. of Norfolk, Va., which operates convenience stores and gas stations.
Miller Oil has said it used to pay about 16 cents per transaction in debit swipe fees.
The Fed rule doesn't apply to credit cards, government-issued debit cards, prepaid cards or cards issued by banks and credit unions with assets under $10 billion.