(AP) -- The Justice Department's rejection of AT&T's proposed purchase of T-Mobile USA will test new federal guidelines on challenging mergers and the companies' resolve in forming the nation's largest wireless carrier.

A courtroom battle is likely and could wring out information that the companies would prefer to keep private. Still, AT&T Inc. has a big incentive to fight: If the deal is called off, the company has to pay a $3 billion breakup fee and surrender some of its unused spectrum for wireless communications.

AT&T is promising to fight the Justice Department's decision. The department filed a lawsuit Wednesday to block the $39 billion deal, saying it would reduce competition and lead to price increases for customers.

If AT&T follows through on that, it could produce the biggest antitrust showdown since business software maker Oracle Corp. squared off with the federal government seven years ago. That dispute, triggered by the government's decision to block Oracle's proposed purchase of rival PeopleSoft Inc., exposed several well-kept corporate secrets and required Oracle CEO Larry Ellison to testify before a packed courtroom.

In the end, Oracle pulled off something few companies have done in the past 30 years: It persuaded a federal judge that the didn't have grounds to block its PeopleSoft deal. Oracle closed its $11.1 billion takeover four months after getting the favorable court ruling.

Usually, not even the most powerful companies bother to fight government regulators in an antitrust dispute. Google Inc., for example, backed off in 2008 when the Justice Department threatened to sue to block a proposed Internet search partnership with Yahoo Inc. Microsoft Corp., the world's largest software maker, pulled out of a deal to buy Intuit Corp. in 1995 after the Justice Department objected.

The Justice Department filed 138 antitrust cases in federal courts from 1999 to 2008 and lost just four of them, according to the latest breakdown from the agency.

One reason that the Justice Department has such a good track record is because it rarely challenges a deal unless it's very confident it can win, said Joseph Bauer, a University of Notre Dame law professor and antitrust expert.

Knowing AT&T would probably go to court, the Justice Department may have wanted to signal that it intends to get tougher on corporate marriages between rivals in markets with few other competitors.

A union between AT&T and T-Mobile USA would leave Verizon and Sprint as the only other major cellphone carriers in the U.S. , a subsidiary of German telecom company Deutsche Telekom AG, is currently the No. 4 wireless carrier, while AT&T is second. Combined, AT&T would be the largest.

In a sign of its confidence, the Justice Department decided to strike down the deal even though it could have taken about three more months to study the pros and cons. The timing stunned AT&T, which said it didn't get any advance warning.

"It was an aggressive and impressive move by the DOJ to take the battle right at AT&T," said Daniel Wall, a San Francisco attorney who represented Oracle in its 2004 fight to win the right to buy PeopleSoft. "It sent a statement that the DOJ intends to fight this one all the way to the finish line."

Wall said AT&T may have a tougher time proving its case than Oracle did against the Justice Department. In the PeopleSoft deal, Wall said, antitrust enforcers seemed to be manipulating the definition of the business software market. "This time, it looks to me that they have a pretty solid market definition," Wall said. "They don't appear to be playing games."

University of Iowa law professor Herbert Hovenkamp said the Justice Department is being guided by a set of new guidelines, issued late last year, which make it clearer when mergers should be challenged on antitrust grounds.

"I don't think they are overreaching here," Hovenkamp said. "If there is a broader message here, it's that the government intends to enforce these new guidelines."

Besides being forced to divulge potentially damaging information, AT&T will face other risks if it doesn't settle with the Justice Department. Going to trial will take months, or even years, leaving the company in a legal limbo that could depress its stock price and cause customers and key employees to defect.

There's another risk to going to trial: as they try to prove their case, antitrust lawyers sometimes obtain confidential e-mails that contain embarrassing snippets and present other evidence that can make companies look bad.

Those are some of the reasons why AT&T mayl try to reach some kind of settlement with the government.

If AT&T persists, antitrust experts said that it's better off going up against the Justice Department than the Federal Trade Commission, which also handles antitrust reviews. That's mainly because lawsuits with the Justice Department are contested in federal courts. By contrast, the threshold for the FTC to block deals is generally lower, and the ensuing legal skirmishes occur in administrative law proceedings that drag on longer.

"The merging parties usually have a better shot when they are going up against the DOJ than the FTC," said D. Daniel Sokol, a University of Florida professor specializing in antitrust law.