For weeks, a bipartisan group of senators wrestled over the question of how their infrastructure ideas would be paid for. Republicans opposed any sort of tax increase on the wealthy with Democrats just as adamant about gas taxes or user fees.

As a compromise, negotiators did neither and instead cobbled together a range of offsets in the deal unveiled last week which included measures like reducing the tax gap, spectrum auctions, and selling oil from the Strategic Petroleum Reserve.

The 10 senators – as well as President Biden – say their framework will cover the additional $579 billion in spending they propose. But the plan’s skeptics have already signaled that these so-called pay-fors are what they’ll be focused on in the weeks aheads.

"How they've arrived at some of these figures are a bit puzzling,” Rep. Kevin Brady (R., Texas), the top Republican on the powerful House Ways and Means committee, told Yahoo Finance.

Brady said he wants to give a recommendation to his fellow Republicans, but “we just don't have the details to make that determination.”

The Biden administration's public fact sheet doesn’t go into much detail about how much money each offset would generate, while senators have reportedly been given a bit more.

Experts have also raised questions about the math behind other parts of the framework. One example is whether it's actually possible to raise $70 billion by targeting fraud and waste in unemployment benefits.

Nonetheless, in a statement announcing the deal last week, the bipartisan group said the deal “would be fully paid for and not include tax increases.” And on Tuesday, during a stop in Wisconsin, Biden again promised “we’re going to get it all done without raising taxes on a single American earning less than $400,000.”

A focus on the IRS

The provisions focusing on the tax gap have already drawn the most attention in different corners of Washington.

About $40 billion in additional funds for the IRS would allow the agency to better chase down tax cheats and - in net negotiators say - bring in $100 billion dollars into the US Treasury over the coming decade.

Prominent voices in the business world like former Treasury Secretary Larry Summers have been pushing Biden since his election to focus on the tax gap as a way to pay for his programs.

[Read more: The Biden tax hikes are coming into view]

In Wisconsin this week, Biden said the deal would give “the Internal Revenue Service the resources it needs to collect taxes on the wealthiest Americans who owe but are currently not paying under the existing tax rules.”

In a call this week with reporters, Brady said he wasn’t convinced more funds for the IRS would generate as much as promised. “I'm very concerned about and skeptical that that number is real,” he said of the tax gap provisions of the deal. Historically, estimates of potential tax gap savings have been based on outdated data and “frankly are very wild guesses as to what the tax gap is,” he said.

Brady and some of his Republican colleagues are skeptical of adding to the IRS’s power in general. A report published last month showed that the top 25 wealthiest Americans paid little, and sometimes nothing, in federal income taxes, leading to some Democratic lawmakers to vow reforms.

Republican Sen. John Barrasso (R., Wyo.) told Axios recently: "spending $40 billion to super-size the IRS is very concerning... Law-abiding Americans deserve better from their government than an army of bureaucrats snooping through their bank statements."

Brady said recently that “Republicans believe we should collect all the taxes owed to the American government,” but that it needed to be pursued ”without significantly intruding on the privacy of Americans.”

The bipartisan deal doesn't appear to include any new powers for the IRS, but would provide new funding for the enforcement priorities of the agency, which has seen its budget shrink by one-fifth in recent years.

‘They want the higher taxes’

Brady reiterated his pledge to prevent any tax increases. Biden dropped tax increases on the wealthiest Americans from last week’s bipartisan deal but is still promising to make them law in a Democrats-only reconciliation bill expected to move in the coming months.

The only way Brady would support a bipartisan infrastructure package is “if it's not tied to the promise of higher crippling taxes” in a reconciliation bill, he said. House Speaker Nancy Pelosi (D., Calif.) and Senate Leader Chuck Schumer (D., N.Y.) have been consistent: “They want the higher taxes," said Brady. "They want trillions of more spending. That will make it difficult for Republicans to support.”

Ben Werschkul is a writer and producer for Yahoo Finance in Washington, DC.

Larry Summers' tax advice for Biden: Collect on the $7 trillion owed by 'the richest taxpayers

The Biden tax hikes are coming into view

Commerce Secretary on corporate tax rate: ‘Many CEOs felt that 21% was too low’

Read the latest financial and business news from Yahoo Finance

Follow Yahoo Finance on Twitter, Facebook, Instagram, Flipboard, LinkedIn, YouTube, and reddit.