Those higher prices haven't derailed a steadily improving economy. But if they surpass $4 or $5 a gallon, experts fear Americans could pull back on spending, and job growth could stall, posing a potentially serious threat to the recovery.
And the longer prices remain high, the more they could imperil President Barack Obama's re-election hopes.
A few weeks ago, economists generally agreed that the economy was in little danger from higher gas prices as long as job growth remained strong. But fears are now mounting that gas prices could begin to weaken consumer confidence.
The average pump price nationwide is $3.83 a gallon. Energy analysts say it's bound to climb higher in the weeks ahead.
"It's a thorn in the side of the consumer and businesses," said Chris Christopher, an economist at IHS Global Insight. The economy this year "would have been better and stronger if we didn't have to deal with this."
So far, higher prices aren't undermining the economic recovery, which is getting a lift from strong job creation. It would take a big jump — to around $5 a gallon — before most economists would worry that growth would halt and the economy would slide into another recession.
That's because an improving economy is somewhat insulated from any threat posed by higher prices at the pump.