The S&P 500 energy sector (.SPNY) was down 4.2%.
The dollar fell alongside U.S. Treasury yields on fears the U.S. economy could slip into recession after Powell, in testimony to the U.S. Senate Banking Committee, said higher rates are painful but are the means the U.S. central bank has to slow inflation.
"Like all Fed commentary, there are positives and negatives, but the overall message is the Fed is not backing away from rate hikes," said Tim Ghriskey, senior portfolio strategist at Ingalls & Snyder in New York.
The Fed recently raised its benchmark overnight interest rate by three-quarters of a percentage point - its biggest hike since 1994.
The Dow Jones Industrial Average (.DJI) fell 47.12 points, or 0.15%, to 30,483.13, the S&P 500 (.SPX) lost 4.9 points, or 0.13%, to 3,759.89 and the Nasdaq Composite (.IXIC) dropped 16.22 points, or 0.15%, to 11,053.08.
Investors are continuing to assess how worried they need to be about central banks potentially pushing the world economy into recession as they attempt to curb inflation with interest rate increases.
Minutes from the Bank of Japan's April policy meeting released Wednesday showed the central bank's concerns over the impact the plummeting currency could have on the country's business environment.
The Japanese yen strengthened 0.27% versus the greenback at 136.24 per dollar.
In Treasuries, benchmark 10-year notes rose in price to yield 3.156%, from 3.305% late on Tuesday.
U.S. crude fell $3.33 to settle at $106.19 a barrel while Brent dropped $2.91 to settle at $111.74.
U.S. President Joe Biden called on Congress to pass a three-month suspension of the federal gasoline tax to help combat record pump prices.
Spot gold added 0.3% to $1,838.03 an ounce.