The growing wealth divide in the U.S. economy is now playing out in the U.S. housing market, and that could be a problem for its future. While many wealthy Americans have been purchasing homes -- increasingly with all-cash deals -- others are having a hard time affording one to buy or rent.

Housing and Urban Development Secretary Shaun Donovan calls it "the worst rental affordability crisis this country has ever known.”

Related: How 10 million underwater mortgages are hurting the housing market

A recent study from the Joint Center for Housing Studies at Harvard University found that almost half of all renters are paying more than 30% of their income in rent--more than double the percentage in 1960.

Lisa Sturtevant, vice president for research at that National Housing Conference, a nonprofit focused on affordable housing for all Americans, explains what's been happening: "A tighter mortgage market is making it increasingly difficult for middle class folks to get home loans, which is pushing a whole group of folks into the renter side, which increases demand on that side and pushes up rents."

The upshot: both lower income and middle class renters are having a hard time, says Sturtevant.

Related: Momentum may be changing in the housing market: Robert Shiller

In contrast, says Sturtevant, there's been "tremendous activity at the higher end," for homes selling for $1 million or more, including second homes. One of the beneficiaries of those sales: Toll Brothers (TOL). The luxury homebuilder Wednesday reported that its net income more than doubled in its latest quarter to $65.2 million due to strong sales and rising home prices. Its executive chairman Robert Toll said he expects sales will rise, and if tight supplies remain, prices could increase rapidly.

Related: Why the housing market is suddenly struggling

But Sturtevant tells the Daily Ticker that home price appreciation overall is "not sustainable" if it continues to be driven by only one segment of the market.

What would help the housing market for the long term, says Sturtevant, are more home construction -- it's been very sluggish -- and more certainty in the mortgage market. She recommends that local governments ease certain regulations in order to increase residential construction and that the federal government clarify its rules for the mortgage market so that banks can make credit more available and "homebuyers can understand what's required of them" to qualify for loans.

The housing market is a cycle that needs buyers coming in at the bottom so others can move up, says Sturtevant. If that chain breaks down out at any point -- because of low supplies, tight lending requirements, or the absence of new buyers due to college debt, low wages or some other reason -- that can affect the whole cycle, says Sturtevant.

Follow The Daily Ticker on Facebook and Twitter (@dailyticker).

More from The Daily Ticker

The bull market is acting like... a bull market

Is the U.S. heading towards a trade war with China?

Harry Potter saves Hogwarts... and Florida