For most of 2013, it looked as if a robust housing recovery was underway. Since housing is a huge part of the overall economy, that bred hope for stronger growth in 2014, along with the hiring and spending that ought to come with it.

But housing has the blahs in 2014, prompting concern that the 2013 spurt may have been a false spring driven by temporary factors or a misreading of events. The latest data show sales of new homes down 13% year over year and sales of existing homes — which is most of them — down 8%. Applications for new mortgages recently hit a 14-year low, as potential buyers seem to be backing away.

The pullback led Federal Reserve chair Janet Yellen to say in Wednesday's Congressional testimony, “The recent flattening out in housing activity could prove more protracted than currently expected, rather than resuming its earlier pace of recovery.” That raises the prospect of a change in Fed policy if an unforeseen housing downturn materializes. The Fed has been winding down its controversial "quantitative easing" policy, which helped pushed interest rates to record lows during the past five years. But Yellen has said all along the Fed may reverse course if new developments give it a reason to.

Nobody disputes that housing has cooled in 2014. The question is whether it’s a temporary chill or the beginning of a deep freeze. “The pace of the recovery is slowing, but housing overall is doing pretty well,” Spencer Rascoff, CEO of research site Zillow (Z), tells Aaron Task in the video above. Rascoff predicts home-price appreciation, which was in the double-digits for most of 2013, will slow to a more normal levels of 3% to 4%, indicating a reasonably healthy market.

A few prominent investors, however, feel the housing bust may have paused but never ended — and is going to get worse “Singe-family housing is overrated,” Jeffrey Gundlach, CEO of investing firm Doubleline Capital, said during a presentation at the recent Sohn Investing Conference in New York. “Where are the first time buyers? The kids are not alright.” Real-estate developer Sam Zell said recently that he expects the homeownership rate — which peaked in 2005 at 69.1% and is currently about 65% — to fall to 55%, which would be the lowest level since the early 1950s.

Here’s the basic rationale behind a negative view of housing:

Last year’s recovery wasn’t real. Home values finally bottomed out and started to rise in 2013, as sales picked up. But much of that activity was driven by investors buying homes at fire-sale prices to hold onto and rent out, profiting from the rental income. Investor purchases began to trail off beginning last summer, as rising interest rates made such investments less profitable. With investors retreating, there may not be enough demand from ordinary buyers to support price gains throughout 2014.

Homes aren’t as affordable as it appears. Affordability isn’t as good as it was early last year, on account of rising prices and interest rates. Yet it’s still much better than it was during the peak years of the housing boom in 2005 and 2006, as the chart below shows. That ought to mean it’s still a good time to buy a house. Gundlach, however, argues that, back then, exotic and poorly underwritten mortgages (many of them headed for default) made affordability look much worse than it was, which means affordability today isn’t nearly as good as it looks. Without fishy mortgages, far fewer people can afford to buy a house.

Would-be first-time buyers have no money. The housing market depends on young buyers to keep demand strong and snap up homes as older owners move out. But young Americans may be under the most financial stress of all. Many recent college grads have thousands of dollars in student loans and weak job prospects, which means they’re not leaving their parents’ basements anytime soon. Those living on their own — and renting — have probably noticed rents are rising sharply, making it harder to save for the down payment on a home. And fewer people are moving around, which dampens demand for homes even more.

Mortgage rates are expected to rise. With rates at or near record lows for much of the past five years, the only direction they seem likely to go is up. Some economists are surprised rates haven’t risen by much this year, given that the Fed has been pulling back on easy-money policies that pushed rates down in the first place. But they also note that, when interest rates do rise, they tend to rise quickly, transforming market dynamics in a hurry. Given that a one-percentage-point rise in 2013 seems to have been enough to puncture home sales, another one-point rise, or worse, could seriously impair affordability. Even then, rates would still be close to historical averages — but too rich for many potential buyers.

Real estate is largely a local concern, as most people ought to know, and there are several markets, largely on the coasts, likely to buck these discouraging trends. The nation as a whole might, too, if the economy grows more than expected, banks become surprisingly loose with lending, or incomes — which have been stagnant for years — begin to rise for some unforeseen reason. Otherwise, it might be prudent to hunker down in the basement, if the twentysomethings will yield some turf.

Rick Newman’s latest book is Rebounders: How Winners Pivot From Setback To Success. Follow him on Twitter: @rickjnewman.