The S&P 500 started this holiday-shortened week at a record high but small caps have been moving in the opposite direction, a signal of trouble ahead to many market watchers. But at least one veteran investor says it's way too soon to head for the exits. Yahoo Finance Editor-in-Chief Aaron Task spoke with Barry Ritholtz, Chief Investment Officer at Ritholtz Wealth Management about the divergence in the market and whether investors should be concerned.

Ritholtz says, “Once the small caps and the mid-caps lose interest - once people are no longer buying them - it’s just the biggest cap, best-known names, the big weights in the index that are driving markets, and that’s usually a very bad warning sign.” But Ritholtz also says we’re not quite at that point yet.

He still says he’s giving this market the benefit of the doubt. “Trust but verify,” says Ritholtz. “We’re always on the lookout for signs that something untoward might be happening.”

High sign

One of those signs Ritholtz is watching is a dip in new 52-week highs, which have started to come back lately. “Those are the early warnings signs that say 'Hey, keep your eye on this,'” said Ritholtz. “It’s still nowhere near a level that you have to say, ‘All right everybody out of the pool’, but it’s the first crack in the foundation.”

Ritholtz says the market has seen this happen before and it can repair itself, but it could also be a very early sign that the bull market is fading.

White House warning sign

Another thing to keep in mind, according to Ritholtz, is the fact that this is the second year of a presidential term. Historically, that is the weakest year of the four-year term for the market. Putting things in perspective, last year the S&P 500 was up more than 30%, so even if this year is nowhere near those levels, it's not a disaster for the market. “That’s not a sustainable level, so if we’re 30% last year and 5% this year… two 17% years is not so bad,” says Ritholtz.

He also says the market could go sideways a couple more months and finish the year a little positive, a little negative.

How to play it

So, how does a smart investor set himself up in this market? For the long-term investor, Ritholtz says use the pullbacks to establish position. Traders, for their part, want to see a return to the volatility which seems to have disappeared lately.

Overall, Ritholtz believes these are “very, very early warning signs that could either rehabilitate themselves with just the passage of time or the small caps catching up, or this could be the first shot on the bow of the bull market.” But he says it’s still way too early to call this bull over.

Related: Is the 'volatility crash' a new reason to worry?