How much of a Random Walk is the Stock Market?

A controversial topic for today. Or maybe not controversial at all – we’ll see. This topic has been on my mind for a long time and I’ve mentioned this in passing in some of my posts over the last few months: The stock market isn’t really precisely a random walk! And just for the record: I am not saying that I am in possession of any kind of formula to perfectly predict tomorrow’s equity performance. My guess is just as good as anyone else’s. On a scale from 0 to 10, where 10 is a Random Walk and 0 is perfectly deterministic and forecastable, I’d still call the stock market a 9.9. But there are few peculiar features in the last 140+ years of equity returns that are clearly at odds with the Random Walk Hypothesis. So, let’s look at some of the small quirks I found and what they mean for us in the Personal Finance community…

Observation 1: “Simple Math” endogenous retirement timing lowers the sustainable Safe Withdrawal Rate

As we showed in Part 22 of the Safe Withdrawal Series, when retirees use the “Simple Math” method, i.e., they save until they reach a certain savings target (e.g., 25x annual expenditures) then the sustainable Safe Withdrawal Rates tend to be lower than for retirees who pick a retirement date at random. Here’s a chart from that post:

SWR-Part22-Chart01
The effect of retiring only during bull markets: Lower Safe Withdrawal Rates!

The top panel is the cumulative real total return of the S&P500. The red dots indicate when cohorts reached their savings goal. On the bottom chart is the sustainable withdrawal rate for each starting date, again with red dots indicating the retirement dates. Notice something? Everybody retires during bull markets and nobody retires at the bottom of the notable bear markets (1932, 1975, 1982, 2002/3, 2009). Which in turn means that if you retire in response to reaching a savings goal you’ll experience a noticeably lower safe withdrawal rate than those who retired without using the “Simple Math” timing.

Well, that’s in complete contradiction to the Random Walk Hypothesis, because with a true “memory-free” stochastic process, it doesn’t matter one little whiff how many months of above-average returns you might have experienced right before retirement. A Random Walk has no memory and thus random draws in the future should not be impacted by those in the past. It’s like in the casino: it doesn’t matter how many times red showed up for the upcoming roll of the roulette wheel. But the stock market appears to display some small degree of “memory” because as the bottom chart shows, the endogenous retirement decision reduces our SWRs.

Observation 2: The real cumulative equity return “hugs” the exponential trend line a bit “too closely”

Side Note: For the mathematically inclined, because of positive returns and compounding, the index level itself can’t be a random walk. Rather the logarithm of the S&P500 Real Total Return would be a random walk with drift. Drift, because stocks go up on average, and we use the logarithm of the Total Return to properly allow for compounding.

Let’s look at the actual returns (red line) the exponential trend line in black (here a straight line due to the log-scale vertical axis) and add 100 draws from simulated random walks with the same annualized risk as the S&P (blue lines). The random walks are rescaled so that they all have the same overall average return over the last 147 years to make the paths easier to compare.

RandomWalkChart02
Actual S&P500 cumulative return vs. 100 simulated Randon Walk (plus drift to match exponential trend).

Notice something peculiar? The red line tracks its trend line much more closely than the typical random walk. Random walks can have wildly different growth rates during any of their 30 or 40-year or even 70-year subperiods, while the red line is tracking its trend closely with a few wiggles around the big macroeconomic events (WW1, Great Depression, WW2, 1973-1982, 2001, 2008/9). It’s almost like there is a rubber band that pulls the red line back to the black trend line. And it should! Equity returns don’t rain down from heaven. They are not pulled from a computer random number generator.

Equity returns are tied to macroeconomic fundamentals!

Let’s look at the table from our equity return history post last year. S&P500 Returns largely came from two components: GDP growth and Dividends. The noise is created because earnings sometimes grow faster/slower than GDP and multiple expansion/contraction. Obviously, over really short horizons, stocks and GDP will deviate wildly. Peak to trough, the U.S. GDP dropped by less than one percent in 2001 but the stock market dropped by over 50%. But over long horizons, everything converges back to economic fundamentals. It’s what we call mean reversion and mean reversion is not something you should see in a true Random Walk!

Big ERN Return Formula Table
Real S&P500 (in logs) by decade and by components of Big ERN’s Equity Return Formula.

Observation 3: Consecutive 15-year window returns are extremely negatively correlated

Here’s another observation that doesn’t quite jive with the Random Walk hypothesis: If real equity returns were above average during one 15-year window then they tend to be below average over the subsequent 15 years. Don’t believe me? Check out this chart below. The red line is the trailing 15-year average annualized S&P500 returns (dividends reinvested, CPI-adjusted) and the blue line is the return over the next 15-year window (and thus the red line is the blue line shifted by 15 years to the right). They seem to be negatively correlated! On the bottom half is a scatterplot with the same data and the correlation is a staggering -0.64.

RandomWalkChart03
Returns over consecutive 15Y windows: Time series plot (top) and Scatterplot (bottom): Stock returns display mean reversion inconsistent with a Random Walk!

How is that possible? Of course, it’s related to Observation 2 because if over long horizons (say 30 years, i.e., multiple business cycles) equity returns have to revert to some macroeconomically justified average return, then 15 years of substantially above average returns make it more likely to have 15 years of below average returns after that.

The correlation becomes even more negative when looking at the 1926-2017 time frame (recall that a lot of the other retirement withdrawal researchers, e.g., Trinity, Bengen, use returns only after 1926), see chart below:

RandomWalkChart04
The same chart but over the 1926-2017 time span: Now the correlation is -0.74!

Such a strong negative correlation is not what you’d normally find with truly random draws. For the math geeks, I simulated several thousand true random walk processes and only less than 2% of them displayed such a negative correlation of consecutive 15Y windows. It would have to be a pretty large coincidence that the observed equity returns came from a mathematically clean Random Walk! And I don’t believe in coincidences!

 

So, what does this all mean for Personal Finance? Here are some implications for us practitioners:

Implication 1: Savers shouldn’t stress out over corrections

When you talk to younger folks who are just starting out in their careers and ask them what’s holding them back from investing in stocks they’ll most likely say the fear of an equity market correction. But that fear is completely misplaced if you are still years away from retirement! In fact, listen to the two consecutive ChooseFI podcasts, Episode 34 (Jim Collins) and Episode 35 (yours truly, Big ERN) and you’ll find one almost identical quote, which I could paraphrase as:

“The best thing that can happen to young savers today is a stock market correction”

That quote, I promise, was not coordinated because the episodes are obviously pre-recorded and I had no idea what Jim Collins was going to say in the other recording. And this statement makes sense only because the stock market doesn’t follow a random walk. Remember, with a random walk, the outlook for future returns is independent of past returns. But the cumulative return chart shows that after a correction, the market would not just recover with some lame long-term average return (6.7% p.a. real) but with strong double-digit returns (see the 1980s, 1990s, 2010s!). God Bless Mean Reversion!

So, if you are just starting out saving for retirement and there is indeed a correction, don’t panic, enjoy the ride and pick up some cheap discounted stocks along the way. Make Sequence of Return Risk work in your favor!

Implication 2: Today’s retirees should be concerned about high equity valuations

Well, unfortunately, the flip side of Implication 1 is that the non-random-walk feature also bites you every once in a while. That “rubber band” behavior also works in reverse when equities are expensive (i.e., today!) and we should probably scale back our equity return expectations just a little bit. In other words, we should all recognize the logical inconsistency in the two statements below:

“Retirees have nothing to worry about; the stock market follows a random walk with 8% annual returns and there is no mean reversion after a long bull market.”

“Young savers have nothing to worry about because the stock market is not really a random walk and tends to mean revert after a correction.”

Both statements cannot be true at the same time but you’ll be surprised how often people in the personal finance community follow this wishful thinking of “Random Walk to the rescue” when equities are expensive and “Mean Reversion to the rescue” when equities are cheap. I wouldn’t go so far and predict 4% (nominal!) returns for equities like Jack Bogle did a while ago but I would certainly give that 8-10% equity return that some people are still using a bit of a “haircut!”

Implication 3: Beware of Monte-Carlo Simulations

I have talked about this issue before, most recently in the SWR Series, Part 20. I prefer to do my simulations with historical data (despite some of the disadvantages) because Monte Carlo cannot properly account for the mean reversion properties of equity returns. Modeling both equity and bond returns as Random Walks misses some of the important interactions between stocks and bonds (changing correlations) and the equity “rubber band.” For example, I showed in the SWR Series (Part 20) that the Kitces/Pfau glidepaths that behaved optimally in a Monte Carlo simulation are some of the worst performers when using historical returns.

Implication 4: Being too passive is going to cost you!

OK, maybe I haven’t said anything controversial enough yet, so let me drop a real bombshell now. The non-Random-Walk nature of equity returns means that there is a case for being more active with our financial decisions. More active than what the bogleheads prescribe, more active than what Jim Collins proposes in The Simple Path to Wealth, and more active than what most people in our personal finance community want to deal with. I’m not saying that anyone has to do any of this but here’s some food for thought if you are interested (if you may remember, these are related to items 3-5 in our post about active vs. passive investing):

  • First, adjusting the safe withdrawal rate in response to equity valuations seems to be a way to somewhat soften the effect of Sequence Risk, see Part 18 of the SWR series. When stocks are expensive, for example, because the past 15Y return was high and the next 15Y return expectations are low, then the Safe Withdrawal Rate should be lower. But the opposite is true as well. After a big drop in stocks, you can probably raise the withdrawal rate to significantly above 4%!
  • Second, glidepaths that increase the equity weight help with Sequence Risk, see Part 19 and Part 20 of the SWR series due to the negative correlation between the consecutive 15-year return windows.
  • Third, since stocks don’t follow a precise Random Walk there is room for actively/tactically timing the stock exposure to manage risk. There is a huge benefit to being able to avoid some of the significant drawdowns like 2001 or 2008/9. Of course, nobody will ever be able to precisely time the market and sell equities at the peak and buy them back at the bottom. But even being able to avoid a small fraction of the drawdown can make the difference between a scary shaky retirement and a safe retirement. How do we do this? I’m not going to tell you today because we’re pushing 2,000 words already. But I’m working on a post, potentially a two-part series, to detail some of my thoughts on how to do this. Stay tuned!

We hope you enjoyed today’s post! Please share your comments below!

Advertisements

65 thoughts on “How much of a Random Walk is the Stock Market?

  1. Definitely piqued my interest for how to avoid part of drawdowns. Looking forward to that. And thanks before being willing to go against some of the dogmas of FI and help us tweak at the edges for better returns and retirements.

    Liked by 2 people

    • Ha, good point! I guess we’d have to politely disagree. For I can use your exact methodology to show that young savers should prefer an early bear market, both mathematically and psychologically. It goes as follows:
      Suppose Bogle is right and we get only 4% p.a. for the next 10Y. As a young saver, would you prefer a straight line up with 4% nominal returns or a 30% drop in year 1 and 8.7% p.a. for 9 years after that?
      First, mathematically, savers are better off with the bear market in year 1, no discussion about that. Sequence Risk.
      But even psychologically it might be the better path. Today’s millennials will probably lose faith after 2-3 years of 4% returns and put their money in CDs or spend it all.
      In contrast, they might get frightened by one bear market, but with returns back to 8.7% for the next 9 years, they will certainly get excited about the stock market!

      Also good point about the time-varying volatility. If you were to simulate a random walk with GARCH risk it would make it even harder to push the simulated runs towards the trend line. So in that sense, the observed path is even more extraordinary and appears more non-random-walk.

      Liked by 1 person

  2. I love your graphs and insight.
    Now if you could just tell me the exact date of the coming rubber band snap back that would help me tremendously!
    Just Kidding. I like the comment about reducing some of the drawdowns. I am at only 40% equity right now. I cut back a bit when stocks started looking pricey to me. The bogleheads dismiss me as a “market timer” but I don’t think that is fair. I’m not saying I can time the market. I just reduce the amount of equity when they get pricey. It is more about value than timing. Also, I’m not saying go 100% to cash. That would be foolish market timing. My strategy is as old as Ben Graham and it has served me well through many crashes.

    Liked by 1 person

    • Haha, good point! Since perfect market timing is impossible the best we can hope for is to soften a little bit of the downside.
      Wow, you’re low on equities! I’m taking a different route: Keep 100% equities (roughly) for now and just wait for an exit point.
      Cheers!

      Like

      • I’m about 90% equities and also waiting for an exit point. Not sure where I will exit to, but I’m about 5 years away from retirement and I want to keep the nice gains I’ve gotten. Being this close to retirement, it’s okay if I miss out on some nice gains but not okay if I loose too much.

        Liked by 1 person

  3. Great points! I’ve often questioned the random walk hypothesis as market returns are definitely effected by human psychology. It seems inevitable that periods of market growth result in exuberance for the average investor baiting them into risking more and then the exact opposite happens in a downturn.

    Mean reversion is something I’ve been thinking a lot about the past few years of this bull market. This bull market is scary for me since all of savings has been accumulated during this time and I’ll likely even transition to “semi-retirement” during it as well. Definitely doesn’t bode well for my next 15 years… I guess that’s the benefit of continuing to have some income coming in to smooth the ride in the next downturn.

    How do you think QE will affect this cycle? It seems possible that the high could be higher than in the past from trillions of dollars being added to the economy which is great… until mean reversion. Maybe much lower lows?

    Awesome post as usual!

    Liked by 1 person

    • Well, so far the bull market has digested all the bad news: rate hikes, reversal of QE (quantitative Tightening). I think it’s possible that after many years of macroeconomic repression due to regulations etc., we have now finally unleashed growth again. I think this can go on for another few years before the next recession/bear market!

      Liked by 2 people

  4. I’ve been “feeling” much of what you write about in this post so I’m very grateful to see these ideas unpacked and analyzed. It gives me confidence in my dis-comfort with the notion that the stock market is entirely random. Having a good idea about what may be significant time scales seems very helpful. I look forward to your thoughts on how to apply the 15 year negative correlation concept to real portfolios. Thank you for your work!

    Liked by 1 person

  5. I wanted to add a comment that related to implication 1, using “conservative” when investing screwed me up big time. I consider myself conservative/frugal with money. So I’ve tended to select conservative investments for the last twenty years. I now know, that is a misleading term to say the least. Conservative and aggressive should be “Close to retirement” and “Long time to retirement”. Using the “wrong” terms has kept me in lower returns when I could most afford the risk.

    Liked by 1 person

    • Yeah, this is the problem of trying to reduce very complicated financial terms into a black/white, 0/1, aggressive/conservative metric.
      For example, you can be very averse to short-term fluctuations but then increase your risk of running out of money in the long-run. Who is risk-averse, who’s not?

      Like

  6. For a bit of my career in engineering, I did some control systems work. The basic thing I learned is that when you try to make a move from point A to point B, the faster you try to get there, the more overshoot there is. I think about that when I see the market and people’s reactions to change. I have always wished to have a way to use this in an investment strategy. I’m really looking forward to your follow-up articles. Great post.

    Liked by 1 person

  7. Is it safe to assume, your market timing approach would be based on trend following of long term moving averages? Not sure if you saw it in ChooseFI but I wrote a post about that a couple weeks back. Interested in your thoughts.

    Liked by 1 person

  8. Hi,

    The graph comparing 100 simulated random walks to the actual S&P 500 and its exponential trend line is an interesting one: the black trend line “looks” like being entirely dependent on its chosen start and end points. Just imagine “sticking” that line to either the 2000 high or 2009 low instead, and you’ll get an entirely different impression on where relative to the mean we currently are. As markets are said to be highly valued right now, intuitively one should think we’re above the trend line, ie. the trend line should be lower. What are your thoughts on that? Can you plot that mean using an actual average, independent of current valuation? Or am I talking bullshit and are we actually close to historic mean right now?

    Liked by 1 person

  9. I subscribe to the idea that the market is a random walk in the short term but in the mid to long is not. I believe this is also Schiller positioning.

    I’m also a fan of Monte-Carlo not because I feel they are more accurate then historical. Instead it’s because fundamentally they should be more conservative while historical may be too rosy. After all stock momentum exists but… as much as it decreases a down year up year random it also reduces the likelihood of many downs in a row Monte-Carlo allows.

    Liked by 1 person

  10. Man, you always give us something interesting to chew on! I became a lawyer after previously planning to pursue a PhD in econ (but began to doubt if i had a taste for academia). Your posts simultaneously make me miss those old days where i knew something of econometrics and humble me by making me realize how little skill I really had. Can’t wait for you to post the series!

    Like

  11. Good arguments for the market not being a 100% random walk event. It would be interesting to see your approach to “tactically” timing the market. If there’s even one bit of a chance that one could possible tell the beginnings of a top in the market (to sell) and the bottom (to buy), that would definitely be a nice “active” addition to an otherwise “passive” investing strategy.

    Liked by 1 person

  12. Hi, I really enjoy reading your blog. Have you done any additional analysis to see if the same holds true on a sector by sector basis with hugging the trend line? For example, if telecom has greatly under performed the trend line for a period of time would you expect it to move back to the trend similar to the graphs you used above? I ask because I have had success investing in sectors that have under performed the markets as a whole over 3-5 year periods. Thanks again for all the work you do.

    Liked by 1 person

    • I doubt that this would work on the sector level. The overall stock market is linked to the overall macroeconomy. But there’s a lot of reshuffling inside the economy. Some sectors will grow more some will disappear over time. There’s too much reshuffling and disruption on the micro level going on.
      Cheers!

      Like

  13. I have a hard time with the emotional part of making adjustments in the market. Staying the course and not reacting to what I “think” will happen has worked best for me. With that said, I look forward to your next few posts on this subject. Great post as always!

    Liked by 1 person

  14. Based on eyeballing the regression, if I’m understanding the 15-year charts, they currently seem to suggest a 6%ish real return over the next 15 years (after 7.5%ish over the past 15 years). That wouldn’t be bad — certainly better than Bogle and even higher than Jeremy Siegel (who was at 5% real last I checked.) The problem with a 15-year return, especially for retirees, is that it may obscure quite a bit of sequence risk.

    Like

    • Well, there are pretty big error bands around the regression. But you are right, the last 15 years are not exactly screaming “mean reversion” but then again, all it takes is 5 bad years and 10 good and we got a sequence risk problem…

      Like

  15. Very insightful post Dr. ERN, very interesting statistical analysis of the S&P500. I really like the table with the real GDP and dividends per decade. Eyeballing it, it does look like there is a downward trend in growth and could therefore reduce overall return on investment in the longer term, or am I missing something?

    Liked by 1 person

    • No, you’re putting your finger right on the sore spot! That’s a good point: Real GDP growth has slowed and this looks like equities have a bit less growth potential. And the Dividend yield has dropped at the same time! But that could also imply that earnings per share can expand a little faster than GDP because you’d think that the IRR of reinvested profits is a lot higher than GDP growth. So, my hope is that the EPS/GDP can expand and make up for some of the low GDP growth and low dividend yield!

      Like

    • No, you’re putting your finger right on the sore spot! That’s a good point: Real GDP growth has slowed and this looks like equities have a bit less growth potential. And the Dividend yield has dropped at the same time! But that could also imply that earnings per share can expand a little faster than GDP because you’d think that the IRR of reinvested profits is a lot higher than GDP growth. So, my hope is that the EPS/GDP can expand and make up for some of the low GDP growth and low dividend yield!

      Like

  16. I think the overshoot/undershoot cycle has been obvious for a while (thanks for quantifying it though!). But I’ve also missed out on literally hundreds of thousands in gains with an AA far lower than 100%. Why? Because stocks always look expensive and there’s always a case for mean reversion. Try periods of time other than 15 years for example. Would the sell signals align the same way? If so, you have an actual timing method on your hands.

    The recent rally has me looking again at insuring the portfolio. This would keep me 100% in stocks and prevent an “event” from derailing my FIRE plans just 4-5 years away. The cost is usually too high to do this consistently, so I’d be looking for a hedge-entry signal like the one you found. Options are not priced based on trendline-deviation, so in theory a signal would provide an edge against option pricing computers that literally assume a random walk.

    SPY at-the-money puts expiring in 2.9 years at the $285 strike cost $27.60. The same puts with only 1.9 years remaining cost $22.52. So if nothing else changed (prices and volatility flat, interest the same), you could buy the 2.9 year option, hold it a year, and then sell it for about $5.08 less than you paid. 5.08/285 = 1.8% cost to have insured a portfolio against any loss. Of course, if the market continued to rally, the cost of the insurance would rise because the put would lose more of its value. I’m trying to figure out if this is a good deal or a good way to lose 6-7% of my portfolio.

    Recommendations/humble requests:

    1) Chart rolling avg. returns at 2, 5, 7, 10, 12, and/or 20 year timeframes. Compare signals for consistency.

    2) Calculate how often it would be profitable to pay x% for full portfolio insurance. Is there an algorithm that could tell us the probability of needing that protection? E.g. if the past x years’ rolling returns are y%, there is a z% chance the market will end the year lower than it started and an option would be in-the-money.

    Liked by 1 person

    • I like the idea of long-dated puts! I personally don’t find it easy to time this, though. Because if there is elevated risk the price for insurance will be high (in fact, that implied vol is probably one of the inputs for timing an upcoming recession and downturn).
      So, what I like to do (eventually) is to set up a plan when to tactically sell equities if the probability of a recession/downturn is high. Option buying might get too expensive around that time.

      Like

  17. Dear ERN,

    Good post. Mean reversion in stocks have been known, e.g. http://www.nber.org/papers/w2343.
    But there’s nothing like making the intellectual journey yourself.

    I’m a market timer, through and through. For me it’s all about logical consistency. I cannot reconcile passive investing with static allocation with my understanding of macro, technical and sentiment analysis. I accept that the future is unknown, or is not known with enough precision. It does not follow that all future outcomes have equal probability. There is a wide chasm between “nobody knows the future” and “nobody knows nothing”.

    I had read your series on retirement withdrawal strategies with great interest. I would like to point out that the variable withdrawal strategy has an element of market timing built-in. Coming from me it is not meant to be derogatory, just trying to be logically consistent.

    I look forward to what you find next.

    Best,
    NR

    Liked by 1 person

    • Excellent points! That NBER paper is a good one. I liked Larry Summers a lot more before he became a politician.
      And agree: timing withdrawal amounts is a form of market timing. In fact, even the Politburo-strict passive folks propose you rebalance your S/B portfolio and as the rationale they use – you guessed it – valuation, i.e., buying the cheaper assets and selling the more expensive one. I wonder if they realize that!? 🙂

      Like

  18. Nothing really new here, but you have a knack for explaining and illustrating complicated things in a clear manner, nice job. I totally agree with your assessment of Monte-Carlo, garbage in, garbage out. Let’s see where the whole thing will bring you with the next post — it’s easy to start from this basis (“rubber band”) and derive flawed conclusions about tactical asset allocation… Also, I’m curious, did you read Mandelbrot (Misbehavior of Markets)?

    Liked by 1 person

  19. Your blog is my favorite reading in all personal finance and I am relying on it in my own planning for my wife and myself.

    You have shown that mean reversion is a fact, contra Random Walk purists and confirming Schiller. I am 61 years old and 41% in common equities, 15% in preferred funds; unsure how much of the latter to allocate to “equity,” but the beta is around 50, so I figure about half. (If you disagree, please say so.)

    My retirement is 3 months to 5 years away and I would prefer sooner. So I’m very eager to hear about tactical market timing, as in my mind I’m waiting for a correction in order to go deeper into equity and maybe then retire.

    Liked by 1 person

    • Thanks for the compliment!!!
      Nice! I would assign the preferred funds a half/half weight between stocks and bonds, just as you did.
      Preferreds will be a bit of a rollercoaster ride with interest rates moving up. I have noticed that in my (small) PFF allocation recently. Best of luck! 🙂

      Like

  20. ERN, I am curious if 15 year periods provide the strongest negative correlations? Did you explore other terms between 10 and 20 years to see which periods are most negatively correlated?

    Liked by 1 person

  21. It’s a shame you don’t have a donate button! I’d most certainly donate to this superb blog!
    I guess for now i’ll settle for book-marking and adding your
    RSS feed to my Google account. I look forward to fresh updates and will talk about this
    blog with my Facebook group. Chat soon!

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s