Trading and Exchanges Chapter 22: Evaluating Trading Performance (Part 1)
Chapter 22 should be required reading before anyone picks a mutual fund, hires a money manager, or brags about stock returns at a dinner party. Harris basically proves, with math, that telling skill from luck in investing is almost impossible.
The Core Problem: Skill vs Luck
Portfolio returns depend on two things: how good the manager is and a thousand random factors nobody can predict. Interest rates, economic shifts, some random E. coli outbreak at a restaurant chain that tanks the stock 35 percent overnight. Harris tells the story of Sizzler International, where analysts did careful research, bought the stock, and then a food poisoning incident wiped out their gains. Were they bad managers? No. They were unlucky.
Good performance can come from a skilled manager or a lucky one. Bad performance can come from an unskilled manager or a skilled one hit by random events. As Harris puts it: “A rising tide lifts all boats.” When the market goes up 30 percent, even terrible managers show positive returns.
How Performance Is Actually Measured
Harris walks through two types of performance measurement.
Absolute performance is just the raw return. Portfolio went from 100 to 120, that is 20 percent. Seems simple, but gets tricky when money flows in and out. Harris tells the hilarious story of the Beardstown Ladies, a retirement-age investment club who wrote a bestselling book claiming 23.4 percent annual returns over 10 years. When accountants checked, their real return was 9.1 percent, well below the S&P 500’s 14.9 percent. They probably forgot to subtract their regular contributions. Oops.
Relative performance compares returns against a benchmark. Much more useful. A portfolio that drops 10 percent when the market drops 20 percent actually did great. Up 15 percent when the market is up 30 percent? Actually terrible.
Attribution Analysis: Breaking Down Returns
Analysts decompose returns into three pieces. Market return is what you get just for being in the market. Stocks go up, you participated, congratulations. Market timing return measures whether the manager correctly shifted between aggressive and conservative positions at the right moments. Risk-adjusted excess return (alpha) is what remains after stripping out market exposure and timing. This is the purest measure of stock-picking skill.
Raw Return = Excess Return + Market Timing Return + Market Return. That decomposition tells you where performance actually came from.
Why Past Performance Does Not Predict Future Returns
This is the most important section. Harris lists three conditions that ALL must be true for past performance to predict future performance:
- Past performance must actually reflect skill, not luck
- Those skills must still work in current market conditions
- The manager must still have those skills
If any one of these fails, looking at past returns is useless.
Harris gives a great example with Regulation FD. Before 2000, some managers were skilled at extracting private info from corporate insiders. Then the SEC banned selective disclosure. That skill became useless overnight. Past performance of those managers suddenly predicted nothing.
Research shows essentially zero correlation between top-performing funds in one year and top performers the next year. The only consistent pattern: the worst funds stay at the bottom, usually because of excessive trading and high fees.
The Statistics Are Brutal
Harris does a power calculation that should make every mutual fund investor uncomfortable. Say a truly skilled manager beats the market by 2 percent per year. With 5 years of monthly data, a 95 percent confidence test will correctly identify that manager as skilled only 15 percent of the time. With 10 years, only 23 percent. You need over 20 years of data for a reasonable chance of proving skill statistically. Most people pick funds based on one year of returns.
He brings up Buffett and Peter Lynch. Buffett beat the market by 11.8 percent annually over 36 years, but in his last decade only 6.8 percent. Lynch averaged 12.7 percent over 13 years, but only 5.1 percent in his last five. Harris suggests their early stellar returns might have been luck on top of genuine skill.
The Strongest Case for Index Funds
Harris runs a cost-benefit analysis. Assume one-third of managers are skilled, skilled ones beat the market by 1 percent after fees, unskilled ones lose 2 percent after fees, and index funds trail the market by only 0.15 percent. With 10 years of data, the expected value of trying to pick a skilled manager versus just indexing is only 8.6 basis points per year. Nearly nothing.
The cost of accidentally picking an unskilled manager is high. The benefit of correctly picking a skilled one is modest. Since you cannot reliably tell them apart, the safe bet is the index fund.
The Lucky Winner Problem
The final point is devastating. When you look at the best performer out of 10,000 managers, even if ALL of them are completely unskilled, the luckiest one will beat the market by about 27 percent in a typical year. Over 10 years, the luckiest unskilled manager will still average 8 percent above the market annually. That beats any reasonable estimate of genuine skill.
When someone shows you the top-performing fund out of thousands, they probably found the luckiest person in a very large room, not a genius.
In Part 2, Harris covers benchmark gaming, survivorship bias, and alternative approaches to predicting performance that do not rely on past returns.
Previous: Chapter 21: Measuring Liquidity and Transaction Costs
Next: Chapter 22: Performance Evaluation and Prediction (Part 2)