Pseudo-mathematics and financial charlatanism

Apr 10, 2014

Your financial advisor calls you up to suggest a new investment scheme. Drawing on 20 years of data, he has set his computer to work on this question: If you had invested according to this scheme in the past, which portfolio would have been the best? His computer assembled thousands of such simulated portfolios and calculated for each one an industry-standard measure of return on risk. Out of this gargantuan calculation, your advisor has chosen the optimal portfolio. After briefly reminding you of the oft-repeated slogan that "past performance is not an indicator of future results", the advisor enthusiastically recommends the portfolio, noting that it is based on sound mathematical methods. Should you invest?

The somewhat suprising answer is, probably not. Examining a huge number of sample past portfolios—-known as "backtesting"—-might seem like a good way to zero in on the best future portfolio. But if the number of portfolios in the backtest is so large as to be out of balance with the number of years of data in the backtest, the portfolios that look best are actually just those that target extremes in the dataset. When an "overfits" a backtest in this way, the strategy is not capitalizing on any general financial structure but is simply highlighting vagaries in the data.

The perils of backtest overfitting are dissected in the article "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance", which will appear in the May 2014 issue of the Notices of the American Mathematical Society. The authors are David H. Bailey, Jonathan M. Borwein, Marcos Lopez de Prado, and Qiji Jim Zhu.

"Recent computational advances allow investment managers to methodically search through thousands or even millions of potential options for a profitable investment strategy," the authors write. "In many instances, that search involves a pseudo-mathematical argument which is spuriously validated through a backtest."

Unfortunately, the overfitting of backtests is commonplace not only in the offerings of financial advisors but also in research papers in mathematical finance. One way to lessen the problems of backtest overfitting is to test how well the investment strategy performs on data outside of the original dataset on which the strategy is based; this is called "out-of-sample" testing. However, few investment companies and researchers do out-of-sample testing.

The design of an investment strategy usually starts with identifying a pattern that one believes will help to predict the future value of a financial variable. The next step is to construct a mathematical model of how that variable could change over time. The number of ways of configuring the model is enormous, and the aim is to identify the model configuration that maximizes the performance of the investment strategy. To do this, practitioners often backtest the model using historical data on the financial variable in question. They also rely on measures such as the "Sharpe ratio", which evaluates the performance of a strategy on the basis of a sample of past returns.

But if a large number of backtests are performed, one can end up zeroing in on a model configuration that has a misleadingly good Sharpe ratio. As an example, the authors note that, for a model based on 5 years of data, one can be misled by looking at even as few as 45 sample configurations. Within that set of 45 configurations, at least one of them is guaranteed to stand out with a good Sharpe ratio for the 5-year dataset but will have a dismal Sharpe ratio for out-of-sample data.

The authors note that, when a backtest does not report the number of configurations that were computed in order to identify the selected configuration, it is impossible to assess the risk of overfitting the backtest. And yet, the number of model configurations used in a backtest is very often not revealed—-neither in academic papers on finance, nor by companies selling financial products. "[W]e suspect that a large proportion of backtests published in academic journals may be misleading," the authors write. "The situation is not likely to be better among practitioners. In our experience, overfitting is pathological within the financial industry." Later in the article they state: "We strongly suspect that such backtest overfitting is a large part of the reason why so many algorithmic or systematic hedge funds do not live up to the elevated expectations generated by their managers."

Probably many fund managers unwittingly engage in backtest overfitting without understanding what they are doing, and their lack of knowledge leads them to overstate the promise of their offerings. Whether this is fraudulent is not so clear. What is clear is that mathematical scientists can do much to expose these problematic practices—-and this is why the authors wrote their article. "[M]athematicians in the twenty-first century have remained disappointingly silent with regard to those in the investment community who, knowingly or not, misuse mathematical techniques such as probability theory, statistics, and stochastic calculus," they write. "Our silence is consent, making us accomplices in these abuses."

Explore further: Investment bankers lead businesses to better mergers, acquisitions

More information: "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance", by David H. Bailey, Jonathan M. Borwein, Marcos Lopez de Prado, and Qiji Jim Zhu, will appear in the May 2014 issue of the on April 10, 2014. The Notices of the American Mathematical Society is freely available without subscription at www.ams.org/notices .

add to favorites email to friend print save as pdf

Related Stories

Rethinking investment risk

Aug 26, 2013

Financial innovation is supposed to reduce risk—in theory, at least. Yes, new financial instruments based on the housing market helped cause the financial crisis of 2008. But in the abstract, those same ...

Mutual funds have 'invisible' costs, study reports

Feb 27, 2013

(Phys.org)—Investors often rely on the commonly published "expense ratio" of mutual funds as a predictor of performance, but this measure ignores "invisible" costs that occur when funds trade portfolio ...

Women's 'safe' choices hinder future funds

Oct 14, 2013

WA researchers have discovered that women tend to choose lower-risk superannuation investment options than men, further compounding the factors that impact their capacity to accrue sufficient savings for ...

Recommended for you

User comments : 19

Adjust slider to filter visible comments by rank

Display comments: newest first

Nik_2213
5 / 5 (1) Apr 10, 2014
After my first, modest investment's gain was eaten by mild inflation, I was very, very wary of getting an endowment mortgage. Remember the hype ? The linked endowment would re-pay your mortgage at term and give you a juicy lump-sum beside ? I did the math my way, came up short.

Took me a grim hour to convince our mortgage advisor that I wanted a traditional, repayment mortgage. If not from them, then elsewhere. We were the only couple among our family, colleagues and friends to have such.

A decade or so down the line, we were the only family in the area not mired in 'negative equity'...

More than a little paranoia when investing really is a Good Thing.
BSD
3 / 5 (2) Apr 10, 2014
I never trust anyone who wears a suit.
antialias_physorg
5 / 5 (5) Apr 10, 2014
Out of this gargantuan calculation, your advisor has chosen the optimal portfolio.

Ah yes. Reminds me of the guy trying to sell investments to all us employees at the company I work at back in 2012. His opening line was: "over the past 23 years our investments have made x percent profit" (where I forget exactly what x was, but it was some astronomical amount).

However, if someone quotes some weird number like "23 years" at me alarm bells start to ring at the back of my mind.

Why 23 years - I asked myself? What happened 23 years ago? Hmm...that was 1989. Wasn't there a stock market crash in 1989? Pretty easy to calculate enormous profits if you calculate from just after a stock market crash. Let's do the calculations from just one month prior to that...Oh your product made an amazing negative y percent if taking THAT as a baseline. Well I'm sold....NOT.
Noumenon
2.3 / 5 (3) Apr 10, 2014
AA, what it would mean is that over the past 23 years it trended up. As an investment it only matters what is likily to occur in the future, not what happened one month prior to 1989.
antialias_physorg
5 / 5 (5) Apr 10, 2014
"Our investors gained money over the past 23 years."
vs
"Our investors lost money over the past 23.1 years (and longer) or 22.9 years (and shorter)".

If someone is trying to tell you how good they are at investing your money for you then you'd evaluate them differently based on these two statments, wouldn't you?

If I could predict the future I wouldn't need anyone to invest my money for me. But if someone has to choose cherrypicked data to make their product look phenomenal then that's just flashes a big "No thank you" sign for me.
julianpenrod
1.8 / 5 (5) Apr 10, 2014
If the stock market weren't all a fraud, there would be more millionaires. And, face it, no system sold accounts for things like stock manipulations, "creative bookkeeping", dummy corporations, companies buying politicos to pass laws forcing people to buy their product. But, in the end, remember, the system is stratified with levels of individuals controlling larger and larger pools of money. In any finite hierarchical system, there has to be a top level. If there are finitely many people on earth, the New World Order members are at the top. They control all the money, it flows by their whim, not by natural laws! They control the disbursing of funds to bring about the social effects they want. They provide the fraudulent stock market numbers to keep the "rank and file" from getting rich.
TegiriNenashi
not rated yet Apr 10, 2014
Didn't PBS run documentary about it? "The Wolf of Wall Street"
Noumenon
3 / 5 (4) Apr 10, 2014
A) Our investors gained money over the past 23 years.
vs
B) Our investors lost money over the past 23.1 years

If someone is trying to tell you how good they are at investing your money for you then you'd evaluate them differently based on these two statments, wouldn't you?


No,... B only matters to those who invested 23.1 years ago, not to you as a new prospective investor. In investing what is important is the trend, A, not the entire history. Most stock money is made during up and down movements, and moved around more frequently than that.

Also, a stock market crash is not necessarily indicative of a particular stocks worthiness. Generally it effects all stocks regardless of future potential. Proof: there has been several crashes and yet people are still making money in the stock market.
antialias_physorg
not rated yet Apr 10, 2014
It's simple: Does the company have a history of making or losing money for the vast majority of their clients? If they have a history of losing money for their clients, would you invest in them?
I wouldn't.

Of course they will make money for some of their clients when THESE invest at an opportune time...but not as much as one might hope, since you have to deduct their profit margin. So such clients who know about opportune times would be better off investing directly and not via such a company, because in effect such clients are investing in a LOSING company whose losses are just being temporarily offset by extraneous factors and not by their investment acumen...(hah...have been waiting for years to use THAT word)

If I invest I want it to be with someone who has a solid history of delivering on promises. Not whose profits depend on MY ability to foresee trends.
Noumenon
3 / 5 (2) Apr 10, 2014
If they have a history of losing money for their clients, would you invest in them?


If they had a particular investment that trended up during the past decade or longer, then yes. I don't need the entire history of the investment. I need to know only what the trend is. Pick any mutual fund and examine its history, you will see many that will be negative if graphed, say, past a given many years, however this does not mean its a bad investment.
ryggesogn2
1.5 / 5 (8) Apr 10, 2014
""Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance",

Suggestions for a new study:

""Pseudo-Mathematics and Global Warming Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance",

Protoplasmix
not rated yet Apr 10, 2014
"I believe that science is the engine of prosperity; that if you look around at the wealth of civilization today, it's the wealth that comes from science." – Michio Kaku

S C I E N C E
is the E N G I N E
of P R O S P E R I T Y
antialias_physorg
not rated yet Apr 11, 2014
If they had a particular investment that trended up during the past decade or longer, then yes.

Well, in the described case it trended up only if you put that decade (in that case two) between two VERY specific point in time over that timeline. If you you almost any other two points it trended down.

It's easiest if you picture a long downslope with a single spike at some point. If you put the points at the beginning of the slope and at the top of the spike then that might constitute a "rising regression line". But that's not how you do regression analysis (which is what you SHOULD do when evaluating such offers)
Noumenon
2.3 / 5 (3) Apr 11, 2014
I could have let you off the hook had the guy stated, "Our investors gained money over the past 23 months",.... (instead of years), then you would have made a sound point, (unless one is in the market as a speculator).

However, as it was, according to your own statement, it must have trended up for over two decades,.... that's a long time. For example, as the facts turned out, the maximum return on investment would have been made had one invested money shortly after the imagined crash, 23.1 years ago,.... which is obviously much closer in time to the negative history you're concerned about than now.

Noumenon
2.3 / 5 (3) Apr 11, 2014
it trended up only if you put that decade (in that case two) between two VERY specific point in time over that timeline....


Are you borrowing that logic from the global warming "deniers"? Have a look at this graph, .... one could zoom into a graph or truncate one side of it (like the prior 17 years of leveling off argument), and say what ever they wanted to say.

This point is, in making money, it's the up/down movement itself that matters. In fact the speculators start to droll when the market goes all wacky, whether it's stock, property, oil, whatever.

If you [put] almost any other two points it trended down.


This does not correlate with the info you posted wrt '23 of making money', trending up. However, is not important to the point I wished to make.
antialias_physorg
not rated yet Apr 11, 2014
This point is, in making money, it's the up/down movement itself that matters.

With long term investments it's the long term trends that matter. Only.
The only people who have access to the arbitrage gains from short term up/down jitters at the stock market are those with the facility to do nanosecond trading.
And as a private person you're not seeing any of those profits (banks, and other major players who have that kind of ability would be fools to let anyone in on that)
The Alchemist
not rated yet Apr 12, 2014
Here's how the stock market works.

Investors, like you and me, (pronounced "idiots"), put their hard-earned money into the stock-market predictably, usually monthly, in 401k s. (Predictability in the market is a good way to make money). This causes the market to rise, especially with hard-core long term investors.

People with money/interests, (pronounced "used food passages") both buy on the rise and sell short to take away hard-earned money from the idiots. After a good "crash" the used food passages start the cycle over again. It is a great and legal way to steal.

Its randomness, like gambling, keeps the idiots coming back.
Ducklet
1 / 5 (1) Apr 13, 2014
This is how climate change activists come up with predictions like the arctic might be completely ice-free by 2013. Or the ecological economist movement of the early 70s came up with the prediction that due to natural resource mismanagement (blamed, of course, on capitalism) four billion people - among them 75 million Americans - would starve to death in 1980-1984. It's what happens when you rely on unproven models that merely fit past data. One only has to see in the comments around here that there's a sucker born every minute, and the financial advisors and private bankers know it. For every one who laughs at their scheme there are nine who believe in it (and yes, belief is the proper term). Education is orthogonal to the issue and it's only tangential to intelligence - the generic human brain is not rationally skeptic.
ryggesogn2
3 / 5 (2) Apr 13, 2014
"I believe that science is the engine of prosperity; that if you look around at the wealth of civilization today, it's the wealth that comes from science." – Michio Kaku

S C I E N C E
is the E N G I N E
of P R O S P E R I T Y

No, it's not.
Tesla was a great scientist. He moved to the USA to obtain capital (money) and the liberty to pursue his ideas.
Of course this was before 'progressive' socialism infected the USA.