Newsletter

The Signal and the Noise – 108

The Signal and the Noise: Why So Many Predictions Fail – But Some Don’t by Nate Silver

Nate silver writes on his blog http://fivethirtyeight.com/ about sports and political forecasting.

Most economists try to predict too accurately and are too confident about their skills.
Every prediction always needs the proper assessment of a human being.
You can use Bayes’ theorem to account for errors in your own predictions.

describes the probability of an event, based on prior knowledge of conditions that might be related to the event. For example, if cancer is related to age, then, using Bayes’ theorem, a person’s age can be used to more accurately assess the probability that they have cancer, compared to the assessment of the probability of cancer made without knowledge of the person’s age.

We are not very good at predicting the future.
Biases decrease accuracy of forecasting. Sometimes the bias is from special interests.
Having a pre-existing narrative or political partisanship bias negatively impacts forecasting accuracy.
The author discussed several examples such as predicting the stock market, poker, weather, earthquakes and other natural phenomenon, and terrorist/economic/political events.
I feel the perfect back-to-back reading is to read with this book –
Superforecasting: The Art and Science of Prediction by Philip E. Tetlock, Dan Gardner
Also relevant is the book –
Ego Is the Enemy by Ryan Holiday
Like the author, Ryan Holiday mentions how the ego can  be ones downfall.
The signal is a metaphor for the correct data.
The noise is a metaphor for inaccurate data and other irrelevant information that misleads and causes predictions to fail.
Prediction is saying that a specific thing (usually with a level of severity) will happen at a specific time.
Forecasting is saying that an event has a statistical likelihood of occurrence within an approximate time frame.
Use common sense and human judgement in forecasting as well as math and statistics. Example is baseball scouts and statistical analysis of players performance.
Reference: Moneyball: The Art of Winning an Unfair Game by Michael Lewis
With the advent of the Internet and big data, the shear volume of data has increased exponentially. This makes it harder to separate the signal from the noise. The internet contains more data now but there’s no guarantee it is correct data.
People are by nature pattern seeking creatures. Often times we see patterns where there are none.
Causation vs. correlation
Occam’s razor – Among competing hypotheses, the one with the fewest assumptions should be selected, or, all things being equal, the simplest solution tends to be the correct one.
False positives are as dangerous if not more so than false negatives. For example, the odds of a test being wrong can be greater than the odds of having the condition to begin with. Example, getting cancer may be 1% while a false positive for having cancer may be 10%. This was actually seen with breast and prostate cancer.
The boy crying wolf syndrome where forecasting is ignored then something bad happens.
“The fox knows many little things, but the hedgehog knows one big thing”.
To predict the behavior of a system requires a thorough understanding of it. Weather, stock market, political predictions, earthquakes. These complex systems have so many moving parts, makes it nearly impossible to predict with perfection. The farther out in time one tries to predict, the less accurate the prediction becomes (and quickly).
Weather programs on TV predict on the “wetter” side because if they are wrong and you get wet, people are pissed while if they predict rain and you get sun the people are pleasantly surprised. There is bias in predictions, even weather.
earthquakes and terrorist attacks follow a power law distribution – smaller events occur more frequently and significant events occur infrequently.
Stock market example – if there is a pattern that can be identified by people, they will take advantage of it and effectively work it out of the system (make it disappear). I held this theory for a while then heard the author describe something similar. The market is driven to total chaos and unpredictability because of this – noise injected into the system by human behavior.
How bubbles work – people stay in the market too long and don’t know when it will pop. You know when you are in a bubble because you can’t believe prices keep rising. But FOMO keeps you in the market and it becomes a game of chicken.
Predictions can be self-fulfilling (elections) or self-canceling (flu)
Overfitting (too much noise) vs. underfitting (not enough data) data. http://docs.aws.amazon.com/machine-learning/latest/dg/model-fit-underfitting-vs-overfitting.html
There is always some uncertainty in models and forecasting
model accuracy depends on our understanding of how the specific situation works, complexity of the situation, and time
timing the stock market is rarely profitable due to following wrong trends and being charged fees. Investing long term is more profitable, i.e. buy and hold or using index funds. Use low fee index funds that track the market (i.e. S&P). Tony Robbins in Money, Master the Game agrees with this philosophy.
80/20 rule, one can get good at something quickly. law of diminishing returns.
In some predictions one is competing with the model, in others they are also competing with other forecasters. (poker example)
If one is competing against other humans, heuristics and strategies used can be used against them. Not applicable to non-human models such as weather and earthquakes.
Inside view – considering all the factors related to one particular model and forecast
Outside view – considering factors related to several instances grouped by similarity
Past performance is not a predictor of future events. (stock market and investing)
In seismology, the Gutenberg–Richter law (GR law) expresses the relationship between the magnitude and total number of earthquakes in any given region and time period of at least that magnitude. http://en.wikipedia.org/wiki/Gutenberg%E2%80%93Richter_law
No other forecasting has been able to reliably beat the accuracy of the GR law because more complicated models overfit data. Simplest is usually best. Same for simple mode of baseball player vs. age compared to more complex models.
Weather forecasters have access to a vast amounts of data, and weather happens constantly which provides them rapid feedback loops that allow them to repeatedly test their hypotheses. Same for baseball.
The combined use of modeling the system and human judgment does notably better than modeling alone (for weather and baseball).

The more famous a political pundit/expert is the more likely they are to be incorrect on average.

Averaging across individual experts’ forecasts provides better forecasts than the average for any one individual, the difference being about 15-20%

Spaghetti model for hurricane tracking.

Some experts are better than others. Experts who do better tend to be multidisciplinary, pursue multiple approaches to forecasting at the same time, be willing to change their minds, offer probabilistic predictions, and rely more on observation than on theory.

2017-06-02T03:06:10+00:00