Monday, January 27, 2014

Math's for Winners - Regression Analysis and an Upset Status Quo

I tend to make some people really mad because of my affinity for data. If in doubt, just look at the feedback I received from a reader to my previous post. He was pretty worked up by the fact that I used statistical distribution to show that his experience agreed with mine. He even went as far as calling me a bourgeois; implying that my methods were snobbish. This comment surprised me since the term bourgeois describes business oriented middle class people instead.
image with various chart types of statistical distributions
Different Statistical Distributions
To him, my use of data destroyed the human side of business. I can see why, but I disagree. As a result he will not be the last one I upset.
To those like him, their experience tells them that data is the flag of the insecure. Data, or numbers, make anyone sound smarter than they are and keeps others form asking further proving questions. Perhaps worse, data also fails on an almost periodic basis. Who can forget the market catastrophe that resulted from Long Term Capital Management's misguided deployment of Nobel Prize winning mathematics? In 1998, the famous algorithms created a global economic meltdown.
Image of the late richard feynman
Late Richard Feynman
But this isn't the only example of high-math. Anyone who has studied the work of the late Richard Feynman knows that math can be a structural part of great progress, for example. As speculative as it may have seemed at the time, Feynman's work continues to prove its accuracy and utility.
To me, what makes data fail is not data itself, but our inability to use it properly. Whether an ignorance of what to measure or of how to interpret what was measured, data can prove to be quite difficult for even the most committed. This is especially true for the many who think that math is something that was left behind as we departed from school and that it should have no place in the real wold of business. But science offers this fantastic tool to help us make desired results predictable and repeatable. We just need to learn how to use it properly.
image of Moneyball book by author Michael Lewis over an old table
Moneyball by Michael Lewis
A great example of this is illustrated in the great book Moneyball; The Art of Winning an Unfair Game by Michael Lewis. Yes, the same Michael Lewis who used to be a bonds trader and who wrote the Wall Street favorite Liar's Poker.
As a science optimist, I very much loved the book. Reading that Baseball has already demonstrated that regression analysis and statistical rigor can be used for unbelievable performance left me as happy as a child in a candy store. And that says nothing about the fact that the benefits can be quantified in real dollars. Baseball teams pay for success with either dollars or math.
The book is not a dry regurgitation of data. No, it is a fun narration by a writer who has demonstrated tremendous bandwidth and an ability to communicate in the most fluid fashion. If you love baseball, you will enjoy the book. On the other hand, if you are religious about the "proper" way the sport of baseball should be played, then this book will piss you off. The dividing line is quite fine.
image from Wikipedia of the chart showing the salaries of the different great leagues baseball teams.
Moneyball 2002 Year; source: Wikipedia
Likewise, I know that my insistence in mathematical rigor is welcomed by those with an open mind and hated by the ones who "know how things should be done". Thus, it is helpful to have a very thick skin and a lot of confidence in one's methods.
In the end, math helps the underdog win. And who in America doesn't like to root for the underdog?

No comments:

Post a Comment