In my last post I claimed that defining the analyses required to prove or disprove an idea is often far more difficult than it might first appear. As food-for-thought, I then suggested you consider how to test the idea that a baseball player’s performance suffers as his weight increases. It later occurred to me that most readers might only have had a moment to do this, and thus never really recognized the difficulties inherent in defining a sound analysis, the very point I was trying to make. So, I thought I’d use this post to share a few thoughts and better illustrate last week’s message.
So, baseball performance as a function of player weight, what to do?
I always like to start by picturing the resulting output. Two charts come immediately to mind.
The first would compare the performance of a particular player (Pablo Sandoval, in last week’s example) as a function of his weight throughout his career, as illustrated here.
While this chart is a helpful start in evaluating the performance vs. weight argument, it leads to more issues than answers. Specifically:
• We have a very small sample size (7 years)
• There are many other factors (e.g., player age) that may be determine performance
• Player performance is not easily defined by a single measure. Here, I use Batting Average, but why not Home Runs? Fielding Percentage? OPS (On Base % Plus Slugging %)? Or WAR (Wins Above Replacement level)?
• Player weight data is very unreliable (it is often over or under stated and may change significantly during the season).
A second analysis we might perform is to examine performance as a function of player weight for a large set of players, as in the scatter plot below.
This analysis solves our earlier sample size issue, and we could choose to filter our data to include only players of equal age to better isolate the relationship between performance and weight. But it still leaves us wrestling with the best measure of player performance. And it actually introduces another complication with weight: since our analysis now includes multiple players, we have to recognize that not all pounds are equal. Two players of equal weight may have gotten there by entirely different means, one by spending time in the gym, the other by spending time at the dinner table.
I could continue, but the point of this post is not to arrive at the best analysis of player performance vs. player weight, but rather to illustrate that most of the analysis we perform is far trickier than it might initially appear.
Our response should not be to take the easiest short cut, or to give up in frustration, but instead to recognize the limits of each source of data we use and each analysis we perform, and to strengthen our overall argument by basing it on a set of analyses that explore an issue from multiple perspectives.