Scofield
Statistics May Mislead
To the Editors:
Darrell Huff’s classic book, How to Lie with
Statistics, (published in 1954!)gives a wonderful warning against
a classic statistical blunder:
“If you can’t prove what you want to prove, demonstrate
something else and pretend that they are the same thing. In the
daze that follows the collision of statistics with the human mind,
hardly anybody will notice the difference.” (p. 74)
The passage is particularly cogent given the letter that was published
in last week’s Review regarding the performance of the Oberlin
endowment. In this letter, we learn that the stock market had a
seven year (‘94-’01) increase of 193 percent, while
over the same period, the college endowment rose 124 percent. The
obvious implication is that the endowment has done poorly because
it has risen less than the stock market. Furthermore, the letter
points out that when compared to peer institutions, Oberlin’s
endowment has risen at a slightly below average rate. And, of course,
there is a glitzy graph presented to enforce these points.
Huff is probably spinning in his retirement.
There are at least 90 students on campus who could explain the fallacy.
(At least I hope they can, because they are going to be tested on
this sort of issue on their first exam.) For those of you not in
my class, the problem is one of confounding variables, which are
variables not under study that may affect the relationship of interest.
In this case the confounding variables are easy to find. Why? Because
the author of the letter points them out for us. He states that
the endowment is affected not only by investment decisions, but
also by payouts and fundraising.
How could this affect our conclusions? Let’s do an example.
Let’s say that College A spends a bunch of money from the
endowment to build new facilities and has an endowment increase
of 150% over 7 years. In comparison, College B spends nothing and
has an increase of 175%. Which college had better endowment performance?
Which college president was better at managing the endowment? A
reasoned conclusion would consider the amount of the payout(s) and
the impact not only on the endowment but on the school in general.
The naïve conclusion made from the “gee-whiz” graph
is that college B is the superior institution (simply because 175
is bigger than 150). Is that a reasonable conclusion?
The comparison of Oberlin’s endowment to that of peer institutions
based solely on percent increase is simply not an apples-to-apples
comparison (comparing the endowment to the stock market has the
same problem).
Please note that this letter is not about the performance of the
endowment. This letter is about making unwarranted conclusions based
on inappropriate comparisons. My point is that a much more thoughtful
analysis would be needed to say anything definitive about the performance
of the endowment, both in absolute terms and in reference to peer
institutions.
-Brian Schmotzer
Visiting Professor of Statistics
|