About those Statistics
Posted by greg2213 on March 21, 2010
From Watt’sUpWithThat –
The quote in the headline is direct from this article in Science News for which I’ve posted an excerpt below. I found this article interesting for two reasons. 1- It challenges use of statistical methods that have come into question in climate science recently, such as Mann’s tree ring proxy hockey stick and the Steig et al statistical assertion that Antarctica is warming. 2- It pulls no punches in pointing out an over-reliance on statistical methods can produce competing results from the same base data. Skeptics might ponder this famous quote:
“If your experiment needs statistics, you ought to have done a better experiment.” – Lord Ernest Rutherford
Also in the article is part of the Science News article and a link to The Reference Frame Defending statistical methods
Leif Svalgaard comments:
It’s science’s dirtiest secret: The “scientific method” of testing hypotheses by statistical analysis stands on a flimsy foundation
Clearly not written by a scientist. We validate a hypothesis by its predictions or explanatory power or even ‘usefulness’ [even if actually not the correct one – e.g. the Bohr atom]. Statistics is only used as a rough guide to whether the result is worth looking into further. Now, if a prediction has been made, statistics can be used as a rough gauge of how close to the observations the prediction came, but the ultimate test is if the predictions hold up time after time again. This is understood by scientists, but often not by Joe Public [his dirtiest secret perhaps 🙂 ].
One of the issues in climate “science” and all the arguments that go back and forth is about the meaning of short term trends. Bascially, do they mean anything other than, “nice weather this year (and in Spokane, it is) …?”
So statistics can be used to tease all sorts of trends and numbers out of the data (regardless of the quality of the data.) So here’s a comment from the same thread that, I think, really illustrates the whole point:
steveta_uk (04:31:35) :
After reading some of the “random walk” posts recently, I thought I’d try a little experiment, which consisted of writing a bit of C code which generated pseudo-temperature records, based on a random +- 0.1 annual deviation from the previous year, centered around 15C, and with a bias factor that made the temperature drift towards 15C if it starts drifting away.
So this 15-minute job produced 10,000 years of temperature records which I imported into a spreadsheet and drew some pictures.
There’s basically with a boring average close to 15, and lots of apparent noise between 13 and 17C. But zoom in a bit, and you see features like little ice ages, medieval warm periods, “hockey stick” features, and all sorts.
And apply some of the trend analysis functions to selected parts of the “noise” and it finds all sorts of things.
And it’s all random.
From another comment. Take a busy street, with traffic moving nicely. Consider crossing the street (J walking) and ask yourself some questions. Can you get a definite answer?
- Is it dangerous to cross that street? Most of us might say something like, “hell yes.”
- Will I be killed if I cross the street (make some assumptions about how, when, speed of crossing, etc.?) The answer isn’t yes or no, it’s only something like, “Possibly, yes.”
- Change the previous to Will I be killed or injured… Then the answer still can’t be yes or no, but is still, “possibly, yes” though it’s more likely than the previous.
- You are now across the street. Are you dead, yes or no?
So really, 2 & 3 are statistical answers while 1 & 4 aren’t. While the answers to 2 & 3 aren’t certain (ie: yes or no) they are correct. The point, obviously, is that the answer really depends on the question and all the assumptions around that question.
Here’s a great example, from another comment.
Smokey (07:01:15) :
So yeah, while we need the statistics and while they can provide useful info, they aren’t the only thing and can certainly be used to excess. It’s put very nicely in that old quote, “Lies, damn lies, and statistics.”