Peer-Reviewing the Software
Shannon Love at ChicagoBoyz has a nice book end to my lengthy post on the Computer Generated Climate Models. He focuses on lack of peer review in the climate software (whereas I focus on the lack of any apparent software discipline, life cycle or code audit). He notes a vital point about how the “weight” of the peer review process is overstated:
Too often these days when people want to use a scientific study to bolster a political position, they utter the phrase, “It was peer reviewed” like a magical spell to shut off any criticism of a paper’s findings.
Worse, the concept of “peer review” is increasingly being treated in the popular discourse as synonymous with “the findings were reproduced and proven beyond a shadow of a doubt.”
This is never what peer review was intended to accomplish. Peer review functions largely to catch trivial mistakes and to filter out the loons. It does not confirm or refute a paper’s findings. Indeed, many scientific frauds have passed easily through peer review because the scammers knew what information the reviewers needed to see.
Peer review is the process by which scientists, knowledgeable in the field a paper is published in, look over the paper and some of the supporting data and information to make sure that no obvious errors have been made by the experimenters. The most common cause of peer review failure arises when the peer reviewer believes that the experimenters either did not properly configure their instrumentation, follow the proper procedures or sufficiently document that they did so.
In other words, peer review isn’t necessarily a 2nd opinion, like an athlete gets before going under the knife. It is meant to be validation of the scientific process, as opposed to an agreement of the data inputs and conclusions.
He goes on to discuss the real issue with the software:
Unfortunately, that attitude has persisted even as software has grown from a minor accessory into the tool that actually performs the experiment. Today many papers are nothing but reports of what a unique piece of software spit out after processing this or that great glob of data. These software programs are so huge, so complex and so unique that no one who wasn’t directly involved in their creation could hope to understand them without months of study.
Just about everyone in business has had the experience of having to puzzle out a spreadsheet created by someone else who left the company unexpectedly. Although we seldom think of them this way, in reality each individual spreadsheet is a custom piece of computer software written in language of the spreadsheet program. (Technically, all spreadsheets are scripts.) Everybody else knows that you can’t trust the output of a spreadsheet just because the person who made it tells you, “It’s done in Excel.” To trust the output, you either have to compare it against known good data or you have to look at the individual spreadsheet itself to find any places it might go wrong. People create very complex spreadsheets and then leave and some poor schmuck gets stuck trying to figure out what the %$#@! the creator of the spreadsheet was trying to do.
I’ve been lucky enough (uh…sure) to test and and analyze a production costing system. I had to figure out what it was doing before I could then verify that our add-on module was correctly calculating our industry specific modifications. I used a spreadsheet program as my “backup” or test mechanism. This took me weeks of buddy testing with the developer who was deciphering the base code. We had access to the underlying code and it still took us a very long time to do it. This was mission critical, so we did it. And of course, we were paid to do so. But we knew the target we were trying to hit.
Climate models appear to be based on programming the science into the system and then running simulations and trusting the simulation results. The source code, data and methodology was not made generally available until after the CRU leaks (and even then, its only been promised to be made available). Anyone whose ever played a modern video game knows that it takes teams of world class developers to build a physics engine. But, those physics engines only give the perception of reality. They are not hitting the rules exactly. A virtual world is only going to be as good as the rules defined in its programming. Why are we so certain that climate models that predict the future would be so scientifically sound (when the code hasn’t been peer reviewed or put through rigorous quality control). Our company’s developers often say “sure, we can program anything you want, its just going to take this much time and expense.” Are we really so sure that these computer programs weren’t developed with reaching a desired result?
Lastly, Mr. Love mentions the concept of Open Source Science. I’m all for this but I don’t believe that all scientists should be compelled to share their research since it is their intellectual property (and that of their universities and/or employers). I do, however, feel very strongly that if your research is taxpayer funded AND your research and computer models will compel dramatic government expansion and intervention, then you better be an open book. Your science can be analyzed by anyone, of any specialty, regardless of whether or not they get to hide up in the “peer reviewed” ivory tower.