Missing Methods

Daniel Shriner nails it in last week's Science.

Some journals have opted for placing the Materials and Methods section after the Results and the Discussion sections. Other journals have almost completely moved the Materials and Methods section from the main text to online supplements. These journals are conveying the message, however inadvertent, that the sine qua non of the scientific method, the Materials and Methods, is the least important part of a scientific publication.
Reproducing results from another group has become downright impossible in the modern era, and the journals aren't helping matters by marginalizing the section. Besides sending the message that methods aren't important, they're allowing authors to get away with not reporting half of the steps they take.

I mean, have you ever tried to recreate another group's computational analysis, based on a paper alone? Even with the extra info from the Supplementary Materials, it's generally impossible. They leave out key parameters, or neglect to mention entire steps. I'd love to see a simple schema for representing computational methods become widely used. Preferably, it'd be machine-readable, so that the analysis could be recreated semi-autonomously.

At the very least, let's require authors to create batch scripts that run the specified tools with the correct parameters, so that reproducibility is no longer a myth.


Written by Ward -

I'm sure I'm guilty of this. But I agree, any custom code used to produce scientific data should be open source. A shell script to reproduce all experiments would be even better. Ironically I can't read the original article because I'm at home and it's in Science...

comments powered by Disqus