Zen and the art of STM publishing (with apologies to Robert Pirsig - 1)

If quality were a measurable parameter, it would not have a normal distribution (in the statistical sense). At best it would be nominal, i.e. categorical. In a recent PLoS ONE article by Liz Allen et al (2), the authors define a four point scale with the quality of a paper being assessed as “landmark”, “major addition to knowledge”, “useful step forward”, or just “for the record”. But this is, as Pirsig would describe, still a romantic view of reality. Allen et al are after all ranking the publications sourced by their own institute and all estimates of quality are very much in the eye of the beholder.

John Ioaniddis and colleagues (3) have argued, most current published research findings are false. For a research finding to be true, as Liz Allen and her colleagues define it, two things must be true. First, the hypothesis underpinning the study must be correct, and secondly the experimental methodology must be powerful enough to provide a conclusive result. Allen et als paper only reviews cases where they believe both of these test are positive. But there are three other possibilities: that the hypothesis is correct but the design is flawed and yields a negative result (“false negatives”), that the hypothesis is wrong but the statistical result is positive (“false positives”), or that neither is true (“noise”). It is likely that in terms of published articles, the “noise” category is the most numerous, followed by the “false positives”. By contrast, “false negatives” will be quite rare, and articles detailing real progress will be fewer still.

Peter Binfield, Managing Editor at PLos argues (4) that most journals make the peer review process unnecessarily complex and time-consuming by trying to assess whether a paper will have “substantial impact” or “significant advance”, rather than just focussing on methodological rigor., and allowing posterity to be the judge of significance. In other words, the classical STM peer-review process supports the view that quality is a continuous parameter, whereas, in reality, with the benefit of sufficient hindsight, the probabilities of either the design of the hypothesis being correct are either 0 or 1.

So article quality doesn’t end in a decimal point, and if it is deemed to be an important factor, then it should be measured from within the research program that funded the work. After all, as a tax payer, scientific progress means wealth, health and a better world for my grandchildren. Impact Factors, Eigenfactors, and Hirsch Indices aren’t really going to fire me up at the next Election…

Notes:
1. Zen and the art of motorcycle maintenance, Robert Pirsig.
http://en.wikipedia.org/wiki/Zen_and_the_Art_of_Motorcycle_Maintenance

2. Looking for Landmarks: The Role of Expert Review and Bibliometric Analysis in Evaluating Scientific Publication Outputs. PLoS ONE 4(6): e5910
http://www.plosone.org/article/info:doi/10.1371/journal.pone.0005910

3. Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124.
http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124

4. PLoS ONE: Background, future, development and article-level metrics, Peter Binfield
http://conferences.aepic.it/index.php/elpub/elpub2009/paper/view/114/51

Measuring the quality of success

In a recent article in PLoS ONE, Liz Allen, Ceri Jones, Kevin Dolby, David Lynn, Mark Walport, all from the Wellcome Trust, make a good case for believing that bibliometric indicators such as Impact Factors and number of citations are not good ways of assessing the quality of research outputs, as measured independently by experts.

Shortly after their publication in 2005, a team of referees assessed the importance of about 700 papers describing research funded by Wellcome. Four categories of descending quality were identified: landmark; major addition to knowledge; useful step forward; and for the record. Then in 2008 the group compared these gradings with the impact factors of the journals the papers appeared in, the number of citations each paper had received, and, where possible, an independent assessment made by Faculty of 1000.

Significant correlations were seen between the Wellcome Trust (WT) ratings and the Impact Factor of the journal the articles were published in, the number of citations each paper received, and the 2008 F1000 rating.

The correlation with journal impact factor is fairly convincing, but perhaps this does not tell us very much as you would expect authors to know if their research had a chance of being published in a highly cited journal or if it was only worth sending to an unspecialised journal. Which is the independent variable ?

The correlations between WT ratings and the number of citations and the later F1000 rating, however, are more contentious. You could argue that there is a correlation. But alternatively, the results could be skewed by a small number, perhaps a dozen, of truly excellent articles, which do well in any of the grading methods used, but that for the vast majority of papers the relationship between the various measures is random.

True impact is presumably a product of experimental design and the quality of the initial hypothesis. The former can be measured before publication, but the latter can only be assessed with the passage of time. It is a pity such a small proportion of the article cohort analysed went on to be featured in F1000, but perhaps the strongest message that the PLoS ONE article delivers is that progress comes from only a small fraction of the research performed.

Identifying reasons for failure in biomedical research and publishing

The regular assessment of Brazilian scientific output means that individual university departments need to constantly improve the quantity and quality of their scientific output. A significant proportion of this output involves the work of Master’s and Doctoral students, but getting this work published in a suitable journal can often prove to be a challenge. Although students’ lack of fluency in English is a contributing factor, many of the problems observed have an early origin in the formulation of the research problem and its relevance to current research trends in the international literature. In short, more time needs to be spent in the library and less in the laboratory, and more effort needs to be made in teaching students basic research skills such as the effective use of bibliographic databases like PubMed, Web of Science and Scopus.

Follow this link to read the full length paper.