Reproducibility and design

Duration ca. 3 hours

No research paper can ever be considered to be the final word, and the replication and corroboration of research results is key to the scientific process.

However, some recent observations do suggest that the quality of research output in some areas is not as strong as it should be. See here for details of the Amgen study mentioned in the lecture. A second Nature report can be found here. This explains why scientists and journal editors must work together to ensure that eye-catching artefacts are not trumpeted as important scientific insights.

In studying complex entities, especially animals and human beings, the complexity of the system and of the techniques can all too easily lead to results that seem robust in the lab, and valid to editors and referees of journals, but which do not stand the test of further studies.

The journals Nature and Ciência Hoje have both published articles about the worrying extent to which research results have been found wanting in this respect.

Nature has developed a checklist intended to prompt authors to disclose technical and statistical information in their submissions. Go here to download a copy.

Read these articles and then discuss as a group how you think these problems apply to your own fields of research.

As individuals, write a short half-page report itemizing how these conclusions might apply to your research area specifically, then send this to me.

What happened next?

Duration ca. 1 hour

Learning objectives:
Summarising complex documents,
Using online resources to identify key areas of scientific progress.

  1. Use the main summary you created in Exercise 1. Find out what happened next using Google/PubMed/Scopus/WoS.
  2. Categorise and list which you think are the most important recent advances that link back to the science highlighted by the press release and identify key papers in these areas that have been published since the date of the press release.
  3. Make a brief comparative statement about how useful you found the four research resources used in step 2.
  4. Write your own short, updated press release based upon on one or more of the areas of advancement you have identified. [This can be a lot shorter than the original!]
  5. Summarise your summary further as a tweet [140 characters max]
  6. Send a copy of your work to me.

Optional additional activities

Experimental design

Total duration – ca. 3 hours.

First read the following five full-text articles linked to below:

“Bias is an inescapable element of research, especially in fields such as biomedicine that strive to isolate cause–effect relations in complex systems in which relevant variables and phenomena can never be fully identified or characterized. Yet if biases were random, then multiple studies ought to converge on truth. Evidence is mounting that biases are not random. A Comment in Nature in March 2012 reported that researchers at Amgen were able to confirm the results of only six of 53 ‘landmark studies’ in preclinical cancer research.”

“Empirical study of the quality of animal experiments is an emerging field, but it does suggest that all is not well. The most reliable animal studies are those that: use randomization to eliminate systematic differences between treatment groups; induce the condition under investigation without knowledge of whether or not the animal will get the drug of interest; and assess the outcome in a blinded fashion. Studies that do not report these measures are much more likely to overstate the efficacy of interventions.” See main article here.

“Clinical research on important questions about the efficacy of medical interventions is sometimes followed by subsequent studies that either reach opposite conclusions or suggest that the original claims were too strong. Such disagreements may upset clinical practice and acquire publicity in both scientific circles and in the lay press.” See main article here.

“Genomic medicine is poised to deliver a broad array of new genome-scale screening tests. However, these tests may lead to a phenomenon in which multiple abnormal genomic findings are discovered, analogous to the “incidentalomas” that are often discovered in radiological studies. If practitioners pursue these unexpected genomic findings without thought,
there may be disastrous consequences.” See main article here.

“When a study of the genomes of centenarians reported genetic variants strongly associated with exceptional longevity, it received widespread media and public interest. It also provoked an immediate sceptical response from other geneticists. That individual genetic variants should have such large effects on a complex human trait was totally unexpected. As it turned out, at least some of the results from this study were surprising simply because they were wrong.See main article here.

Now, as groups, answer the following questions located here. You may need to do some additional research in order to do this.