Honest Error: A Look at the Literature

Ordinary negligence plays a role too. The results remain unreliable and irreproducible, but the persons involved may be innocent of intentional wrongdoing. This column looks at the scholarly literature on “honest” errors.

​

Classification Issues

Resnik (2012) explains that recognizing honest error is important but hard:

“It is important to distinguish between misconduct and honest error or a difference of scientific opinion to prevent unnecessary and time-consuming misconduct proceedings, protect scientists from harm, and avoid deterring researchers from using novel methods or proposing controversial hypotheses. … the line between misconduct and honest error or a scientific dispute is often unclear,”

 

Precisely what constitutes honest error may depend on personal judgment. An older study by Nath (2006) in the Medical Journal of Australia looked at ” [a]ll retractions of English language publications indexed in MEDLINE between 1982 and 2002…” and “[t]wo reviewers categorised the reasons for retraction of each article…”. Nath concluded that:

“Of the 395 articles retracted between 1982 and 2002, 107 (27.1%) were retracted because of scientific misconduct, 244 (61.8%) because of unintentional errors, and 44 (11.1%) could not be categorised.”

 

The percentage of unintentional errors suggests surprisingly high rate of unintentional error. While it is possible that misconduct has increased significantly over time (see below for more recent numbers), the more likely lesson here is that it matters how the classification is made. It is hard to know how accurate the classifications of misconduct are under circumstances where the assumption of innocence is not always strictly observed after an accusation has been made.

​

Estimates of Size

Later studies do not confirm the Nath estimate about the number of unintentional errors. An article by Arturo Casadevall (2014) argues that

“Analysis of the retraction notices for 423 articles indexed in PubMed revealed that the most common causes of error-related retraction are laboratory errors, analytical errors, and irreproducible results. … The database used for this study includes 2047 English language articles identified as retracted articles in PubMed as of May 3, 2012…“

 

This suggests that the cause of just under 12% of the PubMed retractions are essentially ordinary human error. A different study by Moylan and Kowalczuk (2016) looks at the BioMed Central journals finds a similar percentage:

“Honest error accounted for 17 retractions (13%) of which 10 articles (7%) were published in error. … A total of 13 articles (10%) of retractions were due to problems with the data. Often these issues occurred through honest error in how the data were handled, for example … although in some cases it is difficult to determine whether honest error or misconduct was the cause. “

 

Daniele Fanelli (2016) offers a somewhat higher percentage of honest error:

“However, retractions reliably ascribed to honest error account for less than 20% of the total, and are often a source of dispute among authors and a legal headache for journal editors. The recalcitrance of scientists asked to retract work is not surprising. Even when they are honest and proactive, they have much to lose: a paper, their time and perhaps their reputation. Much reluctance to retract errors would be avoided if we could easily distinguish between ‘good’ and ‘bad’ retractions.

 

In this case good retractions are generally ones where the authors recognize their own mistake and ask for the paper to be withdrawn. Fanelli (2016) makes the further argument that:

“Self-retractions should be considered legitimate publications that scientists would treat as evidence of integrity. Self-retractions from prestigious journals would be valued more highly, because they imply that a higher sacrifice was paid for the common good.“

 

This could, as he notes, be open to abuse, but some abuse could well be tolerable in the interests of providing an incentive for researchers to withdraw misleading results so that they do not mislead other scholars. Considering present publication pressure and the effect of public opinion, researchers may be unwilling to admit honest errors because they will be thought guilty of misconduct. It may be hard to escape censure regardless of the choice.

​

Greyscale Measurement

One of the measurements that can help define honest error is the degree to which errors confirm the desired conclusions. This is not to say that every error in favor of the authors’ arguments is dishonest, but errors that weaken the conclusion are more likely unintentional. There is of course a human tendency to believe confirming results and to doubt disruptive ones, and a part of research training that may need more emphasis is a healthy skepticism toward desired results. Another form of measurement has to do with the frequency of error. Everyone makes some errors. When authors repeatedly make errors, it may be reasonable to think that the errors follow a standard distribution where some are for and some against the conclusions. A pattern that is consistently in favour of the desired conclusion may imply more bias than honesty.

Those judging integrity should not forget that honest errors exist, and that people under career or social pressure may be more error prone without particular ill intent.

​

References

Casadevall, Arturo, R. Grant Steen, and Ferric C. Fang. 2014. “Sources of Error in the Retracted Scientific Literature.” FASEB Journal 28 (9): 3847–55. Available online.

Fanelli, Daniele. 2016. “Set up a ‘self-Retraction’ System for Honest Errors.” Nature. Available online.

Moylan, Elizabeth C., and Maria K. Kowalczuk. 2016. “Why Articles Are Retracted: A Retrospective Cross-Sectional Study of Retraction Notices at BioMed Central.” BMJ Open 6 (11). Available online.

Nath, Sara B., Steven C. Marcus, and Benjamin G. Druss. 2006. “Retractions in the Research Literature: Misconduct or Mistakes?” Medical Journal of Australia. Available online.

Resnik, David B., and C. Neal Stewart. 2012. “Misconduct versus Honest Error and Scientific Disagreement.” Accountability in Research. Available online.