The Variations of Replication

Abstract

Strong scientific inference requires a repeatable process with replicate measurements and produces reproducible results across a range of variation. [...]there is a never‐ending tug of war between “sameness” of replicates, the range of variation described the replicates and the repeatability allowing generalization of results to a broader and variable context. The challenges for systematic reviews and meta‐analyses that compile results among multiple studies are examined in the context of medical research. Ellison makes the important point that variation is never absent in biological systems, and results from repeated experiments are always likely to be somewhat different. [...]quantifying naturally occurring variation is an important component of experiments.

A Review of Stepping in the Same River Twice: Replication in biological research, Edited by Ayelet Shavit and Aaron M. Ellison. Yale University Press, Hartford, Connecticut, 2017.

In Stepping in the Same River Twice, a philosopher and an ecologist examine the complex notions around replication, repeatability, and reproducibility in ecology—in essence describing the differences and similarities among these features of repetition and their role in drawing conclusions from investigations. As the title implies, even events that appear to be identical will vary; variation among replicates is crucial. Integral to replication is the notion that replicates represent the range of variation over which scientific inference may be made. The editors show that repetition is defined and limited to the spatial and temporal context of any investigation. Stepping in the Same River Twice brings the philosophy of replication together with a wide range of specific examples from biological science that demonstrate the complexity and detail of this superficially simple idea.

The authors demonstrate the intricacy and complexity of ideas surrounding replication in easily accessible language—not a simple task—and the application of the ideas in practice is equally accessible. This book will be an interesting read for non‐scientists that illustrates the complexities that scientists face. Researchers will likely nod their heads in agreement as they read, and be grateful to have all these ideas in one volume. The book will be useful tool for researchers who mentor students and who need to explain issues around replication, variation, and inference. It could easily form basis of a reading and conference course or be used for in‐class discussions.

The book is divided into three sections, each of which contains contributed essays that expand and detail the concepts. The first section is philosophical and illustrates that replication is a fundamental feature that affects everyone. In the second section, each chapter exemplifies the broad concepts in specific investigative settings. Each contributing author unveils replication in a new way, and details of the broader problem illustrate the importance of repetition. In the final section, the editors circle back to challenges with repetition in time and space and provide advice to improve transparency and replicability in research.

Three different aspects of repetition are defined and examined. Replication is a copy of a result using the same tools with the same spatial and temporal scales. Repeatability is the reoccurrence of a spatially and temporally distinct process, and reproducibility indicates the duplication of a result. Strong scientific inference requires a repeatable process with replicate measurements and produces reproducible results across a range of variation. Thus, there is a never‐ending tug of war between “sameness” of replicates, the range of variation described the replicates and the repeatability allowing generalization of results to a broader and variable context.

Natural history collections, which are physical demonstrations of inherent variation present in individuals of the same or closely related species, are the first example of replication. They provide a record of historical variation and allow us to view changes in variation among individuals of a species over time and the potentially observe evolution in action. Monitoring programs, the second example, extend the idea of repeated observations of preserved individuals to repeated observations of dynamic systems over time to identify changes in indicators. This variation over time dictates that monitoring programs identify temporal scales over which normal variation occurs and over which large or smaller variation can be identified as a “change.” Spatial variation among components of a dynamic system (e.g., oak forests or coastal dunes) may also need representation via replicate observations of similar components. Using controlled scientific experiments, the authors demonstrate how the variation among replicates affects the repeatability of experiments. Any single experiment is ultimately contingent on the conditions under which it was conducted. Careful study is needed to determine whether results from two investigations are truly yielding reproducible results. The challenges for systematic reviews and meta‐analyses that compile results among multiple studies are examined in the context of medical research. Ellison makes the important point that variation is never absent in biological systems, and results from repeated experiments are always likely to be somewhat different. Thus, quantifying naturally occurring variation is an important component of experiments.

Beyond its quantification, identifying changes in naturally occurring variation can signal important scientific results. Many research findings focus solely on changes in the average response. However, this volume devotes an entire chapter to the idea that the average response may not change at all, while a large change in the magnitude of background variation may be a notable and highly relevant effect. This is a particularly relevant idea as many ecological systems now encounter changing climates, which can lead to changes in variation. Replicates and repeatable methods are needed to observe such changes in background variation.

In the final section, the editors bring together other complicated ideas related to replication, repeatability, and reproducibility. Tools and techniques can change over time or from lab to lab and researcher to researcher. Such changes can bias scientific investigations or produce incorrect conclusions. Scientists have invented methods and models for accounting for known biases and minimizing unknown biases. Chapter 16 does an excellent job describing the many ways that “time” and “space” add variation that may reduce repeatability. Definitions of time that are not based on calendar date, such as growing degree days, larval growth stage, or time since bud burst, are all legitimate depending on the perspective of the investigation. Seasonality, or cycles in biological data, are periodic, reoccurring patterns that exist in time, and whose presence adds variation that may case potential replicates to differ in undesirable ways. Yet, in other settings, these cyclic patterns may be the object of study. The section provides a wonderful discussion for any investigator who is contemplating (and who should contemplate) how the scales and patterns of time and space are relevant in their future research. The final flowchart in the book identifies steps in the research process when replication should be addressed. It makes clear that addressing replication requires attention throughout the research process; and failure to do so has significant risk.

As most biological researchers can attest, addressing replication in an investigation is difficult, time‐consuming and often feels like aiming at a moving target. This book provides a broad context for replication and lays out its interacting components and roles. As such, it forms a strong basis for further contemplation.