Venus is in the east.
Venus is in the west.
I can’t see Venus at all.
Here are three observations about the position of Venus in our sky that are about as divergent as can be, but all are true. The movement of Venus and the movement of our viewpoint from earth mean that these seemingly conflicting observations can all be true and fit into a greater complex reality. Discovery begins with initial observations. These may appear to be quite variable without understanding of the complex factors that influence them. We should not discount observations merely because they are variable. Rather, we should use this initial variable information to learn more about the more complete reality.
Non-replication in science is of great concern. There are important incidences of fraud, slipshod methodology and overexuberant interpretation that need to be identified and corrected. But in addition to these problems, there are other important and fundamental contributors to non-replication. These include limitations of statistical evaluation as well as challenges of trying to interpret phenomena in a complex environment. Most common statistical methods control for alpha errors, that is, falsely accepting the appearance of a phenomenon which is not actually true. However, we do not typically control for beta errors, falsely missing a real phenomenon. Repeated experiments may be conducted with some studies detecting significant effects and others not detecting significant effects in similar circumstances, because beta errors are not controlled and we cannot be confident that we are not overlooking important phenomena.
Limits of statistical methodology can be addressed with large enough studies and meta-analyses across studies. A greater challenge comes from the inevitable paucity of knowledge at the edges of understanding. This is where most scientific research is conducted. Scientific research is an endeavor of discovery. Quite often we observe phenomena without knowing the important context of those phenomena. The context can make all the difference in how or whether the phenomenon is seen. For example, one study may analyze samples taken from a river for a contaminant and find a certain concentration and another study later may take a sample from the same site and find a much lower level because of a rainstorm that occurs out of sight, well upstream.
That’s why it is important for scientific progress and communication of science with society to consider non-replication not as just as an indictment, but also as a starting point for further research to know the fuller context behind the phenomena we observe so that we can understand complex processes of the world.
By Edward D. Levin, Ph.D., Duke University edlevin@duke.edu
