Study finds that ecological evidence used by government policy-makers is unstable over time

A large study, led by academics from Royal Holloway, University of London, found research used by policy-makers regarding conservation and environmental management is unstable over time and can quickly become out of date: presenting a serious potential threat to evidence-based decision-making.

The research was led by Ph.D. student Lizzie Brisco and Professor Julia Koricheva from the Department of Biological Sciences at Royal Holloway, in collaboration with Professor Elena Kulinskaya from the University of East Anglia.

Policy-making in managing the environment and conservation planning increasingly relies on quantitative reviews of results from previous studies—otherwise known as “meta-analysis.”

Meta-analysis estimates the size of the effect of an intervention, its statistical significance and the direction of the effect (i.e., whether the intervention effect is large or small, positive or negative). This effect estimate informs environmental management recommendations.

However, these can only be formed from a snapshot of the evidence available at the time when recommendations are made. If the magnitude, statistical significance, or the direction—becoming beneficial when it was once thought to be harmful, or vice versa—of the estimated effect changes after this point (as more evidence is included), then recommendations derived from these reviews could quickly become outdated.

The new study, published in the journal Research Synthesis Methods, assessed 121 meta-analyses from a range of applied ecology and conservation topics, such as impacts of agri-environmental schemes on bees, and the impact of different water temperatures and CO₂ levels on coral.

The analysis found that the effects in most of the studies created “temporal trends” that were unstable over time and could change the magnitude and statistical significance of these effects. For example, whether the effect decreased in size over time, became significant, or changed direction (becoming beneficial when it was once thought to be harmful, or vice versa).

Ph.D. student Lizzie Brisco, the lead author of the study, said, “Temporal instability of the environmental evidence base is extremely concerning because environmental policy interventions are expensive, and their consequences are important both economically and for vulnerable species.

“Changes in evidence over time also add to concerns about a broader ‘reproducibility crisis’ in science—failure to replicate findings reduces public confidence in the scientific method and its conclusions, which has broad implications in society.”

Professor Julia Koricheva, from the Department of Biological Sciences at Royal Holloway, said, “Temporal trends in effect magnitude and significance have been often observed in other research fields, e.g., medicine and social sciences. Our study adds to the growing evidence of temporal trends in effect sizes across scientific disciplines.

“There was a recent disagreement about the prevalence of such temporal trends in ecology, but the focus of previous studies was largely on the so-called ‘decline effect’ when evidence for the phenomenon decreases over time.

“Our study has shown that temporal patterns for environmental evidence are much more complex, which means that policies based on this evidence may have lesser, greater, or opposite consequences than expected.”

In the study, 93% of the meta-analyses that the researchers examined showed temporal trends in effect magnitude or statistical significance, with 27% of the datasets exhibiting temporal trends in both effect size and significance. Extreme early study effects represented the most common pattern (where studies early in the time span of evidence for the intervention show more extreme effects), but many other trends including diminishing and increasing magnitudes of effect and changes in the sign of the effect were detected.

The researchers established that, in most cases, these temporal fluctuations in effects were not due to changes in sample size and were not related to changes in study characteristics such as geographic location or study species.

This led the researchers to conclude that the observed temporal changes in effects might either reflect genuine biological changes over time or be caused by publication bias—where results of studies can affect how quickly the research is published.

Regardless of the underlying causes, temporal instability in estimates of effects in ecology and conservation represents a serious potential threat to policy decision-making, as statistical estimates of the effect of an intervention may not accurately reflect the real-world impacts.