Risks to Replication of Credibility Revolution

While the advanced causal inference methods increases credibility of causal results, there are two concerns that make studies hard to replicate:

decline in reporting of null results
increase in use of private sector data

1. Decline in Reporting of Null Results

The Importance of Transparency in Research Findings

The reporting of null results is critical for scientific transparency and the accumulation of knowledge.
Null results provide valuable information about the absence of expected effects, helping to prevent publication bias and ensuring a more accurate understanding of economic phenomena (Rosenthal 1979, Sterling 1959).
Despite their importance, null results are often underreported due to the perceived lower likelihood of publication and the undervaluation of such findings in the academic community (Brodeur et al. 2016, Chopra et al. 2024).

Decline Over Time: The reporting of null results has decreased from approximately 15% in 1980 to around 8.6% in 2023.

A reflection of research practices and publication norms across field

We observe a general decrease across most fields in the post-2000 period.
Highest levels: Fields such as Econometrics and Behavioural Economics report higher shares of null results.
Health Economics shifted from least to one of the highest reporter of Nulls.
Finance and IO report lowest shares post-2020

Large variations across methods

Causal inference design methods like RCTs, DiD, and RDD are associated with higher shares of null results.
Methods like Structural Estimation, IV and Simulations see the lowest share of Null result claims.
These patterns may reflect the differing nature of these methods.
Experimental and quasi-experimental methods like RCTs and RDDs, designed for rigorous causal inference, may often result in null findings when interventions do not produce significant effects.
Transparent reporting of such results is important to avoid publication bias.

Large variations across methods

RCTs and RDDs have relatively high null results across all fields.
Opposite is true for IV, with the exception of papers that are primarily about econometrics.

2. Use of Private Data in Economics Research

Implications for Data Accessibility and Replicability

The use of proprietary data exacerbates the problem, limiting other researchers’ ability to replicate studies or test alternative hypotheses.
Open data is not widely practiced in economics (Andreoli-Versbach & Mueller-Langer 2014).
The need for policies that balance privacy concerns with the benefits of data accessibility for scientific advancement (Fetzer 2022).
Data privacy regulations like the GDPR have introduced additional barriers to data sharing.
In response to these challenges, Miguel (2021) documents the adoption of open science practices in economics, such as pre-registration and data sharing, noting a rapid transition toward increased transparency.

Increase Over Time: Use of private or proprietary data rose from about 4% in 1980 to approximately 8.6% in 2023.

This trend reflects the greater availability of granular, individual-level data collected by private companies, as well as increased collaboration between researchers and private entities.

A reflection of research practices and publication norms across field

Fields such as Finance, IO, and Behavioral Economics exhibit higher proportions of papers using private data in the post-2000 period.
These increases may reflect the nature of research in these fields, which often relies on firm-level or experimental data that is not publicly available.
The use of proprietary datasets allows researchers to conduct detailed analyses of financial markets, consumer behaviour, and firm dynamics.

By Methods

We find that methods such as Event Studies, Difference-in-Differences (DiD), and Instrumental Variables (IV) are associated with higher proportions of private data usage
These methods often require detailed data on firm events, policy changes, or instrumental variables that may be proprietary or collected by private companies.
The increasing reliance on these methods may contribute to the greater use of private data in economic research.

Field x Method

We observe that certain combinations of fields and methods are associated with higher proportions of private data usage.
For example, DiD in Behavioral Economics (29%) and Finance (20%), and Structural in IO (28%), are associated with higher private data usage.

Page updated

Google Sites

Report abuse