repframe: A Reproducibility and Replicability Indicator Framework for R and Stata

repframe (v1.8.0) is an R and Stata package for assessing the reproducibility, replicability, and robustness of empirical results across multiverse analyses, specification curves, and other sets of robustness specifications.

The repframe Reproducibility and Replicability Indicator Framework provides complementary numerical and graphical outputs for individual studies and pooled analyses. Its main outputs are a Reproducibility and Replicability Indicators table and a Robustness Dashboard. Additional coefficient, contribution, and decision-combination plots help diagnose how particular analytical choices shape the results.

repframe is designed for robustness analyses in which alternative empirical specifications vary analytical choices while targeting the same underlying estimand. Differences between results can therefore be interpreted as arising from analytical choices rather than from changes in the parameter of interest.

The overview below summarizes the repframe workflow, from path-level or pooled input data through the calculation of reproducibility, replicability, and robustness indicators to the principal tables and graphical outputs.

Tip

Explore repframe without writing code: use the interactive repframe Shiny app to upload or select example data, adjust the main analysis settings, and inspect the resulting indicators and visualizations. The app is intended for interactive exploration; the R and Stata packages remain the appropriate interfaces for fully scripted and reproducible analyses.

Documentation roadmap

• Run a first analysis: follow Installation and then go directly to Quick start.

• Understand the outputs and their interpretation: see Main outputs, Scope and limits, and the Indicator overview.

• Combine evidence across studies: see Data structure for analyses across studies.

• Explore extensions and specialized analyses: see Complementary analytical-decision plots, Additional indicators, and Weak-IV robust inference.

• Use repframe alongside related software: see Related multiverse and specification-curve software for joint-use guidance covering 15+ related R, Stata, and Python tools.

Tip

To interactively learn more about repframe and its indicators, you can use this model-agnostic AI prompt with an AI assistant of your choice.

Installation

Install the Stata package with:

net install repframe, from("https://raw.githubusercontent.com/guntherbensch/repframe/main/stata") replace

Install the R package from GitHub with:

install.packages("remotes") # only needed if remotes is not installed
remotes::install_github("guntherbensch/repframe")

For complete syntax and option definitions, use help repframe in Stata or ?repframe in R.

Main outputs produced by repframe

repframe calculates indicators both within individual studies and across pooled studies. Its two principal outputs can be produced at either level:

• Reproducibility and Replicability Indicators table: a structured table containing one set of main indicators in .csv, .xlsx, or .tex format. See below and Dreber and Johannesson (2025).

• Robustness Dashboard: an integrated visualization of a second set of main indicators. The Dashboard is versatile and can accommodate different analysis settings and display more or less extensive sets of indicators depending on the application and selected options. Circle sizes show the share of robustness paths in each result category, while colours distinguish confirmatory from non-confirmatory results under the selected significance-classification convention. See below and Bensch et al. (2025).

As additional output, repframe transforms analysis-path input data into a Harmonized analysis-path dataset and creates Study-level indicator data that can be reintroduced into repframe to calculate indicators across studies. In Stata, these datasets are saved according to datasave(); in R, they are components of the returned object and are written only when the user calls write_repframe_analysispaths() or write_repframe_studies().

Both implementations can additionally generate plots comparing original and preferred coefficients and showing how individual and joint analytical choices contribute to robustness patterns. See below.

The following diagram presents the underlying data workflow. It depicts the Stata file-based workflow; the R implementation returns the corresponding objects in memory and allows users to export them explicitly.

 

_{Note: The numbered labels identify the principal data and output components generated by repframe: (1) the Harmonized analysis-path dataset; (2) the Study-level indicator data used for analyses across studies; (3) the Reproducibility and Replicability Indicators table; (4) the Robustness Dashboard; and (4+) the complementary coefficient, contribution, and decision-combination heatmap plots.}

Quick start

For a standard robustness analysis within one study, the input data contain one observation per analytical path, including one original specification for each result. A basic application specifies the coefficient, a standard error and/or p-value, the original path, and the significance level used in the robustness analysis.

Stata

repframe outcome, beta(bvar) se(sevar) pval(pvar) origpath(origpathvar) siglevel(5) shortref("Author et al. (year)")

R

library(repframe)

dat <- repframe_example("single")

result <- repframe(
  dat,
  mainvar = "result",
  beta = "beta",
  se = "se",
  pval = "pval",
  origpath = "origpath",
  siglevel = 5,
  shortref = "Example study"
)

plot(result)       # Robustness Dashboard
result$table       # Reproducibility and Replicability Indicators
summary(result)

The Stata call creates the configured files and returns key information through r(). The R call returns a repframe object without writing files; the object contains the harmonized data, indicators, tables, Dashboard data and plot, settings, and diagnostic messages. This separation also lets users fine-tune exported output, for example by adjusting the Dashboard width and height when saving it. For example:

export_repframe_dashboard(result, "repframe_dashboard_example.png")
export_repframe_dashboard(result, "repframe_dashboard_example_custom.png",  width = 11, height = 6)
export_repframe_table(result, "repframe_indicators_example.xlsx")
write_repframe_analysispaths(result, "repframe_paths_example.dta")
write_repframe_studies(result, "repframe_studies_example.dta")

The .xlsx and .dta exports require the suggested R packages writexl and haven, respectively. The quick start gives the minimum working pattern. The getting-started vignette expands it into a self-contained executable walkthrough of the main R workflows, whereas this README also serves as the broader conceptual reference for the indicators and their interpretation.

Optional: ask an AI assistant about repframe

The prompt below tells an AI assistant to ground its answers in the current documentation, distinguish Stata from R, and identify uncertainty rather than guess. It can be useful when choosing a pooling mode, mapping your variables to repframe, or interpreting a particular indicator.

Show the model-agnostic prompt

Copy the prompt below into an AI assistant with web access to initiate a conversation about repframe:

I am using repframe, a Stata/R framework for Reproducibility and Replicability Indicators based on multiverse robustness analyses.

Use the documentation for the current repframe release at https://github.com/guntherbensch/repframe as the primary and authoritative source for this conversation. Read the README and the documentation for my software implementation (Stata or R) before answering. Consult the corresponding source code only where the documentation does not determine the implemented behaviour.

Help me understand and apply repframe, including its indicators, Robustness Dashboard, input-data structure, options, weighting and pooling modes, and the interpretation of robustness and replicability assessments.

When answering:

• distinguish carefully between definitions in the repframe documentation and your own interpretation;
• explain the substantive logic behind an option or indicator, not only its syntax;
• ask whether I am using Stata or R if this is not clear, and do not assume that file handling, returned objects, export behaviour, or auxiliary plots are identical across implementations;
• take account of whether the question concerns a single study, studypooling(1), or studypooling(2);
• do not infer undocumented behavior from variable or option names;
• if the documentation is ambiguous or does not answer the question, say so rather than guessing;
• state explicitly when an answer is an inference rather than documented behaviour;
• point me to the relevant documentation section or source file when useful.

I will now ask questions about repframe.

Software implementations and requirements

repframe is implemented in Stata 14.0 or newer and R 4.1.0 or newer. The two implementations use the same indicator definitions and numerical conventions. Deterministic cross-language tests compare the R calculations with reference outputs generated by Stata. The implementations nevertheless use the strengths and conventions of their respective languages rather than attempting to make every interface detail identical.

Both implementations provide repframe as the main entry point. The Stata package additionally includes repframe_gendata, which creates the datasets used in the Stata help examples. The R package provides equivalent deterministic example datasets through repframe_example().

	Stata	R
Main workflow	File-oriented command	Object-based function
Results	Writes configured tables and figures; saves data according to datasave(); returns key metadata through r()	Returns a reusable repframe object and writes nothing unless an export function is called
Data export	.dta	.rds, .csv, or .dta
Table export	.csv, .xlsx, or .tex	.csv, .xlsx, or .tex
Dashboard	Stata graph export	R graphics export through export_repframe_dashboard()
Complementary coefficient, contribution, and heatmap plots	Stata graph export; heatmaps require Stata 19 or newer	R graphics export through export_repframe_complementary_plots()

Within Stata, functionality is identical across supported versions except that Dashboard presentation uses minor version-specific formatting adjustments from Stata 14.2 and 16 onward, decision-combination heatmaps require Stata 19 or newer, and LaTeX table export via tabfmt(tex) requires Stata 17 or newer. In R, .xlsx and .dta exports additionally require writexl and haven, respectively.

Throughout this README, Stata-style option notation such as siglevel(5) is often used in the conceptual discussion. The corresponding R argument is generally written as siglevel = 5; column options such as Stata's beta(b) become mappings such as beta = "b". The main naming difference is Stata's mean() versus R's outcome_mean argument.

Numerical conventions across implementations

To ensure identical Stata and R output, repframe applies the following conventions:

• Standard deviations use the sample denominator n − 1; even-sample medians average the two middle values; and an all-missing eligible group produces a missing value rather than zero.
• Aggregation weights are renormalized separately for each indicator and original-significance group over nonmissing indicator values with positive finite weights.
• Nonnegative Dashboard percentages are rounded to the nearest integer, with half points rounded upward. If the three rounded shares do not sum to 100, downward adjustments prioritize the smaller displayed share, then the smaller fractional remainder, and then the leftmost cell; upward adjustments use the reverse ordering. Empty aggregate rows remain zero. Secondary circle components are rounded independently and capped at the corresponding cell share.

Defaults and conventions applied by repframe

The following conventions define how repframe interprets robustness analyses. Most users can retain these defaults; use the corresponding options where your application requires a different treatment.

Definition of robustness analyses and analytical paths: A main convention of repframe is to treat robustness analyses as variations in analytical choices that continue to target the same underlying estimand. Following Ankel-Peters et al. (2025), we distinguish between analytical decisions—such as outcome definition, estimator, or sample selection—and the analytical choices available for each decision. Each unique combination of choices across decisions defines an analytical path in the multiverse analysis. Terminology varies across the literature: Steegen et al. (2016), for example, use choices instead of decisions and options instead of choices; Götz et al. (2024) use parameters instead of analytical decisions; and Cantone and Tomaselli (2024) use modalities instead of analytical choices; in Bensch et al. (2025) we refer to decisions as domains. While alternative conventions are possible, our distinction helps structure robustness analyses and supports the interpretation of decision-specific contributions. Accordingly, analyses that shift the estimand or target different estimands—such as heterogeneity analyses across subpopulations, placebo or falsification tests, and broader assessments of an identification strategy—are conceptually distinct from robustness analyses in this framework and are typically not included in the multiverse robustness analysis.

Tests for statistical significance: repframe applies two-sided t-tests to determine which p-values imply statistical significance. By default, it assumes that the original and robustness analyses apply the same significance level, as specified in siglevel(#) in Stata or siglevel = # in R. A different significance level for the original estimates can be specified through siglevel_orig(). If p-values are alternatively based on one-sided tests, multiply them by two before supplying them through pval(), so that they correspond to a two-sided test. If no p-values are available, repframe derives the missing p-value information using the t-test formula based on t/z-scores (zscore()), standard errors (se()), and optionally degrees of freedom (df()). If no degrees of freedom are provided, a normal distribution is assumed. Note that this assumption may be inappropriate, for example in small samples, when p-values are derived using randomisation inference, or in designs using complex estimations such as those that account for survey sampling, e.g. via the Stata command svy:. Conversely, if your input p-values are based on distributional assumptions other than normality, the derived standard errors may be inaccurate. It is therefore recommended to provide both p-values and standard errors.

Units of effect size measurement: repframe assumes that effect sizes in the original and robustness analyses are measured in the same units. If not (e.g. one is in logs and the other is not), supply a sameunits variable that equals 1 if units match and 0 otherwise.

Inclusion of original specification in multiverse: By default, the original specification is assumed not to be part of the robustness-analysis paths. If it is, specify via the option orig_in_multiverse(varname) for which results the original specification belongs to the multiverse. It is then included in the computation of the variation indicators ($I_{4}$, $I_{5}$, and $I'{3}$), which characterize variation across the specifications in the multiverse, in line with the indicator framework of Dreber and Johannesson 2025. Indicators that assess robustness paths relative to the original result (notably $I{1}$–$I_{3}$, $I'{2}$, $I'{4}$, and $I'_{5}$) continue to exclude the original specification, since including it would mechanically increase agreement with the original result. Regardless of whether the original specification is included as one robustness analytical path or not, the original specification for each result must appear as a separate analytical path in the input data.

Weighting of results, analytical paths, and studies: By default, repframe gives equal weight to results with studypooling(0) and studypooling(2). With studypooling(1), studies receive equal weight by construction and rfweight() must not be specified.

For analysis-path-level input, alternative weighting schemes can be specified through rfweight(). rfweight(paths) weights each result by the number of non-original robustness-analysis paths available for that result. The original specification does not contribute to this weight, including where orig_in_multiverse() specifies that it forms part of the multiverse. rfweight(precision) weights results using the inverse squared mean relative standard error across their non-original robustness-analysis paths, with relative standard errors defined using the absolute outcome mean. rfweight(varname) uses a user-supplied positive result-level weighting variable. With studypooling(2), such a custom variable can be added to the appended analysis-path dataset before the pooled call and must be positive, nonmissing, and constant within each study–result combination.

With studypooling(2), rfweight(studies) is additionally available. It gives each study equal total weight by first averaging the eligible result-level indicator values within each study and then averaging these study-level values equally across studies. The within-study averaging is performed separately for each indicator, so that unavailable indicator values do not change the relative weight of studies contributing to that indicator. rfweight(paths) therefore represents weighting by multiverse coverage, not by statistical precision; robustness-analysis paths need not constitute statistically independent observations. The precision weight for result $j$ is $w_j=[mean_{i\in R_j}(se_{ij}/|\overline{Y}_{ij}|)]^{-2}$, where $R_j$ denotes the non-original robustness-analysis paths.

Results represented by two coefficients: The option beta2() allows results to be represented by two coefficients. If the original result contains two coefficients, every robustness analysis path for that result must likewise contain two coefficients; if the original result contains only one coefficient, individual robustness paths may contain either one or two coefficients. For original results with two coefficients, repframe classifies the result as statistically significant only if both coefficients are statistically significant. Statistical-significance agreement likewise requires both robustness coefficients to be statistically significant and to agree with the directions of their corresponding original coefficients. For the relative significance indicator, the two robustness t/z-values are aligned with the corresponding original coefficient directions and then averaged. If the original result contains only one coefficient but a robustness path contains two, both robustness coefficients are evaluated relative to the direction of the single original coefficient. The Indicator table accounts for results represented by two coefficients: significance-based indicators remain available, whereas scalar effect-size indicators that require a single coefficient are not calculated for results containing two original coefficients. The Robustness Dashboard is not produced for analyses using beta2(). rfweight(precision) is based on the standard error of the first coefficient specified through beta(); the second coefficient specified through beta2() does not enter the precision weight.

Required input data structure

Data structure for analyses within one study

The input data for an analysis within one study need to be in a specific format for repframe to calculate the indicators and Dashboards. Each observation should represent one analytical path: one combination of analytical choices in a multiverse robustness analysis.

The dataset must contain the following variables. The table also lists the repframe option used to include the variable as well as the variable name in the example presented below the table:

variable type	repframe option	variable name in snippet	note
result identifier	Stata: main variable before the comma R: mainvar	outcome	In Stata, numeric with value labels; in R, numeric, character, or factor
coefficient	beta()	b
standard error and/or p-value	se() pval()	se p	It is recommended to specify both the information on p-values and on standard errors, as outlined above in the sub-section on defaults applied by the repframe command
binary indicator identifying the original path	origpath()	origpath
level of statistical significance in the robustness analysis	siglevel()	-

 

The Stata help file contains a detailed example that uses repframe_gendata to build such a data structure. R provides ready-to-use examples through repframe_example("single"), repframe_example("pooling1"), repframe_example("pooling2"), and repframe_example("complementary").

For this example, the minimum command input looks as follows:

repframe outcome, beta(b) se(se) pval(p) origpath(origpath) siglevel(5) shortref("Author et al. (year)")

The analogous R call is:

result <- repframe(
  data,
  mainvar = "outcome",
  beta = "b",
  se = "se",
  pval = "p",
  origpath = "origpath",
  siglevel = 5,
  shortref = "Author et al. (year)"
)

repframe gives informative error messages if these requirements are not met.

The specific dataset example shown above includes two additional types of variables that are typically important. First, the variable orig_include indicates whether the original specification is part of the multiverse robustness analysis. Since repframe assumes by default that it is not, we should additionally supply orig_in_multiverse() in Stata or orig_in_multiverse in R.

Second, we see that four analytical decisions vary for the main result. In this example, the corresponding variables are named cov1 to if_cond, where cov1, for instance, indicates the inclusion of a certain covariate set. These variables help define each unique analytical path, as every analytical path is given by a unique combination of analytical choices. repframe does not require these variables. Including them through decisions() in Stata or decisions in R creates the complementary contribution plots and retains the decisions in the harmonized analysis-path data. Decision-combination heatmaps are additionally requested through heatmapdims() in Stata or heatmapdims in R (see below). Each decision variable should identify the analytical choice, with the choice adopted in the original analysis represented by zero.

repframe outcome, beta(b) se(se) pval(p) origpath(origpath) siglevel(5) shortref("Author et al. (year)") orig_in_multiverse(orig_include) decisions(cov1 cov2 cov3 ifcond)

result <- repframe(
  data,
  mainvar = "outcome",
  beta = "b",
  se = "se",
  pval = "p",
  origpath = "origpath",
  siglevel = 5,
  shortref = "Author et al. (year)",
  orig_in_multiverse = "orig_include",
  decisions = c("cov1", "cov2", "cov3", "ifcond"),
  heatmapdims = list(x = "ifcond", y = "cov3", cols = "cov1", rows = "cov2")
)

Data structure for analyses across studies

repframe supports two approaches to compiling Reproducibility and Replicability Indicators across studies. They differ in the level at which the data from individual studies enter the pooled analysis.

With studypooling(1), previously generated study-level indicator data are pooled. Each study enters the pooled analysis with its already aggregated study-level indicators and receives equal weight by construction; rfweight() must not be specified.

With studypooling(2), appended analysis-path-level data are pooled. repframe first calculates indicators for each study–result combination, just as it does for individual results within a single study, and subsequently aggregates these indicators across study–result combinations. A common siglevel() is specified for the pooled robustness analysis, while the original-study significance level stored in siglevel_oa_j is retained for each study. Original-study significance levels may therefore differ across studies, but must be constant within a study. By default, each study–result combination receives equal weight (rfweight(results)). Alternatively, results can be weighted by their number of non-original robustness-analysis paths (rfweight(paths)), by precision (rfweight(precision)), by a user-supplied result-level variable, or studies can receive equal total weight (rfweight(studies)).

The two approaches therefore retain different information from the individual-study analyses: studypooling(1) pools already aggregated study-level indicators, whereas studypooling(2) retains the underlying study–result structure and permits alternative weighting of results, analytical paths, and studies.

Dreber and Johannesson (2025) propose equal weighting for aggregation, which is implemented in repframe by first using aggregation(1) with the default rfweight(results) to average indicators equally across results within each study, and subsequently using studypooling(1) to average the resulting study-level indicators equally across studies. Alternatively, studypooling(2) with rfweight(studies) applies the corresponding two-stage aggregation directly to pooled analysis-path data.

Detailed Stata workflow for pooling across studies

Pooling study-level indicator data in Stata

With studypooling(1), the command aggregates indicators that were previously calculated separately for each study. In Stata, the following steps need to be taken:

1. Run repframe for each individual study to create the study-level indicator data saved as
   repframe_studies_[fileidentifier].dta,
   where [fileidentifier] is defined through fileidentifier(string). For each study, repframe stores the original-analysis and robustness-analysis significance levels in siglevel_oa_stud and siglevel_ra_stud, respectively. These levels may differ across studies; studypooling(1) pools the already calculated study-level indicators and reports the observed threshold pairs.

2. Append the individual study-level indicator data. For example:

use	         "repframe_studies_[fileidentifier1].dta", clear
append using "repframe_studies_[fileidentifier2].dta"
append using "repframe_studies_[fileidentifier3].dta"

3. Encode the study reference, clean and save the appended dataset:

encode ref, gen(reflist)
drop ref
order reflist
save "[filename].dta", replace

Here, [filename] can be freely chosen for the dataset containing the appended study-level indicator data and may include the complete file path.

4. Run repframe again using studypooling(1) to calculate indicators across studies. The basic command is:

repframe reflist, studypooling(1)

Pooling analysis-path-level data in Stata

With studypooling(2), indicators are calculated directly from appended analysis-path-level data. The command first calculates indicators separately for each study–result combination and subsequently aggregates them across all study–result combinations.

In Stata, the following steps need to be taken:

1. Run repframe for each individual study to create the analysis-path-level data saved as
   repframe_paths_[fileidentifier].dta.

   Use a unique shortref() for each study. This reference is stored in the variable ref.

2. Append the individual analysis-path-level data. For example:

use	         "repframe_paths_[fileidentifier1].dta", clear
append using "repframe_paths_[fileidentifier2].dta"
append using "repframe_paths_[fileidentifier3].dta"

3. Verify that ref identifies the study and that every study–result combination contains exactly one original analytical path before saving the dataset:

assert ref != ""

bysort ref mainvar_str: egen byte x_original_paths = total(origpath_i)
assert x_original_paths == 1
drop x_original_paths

save "[filename].dta", replace

4. Run repframe on the appended analysis-path data with studypooling(2) to calculate indicators across studies. The basic command is:

repframe mainvar, studypooling(2) siglevel(5)

Detailed R workflow for pooling across studies

Pooling study-level indicator data in R

Run repframe() separately for each study and retain the returned study-level data:

study_data <- rbind(
  study1_result$studies,
  study2_result$studies,
  study3_result$studies
)

pooled_studies <- repframe(
  study_data,
  mainvar = "ref",
  studypooling = 1
)

Each input object must come from a completed single-study analysis. Studies receive equal weight by construction. The intermediate data need not be written to disk, but can be saved where useful:

write_repframe_studies(study1_result, "repframe_studies_study1.rds")

Pooling analysis-path-level data in R

Alternatively, append the harmonized analysis-path data returned for the individual studies:

path_data <- rbind(
  study1_result$analysispaths,
  study2_result$analysispaths,
  study3_result$analysispaths
)

pooled_paths <- repframe(
  path_data,
  mainvar = "mainvar",
  studypooling = 2,
  siglevel = 5,
  rfweight = "results"
)

The stored ref column identifies studies, and repframe() constructs the study–result units from ref and the supplied result identifier. Replace rfweight = "results" with "paths", "precision", "studies", or a valid custom result-level weight column where appropriate.

Scope and limits of reproducibility and replicability assessments

repframe is designed for settings in which the original and robustness analyses aim to reproduce or probe the same substantive finding and are sufficiently comparable for differences between their estimates to be meaningfully interpreted. For within-study robustness analyses, this will generally require that alternative specifications continue to target the same underlying estimand. What constitutes sufficient comparability is necessarily context-dependent. Alternative specifications may, for example, differ in covariate adjustment, sample definition, variable construction, functional form, or other analytical choices while still providing informative robustness checks of the same substantive finding.

The indicators derived by repframe summarize how conclusions vary across the analyses supplied to repframe; they do not determine whether those analyses are sufficiently comparable to belong in the same assessment. Defining that comparison set remains a substantive responsibility of the user. The indicators are most informative when the included analyses all bear on the same substantive claim and any differences in constructs or estimands are understood and interpretable. Even seemingly modest analytical changes can alter the construct or estimand being studied, and specifications that are individually defensible are not necessarily interchangeable for every interpretive purpose (Del Giudice and Gangestad 2021; Bartoš et al. 2025; Short et al. 2026).

repframe is descriptive rather than a model for estimating a pooled effect or decomposing heterogeneity. When many analyses use the same dataset, single-dataset meta-analysis offers a complementary approach: it estimates an overall effect and between-analysis heterogeneity while accounting for the shared data source (Bartoš et al. 2025).

When replications use new data, populations, settings, or research designs, disagreement can no longer be attributed only to analytical choices. It may also reflect heterogeneity in true effects, statistical power, measurement, treatment implementation, or other design features and can therefore require additional measures and substantive interpretation (Patil et al. 2016; Mathur and VanderWeele 2020; Pawel and Held 2022; Slough and Tyson 2025). Complementary criteria developed specifically to assess replication success while accounting for these issues include meta-analytic aggregation, prediction intervals and within-reproduction confidence intervals, heterogeneity tests, and power-based approaches such as small telescopes (see, e.g., FReD Explorer). Aggregating repframe indicators across studies summarizes observed patterns of agreement; it does not by itself establish external validity or make the empirical targets of the constituent studies equivalent.

When the included analyses have been deliberately selected as meaningful robustness checks of the same substantive finding, the repframe indicators have a straightforward interpretation: high agreement indicates that the finding is stable across the specified analytical alternatives, whereas low agreement identifies sensitivity to those alternatives. As with any robustness analysis, the informativeness of this evidence depends on the relevance, coverage, and quality of the specifications considered. The indicators therefore structure and summarize the resulting robustness evidence while leaving its substantive interpretation to the researcher (Buzbas and Devezer 2024).

The Reproducibility and Replicability Indicator Framework

The Reproducibility and Replicability Indicators table and the Robustness Dashboard provide complementary views of robustness. They are particularly suited for robustness reproducibility assessments in which the same underlying data are analyzed under alternative plausible specifications (Dreber and Johannesson 2025; Short et al. 2026).

The indicators summarize three main aspects of robustness:

• agreement: Do original and robustness analyses lead to comparable conclusions about statistical significance or effect size?
• relative differences: How large are differences between original and robustness results?
• variation: How much do results vary across robustness-analysis paths?

The Reproducibility and Replicability Indicators table implements the indicator framework proposed by Dreber and Johannesson (2025). The Robustness Dashboard draws on these and other established measures but adapts them to its cell-based representation of robustness results and adds Dashboard-specific indicators. Dashboard indicators are therefore denoted by a prime ($I'$). The following expandable table briefly identifies the closest related measures and explains how the Dashboard indicators differ from them.

Relationship of the Robustness Dashboard indicators to other existing measures

Dashboard indicator	Relationship	Related measures and Dashboard adaptation
$I'_1$: significance agreement	close analogue	Repeated significance (FReD; Open Science Collaboration 2015; Camerer et al. 2016) assesses whether a result is again statistically significant in the original direction. Specification-curve analysis (Simonsohn, Simmons, and Nelson 2020) and vibration-of-effects analyses (Patel, Burford, and Ioannidis 2015; Klau et al. 2023) also report shares of positive-significant, insignificant, and negative-significant estimates. $I'_1$ adopts this three-way partition, defines direction relative to the original estimate, and reports the share in each Dashboard cell.
$I'_2$: relative effect size	adapted measure	Relative effect size (Dreber and Johannesson 2025; Camerer et al. 2016; Held, Micheloud, and Pawel 2022) is commonly defined as the replication or mean robustness estimate divided by the original estimate, as in $I_2$. $I'_2$ instead reports the percentage difference between the cell-specific median robustness coefficient and the original coefficient. The median represents the typical estimate within a Dashboard cell, limits the influence of extreme specifications, and provides the reference point for the corresponding median-based variation indicator $I'_3$. Calculating it separately by cell also avoids aggregating substantively different significance–direction outcomes.
$I'_3$: effect-size variation	adapted measure	Existing robustness measures summarize dispersion using, for example, the standard deviation of estimates across specifications (Athey and Imbens 2015; Dreber and Johannesson 2025) or vibration ratios (Patel, Burford, and Ioannidis 2015). $I'_3$ uses the mean absolute deviation from the cell-specific median, scaled by the absolute original coefficient, to provide a robust and directly interpretable measure within each Dashboard cell.
$I'_4$: significance variation	related measure	Multiverse and vibration-of-effects approaches summarize variation in test statistics or p-values (Dreber and Johannesson 2025; Patel, Burford, and Ioannidis 2015; Klau et al. 2021). $I'_4$ instead measures the average absolute change from the original p-value and is restricted to statistically insignificant paths, where variation in p-values is informative but not already summarized by significance classification.
$I'_5$: effect-size agreement	close analogue	Confidence-interval consistency (Open Science Collaboration 2015; FReD) commonly evaluates whether the original estimate lies within the replication confidence interval, while prediction-interval approaches (Patil, Peng, and Leek 2016) ask whether the replication estimate is compatible with the original result. The Dashboard reverses the usual confidence-interval comparison because it evaluates multiple robustness estimates against one fixed original reference: $I'_5$ reports the share of insignificant robustness coefficients that remain within the original estimate's confidence interval. This complements significance agreement particularly when an originally significant result becomes insignificant, by showing whether the loss of significance is accompanied by a substantial shift in the estimated effect size. A high $I'_5$ indicates that many insignificant robustness estimates remain within the uncertainty range around the original estimate, but does not establish statistical equivalence or test differences between estimates.
$I'_6$: classification-only non-agreement	new diagnostic	No close scalar counterpart was identified. Motivated by the sensitivity of binary significance classifications to thresholds (Gelman and Stern 2006), $I'_6$ isolates paths whose apparent non-agreement arises solely because the original and robustness analyses apply different significance thresholds.
$I'_7$: confirmatory share	generalized measure	Repeated-significance and conclusion-agreement measures (FReD; Camerer et al. 2016; Dreber and Johannesson 2025) classify individual replications as confirmatory or non-confirmatory. $I'_7$ generalizes this idea to the aggregation of analytical paths across results or studies and allows either the original-study significance classification ($\kappa^o$) or a common significance threshold ($\kappa^u$) to define confirmation.
$I'_8$ and $I'_9$: significance-switch decomposition	new diagnostics	No direct counterpart was identified. The two indicators jointly distinguish whether a switch in significance can be accounted for by the change in coefficient magnitude ($I'_8$) or by the change in its standard error ($I'_9$), in each case holding the other quantity at its original value.

_{"Close analogue" denotes an existing measure that addresses essentially the same specific comparison or classification but differs in its statistical implementation, reference quantity, or direction of comparison. "Adapted measure" denotes a direct modification of an established indicator—such as a change in aggregation statistic, scaling, or eligible analysis paths—to fit the Dashboard's cell-based structure. "Generalized measure" retains the underlying logic of an established measure but extends its domain, for example across originally significant and insignificant results, alternative classification conventions, or levels of aggregation. "Related measure" addresses the same broad aspect of robustness, such as variation in statistical significance, but evaluates a substantively different comparison or quantity and is therefore not a direct counterpart. "New diagnostic" denotes a Dashboard-specific measure for which no close previously established counterpart was identified.}

 

The overview below summarizes both sets of indicators and the subsequent sections provide the indicators' exact definitions and interpretation.

Indicator overview

Type of indicator		Reproducibility and Replicability Indicators table	Robustness Dashboard	Symbol in Dashboard
significance (sig.)	sig. agreement	$I_{1}$	$I'_{1}$	main Dashboard circles
	relative sig.	$I_{3}$	-	-
	sig. variation	$I_{5}$	$I'_{4}$	$\overline{\Delta p}$
	non-agreement due to sig. classification	-	$I'_{6}$ (if significance levels differ)	nested-circle component under confirmatory(original)
	confirmatory share	-	$I'_{7}$ (if aggregated)	$\overline{\kappa}^{o}$ or $\overline{\kappa}^{u}$
	sig. switch	-	$I'{8}$ & $I'{9}$ (optional)	low/high $\lvert\beta\rvert$ and $se$
effect size (e.s.)	e.s. agreement	-	$I'_{5}$	$\beta$ in $CI(\beta^o)$
	relative e.s.	$I_{2}$	$I'_{2}$	$\widetilde{\beta}$
	e.s. variation	$I_{4}$	$I'_{3}$	$\overline{\Delta\beta}$

Reproducibility and Replicability Indicators table

Reproducibility and Replicability Indicators implemented in repframe follow and operationalize the framework proposed by Dreber and Johannesson (2025). The paper presents three indicator families—statistical significance, relative effect size, and variation—while repframe reports the coefficient- and t/z-based versions of the latter two separately, resulting in five indicators at the level of each assessed result. When indicators are aggregated, Dreber and Johannesson (2025) propose equal averaging across results within a study and, for analyses across studies, equal averaging of the resulting study-level indicators; repframe implements this approach and also permits alternative weighting schemes.

1. The statistical significance indicator ($I_{1j}$) measures the share of robustness paths that agree with the original result's significance classification. For an originally significant result, robustness paths must additionally have the same coefficient direction:

$$ I_{1j} = \begin{cases} mean\left(\mathbb{I}(pval_i \le \alpha)\times\mathbb{I}(\beta_i \times \beta^{orig}_j \ge 0)\right), & \text{if } pval^{orig}_j \le \alpha^{orig}, \\ mean\left(\mathbb{I}(pval_i > \alpha)\right), & \text{if } pval^{orig}_j > \alpha^{orig}. \end{cases} $$

For originally insignificant results, Dreber and Johannesson (2025) propose reporting the share of significant reproducibility tests, irrespective of direction, so that lower values indicate greater reproducibility. repframe reverses this coding and reports the complementary share of insignificant robustness paths, so that higher values consistently indicate greater significance–classification agreement for both originally significant and insignificant results. This is the only modification of the result-level indicator definitions proposed by Dreber and Johannesson (2025).

Interpretation: For an originally significant result, $I_{1j}=0.3$ means that 30% of robustness paths are significant at the robustness-analysis threshold and have the same sign as the original estimate. For an originally insignificant result, it means that 30% of robustness paths are also insignificant. If $\alpha \ne \alpha^{orig}$, part of any disagreement may reflect the different significance thresholds rather than a change in the underlying p-value pattern.

2. The relative effect size indicator ($I_{2j}$) compares the mean robustness coefficient with the original coefficient. It requires comparable effect-size units and is reported only for originally significant results:

$$ I_{2j} = \begin{cases} \dfrac{mean(\beta_i)}{\beta^{orig}_j}, & \text{if } pval^{orig}_j \le \alpha^{orig}, \\ \text{not applicable}, & \text{if } pval^{orig}_j > \alpha^{orig}. \end{cases} $$

Relative effect size indicators that divide by the original coefficient are not defined when the original coefficient equals zero and are reported as not applicable.

Interpretation: For $I_{2j}>1$, the mean robustness coefficient has the same sign as the original coefficient and is larger in absolute magnitude; for $0<I_{2j}<1$, it has the same sign and is smaller. For $I_{2j}<0$, the mean robustness coefficient has the opposite sign; its absolute magnitude is smaller than the original for $-1<I_{2j}<0$, equal at $I_{2j}=-1$, and larger for $I_{2j}<-1$.

3. The relative t/z-value indicator ($I_{3j}$) compares the mean robustness t/z-value with the original t/z-value and is reported only for originally significant results:

$$ I_{3j} = \begin{cases} \dfrac{mean(zscore_i)}{zscore^{orig}_j}, & \text{if } pval^{orig}_j \le \alpha^{orig}, \\ \text{not applicable}, & \text{if } pval^{orig}_j > \alpha^{orig}. \end{cases} $$

Dreber and Johannesson (2025) present this indicator as the t/z-based variant of the relative effect size indicator ($I_{2j}$). They recommend it when effect sizes are not measured in comparable units and note that it may also be reported alongside the coefficient-based indicator because it incorporates differences in standard errors.

Interpretation: For $I_{3j}>1$, the mean robustness t/z-value has the same sign and larger absolute magnitude than the original value; for $0<I_{3j}<1$, it has the same sign and smaller absolute magnitude. For $I_{3j}<0$, the signs differ; absolute magnitude is smaller for $-1<I_{3j}<0$, equal at $I_{3j}=-1$, and larger for $I_{3j}<-1$.

4. The effect size variation indicator ($I_{4j}$) measures the standard deviation of robustness coefficients relative to the standard error of the original coefficient:

$$ I_{4j} = \frac{sd(\beta_i)}{se(\beta^{orig}_j)}. $$

The indicator is calculated separately for originally significant and insignificant results. The robustness coefficients may include the original specification when orig_in_multiverse() specifies that it belongs to the multiverse, and effect sizes must be measured in comparable units.

Interpretation: $I_{4j}>1$ means that variation across robustness coefficients exceeds the sampling variation represented by the original standard error; $I_{4j}<1$ means that it is smaller.

5. The t/z-value variation indicator ($I_{5j}$) reports the standard deviation of robustness t/z-values:

$$ I_{5j} = sd(zscore_i). $$

It is calculated separately for originally significant and insignificant results, and may include the original specification when this is part of the multiverse. Dreber and Johannesson (2025) present this as the t/z-based variant of the variation indicator. Because t/z-values are already expressed relative to their standard errors, no further normalization by the original standard error is required.

Interpretation: Higher values indicate greater variation in statistical significance across robustness paths.

Standard-deviation-based variation indicators are not defined when fewer than two eligible analytical paths are available.

The following shows an example of the Reproducibility and Replicability Indicators table:

 

As additional references, the table reports the original coefficient $\beta^{orig}_j$, the original coefficient relative to the original outcome mean, and the original p-value $pval^{orig}_j$.

Robustness Dashboard

The Robustness Dashboard provides a visual summary of how results change across analytical choices. Its three columns partition robustness paths into statistically insignificant estimates, statistically significant estimates with the opposite sign, and statistically significant estimates with the same sign as the original estimate. These cells are defined using the robustness-analysis significance level $\alpha$.

Indicators are shown only where they are informative. In particular, effect-size location and variation are calculated separately for statistically insignificant paths and for statistically significant same-sign paths; opposite-sign significant estimates are not averaged into these effect-size indicators.

The Dashboard contains a core set of indicators ($I'_1$ to $I'_4$), two conditional indicators ($I'_5$ and $I'_6$), the confirmatory-share indicator $I'_7$ when results or studies are aggregated, and optional significance-switch indicators $I'_8$ and $I'_9$ requested through customindicators(SIGswitch).

1. The significance agreement indicator ($I'_{1j}$) gives the share of robustness paths in each of the three Dashboard cells. For statistically significant same-sign paths,

$$ I'_{1j} = mean\left(\mathbb{I}(pval_i \le \alpha) \times \mathbb{I}(\beta_i \times \beta^{orig}_j \ge 0)\right)\times 100. $$

The opposite-sign significant share is defined analogously using $\mathbb{I}(\beta_i \times \beta^{orig}_j < 0)$; the insignificant share is the remainder to 100%.

As a sensitivity check, the Dashboard also applies an alternative significance threshold: 10% when the main threshold is 5%, and 5% otherwise. It reports the share of robustness estimates that are significant and have the same direction as the original estimate at this alternative threshold. In nonaggregated Dashboards, the insignificant cell may additionally show the share that remains insignificant at the alternative threshold.

Interpretation: The three cell shares show how robustness paths are distributed across insignificant, significant opposite-sign, and significant same-sign results. They therefore summarize technical significance agreement without averaging fundamentally different types of robustness outcomes.

2. The relative effect size indicator ($I'_{2j}$) compares the median robustness coefficient with the original coefficient:

$$ I'_{2j} = \left(\frac{median(\beta_i)}{\beta^{orig}_j}-1\right)\times 100. $$

It is calculated separately for statistically insignificant paths and statistically significant same-sign paths, requires comparable effect-size units, and is not reported for significant opposite-sign paths. As for the relative effect size indicator $I_{2j}$, $I'_{2j}$ is undefined when the original coefficient equals zero and is reported as not applicable.

Interpretation: $I'_{2j}=0$ means that the median robustness coefficient equals the original coefficient. Positive values indicate a larger same-direction effect; negative values indicate a smaller effect. Among statistically insignificant paths, values below $-100%$ indicate that the median robustness coefficient has crossed to the opposite sign.

3. The effect size variation indicator ($I'_{3j}$) measures the mean absolute deviation of robustness coefficients from their median, relative to the absolute original coefficient:

$$ I'_{3j} = \frac{mean\left(\left|\beta_i-median(\beta_i)\right|\right)} {\left|\beta^{orig}_j\right|}\times 100. $$

The same significance, direction, and effect-size-comparability restrictions apply as for $I'_{2j}$. When orig_in_multiverse() includes the original specification in the multiverse, it also enters the median and mean absolute deviation.

Interpretation: $I'_{3j}=10%$ means that the average absolute deviation from the cell-specific median equals 10% of the absolute original coefficient. With exactly one eligible multiverse path, the mean absolute deviation is zero.

4. The significance variation indicator ($I'_{4j}$) is the average absolute difference between robustness and original p-values among statistically insignificant robustness paths:

$$ I'_{4j} = mean\left(\left|pval_i-pval^{orig}_j\right|\right). $$

Interpretation: Larger values indicate larger changes in reported p-values. Because differences in p-values are not a linear measure of differences in statistical evidence, $I'_{4j}$ should be interpreted descriptively rather than as a cardinal distance in evidential strength.

5. The effect size agreement indicator ($I'_{5j}$) measures the share of insignificant robustness coefficients that fall inside the confidence interval around an original estimate that is statistically significant at the robustness-analysis threshold $\alpha$:

$$ I'_{5j} = \begin{cases} mean\left(\mathbb{I}\left(\beta(cilo)^{orig}_j \le \beta_i \le \beta(ciup)^{orig}_j\right)\right)\times 100, & \text{if } pval^{orig}_j \le \alpha \land pval_i > \alpha, \\ \text{not applicable}, & \text{otherwise}. \end{cases} $$

The indicator requires comparable effect-size units. It asks whether robustness coefficients fall within the confidence interval of the original estimate, treating the original estimate as the fixed reference; it does not ask whether the original coefficient lies inside confidence intervals around each robustness estimate.

Interpretation: $I'_{5j}=80%$ means that 80% of eligible statistically insignificant robustness coefficients fall within the confidence interval around the original estimate. A high value therefore indicates effect-size agreement despite a change in statistical-significance classification.

6. The indicator on non-agreement due to significance classification ($I'_{6j}$) isolates non-agreement that arises only because the original and robustness analyses use different significance thresholds. The Dashboard cells themselves are always defined using the robustness-analysis threshold $\alpha$.

If the robustness analysis uses the stricter threshold ($\alpha<\alpha^{orig}$), the indicator applies to an originally significant result and requires the robustness coefficient to have the same sign as the original estimate:

$$ I'_{6j} = mean\left(\mathbb{I}(\alpha < pval_i \le \alpha^{orig}) \times \mathbb{I}(\beta_i\times\beta_j^{orig}\ge0)\right)\times100. $$

These paths lie in the statistically insignificant Dashboard cell. The sign restriction is necessary because an opposite-sign path would disagree with the original result for more than the significance threshold alone.

If the robustness analysis uses the less stringent threshold ($\alpha&gt;\alpha^{orig}$), the indicator applies to an originally insignificant result:

$$ I'_{6j} = mean\left(\mathbb{I}(\alpha^{orig} < pval_i \le \alpha)\right)\times100. $$

These paths are statistically significant at $\alpha$. The total $I'_{6j}$ can therefore be split between the significant same-sign and significant opposite-sign Dashboard cells according to coefficient direction. The indicator is zero when the significance thresholds coincide or when the threshold difference does not generate such classification-only non-agreement.

Numerical $I'_{6j}$ shares are retained in the calculated Dashboard-indicator results; Stata additionally reports them in the Results window. Under confirmatory(original), their cell-specific components also determine the nested-circle split.

Interpretation: With $\alpha=0.05$ and $\alpha^{orig}=0.10$, for example, $I'_{6j}=10%$ means that 10% of robustness paths have $0.05&lt;pval_i\le0.10$, retain the original coefficient direction, and would therefore disagree with the original significant result only because the robustness analysis applies the stricter threshold.

7. The confirmatory-share indicator $I'_7$ summarizes the share of robustness paths that are confirmatory across results or studies and is shown in the aggregated Robustness Dashboard. It is based on result-level confirmatory shares $\kappa_j$, which can be defined according to two significance-classification conventions. The selected convention also determines confirmatory/non-confirmatory Dashboard colouring at the result level.

(i) The original-result confirmatory share, $\kappa_j^o$, retains the significance classification of the original result based on $\alpha^{orig}$ while evaluating robustness paths at $\alpha$. Paths identified by $I'_{6j}$ as disagreeing only because the thresholds differ are therefore counted as confirmatory:

$$ \kappa_j^o = \begin{cases} I_{1j}^{',ssign} + I'_{6j}, &amp; \text{if } pval_j^{orig} \le \alpha^{orig}, \\ 100 - I_{1j}^{',ssign} - I_{1j}^{',nsign} + I'_{6j}, &amp; \text{if } pval_j^{orig} &gt; \alpha^{orig}. \end{cases} $$

(ii) The uniform-threshold confirmatory share, $\kappa_j^u$, applies the robustness-analysis threshold $\alpha$ to both the original result and robustness paths:

$$ \kappa_j^u = \begin{cases} I_{1j}^{',ssign}, & \text{if } pval_j^{orig} \le \alpha, \\ 100 - I_{1j}^{',ssign} - I_{1j}^{',nsign}, & \text{if } pval_j^{orig} > \alpha. \end{cases} $$

Here, ssign and nsign denote statistically significant robustness paths with the same and opposite sign as the original estimate. Operationally, repframe classifies confirmation at the analytical-path level, including the cell-specific $I'_6$ components where thresholds differ, and only then aggregates the resulting shares across results or studies. If $\alpha=\alpha^{orig}$, $\kappa_j^o=\kappa_j^u$.

In the aggregated Dashboard, $I'_7$ is reported as $\bar{\kappa}^o$ under confirmatory(original) and as $\bar{\kappa}^u$ under confirmatory(uniform). Neither convention is unambiguously preferable when the thresholds differ. $\kappa^o$ asks whether robustness paths confirm the significance classification reported for the original result under the thresholds actually used in the two analyses; $\kappa^u$ asks whether original and robustness results agree under a common threshold. The selected convention determines confirmatory/non-confirmatory Dashboard colouring; original is the default.

Interpretation: $\bar{\kappa}^o=80%$ or $\bar{\kappa}^u=80%$ means that, after aggregation across results or studies, 80% of robustness paths are confirmatory under the respective classification rule.

Extended set of Robustness Dashboard indicators

customindicators(SIGswitch) additionally shows two significance-switch indicators. They ask whether changes in the coefficient or standard error are individually large enough to account for a change in significance status when the other quantity is held at its original value.

For the coefficient-based indicator,

$$ I'_{8j} = \begin{cases} mean\left(\mathbb{I}(|\beta_i|<\beta(tonsig)_j)\right)\times100, & \text{if } pval^{orig}_j\le\alpha \land pval_i>\alpha, \\ mean\left(\mathbb{I}(|\beta_i|\ge\beta(tosig)_j)\right)\times100, & \text{if } pval^{orig}_j>\alpha \land pval_i\le\alpha. \end{cases} $$

For the standard-error-based indicator,

$$ I'_{9j} = \begin{cases} mean\left(\mathbb{I}(se_i>se(tonsig)_j)\right)\times100, & \text{if } pval^{orig}_j\le\alpha \land pval_i>\alpha, \\ mean\left(\mathbb{I}(se_i\le se(tosig)_j)\right)\times100, & \text{if } pval^{orig}_j>\alpha \land pval_i\le\alpha. \end{cases} $$

The indicators require comparable effect-size units.

Interpretation: For an originally significant result, $I'_{8j}=30%$ means that 30% of insignificant robustness paths have coefficients small enough to account for the switch to insignificance even if the standard error were unchanged. The other cases are interpreted analogously.

Advanced: Dashboard using weak-IV-robust inference

The Robustness Dashboard can be extended to incorporate adjustments that reflect robust hypothesis testing in the presence of weak instruments. Three approaches are supported, each allowing the dashboard to additionally provide a weak-IV-robust version of the significance agreement indicator.

1. tF adjustment: Developed by Lee, Moreira, and co-authors, the tF approach adjusts conventional standard errors and p-values based on the first-stage F-statistic. The user must supply F-statistics for each specification via the tFinput() option. The adjustment applies a smooth correction factor to standard errors, improving inference under weak instruments relative to the standard 2SLS t-test.
2. VtF adjustment: The VtF approach (also by Lee et al.) requires the user to supply VtF-adjusted critical values, such as those obtained using the user-written VtF command in Stata (link). This is also done via the tFinput() option. Unlike tF, VtF does not yield adjusted standard errors or p-values. Instead, the dashboard uses the critical values to calculate a VtF-adjusted significance indicator at the 5% or 1% level. VtF leverages additional information—specifically, the empirical correlation between residuals from the 2SLS regression—making it more powerful than tF. It yields shorter confidence intervals and improves the likelihood of detecting true effects under weak instruments. When using this option, the dashboard expects that the provided critical values correspond to the 5% level (if the applied significance level is above 1%) or to the 1% level (if the applied level equals 1%). Both tF and VtF ensure correct test size but do not resolve the power asymmetry problem—the tendency of t-based IV inference to favour one direction of effect over the other under weak instruments. As emphasized by Keane & Neal (2024), this can lead to misleading conclusions about the sign of the effect. In addition, both methods are only applicable to just-identified models with a single instrument.
3. Anderson–Rubin (AR) test: The AR test provides a weak-IV-robust alternative that avoids power asymmetry and maintains valid inference regardless of instrument strength. The dashboard can incorporate AR p-values, which the user supplies via the pval_ar() option (e.g., extracted from the weakiv command in Stata). This method is particularly attractive in the single-instrument case, where the AR test has a known finite-sample distribution. However, AR-based inference is typically less powerful than that based on the VtF adjustment.

In addition to these three inference adjustments, the dashboard can include an alternative indicator that interprets weak instruments as less problematically:

4. Sign disagreement based on first-stage direction: As argued in Angrist & Kolesár (2024), IV specifications with a first-stage coefficient in the unexpected direction are the ones most likely to generate misleading inference under weak instruments. The signfirst() option therefore allows the user to flag the proportion of specifications in which the first-stage coefficient has the wrong sign. Rather than adjusting existing dashboard indicators, this additional first-stage-sign diagnostic offers a less conservative approach to diagnosing weak-IV problems. It helps identify the extent to which robustness analyses include specifications that are not just weak, but problematic from the perspective of directional validity of the instrument.

The Dashboard output

The Robustness Dashboard provides a graphical summary of how robustness results compare with the original findings. The horizontal axis separates statistically insignificant paths, statistically significant paths with the opposite sign, and statistically significant paths with the same sign as the original estimate. Circle sizes show the corresponding shares of robustness paths ($I'_{1j}$).

For individual results, each row represents one original result. The original estimate and its p-value are shown on the left. Depending on the selected options, additional indicators displayed around the circles summarize changes in statistical significance and effect size. The figure below illustrates the main Dashboard indicators and where they appear. For illustration, it shows the full set of Dashboard indicators and a case in which the original and robustness analyses use different significance thresholds.

_{Note: $I'_6$, which identifies disagreement attributable to different significance thresholds, is retained in the calculated Dashboard-indicator results rather than displayed as a separate Dashboard label. Stata additionally reports it in the Results window.}

 

Circle colours distinguish paths that are confirmatory or non-confirmatory in terms of their significance classification. Under confirmatory(original) (the default), the significance classification of the original result is based on the original-study threshold $\alpha^{orig}$, while robustness paths are evaluated at the robustness-analysis threshold $\alpha$. Under confirmatory(uniform), $\alpha$ is applied uniformly to both the original result and the robustness paths. Darker circles indicate confirmatory paths and lighter circles non-confirmatory paths under the selected convention.

When $\alpha^{orig}$ and $\alpha$ differ, confirmatory(original) can divide a Dashboard cell into confirmatory and non-confirmatory components, which are shown as nested circles. Under confirmatory(uniform), each cell has a single significance classification and nested circles are therefore not needed.

The Dashboard can also aggregate results within or across studies. In this case, the vertical axis groups original results by their significance status, and the circle sizes represent the corresponding aggregated shares. A histogram below the Dashboard shows the distribution of confirmatory shares ($\kappa^o$ or $\kappa^u$, depending on confirmatory()) and absolute relative effect-size changes across results or studies. The mean confirmatory share is reported as $\bar{\kappa}^o$ under confirmatory(original) and $\bar{\kappa}^u$ under confirmatory(uniform); where the two differ, the alternative value is also reported as a sensitivity reference.

 

The harmonized analysis-path data additionally store the number of eligible paths underlying each Dashboard cell and conditional indicator (dashN_*). Users can inspect these variables to assess whether adjacent cells or indicators are based on substantially different numbers of eligible paths.

Complementary coefficient, contribution, and heatmap plots

repframe can complement the Robustness Dashboard with additional plots that visualize how estimates vary across specifications, which analytical decisions drive that variation, and how robustness indicators differ across combinations of decisions:

• Coefficient Plot: Displays point estimates and confidence intervals for the original estimate and the preferred robustness-analysis path—as specified via prefpath() in Stata or prefpath in R. Estimates are expressed relative to the corresponding outcome mean, facilitating comparison across results.

 

• Contribution Plots: Show how individual analytical decisions and choices contribute to changes in the robustness indicators. Each of the following plots can be created for both the significance agreement indicator ($I'{1}$) and the relative effect size indicator ($I'{2}$). For $I'_{2}$, only statistically significant robustness paths in the same direction as the original estimate are included:

  • Single-Choice Deviation Plot: Shows the change in an indicator when one analytical decision is fixed to a particular choice, while the other decisions vary.

  • Stepwise Deviation Plot: Groups paths by how many analytical decisions deviate from their original choices, with the remaining decisions fixed to the original choices, and shows the corresponding change in the indicator.

  • Single-Decision Reversion Plot: Shows the change in an indicator when one analytical decision is fixed to its original choice, while the other decisions vary.

  The figure below illustrates all three contribution plots for significance agreement and relative effect size. Together, these plots help identify which analytical decisions and choices contribute most strongly to variation in the robustness results.

 

• Decision-Combination Heatmaps: Show significance agreement, analysis-path p-values, changes in p-values relative to the original, and, for originally significant results, relative effect-size changes across joint combinations of analytical decisions. Decision dimensions can be assigned to the x- and y-axes and to panel rows and columns; multiple dimensions can be combined within a panel dimension. This makes interactions and systematic patterns across the multiverse visible that may not emerge from the individual contribution plots.

  The figure below illustrates these four decision-combination heatmaps, separately for each result.

_{Note: The presentation is inspired by Moors and Hesselmann (2019).}

Together, the contribution plots identify influential individual decisions and choices, while the heatmaps reveal patterns and interactions across their joint combinations.

In R, the underlying plot data and corresponding ggplot objects are returned separately. For example:

dat <- repframe_example("complementary")

result <- repframe(
  dat,
  mainvar = "result",
  beta = "beta",
  se = "se",
  pval = "pval",
  siglevel = 5,
  origpath = "origpath",
  shortref = "Complementary plots",
  outcome_mean = "outcome_mean",
  sameunits = "sameunits",
  orig_in_multiverse = "orig_include",
  prefpath = "prefpath",
  decisions = c("covariates", "sample", "estimator", "outcome"),
  heatmapdims = list(
    x = "outcome", y = "estimator", cols = "covariates", rows = "sample"
  )
)

result$coefficient_plot
result$contribution_plots$SIGagr$deviation
result$heatmap_plots$pval$r1

dir.create("repframe-plots", showWarnings = FALSE)
export_repframe_complementary_plots(
  result,
  directory = "repframe-plots",
  format = "png"
)

Advanced: Indicators with RDD robust bias-corrected inference

To combine conventional Regression Discontinuity Design (RDD) point estimates with robust bias-corrected (RBC) inference following Calonico et al. (2014), the following inputs can be used:

• beta() = conventional coefficient, i.e. the conventional point estimate without bias correction
• se() = robust bias-corrected standard error, as reported by the RDD estimation command
• pval() = robust bias-corrected p-value, as reported by the RDD estimation command

For example, these quantities can be obtained from rdrobust, which implements conventional point estimation alongside robust bias-corrected inference in Stata, R, and Python.

This setup retains conventional coefficients for the interpretation of effect sizes while classifying estimates as statistically significant or insignificant based on RBC inference. Users should provide both se() and pval() explicitly, since the RBC standard error and p-value pertain to the bias-corrected estimator, whereas the coefficient supplied to beta() is the conventional estimator.

Effect-size indicators that use coefficients alone retain their conventional effect-size interpretation. Indicators that combine the conventional coefficient with the RBC standard error, however, should be interpreted with caution. In particular, the effect size agreement indicator ($\beta$ in $CI(\beta^o)$) constructs confidence bounds from this combination and therefore does not correspond exactly to either a conventional or an RBC confidence interval.

Related multiverse and specification-curve software

repframe summarizes results from completed analysis paths; it does not estimate the underlying models. It can complement other multiverse and specification-curve tools through three workflows:

• Direct reuse: use the path-level results created by repframe with the other plotting tool after minor transformations.

• Estimate and transfer: run the specification analysis with the other tool, prepare its resulting path-level estimates as repframe input, and interpret both outputs jointly.

• Separate use: since their input structures are not directly convertible, run the other tool on the original data, prepare the corresponding path-level results independently for repframe, and interpret both outputs jointly.

repframe does not install, execute, or depend on any of the tools listed below.

Stata

Command or resource	Main output	Recommended joint use with repframe
mfcurve	Specification curve	Use mfcurve and repframe separately and interpret both outputs jointly; their input structures are not directly convertible.
mvplot	Multiverse plot combining an estimate-density plot with an analytical-choice panel, as known from specification curves, and estimates of how the choices influence the results (see also Krähmer and Young 2026)	Use multivrs to estimate the specifications, visualize its results with mvplot, prepare the same path-level estimates as repframe input, and interpret both outputs jointly.
mrobust, multivrs	Estimate-density plot, summary statistics, and estimates of model-ingredient influence	Use the command to estimate the specifications, prepare the resulting path-level estimates as repframe input, and interpret both outputs jointly.
specc, speccurve, Sievertsen’s speccurve example, specurve	Specification curves	Use the command or example code to estimate the specifications, prepare the resulting path-level estimates as repframe input, and interpret both outputs jointly.

R

Package or function	Main output	Recommended joint use with repframe
rdfanalysis ::plot_rdf_spec_curve()	Specification curve	Use the repframe path data with plot_rdf_spec_curve() after adding the required confidence-interval bounds (see note below this table), and interpret both outputs jointly.
schart() from spec_chart	Specification curve	Use the repframe path data with schart() after selecting the estimate and standard-error columns and encoding the analytical choices as logical indicators, and interpret both outputs jointly.
mvplot R implementation	see mvplot under Stata	see mvplot under Stata
multiverse	Executed multiverse and path-level results.	Use multiverse to estimate the specifications, prepare the resulting path-level estimates as repframe input, and interpret both outputs jointly.
mverse	Specification curve, specification table, and multiverse tree	Use mverse to estimate the specifications and spec_curve() to visualize them, prepare the resulting path-level estimates as repframe input, and interpret both outputs jointly.
mfcurve	Specification curve	Use mfcurve and repframe separately and interpret both outputs jointly; their input structures are not directly convertible.
speccurvieR, specr, starbility	Specification curves, with speccurvieR additionally featuring diagnostic plots, joint inference, and variance decomposition, and specr optionally featuring joint-inference results	Use the package to estimate the specifications, prepare the resulting path-level estimates as repframe input, and interpret both outputs jointly.

data preparation for rdfanalysis

For example, a 95% normal-approximation confidence interval for rdfanalysis can be added with:

paths$lb <- paths$beta_i - qnorm(0.975) * paths$se_i
paths$ub <- paths$beta_i + qnorm(0.975) * paths$se_i

rdfanalysis::plot_rdf_spec_curve(
  paths, "beta_i", "lb", "ub"
)

Python

Python packages can also be called from R through reticulate and from Stata 16 or newer through Stata’s Python integration.

Package	Main output	Recommended joint use with repframe
RobustiPy, specification_curve	Specification curves, RobustiPy additionally features bootstrap inference, model-fit and validation plots, model averaging, and variable-influence results.	Use the package to estimate the specifications, prepare the resulting path-level estimates as repframe input, and interpret both outputs jointly.

Update log

2026-08-31, v1.8.0

Stata

• Refactor the code structure to facilitate translation to R while retaining existing functionality, including clearer separation of processing steps, more systematic variable and indicator generation, and reduced reliance on Stata-specific coding patterns.

• Add and extend functionality:

  • Replace ivarweight() with the more general rfweight() option. Depending on the pooling mode, indicators can be weighted equally across results or studies, by the number of robustness-analysis paths, by precision, or by a user-supplied weighting variable. The earlier inverse-variance terminology is replaced by rfweight(precision) to clarify that the implemented scheme is precision-based rather than conventional inverse-variance meta-analysis weighting. Precision weights are defined using standard errors relative to the absolute outcome mean, preventing the sign of the outcome mean from affecting or offsetting the precision measure.
  • Extend pooling across studies: studypooling(2) allows indicators to be calculated directly from appended analysis-path-level data, retaining study–result combinations as the result-level units before aggregation across studies. Under studypooling(2), rfweight(studies) uses indicator-specific two-stage aggregation, giving equal total weight to each contributing study. Original-analysis significance levels may differ across studies and are retained from the generated path data, while being required to be constant within each study; user-supplied result weights can likewise be added to appended path data before pooling.
  • Generalize the significance-classification agreement indicator $I'_{7}$ into two confirmatory-share variants, $κ^o$ and $κ^u$: the former retains the original-study significance classification, while the latter applies the robustness-analysis significance level uniformly to original and robustness results. Add option confirmatory(original|uniform) to select which definition determines Dashboard colouring and the primary $\bar{\kappa}$, with original as the default.
  • Add LaTeX (.tex) output for the Reproducibility and Replicability Indicators, and a separate Dashboard Indicators (RF2) table for pooled analyses (studypooling(1) and studypooling(2)).
  • Add datasave() and replace options for explicit control over saved outputs. datasave() determines whether the input, generated analysis-path-level, and study-level datasets are saved, with datasave(paths studies) as the default under studypooling(0) and datasave(none) otherwise. replace permits existing user-facing data, table, and figure output files to be overwritten; by default, existing output files are not overwritten.
  • Store eligible-path counts for Dashboard cells and conditional indicators as dashN_* variables in the saved analysis-path-level dataset.
  • Add complementary heatmaps showing path-level significance agreement, p-values, changes in p-values relative to the original, and, for originally significant results, relative effect-size changes across analytical choices, with the original specification highlighted.
  • Add structured returned results through r(). repframe is now declared as an rclass command that returns key settings, sample and specification counts, and generated output paths to facilitate programmatic use and automation.
  • Store repframe version used to create analysis- and study-level datasets under repframe_version.

• Revise the Robustness Dashboard:

  • Show the significance variation indicator only for statistically insignificant robustness paths;
  • Calculate and show the relative effect size indicator and effect size variation indicator separately for statistically significant same-sign and statistically insignificant robustness paths;
  • When original and robustness analyses use different significance thresholds, use nested circles with different colouring to distinguish threshold-induced classification components under confirmatory(original); under confirmatory(uniform), use a single confirmatory/non-confirmatory classification for each Dashboard cell.
  • Show the statistical significance indicator at an alternative conventional significance threshold also for statistically insignificant robustness paths when this directly reflects differing significance classifications in the original and robustness analyses.
  • Correct the calculation and Dashboard representation of $I'_{6j}$ so that threshold-only non-agreement is identified at the analytical-path and Dashboard-cell level, including the required same-sign restriction for originally significant results and the possible split across both significant cells for originally insignificant results. Confirmatory/non-confirmatory nested-circle components are now aggregated directly from these path-level classifications rather than reconstructed algebraically from aggregate indicators.

• Make siglevel_orig() optional with studypooling(0); if omitted, the significance level specified through siglevel() is also applied to the original result. With studypooling(2), original-study significance levels are instead inherited from the appended path data.

• Save data at level of analytical path with uniform set of variables and uniform naming as repframe_paths_[fileidentifier].dta instead of as repframe_data_analysispaths_[fileidentifier].dta.

• Improve consistency and handling of edge cases:

  • Apply the same result weights to aggregated original-estimate reference statistics as to the corresponding result-level indicators;
  • Use original-path degrees of freedom when constructing reference confidence intervals and significance-switch thresholds;
  • Make the existing adjustment of rounded Dashboard cell shares to a total of 100% deterministic by adding explicit tie-breaking when multiple cells qualify for the required one-percentage-point adjustment. Secondary nested-circle components are rounded independently rather than being altered by the main three-cell rounding reconciliation.
  • Make the treatment of exact-zero coefficients, minimum eligible sample sizes, and two-coefficient results explicit and consistent across calculations; align the exact-boundary inequalities of the significance-switch indicators with the convention that $p=\alpha$ is statistically significant.
  • Improve handling of partial effect-size comparability (sameunits()), so that effect-size indicators are calculated and displayed for Dashboard cells with eligible same-unit paths even when other paths for the same result use different units; explicitly identify and report such mixed cases through sameunits_partial_j and sameunits_none_j.
  • Simplify and align output-data variable labels.
  • Fix an error in Dashboard generation at a 1% significance level caused by reference to an unavailable alternative-threshold aggregate indicator.
  • Fix an error in the processing of results represented by two coefficients that could refer to a nonexistent internal original-path variable.

• Improve internal processing, error handling, and consistency checks, including:

  • Simplify studypooling(1) by retaining explicit significance categories and the study-level share of originally significant results rather than reconstructing these quantities from artificial reference values; store the cell-specific confirmatory components required for exact pooled Dashboard colouring, so study-level datasets generated by earlier pre-release v1.8.0 code should be regenerated;
  • Provide informative messages for deprecated options and additional assertions to detect inconsistent input data or internal calculations.
  • Add support for apostrophes in graph-label text. Embedded double quotation marks remain unsupported (e.g., an outcome labelled Student knows how to spell "stupid").
  • Use double precision systematically for internally generated statistical quantities to reduce floating-point discrepancies, including in Stata–R output comparisons.

• Remove a typo from the 5% tF lookup grid used for instrumental-variable estimation input.

R

• Add the first R implementation of repframe, resolving Issue #2. It provides the common indicator calculations, aggregation modes, Reproducibility and Replicability Indicators tables, Dashboard Indicators tables, Robustness Dashboards, weak-IV adjustments, and deterministic example datasets.
• Provide an object-oriented interface: repframe() returns a reusable R object containing harmonized analysis-path data, result- and study-level indicators, aggregate results, tables, Dashboard data and plots, settings, and structured messages. Files are written only through explicit export functions.
• Support explicit export of Dashboards, indicator tables (.csv, .xlsx, and .tex), and analysis-path and study-level data (.rds, .csv, and .dta).
• Validate the R calculations against deterministic golden outputs generated by the Stata implementation, including pooling and weighting modes, significance boundaries, weak-IV adjustments, edge cases, saved datasets, and exported numeric table content.
• Add R versions of the complementary coefficient, contribution, and decision-combination heatmap plots, with reusable plot data, returned ggplot objects, and explicit export through export_repframe_complementary_plots().
• Add repframe_example("complementary") as a ready-to-run example containing the analytical decisions and additional inputs required by the complementary plots.
• Translate, iteratively audit, and debug the initial R implementation with assistance from OpenAI's ChatGPT. The package author reviewed the implementation, determined the substantive and interface choices, and validated it through the automated test suite and cross-language comparisons.

2025-06-21, v1.7.1

• Dashboard update with publication of the working paper The Robustness Dashboard, Bensch et al. (2025):
  • If siglevel(5) is specified, not only show significance agreement indicator for p $\le$ 0.05 as the main value in the bubble, but also include the same indicator calculated for p $\le$ 0.10 in a smaller label beneath the bubble. This provides a sensitivity check that allows users to assess whether agreement in statistical significance is robust to adopting a slightly more lenient threshold, as is often used in original studies.

2025-05-16, v1.7

• Replace option extended(string) by customindicators(string) in order to additionally allow for a "skeleton" version of the Robustness Dashboard that only includes the bubbles with the statistical significance indicator, which can be called via customindicators(SIGagronly).
• Update how Robustness Dashboard can account for weak instruments:
  • Replace option iVF() by tFinput(string varname) to allow for both tF and VtF adjustment as proposed by Lee et al. (2022) and Lee et al. (2023).
  • Add option pval_ar(varname) to also allow for weak-IV adjustment following Anderson and Rubin (1949).
  • If aggregation(0) and studypooling(0) and tFinput(tF ...) or pval_ar() being defined, add to the original estimate the tF- or AR-adjusted p-value resulting from the analytical path of the original study.
• Remove "0%" from dashboards generated with aggregation(1) when the value is not applicable—that is, when no results in that category exist, such as when no original estimates were significant or insignificant.
• Fix minor bugs occurring with studypooling(1).

2024-12-06, v1.6

• Include option decisions() that allows accounting for the analytical decisions taken; if option decisions() is defined, repframe creates a set of three complementary plots that show the contributions of individual decisions to deviations in indicator values.
  • Because of that inclusion, the input data structure had to be revised in that the specification adopted by the original analysis is included as an individual analytical path, and not only via the variables ending with _orig; this allows for a straightforward way of specifying the decisions() taken by the original authors for each of the results. Another advantage of this data structure is that it corresponds to the data structure used by related commands such as speccurve.
  • in the process of this revision, the option origpath() was introduced to specify the specifications in the original analysis, one for each result.
  • relatedly, all _orig options—except siglevel_orig—have been removed from the repframe command.
• Include option prefpath(). If a preferred new specification can be identified in a robustness test, this option allows identifying such a preferred new specification, and repframe creates another complementary plot that compares the original estimate with this preferred estimate of the robustness test.
• The option orig_in_multiverse() now requests a variable instead of a simple 0-1 indicator; this allows for setting this option differently for individual results.
• Resolve Issue #1, that is providing an explanation of beta_rel_orig in the github Readme.
• Data at level of analytical path with uniform set of variables and uniform naming stored as repframe_data_analysispaths_[fileidentifier].dta.
  • in the process of this revision, the name under which the study-level data is stored has been changed to repframe_data_studies_[fileidentifier].dta (from repframe_data_[fileidentifier].dta).
• Adjustments affecting the Robustness Dashboard:
  • inclusion of effect size agreement indicator into the default set of indicators; accordingly, the option extended() can now only be set to "none" or "SIGswitch".
  • split longer results names into two lines.
  • minor adjustment in line spacing of Dashboard for newer Stata versions (version ${\ge}$ 16).
• Text revisions, among others regarding the use of the term "result".

2024-06-03, v1.5.2

• Minor adjustments in Sensitivity Dashboard:
  • rename dashboard to Robustness Dashboard, including the option sensdash(), which is now called dashboard().
  • correct calculation of the effect size variation indicator when sameunits(variable==0) and orig_in_multiverse(1) applies for any analytical path.
  • correct calculation of the indicator on non-agreement due to significance classification when aggregation(1).
  • remove slight inconsistency in rounding if sum of shares would exceed or fall below 100%.
  • adjust colouring of confirmatory and non-confirmatory results.
  • extend indicator on non-agreement due to significance classification to situations in which an original study applied a less stringent classification of what constitutes a statistically significant result than the robustness analysis.
  • show histogram with share of confirmatory results (${\kappa}$) and absolute values of effect sizes ($|{\beta}|$) at bottom of dashboard when aggregation(1).
  • inclusion of the overall significance (and sig. classification) agreement indicator.

2024-03-17, v1.5.1

• Improve MacOS compatibility.
• Revise table output, including added tabfmt() option.
• Fix a bug that occurred when assessing only one result.

2024-03-05, v1.5

• Fix minor bugs occurring with studypooling(1).
• Clarify that repframe only works with Stata 14.0 or higher.
• Introduce option studypooling() to repframe_gendata command and include illustrative example on pooling across studies in Stata help file.

2024-03-04, v1.4.2

• Adjust the option extended() to allow for multiple choices.
• Extend the application of the significance variation indicator in the Sensitivity Dashboard to originally insignificant results with significant robustness-analysis paths.
• Add examples to the Stata help file.
• Minor revisions of the code.

2024-02-29, v1.4.1

• Make options siglevel() and siglevel_orig() compulsory for analyses at study level.
• Add recommendation to include both the information on p-values and standard errors at study level.

2024-02-28, v1.4

• Add option siglevel_orig() to allow testing against significance level adopted in the original analysis; incorporated as an indicator on significance classification into the Sensitivity Dashboard.
• Additional effect size agreement / confidence interval coverage indicator and additional notes to Sensitivity Dashboard and Reproducibility and Replicability Indicators table.
• Produce Reproducibility and Replicability Indicators table for indicators pooled across studies.
• Remove certain requirements to the input data formatting.
• Use NHANES II data for the example in the help file, among others to have multiple results that effectively differ from each other.
• Revise entire command structure and adopt uniform naming convention.

2024-02-13, v1.3.1

• Minor amendments to the code.

2024-01-22, v1.3

• Add the option studypooling() to calculate indicators across studies.

2024-01-19, v1.2

• Incorporate the package sensdash().

2024-01-18, v1.1

• First version of repframe package.

References

Anderson, T. W., & Rubin, H. (1949). Estimation of the parameters of a single equation in a complete system of stochastic equations. The Annals of Mathematical Statistics, 20(1), 46-63.

Angrist, J., & Kolesár, M. (2024). One instrument to rule them all: The bias and coverage of just-ID IV. Journal of Econometrics, 240(2), 105398. doi: 10.1016/j.jeconom.2022.12.012.

Ankel-Peters, J., Brodeur, A., Dreber, A., Johannesson, M., Neubauer, F., & Rose, J. (2025). A protocol for structured robustness reproductions and replicability assessments. Q Open, 5(3), qoaf004. doi: 10.1093/qopen/qoaf004.

Athey, S., & Imbens, G. W. (2015). A measure of robustness to misspecification. American Economic Review, 105(5), 476-480. doi: 10.1257/aer.p20151020.

Bartoš, F., Hoogeveen, S., Sarafoglou, A., & Pawel, S. (2025). Single-Dataset Meta-Analysis For Many-Analysts And Multiverse Studies. arXiv preprint arXiv:2511.17064. doi: 10.48550/arXiv.2511.17064.

Bensch, G., Rose, J., Brodeur, A., & Ankel-Peters, J. (2025). The Robustness Dashboard. I4R Discussion Paper No. 234. Available at https://hdl.handle.net/10419/319180.

Buzbas, E. O., & Devezer, B. (2024). Statistics in service of metascience: Measuring replication distance with reproducibility rate. Entropy, 26(10), 842. doi: 10.3390/e26100842.

Calonico, S., Cattaneo, M. D., & Titiunik, R. (2014). Robust nonparametric confidence intervals for regression‐discontinuity designs. Econometrica, 82(6), 2295-2326. doi: 10.3982/ECTA11757.

Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433-1436. doi: 10.1126/science.aaf0918.

Cantone, G. G., & Tomaselli, V. (2024). Theory and methods of the multiverse: an application for panel-based models. Quality & Quantity, 58(2), 1447-1480. doi: 10.1007/s11135-023-01698-5.

Del Giudice, M., & Gangestad, S. W. (2021). A traveler’s guide to the multiverse: Promises, pitfalls, and a framework for the evaluation of analytic decisions. Advances in Methods and Practices in Psychological Science, 4(1). doi: 10.1177/2515245920954925.

Dreber, A., & Johannesson, M. (2025). A framework for evaluating reproducibility and replicability in economics. Economic Inquiry, 63(2), 338-356. doi: 10.1111/ecin.13244.

Gelman, A., & Stern, H. (2006). The difference between "significant" and "not significant" is not itself statistically significant. The American Statistician, 60(4), 328-331. doi: 10.1198/000313006X152649.

Götz, M., Sarma, A., & O'Boyle, E. H. (2024). The multiverse of universes: A tutorial to plan, execute and interpret multiverses analyses using the R package multiverse. International Journal of Psychology, 59(6), 1003-1014. doi: 10.1002/ijop.13229.

Held, L., Micheloud, C., & Pawel, S. (2022). The assessment of replication success based on relative effect size. The Annals of Applied Statistics, 16(2), 706-720. doi: 10.1214/21-AOAS1502.

Keane, M. P., & Neal, T. (2024). A practical guide to weak instruments. Annual Review of Economics, 16, 185-212. doi: 10.1146/annurev-economics-092123-111021.

Klau, S., Hoffmann, S., Patel, C. J., Ioannidis, J. P. A., & Boulesteix, A.-L. (2021). Examining the robustness of observational associations to model, measurement and sampling uncertainty with the vibration of effects framework. International Journal of Epidemiology, 50(1), 266-278. doi: 10.1093/ije/dyaa164.

Klau, S., Schönbrodt, F. D., Patel, C. J., Ioannidis, J. P. A., Boulesteix, A.-L., & Hoffmann, S. (2023). Comparing the vibration of effects due to model, data pre-processing and sampling uncertainty on a large data set in personality psychology. Meta-Psychology, 7, MP.2020.2556. doi: 10.15626/MP.2020.2556.

Krähmer, D., & Young, C. (2026). Visualizing vastness: Graphical methods for multiverse analysis. PLoS One, 21(2), e0339452. doi: 10.1371/journal.pone.0339452.

Lee, D. S., McCrary, J., Moreira, M. J., & Porter, J. (2022). Valid t-ratio Inference for IV. American Economic Review, 112(10), 3260-3290. doi: 10.1257/aer.20211063.

Lee, D. S., McCrary, J., Moreira, M. J., Porter, J. R., & Yap, L. (2023). What to do when you can't use '1.96' Confidence Intervals for IV. National Bureau of Economic Research Working Paper No. w31893.

Mathur, M. B., & VanderWeele, T. J. (2020). New statistical metrics for multisite replication projects. Journal of the Royal Statistical Society Series A: Statistics in Society, 183(3), 1145-1166. doi: 10.1111/rssa.12572.

Moors, P., & Hesselmann, G. (2019). Unconscious arithmetic: Assessing the robustness of the results reported by Karpinski, Briggs, and Yale (2018). Consciousness and Cognition, 68, 97–106. doi: 10.1016/j.concog.2019.01.003.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. doi: 10.1126/science.aac4716.

Patel, C. J., Burford, B., & Ioannidis, J. P. A. (2015). Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. Journal of Clinical Epidemiology, 68(9), 1046-1058. doi: 10.1016/j.jclinepi.2015.05.029.

Patil, P., Peng, R. D., & Leek, J. T. (2016). What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspectives on Psychological Science, 11(4), 539-544. doi: 10.1177/1745691616646366.

Pawel, S., & Held, L. (2022). The sceptical Bayes factor for the assessment of replication success. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3), 879-911. doi: 10.1111/rssb.12491.

Short, C. A., Breznau, N., Bruntsch, M., Burkhardt, M., Busch, N. A., Cesnaite, E., ... & Hildebrandt, A. (2026). Multicurious: A multidisciplinary guide to multiverse analysis. Advances in Methods and Practices in Psychological Science, 9(2), 25152459261434881. doi: 10.1177/25152459261434881.

Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020). Specification curve analysis. Nature Human Behaviour, 4(11), 1208-1214. doi: 10.1038/s41562-020-0912-z.

Slough, T., & Tyson, S. A. (2025). Sign-congruence, external validity, and replication. Political Analysis, 33(3), 195-210. doi: 10.1017/pan.2024.26.

Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702-712. doi: 10.1177/1745691616658637.