02-select.Rmd

```{r setup-select, echo = FALSE, warning=FALSE, message=FALSE}
library(tidyverse)
library(knitr)
library(kableExtra)
```

# Selecting a paper {#select}

The goal of this stage is to help you select the paper that you will do your reproduction on. You might first consider multiple papers without analyzing them more closely (we refer to these as **candidate papers**) before moving forward with your **declared paper**. 

The main difference between a candidate and a declared paper is the availability of a reproduction package. A **reproduction package** is the collection of materials that make it possible to reproduce a paper. This package may contain data, code, or documentation. If you are unable to independently locate the reproduction package for your paper, you can ask the paper's author for it (find guidance on this in [Chapter 7](https://bitss.github.io/ACRE/guidance-for-a-constructive-exchange-between-reproducers-and-original-authors.html)) or simply choose another candidate paper. If you still want to explore the reproducibility of a paper with no reproduction package, these guidelines provide instructions for requesting materials from authors to create a public reproduction package, or if this proves unsuccessful, for building your reproduction package from scratch.

To avoid duplicating the efforts of others who may be interested in reproducing one of your candidate papers, **we ask that you record your candidate papers in the SSRP**.

Note that in this stage, *you are not expected to review the reproduction materials in detail*, as you will dedicate most of your time to this in later stages of the exercise. 

## From candidate to declared paper {#declare}

At this point of the exercise, you are *only verifying the availability* of (at least) one reproduction package and not assessing the quality of its content. Follow the steps below to verify that a reproduction package is available, and stop whenever you find it (this may mean that you have found your declared paper).

  1. Check whether previous reproduction attempts have been recorded in the [SSRP](https://www.socialsciencereproduction.org/) for the paper.
  2. Check the journal or publisher's website, looking for materials named "Data and Materials," "Supplemental Materials," "Reproduction/Replication Package/Materials," etc.  
  3. Look for links in the paper (review the footnotes and appendices).  
  4. Review the personal websites of the paper's author(s).
  5. Contact the author(s) to request the reproduction package using [this](#contacting-the-original-authors-when-there-is-no-reproduction-package) email template. In this and future interactions with authors, we encourage you to follow our guidance outlined in [Chapter 7](#contacting-the-original-authors-when-there-is-no-reproduction-package).
  6. Deposit the reproduction package in a trusted repository (e.g., [Dataverse](https://dataverse.org/), [Open ICPSR](https://www.openicpsr.org/openicpsr/), [Zenodo](https://zenodo.org/), or the [Open Science Framework](https://osf.io/)) under the name `Original reproduction package for - Title of the paper`. You will be asked to provide the URL of the repository in Survey 1.

In case you need to contact the authors, make sure to *allocate sufficient time for this step* (we suggest at least three weeks before the date you plan to start the reproduction). Instructors should also plan to accordingly (e.g., if the ACRE exercise is expected to take place in the middle of the semester, students should review candidate papers and (if applicable) contact the authors in the first few weeks of the semester).

Review the decision tree (Figure \@ref(fig:candidate-paper-dec-tree)) below for a more detailed overview of this process. Remember, *if at any step of the process you decide to abandon the paper, make sure to record the candidate paper in the SSRP* before moving on to another candidate paper. Once you have obtained the reproduction package, the *candidate paper* becomes your *declared paper* and you can move forward with the exercise! Do not invest time in doing a detailed read of any paper until you are sure that it is your declared paper.

### Candidate paper entries in the SSRP {#check-acre}

If the SSRP  contains previous reproduction attempts of the paper, you will see a report card with the following information:   

>**Box 1:** Summary Report Card for SSRP Paper Entry     
> **Title:**  Sample Title   
> **Authors:**  Jane Doe & John Doe  
> **Original Reproduction Package Available:** Yes (link)/No.  
> [If "No"] **Contacted Authors?:** Yes/No   
> [If "Yes(contacted)"] **Type of Response:** Categories (6).   
> **Additional Reproduction Packages:** Number (eg., 2)     
> **Authors Available for Further Questions for SSRP Reproductions:** Yes/No/Unknown     

If after taking steps 1-5 above (or for some other reason) you are unable to locate the reproduction package, record your candidate paper (and if applicable, the outcome of your correspondence with the original authors) in the SSRP database following the example above.


```{r candidate-paper-dec-tree, eval=TRUE, echo=FALSE, fig.cap="Decision tree to move from candidate to declared paper", out.width = '80%'}
#Bug1: help to get this to run within the code chunk
#Bug2: cannot call .svg and get pdf (latex) to run
#
if (FALSE) {
mermaid("
graph TD
	A0[Select Candidate Paper] --> A[Check SSRP]
  A --> |No paper entry|B1[Reproduction Package-RP-?]
  B1 --> |Yes|F1[Declare & Read]
  B1 --> |No|C1[Contact Authors?]
  C1 --> |Yes|D1[Found RP?]
  C1 --> |No|R[Brief Record]
  D1 --> |Yes|F1
  D1 --> |No|E1[Build RP?]
  E1 --> |Yes|F1[Declare & Read]
  E1 --> |No|R

  A --> |Paper entry|B2[Status on SSRP]
  B2 --> |RP exists|F1
  B2 --> |No RP. No Contact to Authors|C1
  B2 --> |No RP. Authors contacted|D2[Follow up?]
  D2 --> |Yes|C1
  D2 --> |No|R

  R -->A0
")  
}
# [![](https://mermaid.ink/img/eyJjb2RlIjoiXG5ncmFwaCBURFxuXHRBMFtTZWxlY3QgQ2FuZGlkYXRlIFBhcGVyXSAtLT4gQVtDaGVjayBTU1JQXVxuICBBIC0tPiB8Tm8gcGFwZXIgZW50cnl8QjFbXCJSZXByb2R1Y3Rpb24gUGFja2FnZSA8YnI-IChSUCkgQXZhaWxhYmxlP1wiXVxuICBCMSAtLT4gfFllc3xGMVtEZWNsYXJlICYgUmVhZF1cbiAgQjEgLS0-IHxOb3xDMVtDb250YWN0IEF1dGhvcnM_XVxuICBDMSAtLT4gfFllc3xEMVtGb3VuZCBSUD9dXG4gIEMxIC0tPiB8Tm98UltCcmllZiBSZWNvcmRdIFxuICBEMSAtLT4gfFllc3xGMVxuICBEMSAtLT4gfE5vfEUxW0J1aWxkIFJQP11cbiAgRTEgLS0-IHxZZXN8RjFbRGVjbGFyZSAmIFJlYWRdXG4gIEUxIC0tPiB8Tm98UlxuXG4gIEEgLS0-IHxQYXBlciBlbnRyeXxCMltTdGF0dXMgb24gU1NSUF1cbiAgQjIgLS0-IHxSUCBleGlzdHN8RjFcbiAgQjIgLS0-IHxObyBSUC4gTm8gQ29udGFjdCB0byBBdXRob3JzfEMxXG4gIEIyIC0tPiB8Tm8gUlAuIEF1dGhvcnMgY29udGFjdGVkfEQyW0ZvbGxvdyB1cD9dXG4gIEQyIC0tPiB8WWVzfEMxXG4gIEQyIC0tPiB8Tm98UlxuXG4gIFIgLS0-QTAiLCJtZXJtYWlkIjp7InRoZW1lIjoiZGVmYXVsdCJ9LCJ1cGRhdGVFZGl0b3IiOmZhbHNlfQ)](https://mermaid-js.github.io/mermaid-live-editor/#/edit/eyJjb2RlIjoiXG5ncmFwaCBURFxuXHRBMFtTZWxlY3QgQ2FuZGlkYXRlIFBhcGVyXSAtLT4gQVtDaGVjayBTU1JQXVxuICBBIC0tPiB8Tm8gcGFwZXIgZW50cnl8QjFbXCJSZXByb2R1Y3Rpb24gUGFja2FnZSA8YnI-IChSUCkgQXZhaWxhYmxlP1wiXVxuICBCMSAtLT4gfFllc3xGMVtEZWNsYXJlICYgUmVhZF1cbiAgQjEgLS0-IHxOb3xDMVtDb250YWN0IEF1dGhvcnM_XVxuICBDMSAtLT4gfFllc3xEMVtGb3VuZCBSUD9dXG4gIEMxIC0tPiB8Tm98UltCcmllZiBSZWNvcmRdIFxuICBEMSAtLT4gfFllc3xGMVxuICBEMSAtLT4gfE5vfEUxW0J1aWxkIFJQP11cbiAgRTEgLS0-IHxZZXN8RjFbRGVjbGFyZSAmIFJlYWRdXG4gIEUxIC0tPiB8Tm98UlxuXG4gIEEgLS0-IHxQYXBlciBlbnRyeXxCMltTdGF0dXMgb24gU1NSUF1cbiAgQjIgLS0-IHxSUCBleGlzdHN8RjFcbiAgQjIgLS0-IHxObyBSUC4gTm8gQ29udGFjdCB0byBBdXRob3JzfEMxXG4gIEIyIC0tPiB8Tm8gUlAuIEF1dGhvcnMgY29udGFjdGVkfEQyW0ZvbGxvdyB1cD9dXG4gIEQyIC0tPiB8WWVzfEMxXG4gIEQyIC0tPiB8Tm98UlxuXG4gIFIgLS0-QTAiLCJtZXJtYWlkIjp7InRoZW1lIjoiZGVmYXVsdCJ9LCJ1cGRhdGVFZGl0b3IiOmZhbHNlfQ)

#BUG: bookdown cannot seem to load the svg versio of the figure below
knitr::include_graphics("candidate-to-declared-paper.png") 
```



## Identifying the relevant timeline {#identify-timeline}  

Before you begin working on the four main stages of the reproduction exercise (Scoping, Assessment, Improvement, and Robustness), it is important to manage your own expectations and those of your instructor or advisor. Be mindful of your time limitations when defining the scope of your reproduction activity. These will depend on the type of exercise chosen by your instructor or advisor and may vary from a weeklong homework assignment, to a semester-long project (an undergraduate thesis, for example).

Table 1 shows an example distribution of time across three different reproduction formats. The Scoping and Assessment stages are expected to last roughly the same amount of time across all formats (lasting longer for the semester-long activities as less experienced researchers, such as undergraduate students, may need more time). Differences emerge in the distribution of time for the last two main stages: Improvements and Robustness. For shorter exercises, we recommend avoiding any possible improvements to the raw data (or cleaning code). This will limit how many robustness checks are possible (for example, by limiting your ability to reconstruct variables according to slightly different definitions), but it should leave plenty of time for testing different specifications at the analysis level.


```{r example, out.width = '100%', echo=FALSE, eval=TRUE}
df <- data.frame(matrix(ncol = 7, nrow = 0))

kbl(df, col.names = rep(c(" "), times = 7), booktabs = TRUE) %>%
        kable_styling(latex_options = c("hold_position")) %>%
        kable_paper() %>%
        sub("\\\\bottomrule", "", .) %>%
        add_header_above(c("Robustness"=1, "25%"=1, "5%"=1, "25%"=2, "25%"=2)) %>%
        add_header_above(c("Improvement"=1, "25%"=1, "0%"=1, "40%"=2, "20%"=1, "30%"=1)) %>%
        add_header_above(c("Assesment"=1, "35%"=2, "25%"=2, "15%"=2)) %>%
        add_header_above(c("Scoping"=1, "10% (1 day)"=2, "10% (2 day)"=2, "10% (10 days)"=2)) %>%
        add_header_above(c("Stages of\n Reproduction"=1, "analysis\ndata"=1, "raw\ndata"=1,
                           "analysis\ndata"=1, "raw\ndata"=1,"analysis\ndata"=1, "raw\ndata"=1)) %>%
        add_header_above(c(" "=1, "2 weeks\n(~10 days)"=2, "1 month\n(~20 days)"=2, "1 semester\n(~100 days)"=2))
```




## Potential sources of papers to prioritize {#identify-timeline}  

### Prioritize based on higher likelihood of reproduction 

To curate a list of target papers in *economics*, instructors can consult recently published articles in the journals of the [American Economic Association](https://www.aeaweb.org/journals) (AEA), especially articles published since 2019, which are more likely to contain reproduction packages given a change in the [AEA Data and Code Availability Policy](https://www.aeaweb.org/journals/data/data-code-policy) that took effect that year. In *political science*, target papers can be found in the *[American Journal of Political Science](https://ajps.org/ajps-verification-policy/)*, especially those published after 2016, which are also likely to contain reproduction packages. In other social science disciplines, instructors can focus on journals that score 2 or above in data, code, and materials transparency in the [Transparency and Openness Promotion (TOP) Factor](https://www.cos.io/initiatives/top-guidelines) (see scores [here](https://topfactor.org/)), maintained by the Center for Open Science.

### Prioritize based on proxies of paper impact within a field

Instructors and reproducers could prioritize papers based on their impact in their field. We suggest that an explicit criteria should be specified in order to define a pool of "high impact" papers. Here we demonstrate one such criteria for the field of Economics. 

 1. Go to [Google Scholar metrics](https://scholar.google.com/citations?view_op=metrics_intro&hl=en)   
 2. Select [top publications](https://scholar.google.com/citations?view_op=top_venues&hl=en)   
 3. Click on categories and select [Business, Economics & Management](https://scholar.google.com/citations?view_op=top_venues&hl=en&vq=bus). Then select in subcategories, select [Economics](https://scholar.google.com/citations?view_op=top_venues&hl=en&vq=bus_economics)   
 4. Exclude the Journal of Economics Perspectives (a review journal that does little computational analysis).  
 5. Focus on the 10 journals with highest h5-index. In August 4th, 2021, this yields:  
    - American Economic Review
    - The Quarterly Journal of Economics
    - The Review of Financial Studies
    - Journal of Political Economy
    - The Journal of Finance
    - Econometrica
    - The Review of Economic Studies
    - Review of Economics and Statistics
    - The Economic Journal
    - Journal of Public Economics  
 6. For each journal click on the hyperlinked h5-index number (sort highest to lowest). 
 7. Select the first 10 articles in the list as sorted by citations. For example, for the *American Economic Review*, this procedure yields the following articles: 
    - [The Effects of Exposure to Better Neighborhoods on Children: New Evidence from the Moving to Opportunity Experiment](https://scholar.google.com/scholar?oi=bibs&cluster=14593109523353163362&btnI=1&hl=en)  
    - [Railroads of the Raj: Estimating the Impact of Transportation Infrastructure](https://scholar.google.com/scholar?oi=bibs&cluster=9632592800621257435&btnI=1&hl=en)
    - [The Race between Man and Machine: Implications of Technology for Growth, Factor Shares, and Employment](https://scholar.google.com/scholar?oi=bibs&cluster=14882938779698367846&btnI=1&hl=en)
    - [The Surprisingly Swift Decline of US Manufacturing Employment](https://scholar.google.com/scholar?oi=bibs&cluster=12934661864410944134&btnI=1&hl=en)
    - [Monetary Policy According to HANK](https://scholar.google.com/scholar?oi=bibs&cluster=4926806483525389434&btnI=1&hl=en)
    - [The Determinants and Welfare Implications of US Workers' Diverging Location Choices by Skill: 1980-2000](https://scholar.google.com/scholar?oi=bibs&cluster=2965198650429157010&btnI=1&hl=en)
    - [Disruptive Change in the Taxi Business: The Case of Uber](https://scholar.google.com/scholar?oi=bibs&cluster=8446349612486899408&btnI=1&hl=en)
    - [Bartik Instruments: What, When, Why, and How](https://scholar.google.com/scholar?oi=bibs&cluster=6025324097129642737&btnI=1&hl=en)
    - [Importing Political Polarization? The Electoral Consequences of Rising Trade Exposure](https://scholar.google.com/scholar?oi=bibs&cluster=16547462466761479574&btnI=1&hl=en)
    - [Long-Run Impacts of Childhood Access to the Safety Net](https://scholar.google.com/scholar?oi=bibs&cluster=4766165857030680369&btnI=1&hl=en)
  
 8. Starting from the top: select a paper and read the abstract to verify that there is a computational analysis. Discard if there is not.  
 9. If there is, search [socialsciencereproduction.org](https://www.socialsciencereproduction.org/) to verify that there is a recorded reproduction.  
 10. If there is, you can create a new reproduction or move to the next paper in the list. If there is no reproduction recorded, create one.