CGIAR food security dataset scripts

Code used to generate the dataset for the 2026 collation of CGIAR work on food security, nutrition, and food environment.

Items matching the inclusion criteria were retrieved from six CGIAR institutional repositories. This Python-based extract, transform, and load (ETL) pipeline merges and normalizes the metadata to ensure consistent use of date formats, multi-value separators, and identifiers. Naive deduplication is performed first using a combination of normalized titles and types, followed by exact matching of DOIs. Items identified to have been included erroneously due to incorrect repository metadata were excluded.

We used Crossref, Unpaywall, and OpenAlex to fill in gaps for missing metadata such as abstracts, usage (license) and access rights, affiliations, funders, and publishers because this information can be valuable to researchers. Minor normalization was performed on some metadata in order to reduce variation in controlled terms, but all other metadata was used as-is from the respective repositories.

This dataset supports secondary analyses of CGIAR’s thematic coverage over time and geography, enables the extraction of indicators for food and nutrition security research, and provides a reusable pipeline that can be extended to future harvests as repositories evolve.

See:

Orth, Alan; Oluchi, Ezekannagha, 2026, "A comprehensive metadata dataset of CGIAR food security, nutrition, and food environment research outputs (2012–2024)", https://hdl.handle.net/20.500.11766.1/FK2/BXBRF9, MELDATA, V2

Methodology

Search CGIAR institutional repositories to find items matching the following criteria:

• Issue date: 2012 to 2024
• Output type: Journal Article, Working Paper, Preprint, Thesis, Conference Paper, Report, Book, or Book Chapter
• CGIAR affiliation
• Food security and related terminology in metadata or fulltext

For the exact search terms used with each repository, see src/update-sources.sh.

Data Sources

CGIAR institutional repositories used in this dataset (sorted by total number of records as of April, 2026):

Name	URL	Technology	Total Records
CGSpace	https://cgspace.cgiar.org	DSpace	157,000
CIFOR–ICRAF	https://data.cifor-icraf.org/dspace	DSpace	43,981
CIMMYT	https://repository.cimmyt.org	DSpace	20,758
MELSpace	https://repo.mel.cgiar.org	DSpace	14,207
WorldFish	https://digitalarchive.worldfishcenter.org	DSpace	6,576
ICRISAT	https://oar.icrisat.org	EPrints	?

Requirements

• Python >= 3.11
• UNIX-like operating system

Usage

This project is managed using uv. You will need to install that first and then initialize the project:

$ uv sync
$ source .venv/bin/activate

Alternatively, you can use a vanilla Python virtual environment to install the dependencies:

$ python -m venv .venv
$ source .venv/bin/activate
$ pip install -r requirements.txt

Once the dependencies are installed you can run the pipeline:

$ ./src/merge_source_csvs.py 2>&1 | tee data/out/run.log

This will use pre-harvested data from the ./data/raw directory and save a run log to ./data/out/run.log. The final dataset is ./data/out/final.csv.

To get fresh date from sources, use the src/update_sources.sh script. Note that this can take many hours (or even days in the worst case). Caches are used where possible to speed up repeated runs.

License

This work is licensed under the GPLv3.

The license allows you to use and modify the work for personal and commercial purposes, but if you distribute the work you must provide users with a means to access the source code for the version you are distributing. Read more about the GPLv3 at TL;DR Legal.