Impresso NER Data Augmentation

Reproducible pipeline for training-data augmentation of Named Entity Recognition models on historical newspaper corpora (French and German), evaluated on the HIPE-2022 benchmark.

🎊 Happy to share that the accompanying paper received the Best Paper Award at SwissText 2026.
See the Citation section for the full reference.

Two complementary augmentation strategies are implemented and systematically compared across three NER tasks and two BERT architectures:

Strategy	Configurations
Mention Replacement	Random · Semantic · Temporal · Semantic + Temporal
LLM Annotation	LLM Strict · LLM Majority · LLM Majority + BERT

Models are evaluated on Coarse Literal NER, Fine-Grained NER, and Metonymic Shift Detection using strict, exact, type, and partial matching strategies.

Repository Structure

impresso-named-entity-data-augmentation/
│
├── configs/
│   ├── llm_models.yaml          # LLM provider and model configuration
│   └── training_config.yaml     # Training hyperparameters
│
├── data/
│   ├── corpus_sample/           # Raw corpus samples (de/, fr/)
│   └── experiments/
│       ├── exp1_mention_replacement/
│       ├── exp2_llm_annotation/
│       ├── training_data/       # Augmented TSV datasets
│       └── training_results/    # Per-model metrics (aggregated.parquet)
│
├── models/
│   ├── augmented/               # Fine-tuned augmented models (de/, fr/)
│   └── baseline/                # Baseline models (de/, fr/)
│
├── notebooks/
│   ├── evaluation/              # Per-language analysis notebooks
│   │   ├── 00_llm_hipe_test_evaluation.ipynb
│   │   ├── 01_global_overview.ipynb
│   │   ├── 02_best_model_comparison.ipynb
│   │   ├── 03_performance_by_attributes.ipynb
│   │   ├── 04_best_model_augmented.ipynb
│   │   └── 05_augmented_dataset_analysis.ipynb
│   └── bilingual_analysis/      # Cross-language synthesis
│       ├── 01_overall_results.ipynb
│       ├── 02_best_augmented.ipynb
│       └── 03_augmented_dataset_analysis_bilingual.ipynb
│
├── prompts/
│   ├── system_prompt_de.txt     # LLM annotation system prompt (German)
│   └── system_prompt_fr.txt     # LLM annotation system prompt (French)
│
├── scripts/
│   ├── 01_exp1_mention_replacement/
│   │   ├── 01_precompute_enriched_pools.py
│   │   ├── 02_generate_augmentations.py
│   │   └── 03_convert_to_tsv.py
│   ├── 01_exp2_llm_annotation/
│   │   ├── 00a_prepare_hipe_test_sentences.py
│   │   ├── 00b_build_evaluation_pairs.py
│   │   ├── 01_run_corpus_inference.py
│   │   ├── 02_merge_predictions.py
│   │   ├── 03_preprocess_corpus.py
│   │   ├── 04_build_corpus_sample.py
│   │   ├── 05_annotate_corpus_async.py
│   │   ├── 06_convert_to_eda_format.py
│   │   ├── 07_convert_to_tsv.py
│   │   └── 08_ensemble_annotations.py
│   ├── 02_dataset_creation/
│   │   └── 01_create_training_subsets.py
│   ├── 03_training/
│   │   ├── 01_run_baseline_training.sh
│   │   ├── 02_run_baseline_inference.sh
│   │   ├── train_batch.py
│   │   └── train_single.py
│   └── 04_evaluation/
│       ├── 00_preprocess_augmented_predictions.py
│       ├── 00_preprocess_baseline_predictions.py
│       └── 01_compute_all_metrics.py
│
├── src/
│   ├── augmentation/            # Mention replacement logic
│   │   ├── candidate_selection.py
│   │   ├── mention_replacement.py
│   │   └── quality_controls.py
│   ├── inference/               # NER inference on raw corpus
│   │   ├── corpus_reader.py
│   │   └── ner_inference.py
│   ├── llm/                     # LLM client wrappers and response schemas
│   │   ├── clients.py
│   │   └── schemas.py
│   ├── training/                # Dataset, model, trainer, utilities
│   │   ├── dataset.py
│   │   ├── models.py
│   │   ├── trainer.py
│   │   └── utils.py
│   ├── utils/                   # Annotation converters and TSV helpers
│   │   ├── annotation_converter.py
│   │   └── create_hipe_tsv.py
│   └── config.py                # Central config — loads .env, exposes PROJECT_ROOT
│
├── eda/                         # Standalone EDA sub-project (own pyproject.toml)
│
├── .env.example                 # Environment variable template (copy → .env)
├── pyproject.toml
└── README.md

Prerequisites

Requirement	Version
Python	≥ 3.9 (3.11 recommended)
PyTorch	2.9.1
CUDA	≥ 11.8 (optional, CPU fallback available)
HIPE-2022 data	v2.1 / hipe2020 split

Installation

# 1. Clone
git clone https://github.com/impresso/impresso-named-entity-data-augmentation.git
cd impresso-named-entity-data-augmentation

# 2. Create and activate conda environment
conda create -n ner-da python=3.11 -y
conda activate ner-da

# 3. Initialize submodules (required: eda/)
git submodule update --init --recursive

# 4. Install the package in editable mode (installs all dependencies)
pip install -e .

If you are introducing the submodule in a fresh clone/fork (maintainer workflow):

git submodule add git@github.com:impresso/impresso-named-entity-eda.git eda
git submodule update --init --recursive

GPU note: PyTorch 2.9.1 with CUDA support must be installed separately if the default wheel does not match your driver. See the PyTorch installation matrix.

Configuration

Copy the provided template and fill in the required values:

cp .env.example .env

Clone HIPE-2022 data once (outside or inside this repo), then point .env to the data/v2.1/hipe2020 directory:

git clone git@github.com:hipe-eval/HIPE-2022-data.git

Create a writable models directory before running training:

mkdir -p models

.env.example:

# ── Required paths ────────────────────────────────────────────────────────────
# Root directory where trained models are stored / will be written.
IMPRESSO_MODELS_DIR=/path/to/models

# Root of the HIPE-2022 data repository (v2.1, hipe2020 split).
HIPE_DATA_DIR=/path/to/HIPE-2022-data/data/v2.1/hipe2020

# ── LLM API keys (only required for Exp 2 — LLM Annotation) ──────────────────
OPENAI_API_KEY=
MISTRAL_API_KEY=
QWEN_API_KEY=

config.py loads this file at import time and raises a clear EnvironmentError if a required variable is missing. All scripts and notebooks consume PROJECT_ROOT and IMPRESSO_MODELS_DIR exclusively through this module — never through hardcoded paths.

Production Setup (Recommended)

This is the robust sequence to prepare a fresh machine and run the full workflow.

# From repo root
conda create -n ner-da python=3.11 -y
conda activate ner-da

git submodule update --init --recursive
pip install -e .

# Clone HIPE data repository (once)
git clone git@github.com:hipe-eval/HIPE-2022-data.git

# Prepare runtime config
cp .env.example .env
mkdir -p models

Then edit .env:

Set HIPE_DATA_DIR to <path>/HIPE-2022-data/data/v2.1/hipe2020
Set IMPRESSO_MODELS_DIR to a writable models directory (for example ./models)

Run baseline training first (recommended before EDA metrics refresh):

bash scripts/03_training/01_run_baseline_training.sh --lang fr

Then run EDA baseline preparation:

cd eda/scripts

python 00_enrich_hipe_with_led.py             --lang fr
python 01_setup_baseline_predictions.py       --lang fr
python 02_preprocess_baseline_predictions.py  --lang fr
python 03_compute_baseline_metrics.py         --lang fr

After this, continue with the standard pipeline sections below (Exp 1, Exp 2, dataset creation, training, evaluation).

Pipeline

All scripts accept --lang {fr,de} as their primary parameter. Run python <script>.py --help for the full argument list.

0 — EDA and Baseline Setup

Enriches the HIPE dataset and computes baseline predictions used throughout the pipeline.

cd eda/scripts

python 00_enrich_hipe_with_led.py          --lang fr
python 01_setup_baseline_predictions.py    --lang fr
python 02_preprocess_baseline_predictions.py --lang fr
python 03_compute_baseline_metrics.py      --lang fr

Repeat with --lang de.

1 — Mention Replacement (Exp 1)

Generates augmented training sets by replacing entity mentions with candidates selected from a pre-computed enriched pool.

cd scripts/01_exp1_mention_replacement

# Build the candidate pool (run once per language)
python 01_precompute_enriched_pools.py   --lang fr

# Generate augmented variants for all configurations and levels
python 02_generate_augmentations.py      --lang fr

# Convert to HIPE-compatible TSV for training
python 03_convert_to_tsv.py              --lang fr

2 — LLM Annotation (Exp 2)

Annotates unseen corpus sentences with three LLMs and ensembles the results. Requires OPENAI_API_KEY, MISTRAL_API_KEY, and QWEN_API_KEY to be set.

cd scripts/01_exp2_llm_annotation

# Prepare HIPE test sentences and evaluation pairs
python 00a_prepare_hipe_test_sentences.py  --lang fr
python 00b_build_evaluation_pairs.py       --lang fr

# Run baseline NER inference on the corpus
python 01_run_corpus_inference.py          --lang fr
python 02_merge_predictions.py             --lang fr

# Build and annotate the corpus sample
python 03_preprocess_corpus.py             --lang fr
python 04_build_corpus_sample.py           --lang fr
python 05_annotate_corpus_async.py         --lang fr   # calls LLM APIs

# Convert and ensemble
python 06_convert_to_eda_format.py         --lang fr
python 07_convert_to_tsv.py                --lang fr
python 08_ensemble_annotations.py          --lang fr

3 — Dataset Creation

Builds training subsets at each augmentation level (+25 % → +200 %).

cd scripts/02_dataset_creation

python 01_create_training_subsets.py --lang fr

4 — Training

Fine-tunes BERT-base and BERT-historical on each augmented dataset.

cd scripts/03_training

# Augmented models (all configs × all levels)
python train_batch.py --lang fr

For a single model:

python train_single.py \
    --lang         fr \
    --method       mention_replacement \
    --config       semantic \
    --level        train_100 \
    --architecture hist_base_extended \
    --seed         2025

5 — Evaluation

Preprocesses predictions and computes all metrics (strict / exact / type / partial).

cd scripts/04_evaluation

python 00_preprocess_baseline_predictions.py  --lang fr
python 00_preprocess_augmented_predictions.py --lang fr
python 01_compute_all_metrics.py              --lang fr

Outputs metrics_aggregated.parquet for each model under $IMPRESSO_MODELS_DIR/augmented/{lang}/.

Notebooks

All notebooks read LANG from a single constant in the second cell. Set LANG = "fr" or LANG = "de" to switch languages.

`notebooks/evaluation/`

Notebook	Prerequisite scripts	Produces
`00_llm_hipe_test_evaluation.ipynb`	`00b_build_evaluation_pairs.py`	LLM vs HIPE gold evaluation (F1, error distribution)
`01_global_overview.ipynb`	`01_compute_all_metrics.py`	Rankings, learning curves, stability heatmaps — all tasks
`02_best_model_comparison.ipynb`	`01_compute_all_metrics.py`	Deep error-pattern analysis for the best augmented models
`03_performance_by_attributes.ipynb`	`01_compute_all_metrics.py`	Performance breakdown by entity attributes
`04_best_model_augmented.ipynb`	`01_compute_all_metrics.py`	Head-to-head comparison of the best augmented models
`05_augmented_dataset_analysis.ipynb`	`01_create_training_subsets.py`	Dataset statistics: entity density, TTR, novel entities

`notebooks/bilingual_analysis/`

Notebook	Prerequisite	Produces
`01_overall_results.ipynb`	Both languages evaluated	Cross-language performance overview
`02_best_augmented.ipynb`	Both languages evaluated	Bilingual best-model comparison
`03_augmented_dataset_analysis_bilingual.ipynb`	Both languages evaluated	Bilingual dataset statistics

Reproducing Results End-to-End

The commands below reproduce all results from a fresh clone. Estimated total runtime depends on GPU availability and LLM API throughput.

# 0. Environment
git submodule update --init --recursive
cp .env.example .env          # fill in paths and API keys
pip install -e .

# 0b. Data and output paths
git clone git@github.com:hipe-eval/HIPE-2022-data.git
mkdir -p models

# 1. EDA and baseline (both languages)
for LANG in fr de; do
    python eda/scripts/00_enrich_hipe_with_led.py            --lang $LANG
    python eda/scripts/01_setup_baseline_predictions.py      --lang $LANG
    python eda/scripts/02_preprocess_baseline_predictions.py --lang $LANG
    python eda/scripts/03_compute_baseline_metrics.py        --lang $LANG
done

# 2. Augmentation
for LANG in fr de; do
    python scripts/01_exp1_mention_replacement/01_precompute_enriched_pools.py --lang $LANG
    python scripts/01_exp1_mention_replacement/02_generate_augmentations.py    --lang $LANG
    python scripts/01_exp1_mention_replacement/03_convert_to_tsv.py            --lang $LANG

    python scripts/01_exp2_llm_annotation/00a_prepare_hipe_test_sentences.py  --lang $LANG
    python scripts/01_exp2_llm_annotation/00b_build_evaluation_pairs.py       --lang $LANG
    # ... (steps 01–08, see Pipeline section above)
done

# 3. Dataset creation
for LANG in fr de; do
    python scripts/02_dataset_creation/01_create_training_subsets.py --lang $LANG
done

# 4. Training (GPU recommended)
for LANG in fr de; do
    python scripts/03_training/train_batch.py --lang $LANG
done

# 5. Evaluation
for LANG in fr de; do
    python scripts/04_evaluation/00_preprocess_augmented_predictions.py --lang $LANG
    python scripts/04_evaluation/01_compute_all_metrics.py              --lang $LANG
done

# 6. Notebooks — open in Jupyter and run top-to-bottom
jupyter lab notebooks/

Citation

This repository accompanies the following paper (Best Paper Award at SwissText 2026):

Blinière, L., Ehrmann, M., Boros, E., Clematide, S., & Kaplan, F. (2026).
Data Augmentation for Historical NER: A Systematic Comparison of Lexical and LLM-based Approaches.
Proceedings of the 11th Edition of the Swiss Text Analytics Conference, pp. 154–170. Zurich, Switzerland. ACL Anthology.
🔗 https://aclanthology.org/2026.swisstext-1.14/

BibTeX - click to expand, then copy

@inproceedings{bliniere-etal-2026-data,
    title = "Data Augmentation for Historical {NER}: A Systematic Comparison of Lexical and {LLM}-based Approaches",
    author = "Blini{\`e}re, L{\'e}a  and
      Ehrmann, Maud  and
      Boros, Emanuela  and
      Clematide, Simon  and
      Kaplan, Frederic",
    editor = "Sennrich, Rico  and
      Schneider, Gerold  and
      Ellendorff, Tilia  and
      Gao, Yingqiang  and
      Vamvas, Jannis  and
      Cieliebak, Mark",
    booktitle = "Proceedings of the 11th Edition of the {S}wiss Text Analytics Conference",
    month = jun,
    year = "2026",
    address = "Zurich, Switzerland",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.swisstext-1.14/",
    pages = "154--170"
}

Licence

This work is licensed under the GNU Affero General Public License v3.0.

About

This work stems from the Master thesis of Léa Blinière (Fall 2026). It was conducted in the context of Impresso - Media Monitoring of the Past, a interdisciplinary research project that aims to develop and consolidate tools for processing and exploring large collections of media archives across modalities, time, languages and national borders. The first project (2017-2021) was funded by the Swiss National Science Foundation under grant No. CRSII5_173719 and the second project (2023-2027) by the SNSF under grant No. CRSII5_213585) and the Luxembourg National Research Fund under grant No. 17498891.

Navigation Menu

Uh oh!

Repository files navigation

Impresso NER Data Augmentation

Table of Contents

Repository Structure

Prerequisites

Installation

Configuration

Production Setup (Recommended)

Pipeline

0 — EDA and Baseline Setup

1 — Mention Replacement (Exp 1)

2 — LLM Annotation (Exp 2)

3 — Dataset Creation

4 — Training

5 — Evaluation

Notebooks

`notebooks/evaluation/`

`notebooks/bilingual_analysis/`

Reproducing Results End-to-End

Citation

Licence

About

Copyrights

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
configs		configs
notebooks		notebooks
prompts		prompts
scripts		scripts
src		src
.env.example		.env.example
.gitignore		.gitignore
.gitmodules		.gitmodules
LICENSE		LICENSE
README.md		README.md
pyproject.toml		pyproject.toml

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Impresso NER Data Augmentation

Table of Contents

Repository Structure

Prerequisites

Installation

Configuration

Production Setup (Recommended)

Pipeline

0 — EDA and Baseline Setup

1 — Mention Replacement (Exp 1)

2 — LLM Annotation (Exp 2)

3 — Dataset Creation

4 — Training

5 — Evaluation

Notebooks

notebooks/evaluation/

notebooks/bilingual_analysis/

Reproducing Results End-to-End

Citation

Licence

About

Copyrights

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`notebooks/evaluation/`

`notebooks/bilingual_analysis/`