Multi-Agent AI System for Radiology Report Structuring and Quality Assurance

A locally deployed multi-agent AI pipeline for automated radiology report structuring and quality assurance (QA).

The system combines deterministic regular-expression rules with locally deployed large language models (LLMs) to reorganize the Findings section of radiology reports into predefined anatomical sections while preserving the original report text. A separate QA agent evaluates the full report for clinically meaningful inconsistencies and potential communication issues.

All LLM inference is performed locally, allowing radiology reports to remain within the institutional computing environment.

---

Overview

Radiology reports can vary substantially in organization and reporting style across radiologists. This project provides a multi-agent workflow that performs two complementary tasks:

1. Report structuring

   • Assigns sentences from the Findings section to predefined anatomical sections.
   • Preserves the original sentence text rather than rewriting or summarizing the report.

2. Quality assurance

   • Evaluates the complete report for potential inconsistencies or reporting errors.
   • Produces structured outputs that can be processed automatically downstream.

The pipeline was developed and evaluated using CT chest, abdomen, and pelvis radiology reports.

---

Pipeline

The system consists of four agents. The first three agents perform sentence-level report structuring, while the fourth agent performs report-level quality assurance.

• Agent 1 — Rule-based classification: Uses a domain-specific dictionary of approximately 600 regular-expression patterns to assign findings to predefined anatomical sections.
• Agent 2 — LLaMA-3 8B: Classifies findings not resolved by the rule-based agent using the full Findings section and local sentence context.
• Agent 3 — DeepSeek-R1 70B: Handles selected sentences requiring additional contextual interpretation, including unresolved, ambiguous, and device-related findings.
• Agent 4 — DeepSeek-R1 70B QA: Evaluates the full report for clinically meaningful inconsistencies, including section mismatches, laterality or polarity conflicts, gender–anatomy discrepancies, and undocumented communication of critical findings, and assigns an overall severity level.

Repository Structure

├── helpers/
│   ├── config.py             # Model names (Agents 2, 3 and 4)
│   ├── model.py               # Model settings (temperature, LLM instantiation via Ollama)
│   ├── schema.py               # JSON output structure: required anatomical section keys and field descriptions
│   ├── keywords.py              # Regex rules: ~600 patterns used by Agent 1 (rule-based classification)
│   ├── json_utilities.py         # JSON parsing/validation helpers for LLM outputs
│   ├── text_utilities.py
│   ├── csv_processor.py
│   └── findings_formatter.py
├── report_structuting/
│   ├── sentence_classifier.py   # Prompts for Agent 2 and Agent 3
│   └── structuring_pipeline.py  # Orchestrates Agents 1–3
├── quality_assurance/
│   └── qa_task.py               # Prompt and retry logic for Agent 4 (QA)
├── figures/
├── main.py
└── README.md

Where to find things

Component	Location
Model settings (model names, temperature)	helpers/config.py, helpers/model.py
Prompts (Agent 2, 3 structuring prompts)	report_structuting/sentence_classifier.py
Prompts (Agent 4 (QA) prompt + retry logic)	quality_assurance/qa_task.py
JSON output structure (required keys, field descriptions, critical-findings hints)	helpers/schema.py
JSON parsing/validation	helpers/json_utilities.py
Regex rules (~600 patterns for rule-based classification)	helpers/keywords.py

Citation

If you use this pipeline, please cite:

Hartsock I, Lam C, Otteni C, Qayyum A, Gatenby R, Araujo C, Rasool G. Multi-Agent AI System for Radiology Report Structuring and Quality Assurance with Independent Radiologist Evaluation. arXiv preprint arXiv:2608.18072, 2026. https://arxiv.org/abs/2608.18072

@article{hartsock2026multiagent,
  title={Multi-Agent AI System for Radiology Report Structuring and Quality Assurance with Independent Radiologist Evaluation},
  author={Hartsock, Iryna and Lam, Cesar and Otteni, Christopher and Qayyum, Aliya and Gatenby, Robert and Araujo, Cyrillo and Rasool, Ghulam},
  journal={arXiv preprint arXiv:2608.18072},
  year={2026},
  url={https://arxiv.org/abs/2608.18072}
}