Skip to content

README.md

Latest commit

 

History

History
152 lines (100 loc) · 5.02 KB

README.md

File metadata and controls

152 lines (100 loc) · 5.02 KB
Raw

(In)equality in school statutes - generated AI-assisted analysis toolkit

Paper: https://press.amu.edu.pl/pl/nie-rownosci-w-statutach-szkolnych-raport-z-badan.html
DOI: 10.14746/amup.9788323244325

A research-grade toolkit for analysing Polish school bylaws/statutes at scale - combining document processing, NLP-style keyword pipelines, and generative AI to turn dense PDFs into structured datasets ready for rigorous qualitative and quantitative work.

This repository supports an end-to-end workflow:

  • PDF → structured JSON using OpenAI Assistants
  • PDF → sentences CSV → keyword hits CSV for downstream qualitative analysis
  • Context-code enrichment (optional) using OpenAI (questionnaire-style coding)

The codebase has been refactored into importable modules under src/research_code/ with thin CLI entrypoints.

1. Research: (in)equality in school statutes

This project accompanies research on (nie)równość / (in)equality in the content of Polish school statutes (bylaws). A statute is a key document regulating everyday life in a given institution; it shapes the experience of students and the wider school community - often even when that influence is not consciously noticed.

The research presents the course and results of an academic, intersectional analysis of school statutes in the light of:

  • equality law and equality values,
  • principles of anti-discrimination and inclusive education,
  • respect for human rights and children’s rights.

Through quantitative and qualitative critical document analysis of norms present in statutes, the research team sought to answer:

To what extent do provisions in school statutes address equal treatment and social inclusion?

This repository operationalizes that research workflow using reproducible data processing and generative AI assistance to scale analysis across many documents while keeping outputs auditable (CSV/JSON artifacts).

2. Package layout

All reusable code lives in src/research_code/:

  • config/
    • settings.py – centralized config (env vars + defaults)
    • logging.py – shared logging setup
  • io/
    • file_utils.py – file discovery, path helpers
    • prompts.py – prompt loader for src/prompts/**
  • nlp/
    • citations.py – citation expansion (context around keyword)
    • keywords.py – keyword/lexicon matching and CSV helpers
  • openai/
    • client.py – OpenAI client creation + API key checks
  • pipelines/
    • chunky_runner.py – Assistants PDF→JSON pipeline
    • keyword_scan.py – sentences.csv → keyword_hits.csv
    • dedup.py – keyword hits CSV de-duplication
    • context_codes.py – enrich keyword hits with context codes (OpenAI)
  • cli/ – thin entrypoints runnable via python -m ...

Legacy scripts have been moved into src/legacy/ to keep the root src/ clean.

Installation

Create a virtual environment (recommended) and install dependencies:

python -m pip install -r .\requirements.txt

Note: src/scrapSentences.py uses Apache Tika (tika Python package), which is currently treated as a legacy/optional tool and is not part of requirements.txt.

Configuration

Environment variables

Required for OpenAI-based pipelines:

  • OPENAI_API_KEY

Optional:

  • OPENAI_MODEL (default: gpt-4o-mini)
  • OPENAI_TEMPERATURE (default: 0.2)

Optional path overrides:

  • PB_DATA_DIR
  • PB_OUTPUT_DIR
  • PB_KEYWORDS_DIR
  • PB_PROMPTS_DIR

Defaults are computed relative to the repository root.

Workflow A - PDF → JSON (OpenAI / Assistants) "chunky"

Single file

python -m research_code.cli.chunky .\data\examples\BPOMMPB_1.pdf .\output\BPOMMPB_1.json .\data\template.json

Batch

python -m research_code.cli.supervisor .\data\examples .\output .\data\template.json

Workflow B - PDF → sentences.csv → keyword_hits.csv

1) Extract sentences from PDFs (legacy)

python .\src\scrapSentences.py .\data\examples .\output\sentences.csv

2) Scan keywords

python -m research_code.cli.find_keywords .\output\sentences.csv --keywords-dir .\data\keywords .\output\keyword_hits.csv

3) De-duplicate

python -m research_code.cli.dedup .\output\keyword_hits.csv .\output\keyword_hits_dedup.csv

Workflow C - Context code enrichment (OpenAI)

This step takes (deduped) keyword hits and evaluates each citation against a set of questions (context codes).

python -m research_code.cli.context_codes .\output\keyword_hits_dedup.csv .\output\keyword_hits_context.csv --questions-csv .\data\contextcodes\contextcodesquestions.csv --prompt contextcodes/v2_0.txt

The prompt file is loaded from src/prompts/ and should contain placeholders:

  • [SENTENCE]
  • [QUESTION]

What’s legacy / what’s maintained?

Maintained (recommended):

  • research_code.cli.* + research_code.pipelines.*

Legacy (kept for reference):

  • moved into src/legacy/ (including old script-style OpenAI + keyword pipelines)