Evaluation of a System Architecture for LLM-based Exploratory Process Mining

This repository contains the code for the paper "Evaluation of a System Architecture for LLM-based Exploratory Process Mining".

It implements a modular system using the DSPy framework where a Large Language Model (LM) orchestrates process analysis based on natural language questions about an event log. The core idea is an LM deciding between enriching the log (via Python/Pandas code generation) and querying it (via SQL/SQLite code generation).

The system uses RAG (ChromaDB) for column context and a Column Dependency Graph to manage enrichment steps. It was benchmarked on the Road Traffic Fine Management dataset. For detailed methodology, findings, and analysis, please refer to the thesis document.

Code - Paper differences

The following renamings have been made from Code -> Paper for better readability:

• SQL Program -> SQL Query component
• Python Program -> Column Enricher component
• Class Answer in Column Desc module -> Describe component

Architectural Variants:

• Bootstrap_fewshot_1 -> SQL_SR_FS
• Bootstrap Few-shot + Val -> CE_FS_Val
• Zero-shot + Val -> CE_ZS_Val
• Bootstrap Few-shot w/o Val -> CE_FS

Repository Structure

• /benchmark: Evaluation datasets derived from the Road Traffic Fine Management log (Process Mining Questions, Q&A pairs, Judge labels).
• /Optimized_prompts: JSON files containing optimized DSPy Signatures and few-shot demonstrations, generated by DSPy optimizers (e.g., BootstrapFewShot). These are loaded by the programs to run "compiled" evaluations.
  • Subdirectories (combined, judge, python, sql) categorize prompts by task.
• /Programs: Core Python scripts (.py) and Jupyter notebooks (.ipynb) implementing the DSPy programs and evaluation workflows.
  • Notebooks (*.ipynb): The primary way to run evaluations and experiments (e.g., combined_program.ipynb, optimizing_*.ipynb, SQL_generic.ipynb, python_generic.ipynb).
  • Python Modules (Combined_programs/, PY_programs/, SQL_programs/): Contain the underlying DSPy program logic imported by the notebooks.
    • Combined_programs/: Orchestrator logic (combined.py: LM decides, combined_perfect_decision.py: Oracle decision).
    • PY_programs/: Python enrichment logic (e.g., python_tables.py).
    • SQL_programs/: SQL querying logic variants:
      • sql_reasoning.py (Class PM_SQL_multi_sp): Implements Separate Reasoning (SR). Corresponds to _sp_ in result filenames.
      • sql_no_reasoning.py (Class PM_SQL_multi_nr): Implements No Reasoning (NR). Corresponds to _nr_ in result filenames.
      • sql_coi.py (Class PM_SQL_multi_COI): Implements Chain-of-Thought (COT/COI). Corresponds to _COI_ in result filenames.
      • sql_simple.py (Class PM_SQL_multi_simple): Implements the Simple variant. Corresponds to _simple_ in result filenames.
      • sql_llm_judge.py: Implements the LM_EVAL judge.
  • Utils/: Helper modules (column_dependency.py, saving_functions.py).
  • chroma_retriever.py: RAG implementation.
  • .env: (Add to .gitignore) For API keys.
• /Results_*: CSV outputs from experimental runs, categorized by program type.
  • Filenames often indicate the program variant (sp, nr, coi, simple), model (4o, mini), and prompt configuration.
  • compiled in filename: Run used optimized prompts loaded from a JSON file in /Optimized_prompts.
  • uc (uncompiled) in filename: Run did not use optimized prompts (e.g., zero-shot or base signatures only).

Setup

1. Clone Repository:

   git clone https://github.com/timothyvinzent/nl-exploratory-process-mining.git
   cd nl-exploratory-process-mining

2. Python Environment:
   • The project uses Python 3.12.8.
   • Install dependencies:

     pip install -r requirements.txt

3. Download Data:
   • Download the Road Traffic Fine Management event log .xes file from: https://data.4tu.nl/articles/_/12683249/1
   • Place the .xes file in a location accessible by the notebooks.
4. Configure Paths & API Keys:
   • CRITICAL: Modify Paths: The Jupyter notebooks in /Programs/ contain hardcoded absolute paths for data files (INPUT_FILE_NAME, etc.), prompt files (PM_PY_PATH, etc.), and database names (SQLITE_DB_NAME). You MUST update these paths within the notebooks to match your local file structure before running them.
   • API Key: Create a .env file in the root directory and add your OpenAI API key:

     OPENAI_API_KEY="sk-..."
     

   • ChromaDB Path: Ensure the path used for ChromaDB initialization exists and is writable. Modify if necessary in the relevant scripts/notebooks.
5. Initialize Database (First Time):
   • Running a notebook like combined_program.ipynb for the first time involves executing initial cells that:
     • Read the .xes file using pm4py.read_xes().
     • Perform initial preprocessing (type conversions, renaming, handling NaNs).
     • Create the SQLite database (e.g., combined.db) using Pandas df.to_sql().
     • Create necessary SQL indexes for performance.
   • Ensure the INPUT_FILE_NAME path in the notebook is correct before running these initial database setup cells.

Usage (via Notebooks)

All experiments and evaluations are run via the Jupyter Notebooks located in the /Programs/ directory. There are no separate command-line tools.

The combined_program.ipynb notebook serves as the main example, demonstrating the workflow for evaluating the Combined_Standard and Combined_Perfect programs. Adapt this notebook or use others for different evaluations:

1. Open a Notebook: Launch Jupyter Lab/Notebook and open the desired .ipynb file from the /Programs/ directory (e.g., combined_program.ipynb).
2. Modify Paths: Ensure all file paths within the notebook cells are updated for your system (see Setup Step 4).
3. Run Setup Cells: Execute cells for imports, configuration, LLM setup, data loading/DB initialization (if first time), retriever initialization, and dependency graph setup.
4. Select Program & Prompts: Instantiate the desired DSPy program class (e.g., PM_combined, PM_isolated, PM_SQL_multi_sp) and optionally load optimized prompts using .load("path/to/Optimized_prompts/... .json") for a "compiled" run. Wrap program instances with assert_transform_module to enable backtracking.
5. Configure Evaluation: Set up the Evaluate object, specifying the testset (loaded from benchmark CSVs) and the judge metric (judge_adjusted).
6. Reset State: Before each evaluation run, execute the cells that reset the SQLite database and potentially the Chroma retriever to their initial states. This ensures consistent starting conditions and prevents results from one run affecting the next.
7. Run Evaluation: Execute the cell calling evaluate(program=...).
8. Save Results: Execute the cells using helper functions (save_report_v2, etc.) to save detailed outputs and scores to the /Results_* directories.

• Use optimizing_*.ipynb notebooks to understand the prompt optimization process (generating the JSON files in /Optimized_prompts).
• Use SQL_generic.ipynb or python_generic.ipynb to explore the basic flow of individual program components.