🏭 Secure Engineering Document Intelligence RAG System

An enterprise-grade, fully offline Document Intelligence platform built to parse, search, query, and audit complex industrial engineering standard specifications. The system leverages high-precision hybrid retrieval (dense vector + sparse BM25), Cross-Encoder reranking, coordinate-aware table intelligence parsing, dynamic unit conversion, and visual PDF grounding with bounding-box highlights.

Every query is automatically audited against a zero-tolerance hallucination and numeric compliance guard, making it ideal for safety-critical industrial engineering workflows.

---

🚀 Key System Features

1. Auto-Ingestion Watchdog Pipeline: Scans the uploaded_documents/ folder in real-time. Automatically parses, chunks, embeds, and indexes PDFs using SHA-256 fingerprint deduplication, moving processed files to processed_documents/ and corrupted ones to failed_documents/.
2. Layout-Aware Hierarchical Parser: Captures PDF layout boundaries, headings, and outlines. Combines PyMuPDF layout analysis with Gaussian-blurred contour preprocessing and Tesseract OCR fallbacks for scanned sheets.
3. Advanced Table QA & Filtering Engine: Extracts borderless data tables programmatically by character coordinate grids. Parses comparisons (above 25 mm, between 10 and 20 mm), expands metric lookups (mapping generic parameters to Root Bead and Top Bead combinations), and performs automatic unit conversions (mm ↔ cm ↔ inch, bar ↔ psig ↔ °C ↔ °F).
4. Conversational Context Resolution: Automatically extracts the previously active table from the chat history and appends it to implicit follow-up queries (e.g. resolving "Show rows above 25 mm" to "Show rows above 25 mm for DOUBLE BEVEL BUTT WELD").
5. Zero-Tolerance Compliance Guard: Runs an inline check verifying that every response is 100% grounded in the retrieved context. Automatically intercepts, filters, and logs hallucinations, acronym conflicts, or numeric mismatches before rendering the answer.
6. Layout Grounding & PDF Viewer: Invokes transparent blue bounding-box highlights over the cited source paragraphs, rendering the vertical page coordinate crops directly inline in the UI.

---

📐 System Architecture

1. Ingestion & Watchdog Pipeline

graph TD
    A[New PDF Uploaded] -->|Watchdog Detection| B[uploaded_documents/]
    B -->|Check SHA-256| C{Has Hash Changed?}
    C -->|No: Duplicate| D[Move to processed_documents/ without re-indexing]
    C -->|Yes: New/Modified| E[Copy to data/raw/ and Enqueue Task]
    E -->|Background Worker| F[Process PDF parsing & indexing]
    F -->|Success| G[Move original PDF to processed_documents/]
    F -->|Failure| H[Retry 2x, if fails move to failed_documents/]

Loading

2. Chat Query Runtime & Verification Flow

graph TD
    A[User Query] -->|Expand Abbreviations & Context| B[Hybrid Retrieval]
    B -->|RRF & Reranker| C[Top Chunks Context]
    C -->|Format Prompt| D[Ollama Qwen2.5:7b Engine]
    D -->|Generate Answer| E[Hallucination Compliance Guard]
    E -->|Verify Numeric Matches & Hallucinations| F[Compliance Report & Citation Extraction]
    F -->|Persist Turn| G[Render Chat Message]

Loading

---

📁 Repository Structure

rag_project/
├── app.py                      # Main Streamlit web application entrypoint
├── setup.py                    # Primary environment bootstrap, system check, and indexing entrypoint
├── requirements.txt            # System dependencies pinned to exact versions
├── PROJECT_ARCHITECTURE.md     # In-depth technical architecture documentation
├── FINAL_READINESS_REPORT.md   # System readiness, validation matrices, and performance summaries
├── config/
│   ├── __init__.py
│   └── config.py               # Centralized layout paths, model names, and hyper-parameters
├── data/
│   ├── raw/                    # Baseline source technical specification PDFs
│   ├── raw_backup/             # Safe backup copy of active standards
│   └── processed/              # Persistent metadata (manifest, abbreviation glossary)
├── db/
│   └── qdrant_storage/         # SQLite transactional local Qdrant database files
├── failed_documents/           # Watchdog sink for invalid or unreadable PDFs
├── logs/                       # Runtime logs folder
├── processed_documents/        # Watchdog sink for successfully indexed PDFs
├── uploaded_documents/         # Watchdog drop-folder monitored for automatic ingestion
├── tests/
│   ├── __init__.py
│   ├── test_app.py             # Streamlit AppTest simulation verification suite
│   ├── test_compliance.py      # Hallucination Guard compliance verification suite
│   ├── test_table_intelligence.py # Table QA filtering, lookups, and unit conversion checks
│   └── test_table_discovery.py # Table Discovery intent and routing verification checks
└── src/
    ├── __init__.py
    ├── ingestion/
    │   ├── glossary.py         # Dynamic engineering abbreviation glossary
    │   └── ingestion_queue.py  # Watchdog handler and concurrent task queue
    ├── parsing/
    │   ├── chunking.py         # Hierarchical text & table chunk splitter
    │   └── parser.py           # Layout-aware PDF plumber and Tesseract OCR engine
    ├── vectorstore/
    │   ├── embeddings.py       # SentenceTransformer vector embedding wrapper
    │   └── vector_store.py     # Qdrant client collection managers and vector operations
    ├── retrieval/
    │   ├── citation_engine.py  # Inline citation extractor and page highlight crop generator
    │   ├── generator.py        # Ollama local LLM chat coordinator
    │   ├── reranker.py         # Cross-Encoder candidate sequence evaluator
    │   ├── retrieval.py        # Hybrid (RRF vector + BM25) retriever
    │   └── table_qa.py         # Structured engineering lookup extractor
    ├── monitoring/
    │   ├── answer_grounding.py # LLM-based answer-to-context evaluator
    │   ├── faithfulness_eval.py# Groundedness/faithfulness evaluator
    │   ├── hallucination_check.py# Hallucination score verification
    │   ├── hallucination_guard.py# Centralized compliance auditor orchestrator
    │   └── retrieval_eval.py   # Retrieval latency and score evaluator
    └── ui/
        ├── animations.py       # HSL theme CSS stylesheet injections and micro-animations
        ├── chat_panel.py       # Chat execution loop and conversation UI
        ├── engineering_tables.py# Inline table rendering component
        ├── ingestion_dashboard.py# Ingestion queue live progress panel
        ├── pdf_viewer.py       # Bounding box highlighted visual crop viewer
        ├── query_suggestions.py# Suggested query widgets
        ├── sidebar.py          # Command center sidebar (system status & purging buttons)
        ├── system_monitor.py   # Production Readiness Audit & loaded components details
        └── ui_helpers.py       # Streamlit helper utilities

---

🛠️ Installation & Setup

Prerequisites

1. Python: Python 3.10 or higher (verified on 3.12).
2. Ollama: Download and install Ollama. Fetch the local model:

   ollama pull qwen2.5:7b

3. Tesseract OCR: Install Tesseract on your local machine:
   • macOS: brew install tesseract
   • Linux: sudo apt-get install tesseract-ocr

Installation

1. Clone the repository and navigate to the directory:

   git clone <your-repository-url>
   cd rag_project

2. Create and activate a virtual environment:

   python3 -m venv venv
   source venv/bin/activate

3. Install system dependencies:

   pip install -r requirements.txt

4. Run the setup bootstrap script to verify installations and seed the database:

   python setup.py

Running the Web Application

Launch the local Streamlit dashboard server:

streamlit run app.py

Open http://localhost:8501 in your browser.

---

🧪 Verification & Testing

The system is equipped with modular automated verification suites:

• Run Table Intelligence Suite:

  python tests/test_table_intelligence.py

• Run Table Discovery Intent Suite:

  python tests/test_table_discovery.py

• Run Compliance Guard Suite:

  python tests/test_compliance.py

• Run Streamlit app tests:

  pytest tests/test_app.py

---

📊 Evaluation & Demo Script

Here is a quick walkthrough to demonstrate the system capabilities:

1. Auto-Ingestion Watchdog:
   • Go to the System Status tab. Verify the watchdog status is COMPLIANT.
   • Drag and drop a specification PDF into uploaded_documents/.
   • View the live task progress. Once completed, notice the file has moved to processed_documents/.
2. SHA-256 Deduplication:
   • Drag and drop the same file into uploaded_documents/ again.
   • The system instantly detects the duplicate hash, skips re-indexing, and removes the duplicate file.
3. Grounded QA & Bounding Boxes:
   • In the Engineering Chat tab, ask: "What is the minimum storage capacity required for each dust hopper?"
   • Once the response renders, expand View Sources and click one of the citation buttons (e.g. 📖 [1] 6-23-0002_0.pdf).
   • Open View PDF Evidence to view the highlighted page crop inline.
4. Table Intelligence:
   • Ask: "What is the tracer length at 150 PSIG?"
   • Expand the View Table Evidence panel to view the matching table row highlighted inside a clean markdown card.
5. Conversational Table QA:
   • Ask the follow-up query: "Show rows above 25 mm thickness."
   • The system dynamically extracts the active table title from your conversation history, scopes the filter query to the Double Bevel Butt Weld table, and outputs the filtered row subset inside the 📋 Filtered Rows collapse widget.