Mishkat

Executive Summary

Mishkat is a scalable Retrieval-Augmented Generation (RAG) backend system designed for storing, embedding, and querying Hadith texts. The system leverages state-of-the-art multi-lingual embedding and generative models to allow semantic search over religious texts, specifically Sahih al-Bukhari, in Arabic. Built with FastAPI, the system relies on MongoDB for persistent document storage and Qdrant for high-performance vector search. The architecture handles data ingestion, chunking, vectorization, and conversational querying via simple REST APIs.

System Architecture

The architecture consists of four core components:

1. Application Server (FastAPI): Exposes RESTful APIs, handles RAG business logic (query rewriting, context retrieval, prompt generation), manages user authentication, chat history, and reference collections.
2. Document Database (MongoDB): Stores the raw Hadith objects, textual chunks, users, chat sessions, conversational messages, and references.
3. Vector Database (Qdrant): Stores high-dimensional text embeddings and provides rapid semantic search capabilities using cosine similarity.
4. LLM Provider (Ollama / External Providers): Embeds chunks into vectors and generates conversational responses based on retrieved context.

flowchart TD
    User([User / Client])
    App[FastAPI Application Server]
    Mongo[(MongoDB\nDocument Store)]
    Qdrant[(Qdrant\nVector Database)]
    LLM[LLM Provider\nOllama / External API]

    User -- "REST API (JSON) + Auth" --> App
    App -- "Store/Retrieve Hadiths, Users, Chats & References" --> Mongo
    App -- "Store/Search Vector Embeddings" --> Qdrant
    App -- "Generate Embeddings & Answers" --> LLM

Loading

API Documentation

1. User Endpoints (/api/v1/users)

• POST /api/v1/users/register: Registers a new user.
• POST /api/v1/users/login: Authenticates a user and starts a session.
• GET /api/v1/users/{user_id}: Retrieves a user's profile.

2. Chat Endpoints (/api/v1/chats)

• GET /api/v1/chats/: Retrieves paginated chats for the authenticated user.
• POST /api/v1/chats/: Creates a new chat session using first_message_content.
• GET /api/v1/chats/{chat_id}: Retrieves a specific chat by ID.
• DELETE /api/v1/chats/{chat_id}: Deletes a specific chat.

3. Message Endpoints (/api/v1/messages)

• GET /api/v1/messages/?chat_id={chat_id}&page={page}&size={size}: Retrieves paginated messages for a specific chat.

4. Query Endpoints (/api/v1/query)

• POST /api/v1/query/
  • Description: Submits a query, retrieves context from Qdrant, and generates an answer using the LLM. Supports multi-reference filtering and conversational memory via chat_id.
  • Expected JSON Payload:

    {
      "query": "ما هو فضل الصلاة على النبي؟",
      "references": ["bukhari"],
      "language": "ar",
      "limit": 5,
      "chat_id": "optional_chat_id"
    }

• POST /api/v1/query/stream
  • Description: Same as above, but returns the answer as a Server-Sent Events (SSE) stream.
  • Expected JSON Payload: Same as /query/.
• POST /api/v1/query/search-chunks
  • Description: Searches for text chunks directly based on query text.

5. Reference Endpoints (/api/v1/ref)

• POST /api/v1/ref/: Creates a new reference source.
• GET /api/v1/ref/: Retrieves all references.
• GET /api/v1/ref/{ref_name}: Retrieves a reference by its name.

6. Data Endpoints (/api/v1/data)

• POST /api/v1/data/
  • Description: Adds a single Hadith object to MongoDB.
  • Expected JSON Payload: Hadith JSON object.
• POST /api/v1/data/batch/{batch_size}
  • Description: Batch ingests multiple Hadiths.
  • Expected JSON Payload: List of Hadith JSON objects [...].
• GET /api/v1/data/?hadith_id={id}
  • Description: Retrieves a single Hadith by its ID.
• GET /api/v1/data/hadiths/{page}/{size}
  • Description: Retrieves a paginated list of Hadiths.

7. Chunking Endpoints (/api/v1/chunk)

• POST /api/v1/chunk/
  • Description: Generates text chunks for specific Hadith IDs and stores them in MongoDB.
  • Expected JSON Payload:

    ["hadith_id_1", "hadith_id_2"]

• GET /api/v1/chunk/search/{chunk_id}
  • Description: Retrieves a specific chunk by its ID.
• GET /api/v1/chunk/{page}/{size}
  • Description: Retrieves a paginated list of stored chunks.

8. Vector Endpoints (/api/v1/vector)

• POST /api/v1/vector/
  • Description: Embeds stored chunks for the given Hadith IDs and saves them into Qdrant.
  • Expected JSON Payload:

    ["hadith_id_1", "hadith_id_2"]

• POST /api/v1/vector/search
  • Description: Performs a direct semantic search on Qdrant vectors.
  • Expected JSON Payload:

    {
      "text": "فضل الصلاة",
      "limit": 5
    }

Embedding Model and Chunking Strategy

Chunking Strategy

• Methodology: The system utilizes the RecursiveCharacterTextSplitter from LangChain.
• Parameters: chunk_size = 500 characters, chunk_overlap = 80 characters.
• Justification: Hadith texts often consist of a core text (Matn) and an associated scholarly explanation (Sharh). Chunking by characters recursively ensures that paragraphs and sentences are not forcefully broken in the middle of a concept. A chunk size of 500 provides a well-balanced granularity: it is large enough to contain full context for a specific ruling or explanation, but small enough to remain highly relevant during vector similarity search. The overlap of 80 characters prevents the loss of context at chunk boundaries.

Embedding Model

• Model Selection: bge-m3 (BAAI General Embedding, Multi-lingual) running via Ollama.
• Dimensions: 1024
• Justification: The bge-m3 model is highly optimized for multi-lingual tasks, boasting state-of-the-art performance on Arabic text semantics. Hadith texts contain dense classical Arabic phrasing that simpler models often misinterpret. The 1024 dimensionality provides the required expressiveness to capture fine-grained religious semantics. Furthermore, BGE-M3 handles multiple granularities well, mapping both short search queries and longer explanatory chunks effectively into the same vector space, which drastically improves retrieval accuracy over models that only support symmetric text lengths.

Docker Deployment Instructions

The system is fully containerized using Docker Compose.

1. Clone the repository and navigate to the root directory.
2. Environment Configuration: Navigate to the src directory and copy the example environment file:

   cp src/.env-example src/.env

   Modify .env to include your provider configurations (e.g., set up your external LLM keys if you are not using a local Ollama instance).
3. Build and Start Services: Run the following command from the root directory to spin up FastAPI, MongoDB, and Qdrant in detached mode:

   docker-compose -f docker/docker-compose.yml up -d --build

4. Access the Application:
   • The FastAPI backend will be accessible at: http://localhost:8000
   • You can view the automatically generated Swagger API documentation at: http://localhost:8000/docs
5. Shutting Down: To tear down the containers and networks (keeping volumes intact):

   docker-compose -f docker/docker-compose.yml down