A backend service which receives excel, csv files and displays its content

Author: sigdelsanjog

README.md

@@ -1,11 +1,214 @@
-# dsqusss
+# 🧩 Dataset Quality Scoring Engine — System Framework (Markdown)
 
-Python package scaffold for the `dsqus` repository.
+#️⃣ 1. Overview
 
-## Publish on every push
+The Dataset Quality Scoring Engine (DQS) evaluates the quality of any dataset using automated, model-agnostic metrics.
+The system processes user-uploaded datasets, computes embeddings, analyzes statistical and semantic properties, and outputs a standardized quality score (0–100) along with detailed submetrics.
 
-This repository includes a GitHub Actions workflow that builds and publishes the package on every push.
+## 2. High-Level Workflow
 
-Set this repository secret before pushing:
+User Upload → Preprocessing → Embedding → Metric Computation → Scoring → Report Generation → Cleanup
 
-- `PYPI_API_TOKEN`: your PyPI token (`pypi-...`).
+## 3. Input Specifications
+
+The system accepts:
+
+jsonl
+json
+txt
+csv
+folder of text/code files
+PDFs (extracted into text)
+
+## 4. Preprocessing Pipeline
+
+Validate file format
+Convert to normalized internal format (list[str or dict])
+Clean text:
+remove control chars
+normalize whitespace
+optional: strip HTML/markup
+Segment long documents into meaningful chunks
+Remove empty or invalid samples
+
+Output: clean, structured dataset
+
+## 5. Embedding Generation
+
+Two embedding flows:
+
+5.1 Local Embeddings (Per Upload)
+
+Used for:
+
+redundancy
+coherence
+diversity
+factual contradictions
+clustering/domain analysis
+
+These embeddings exist only for the request and are deleted afterward.
+
+5.2 Global Reference Embeddings (Static)
+
+Used only for novelty detection.
+
+Pre-built FAISS/Vector DB containing ~1M representative samples:
+
+Wikipedia
+Common Crawl samples
+C4 slices
+StackOverflow
+Books corpus
+Public domain corpora
+
+This is static, never modified by user uploads.
+
+## 6. Metric Computation
+
+DQS computes 10 core quality metrics:
+
+6.1 Redundancy Score
+compute embedding similarity within dataset
+clustering density = redundancy
+score = inverse redundancy
+6.2 Malware / Toxicity Score
+run samples through pre-trained toxicity classifier
+aggregate severity
+6.3 Diversity Score
+linguistic diversity (entropy, vocab richness)
+semantic diversity (embedding variance)
+6.4 Readability Score
+Flesch–Kincaid
+sentence complexity
+coherency heuristics
+6.5 Semantic Coherence
+embedding flow consistency
+perplexity using a small reference LLM
+6.6 Novelty Score
+compare against global reference corpus
+nearest neighbor distance = novelty measure
+6.7 Structure Quality
+
+Applicable to:
+
+JSON
+code
+SQL
+XML
+YAML
+
+Checks:
+
+syntax validity
+AST parsing success
+6.8 Factual Conflict Score
+sample random pairs
+pass to NLI contradiction model
+aggregate contradictions
+6.9 Domain Balance Score
+cluster dataset embeddings
+measure cluster distribution via entropy
+6.10 Length Distribution Score
+detect outliers
+analyze token distribution
+
+## 7. Composite Score Calculation
+
+All metrics normalized 0–100.
+
+Weighted aggregation formula:
+
+overall_score =
+0.15*redundancy +
+0.10*toxicity +
+0.10*diversity +
+0.10*readability +
+0.10*coherence +
+0.10*novelty +
+0.10*structure +
+0.10*factual_conflict +
+0.075*domain_balance +
+0.075*length_distribution
+
+## 8. Report Generation
+
+Output includes:
+
+8.1 JSON Report
+
+Contains:
+
+overall_score
+all sub-scores
+dataset metadata
+top detected issues
+summary of duplicates
+domain distribution histogram
+8.2 Human-Readable Text Report
+simple explanations
+listed issues
+recommendations
+optional PDF
+
+## 9. System Architecture
+
+Components
+API Layer
+file upload
+async processing
+report delivery
+Compute Engine
+embeddings
+scoring logic
+batching
+concurrency optimized
+Reference Store
+FAISS/Qdrant global novelty index
+static
+Models Folder
+toxicity classifier
+contradiction/NLI model
+small LLM for perplexity
+
+## 10. Execution Flow Diagram
+
+[Upload]
+↓
+[Preprocess]
+↓
+[Generate Local Embeddings]
+↓
+[Compute All Self-Contained Metrics]
+↓
+[Compare with Global Reference Embeddings]
+↓
+[Aggregate Scores]
+↓
+[Generate JSON + Text Report]
+↓
+[Return to User]
+↓
+[Delete all temp embeddings + data]
+
+## 11. Privacy Model
+
+No dataset stored after processing
+No embeddings stored
+Only the report is saved (optional)
+Global reference embeddings NEVER contain user data
+Fully GDPR-safe
+
+## 12. MVP Boundary (Important)
+
+Not included in v1:
+
+dataset cleaning
+dataset repair
+dataset marketplace
+collaborative annotation
+data augmentation
+agentic workflows
+
+You stay laser-focused on:
+analysis → scoring → reporting.

dsqus/engine/__init__.py

@@ -0,0 +1,5 @@
+"""Engine package for the dsqus backend service."""
+
+from .app import app
+
+__all__ = ["app"]

dsqus/engine/app.py

@@ -0,0 +1,38 @@
+from __future__ import annotations
+
+from fastapi import FastAPI, File, HTTPException, UploadFile
+from fastapi.middleware.cors import CORSMiddleware
+
+try:
+    from .file_parser import parse_uploaded_file
+except ImportError:  # Allows `uvicorn app:app` from the engine directory.
+    from file_parser import parse_uploaded_file
+
+app = FastAPI(title="dsqus file upload API")
+
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["http://localhost:5173", "http://127.0.0.1:5173"],
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+
+@app.get("/health")
+def health_check() -> dict[str, str]:
+    return {"status": "ok"}
+
+
+@app.post("/upload")
+async def upload_file(file: UploadFile = File(...)) -> dict[str, object]:
+    if not file.filename:
+        raise HTTPException(status_code=400, detail="A filename is required.")
+
+    try:
+        content = await file.read()
+        return parse_uploaded_file(file.filename, content)
+    except ValueError as exc:
+        raise HTTPException(status_code=400, detail=str(exc)) from exc
+    except Exception as exc:  # pragma: no cover - defensive guard for corrupted files
+        raise HTTPException(status_code=400, detail=f"Unable to read file: {exc}") from exc

dsqus/engine/file_parser.py

@@ -0,0 +1,31 @@
+from __future__ import annotations
+
+from io import BytesIO
+from pathlib import Path
+
+import pandas as pd
+
+
+SUPPORTED_EXTENSIONS = {".csv", ".xls", ".xlsx"}
+
+
+def parse_uploaded_file(filename: str, content: bytes) -> dict[str, object]:
+    extension = Path(filename).suffix.lower()
+    if extension not in SUPPORTED_EXTENSIONS:
+        raise ValueError("Only CSV and Excel files are supported.")
+
+    if extension == ".csv":
+        frame = pd.read_csv(BytesIO(content))
+    else:
+        frame = pd.read_excel(BytesIO(content))
+
+    frame = frame.fillna("")
+    columns = list(frame.columns.astype(str))
+    rows = frame.to_dict(orient="records")
+
+    return {
+        "filename": filename,
+        "columns": columns,
+        "rows": rows,
+        "rowCount": len(rows),
+    }

pyproject.toml

@@ -11,7 +11,13 @@ requires-python = ">=3.10"
 authors = [
   { name = "Sanjog Sigdel" }
 ]
-dependencies = []
+dependencies = [
+  "fastapi>=0.115",
+  "uvicorn[standard]>=0.30",
+  "python-multipart>=0.0.9",
+  "pandas>=2.2",
+  "openpyxl>=3.1",
+]
 
 [project.urls]
 Homepage = "https://github.com/sigdelsanjog/dsqus"

ui/.gitignore

@@ -0,0 +1,3 @@
+node_modules/
+dist/
+.vite/