Skip to content

CONCEPT.md

Latest commit

 

History

History
598 lines (440 loc) · 13.6 KB

CONCEPT.md

File metadata and controls

598 lines (440 loc) · 13.6 KB
Raw

Query Preprocessing

What is Query Preprocessing?

Query preprocessing is cleaning up user queries before searching for relevant documents.

Think of it like preparing ingredients before cooking:

  • Wash the vegetables (remove noise)
  • Chop everything the same size (normalize format)
  • Use fresh ingredients (expand abbreviations)

Simple Definition:

Query preprocessing transforms messy user input into clean, consistent queries that work better with search.

Why Do We Need It?

The Problem

Users type queries in many different ways:

"What's ML?"
"WHAT IS ML!!!"
"what   is    ml"
"  What is ML?  "

These all mean the same thing, but computers see them as different!

Without Preprocessing

User 1: "What's ML?"     → Gets Result A
User 2: "what is ml"     → Gets Result B  
User 3: "WHAT IS ML???"  → Gets Result C

❌ Same question, different answers = Bad user experience

With Preprocessing

User 1: "What's ML?"     → "what is machine learning" → Result X
User 2: "what is ml"     → "what is machine learning" → Result X
User 3: "WHAT IS ML???"  → "what is machine learning" → Result X

✅ Same question, same answer = Good user experience

What is "Noise" in Queries?

Noise = anything that doesn't help find the right answer.

Examples of Noise

1. Extra Spaces

"what   is    python"  ← Too many spaces
"what is python"       ← Clean

2. SHOUTING

"WHAT IS PYTHON!!!"    ← All caps + punctuation
"what is python"       ← Clean

3. Punctuation Overload

"what's python???"     ← Too much punctuation
"what is python"       ← Clean

4. Messy Formatting

"  What   IS   Python???  "  ← Spaces, caps, punctuation
"what is python"             ← Clean

Why Noise is Bad

Noisy Query:   "   WHAT   IS   Python???  "
                    ↓
              Embedding Model
                    ↓
         [0.234, 0.112, 0.889, ...]  ← Inconsistent embedding
                    ↓
              Poor results ❌


Clean Query:   "what is python"
                    ↓
              Embedding Model
                    ↓
         [0.891, 0.234, 0.567, ...]  ← Consistent embedding
                    ↓
              Good results ✅

Preprocessing Techniques

1. Lowercase (Always Use)

What it does: Converts everything to lowercase

"PYTHON" → "python"
"Python" → "python"
"PyThOn" → "python"

Why it helps:

These should all mean the same thing:

  • "Python programming"
  • "PYTHON programming"
  • "python programming"

But computers see them as different. Lowercase makes them identical!

Example:

// Before
"What is Python?" 
"WHAT IS PYTHON?"

// After lowercase
"what is python?"  ← Both the same now!
"what is python?"

Rule of thumb: Always use lowercase for user queries.

2. Whitespace Normalization (Always Use)

What it does: Fixes spacing issues

"what   is    python"     → "what is python"
"what\tis\npython"        → "what is python"
"  what is python  "      → "what is python"

Why it helps:

Extra spaces confuse the system:

  • Computer sees: "what␣␣␣is␣␣python"
  • Should be: "what␣is␣python"

How it works:

  1. Replace all whitespace (spaces, tabs, newlines) with single space
  2. Remove spaces at start and end

Example:

// Before
"   What   IS   Python???  "

// After normalization
"what is python???"  ← Clean spacing

3. Query Expansion (Recommended)

What it does: Replaces abbreviations with full words

"ML"  → "machine learning"
"AI"  → "artificial intelligence"
"JS"  → "javascript"
"plz" → "please"

Why it helps:

Users use shortcuts, documents use full terms:

User types:       "Tell me about ML"
Document says:    "Machine learning is a subset of AI..."

Problem: "ML" doesn't match "machine learning" well!

Solution: Expand "ML" → "machine learning"

Common Expansions:

Technical:

ml   → machine learning
ai   → artificial intelligence
api  → application programming interface
js   → javascript
py   → python

Chat-speak:

plz  → please
thx  → thanks
abt  → about
diff → difference

Example:

// Before
"tell me abt ML plz"

// After expansion
"tell me about machine learning please"

// Result: Much better match with documents!

4. Special Character Removal (Use Carefully)

What it does: Removes punctuation and symbols

"What's Python???" → "what s python"
"###ML###"        → "ml"
"python!!!"       → "python"

Why it helps:

Special characters add noise:

  • They don't change the meaning much
  • They make embeddings inconsistent
  • Different punctuation = different embeddings

When it helps:

// Excessive noise - removal helps!
"###python### !!! @@@" → "python"
✅ Much cleaner

When it hurts:

// Natural punctuation - removal hurts!
"What's machine learning?" → "what s machine learning"
❌ Created artifact "s"

The Solution:

Expand FIRST, then remove special characters:

// RIGHT ORDER ✅
"What's ML"
  → expand: "what is machine learning"
  → remove special: "what is machine learning"
  ← Clean!

// WRONG ORDER ❌
"What's ML"
  → remove special: "what s ml"
  → expand: "what s machine learning"
  ← Artifact "s" remains!

The Recommended Pipeline

Step-by-Step Process

Here's the best order to clean queries:

function preprocessQuery(query) {
    // 1. Lowercase
    query = query.toLowerCase();
    
    // 2. Trim edges
    query = query.trim();
    
    // 3. Fix whitespace
    query = query.replace(/\s+/g, ' ');
    
    // 4. Expand abbreviations (BEFORE removing special chars!)
    query = expandAbbreviations(query);
    
    // 5. Remove special characters
    query = query.replace(/[^a-z0-9\s]/gi, ' ');
    
    // 6. Fix whitespace again
    query = query.replace(/\s+/g, ' ').trim();
    
    return query;
}

Real Example

Let's see it in action:

Input: "   Hey!!! Tell me abt ML plz???  "

Step 1 - Lowercase:
"   hey!!! tell me abt ml plz???  "

Step 2 - Trim:
"hey!!! tell me abt ml plz???"

Step 3 - Fix whitespace:
"hey!!! tell me abt ml plz???"

Step 4 - Expand abbreviations:
"hey!!! tell me about machine learning please???"

Step 5 - Remove special chars:
"hey tell me about machine learning please"

Step 6 - Fix whitespace again:
"hey tell me about machine learning please"

✅ Final: Clean, searchable query!

Why This Order?

Order matters! Think of it like cooking:

❌ Wrong: Chop vegetables, then wash them

  • Dirt gets inside the cuts!

✅ Right: Wash vegetables, then chop them

  • Clean ingredients from start

In preprocessing:

❌ Wrong: Remove special chars, then expand

"What's ML" → "what s ml" → "what s machine learning"
                            ↑ Artifact remains!

✅ Right: Expand, then remove special chars

"What's ML" → "what is machine learning" → "what is machine learning"
                                           ↑ Clean!

Visual Summary

Before and After

MESSY INPUT:
┌─────────────────────────────────────┐
│ "   WHAT'S   ML???   "              │
│ "what is ml"                        │
│ "###Tell me abt ML plz!!!###"       │
└─────────────────────────────────────┘
         ↓ PREPROCESSING ↓
┌─────────────────────────────────────┐
│ "what is machine learning"          │
│ "what is machine learning"          │
│ "tell me about machine learning     │
│  please"                            │
└─────────────────────────────────────┘
CLEAN OUTPUT - CONSISTENT RESULTS!

The Impact

Without Preprocessing:

Query: "What's ML???"
Embedding: [0.23, 0.45, 0.12, ...]
Top Result: 78% match ❌

With Preprocessing:

Query: "What's ML???"
Preprocessed: "what is machine learning"
Embedding: [0.89, 0.56, 0.78, ...]
Top Result: 92% match ✅

Improvement: 14% better results!

Problem 3: Vocabulary Mismatch

The Gap:

User Query:          Document Content:
"Tell me about ML"   "Machine learning is a subset of AI..."
"JS frameworks"      "JavaScript libraries for building UIs..."
"abt APIs plz"       "About application programming interfaces..."

Impact:

  • User vocabulary ≠ Document vocabulary
  • Abbreviations don't match full terms
  • Chat-speak doesn't match formal writing
  • Semantic similarity is obscured by lexical differences

Problem 4: Stopword Controversy

Traditional View (TF-IDF, BM25):

"what is the best way to learn python"
       ↓ Remove stopwords
"best way learn python"
✓ Focus on content words
✓ Reduce noise

Modern View (Neural Embeddings):

"what is the best way to learn python"
       ↓ Keep stopwords
"what is the best way to learn python"
✓ Context matters: "what is" vs "what"
✓ Grammar provides semantic clues
✓ Models trained on natural language

The Dilemma:

  • Old techniques say remove
  • New models say keep
  • Impact is model-specific
  • Must test empirically

Core Concepts

Concept 1: Embedding Space Stability

Definition: The degree to which semantically identical queries produce similar embeddings.

Visualization:

Unstable Embedding Space (No Preprocessing):
┌────────────────────────────────────┐
│  "python"                          │
│         ●                          │
│                                    │
│                    "Python"        │
│                           ●        │
│                                    │
│  "PYTHON!!!"                       │
│         ●                          │
└────────────────────────────────────┘
Embeddings are scattered despite same meaning

Stable Embedding Space (With Preprocessing):
┌────────────────────────────────────┐
│                                    │
│                                    │
│          "python" cluster          │
│                 ●●●                │
│                                    │
│                                    │
└────────────────────────────────────┘
Embeddings are tightly clustered

Why It Matters:

  • Predictable retrieval
  • Consistent user experience
  • Fewer edge cases
  • Easier to debug

How to Measure:

// Calculate cosine similarity between query variations
const queries = ["What is Python?", "WHAT IS PYTHON", "what is python"];
const embeddings = queries.map(q => embed(preprocess(q)));

// High stability: similarity > 0.99
// Low stability: similarity < 0.95
const stability = cosineSimilarity(embeddings[0], embeddings[1]);

Concept 2: Signal-to-Noise Ratio

Definition: The ratio of meaningful semantic information to irrelevant surface-level noise in a query.

Formula:

SNR = Semantic Information / Total Information

High SNR:   "machine learning algorithms"  (all content)
Medium SNR: "what is ML?"                  (some noise, some content)
Low SNR:    "###ML!!!###"                  (mostly noise)

Impact on Embeddings:

High SNR Query:
[0.23, 0.87, 0.45, ...]  ← Clear semantic signal
         ↓
    Good retrieval

Low SNR Query:
[0.12, 0.09, 0.11, ...]  ← Weak, noisy signal
         ↓
    Poor retrieval

Preprocessing Goal: Increase SNR by removing noise while preserving signal.

Concept 3: Semantic Preservation

Definition: The degree to which preprocessing maintains the original semantic intent of a query.

The Balance:

Too Little Preprocessing:
"###What's ML???###" → Keep all → High noise, low quality

Optimal Preprocessing:
"###What's ML???###" → "what is machine learning" → Low noise, high quality

Too Much Preprocessing:
"###What's ML???###" → "ml" → Lost context, poor quality

Trade-off Curve:

Retrieval Quality
       ↑
   100%│     ╱╲
       │    ╱  ╲
       │   ╱    ╲
       │  ╱      ╲___
       │ ╱           ╲___
     0%└────────────────────→ Preprocessing Aggressiveness
       None  Light  Optimal  Aggressive  Extreme

Key Insight: More preprocessing isn't always better. Find the sweet spot for your use case.

Concept 4: Domain Adaptation

Definition: Customizing preprocessing to match the vocabulary, style, and conventions of your specific domain.

Generic Preprocessing:

expansions = {
    'ml': 'machine learning',
    'ai': 'artificial intelligence',
    'api': 'application programming interface'
}
// Works for general tech queries

Why It Matters:

  • Generic preprocessing misses domain-specific patterns
  • Domain adaptation improves retrieval by 15-30%
  • Users expect domain-appropriate behavior

Final Thought:

Query preprocessing is both an art and a science. The art is understanding your users and domain. The science is measuring what works. Master both, and you'll build RAG systems that delight users with relevant, consistent results.