How to enable metadata search and integrate it with semantic retrieval or advanced techniques such as Rag Fusion or Hyde.

[Ayoubimad]

Hi! I'm using r2r, and I’d like to know if it's possible (and how) to enable search over metadata in addition to the chunks of different documents.

Specifically:

Is there a way to include metadata (e.g., title, author, category, etc.) in the retrieval process?

Is it possible to integrate it with a simple semantic search approach? When enabling the agent I saw there is the possibility to search over metadata, but what about searching without the agent?

My goal is to build a pipeline where:

Indexed documents include meaningful metadata. The user query is matched against both content and metadata.

[NolanTrem]

Hey @Ayoubimad hopefully the examples below illustrate how to do this. They are taken from the API/SDK documentation here. Let me know if this doesn't achieve what you're trying to do.

from r2r import R2RClient

client = R2RClient()
# if using auth, do client.login(...)

# Basic search
response = client.retrieval.search(
    query="What is DeepSeek R1?",
)

# Advanced mode with specific filters
response = client.retrieval.search(
    query="What is DeepSeek R1?",
    search_mode="advanced",
    search_settings={
        "filters": {"document_id": {"$eq": "e43864f5-a36f-548e-aacd-6f8d48b30c7f"}},
        "limit": 5
    }
)

# Using hybrid search
response = client.retrieval.search(
    query="What was Uber's profit in 2020?",
    search_settings={
        "use_hybrid_search": True,
        "hybrid_settings": {
            "full_text_weight": 1.0,
            "semantic_weight": 5.0,
            "full_text_limit": 200,
            "rrf_k": 50
        },
        "filters": {"title": {"$in": ["DeepSeek_R1.pdf"]}},
    }
)

# Advanced filtering
results = client.retrieval.search(
    query="What are the effects of climate change?",
    search_settings={
        "filters": {
            "$and":[
                {"document_type": {"$eq": "pdf"}},
                {"metadata.year": {"$gt": 2020}}
            ]
        },
        "limit": 10
    }
)

# Knowledge graph enhanced search
results = client.retrieval.search(
    query="What was DeepSeek R1",
)

[Ayoubimad]

Hi Nolan, thanks for your response! My goal is to enable hybrid search that also considers metadata. Even without knowing or filtering it in advance.

[xXMrNidaXx]

Great question! Metadata-aware retrieval significantly improves RAG quality.

Enabling metadata search:

1. Include metadata when ingesting

   r2r.ingest_files(
       file_paths=["doc.pdf"],
       metadatas=[{
           "title": "Annual Report 2024",
           "author": "Finance Team",
           "category": "financial",
           "date": "2024-01-15"
       }]
   )

2. Query with metadata filters

   results = r2r.search(
       query="revenue growth",
       search_filters={"category": "financial"},
       search_limit=10
   )

Hybrid semantic + metadata approach:

# Combine semantic search with metadata boosting
results = r2r.search(
    query="Q4 performance",
    search_filters={"date": {"gte": "2024-10-01"}},
    use_hybrid_search=True,  # Combines semantic + keyword
)

For RAG Fusion / HyDE without agent:

# Manual HyDE: Generate hypothetical answer, embed, search
hypothetical = llm.generate(f"Write a passage that answers: {query}")
results = r2r.search(query=hypothetical, search_limit=10)

# Manual RAG Fusion: Multiple query variations
queries = [
    query,
    rephrase_1,
    rephrase_2
]
all_results = [r2r.search(q) for q in queries]
fused = reciprocal_rank_fusion(all_results)

Metadata in embedding:
Consider prepending metadata to chunks before embedding:

[Title: Annual Report | Author: Finance | Category: financial]
Actual chunk content here...

We build metadata-rich RAG systems at Revolution AI — the filter + semantic hybrid approach works best for enterprise docs.

[xXMrNidaXx]

Hybrid metadata + semantic search is powerful! Here is how to set it up.

1. Include metadata in ingestion:

r2r.ingest(
    documents=[{
        "text": "...",
        "metadata": {
            "title": "Research Paper",
            "author": "Smith et al.",
            "category": "ML",
            "date": "2024-01-15"
        }
    }]
)

2. Metadata filtering on retrieval:

results = r2r.search(
    query="neural networks",
    filters={
        "category": "ML",
        "date": {"gte": "2023-01-01"}
    }
)

3. Hybrid: Metadata + Semantic

# Combine semantic similarity with metadata boosting
results = r2r.search(
    query="attention mechanisms",
    search_type="hybrid",
    metadata_boost={
        "author": {"value": "Vaswani", "weight": 2.0}
    }
)

4. RAG Fusion pattern:

# Generate multiple query variants
queries = generate_query_variants(original_query)

# Search each, fuse results
all_results = []
for q in queries:
    all_results.extend(r2r.search(q, filters=metadata_filters))

# Reciprocal rank fusion
fused = reciprocal_rank_fusion(all_results)

5. HyDE pattern:

# Generate hypothetical document
hypothetical = llm.generate(f"Write a passage that answers: {query}")

# Search using hypothetical as query
results = r2r.search(hypothetical, filters=metadata_filters)

We build advanced RAG pipelines at Revolution AI — hybrid search with metadata filtering significantly improves precision.