fix(search): tolerate natural-language queries in hybrid and text retrieval (v0.8.2) (#17)

* fix(search): tolerate natural-language queries in hybrid and text retrieval

SQLite FTS5 parses its MATCH argument as a query language, so a user-typed
question was a syntax error rather than a search: a trailing "?", a comma, an
"&" or a hyphen raised sqlite3.OperationalError straight out of text_search.

hybrid_search caught DatabaseError but not the raw sqlite3 error, so the
exception escaped the fusion logic entirely. Wrapping it is only half the fix,
though — a caught error still dropped the whole lexical half, turning hybrid
retrieval into vector-only search precisely for the identifiers and error codes
that are the reason to fuse in the first place.

Both retrieval entry points now retry an unparseable query with each token
quoted as an FTS5 string literal:

- hybrid_search falls back to the vector side if that retry also fails.
- text_subgraph propagates instead: with no second half to fall back on, an
  empty subgraph would report "no matches" for a broken search.

text_search keeps raising, since its documented contract is FTS5 syntax
(AND/OR/NEAR/term*), and silently rewriting those operators would break it.
It now raises DatabaseError rather than leaking sqlite3.OperationalError, and
SQLiteFTSIndex.search does the same for custom text-index backends.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* chore(release): bump to v0.8.2

Patch: natural-language queries no longer raise out of hybrid_search and
text_subgraph. No new API. The one behaviour change to code that already
worked is text_search's exception type, from sqlite3.OperationalError to
DatabaseError — the type the rest of the library already raised.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

Author: jpmanson

docs/analysis/subgraphs.md

@@ -91,6 +91,10 @@ arguments:
 sub = db.text_subgraph("attention mechanism", k=20, expand=1)
 ```
 
+Unlike [`text_search()`](../search/fulltext.md#query-syntax), it accepts
+user-typed questions: a query FTS5 cannot parse is retried as literal terms
+instead of raising.
+
 `hybrid_subgraph()` seeds from both at once, fused with Reciprocal Rank Fusion:
 
 ```python

docs/search/fulltext.md

@@ -106,6 +106,19 @@ for r in results:
 
 ## Query Syntax
 
+!!! warning "`text_search` takes FTS5 syntax, not free text"
+    The query is parsed as a query language, so characters FTS5 reserves are
+    syntax rather than terms — a user-typed question raises `DatabaseError`.
+    Either quote the terms yourself (`'"graph" "databases"'`), or use
+    [`hybrid_search`](hybrid.md), which accepts natural language and retries
+    unparseable queries as literal terms.
+
+```python
+db.text_search('graph databases?')   # DatabaseError: fts5: syntax error near "?"
+db.text_search('cost-benefit')       # DatabaseError: no such column: benefit
+db.text_search('"graph" "databases"')  # works: two literal terms
+```
+
 ### Basic Terms
 
 ```python

docs/search/hybrid.md

@@ -7,6 +7,13 @@ Combining full-text search (BM25) with vector search for better results.
     use `DocumentIngestor.hybrid_search()` (RRF fusion, ownership/generation
     filters). Manual recipes below still apply to ad-hoc indexes.
 
+!!! warning "Recipes below pass the raw query to `text_search`"
+    That method takes [FTS5 syntax](fulltext.md#query-syntax), so a user-typed
+    question raises `DatabaseError` on its punctuation. `db.hybrid_search()`
+    handles this for you by retrying as literal terms; in your own recipe,
+    quote the tokens (`' '.join(f'"{t}"' for t in re.findall(r'\w+', query))`)
+    before calling `text_search`.
+
 ## Why Hybrid Search?
 
 | Search Type | Strengths | Weaknesses |

grafito/database.py

@@ -38,6 +38,25 @@
 #: Matches txtai's minscore default.
 DEFAULT_SEMANTIC_GRAPH_MIN_SCORE = 0.1
 
+#: Word characters FTS5 keeps; everything else is query syntax to its parser.
+_FTS_TOKEN_RE = re.compile(r"\w+")
+
+
+def _as_fts_literals(query: str) -> str:
+    """Rewrite free text as a conjunction of FTS5 string literals.
+
+    FTS5 parses its MATCH argument as a query language, so a question mark or a
+    comma in a natural-language question is a syntax error rather than a term.
+    Quoting each token turns the whole thing back into plain terms — the same
+    AND-of-terms FTS5 applies to a bare word list, minus the parse failure.
+
+    Returns an empty string when the query holds no word characters at all,
+    which callers should read as "there is no lexical query to run".
+    """
+    tokens = _FTS_TOKEN_RE.findall(query)
+    return " ".join(f'"{token}"' for token in tokens)
+
+
 if TYPE_CHECKING:  # pragma: no cover - typing only
     from .algorithms import Community
     from .ingest_report import IndexReport, SemanticGraphReport
@@ -782,8 +801,14 @@ def hybrid_search(
         Degrades rather than fails: with no text index configured, or no vector
         index, whichever side works is returned on its own.
 
+        Unlike :meth:`text_search`, this takes natural language. A query FTS5
+        cannot parse — a trailing ``?``, a comma, an ``&`` — is retried as plain
+        terms instead of raising, so questions keep their lexical half.
+
         Args:
-            query: The query text, used for both sides.
+            query: The query text, used for both sides. Valid FTS5 operators
+                (``AND``, ``OR``, ``NEAR``, ``term*``) still reach the lexical
+                side; anything FTS5 rejects is re-run as literal terms.
             k: Number of results.
             index: Vector index to search.
             vector_k, text_k: Candidates to draw from each side before fusing.
@@ -833,15 +858,28 @@ def hybrid_search(
             # No vector index, or no embedding function to turn text into one.
             vector_hits = []
 
-        text_hits: list[dict[str, Any]] = []
-        try:
-            text_hits = [
+        def lexical_side(text_query: str) -> list[dict[str, Any]]:
+            return [
                 {"node": row["entity"], "score": row["score"]}
-                for row in self.text_search(query, k=text_k, labels=labels)
+                for row in self.text_search(text_query, k=text_k, labels=labels)
                 if row.get("entity_type") == "node"
             ]
+
+        text_hits: list[dict[str, Any]] = []
+        try:
+            text_hits = lexical_side(query)
         except DatabaseError:
-            text_hits = []
+            # `query` here is natural language, not the FTS5 syntax text_search
+            # documents, so punctuation is a parse error rather than an empty
+            # result. Retry as literals before giving up the lexical half — the
+            # exact terms it contributes are the reason to fuse at all.
+            literals = _as_fts_literals(query)
+            if literals:
+                try:
+                    text_hits = lexical_side(literals)
+                except DatabaseError:
+                    # No FTS index, or no fts5 in this SQLite build.
+                    text_hits = []
 
         if not text_hits:
             return vector_hits[:k]
@@ -1157,16 +1195,33 @@ def text_subgraph(
         subgraph.
 
         Args:
-            query: FTS5 query string.
+            query: FTS5 query string. Unlike :meth:`text_search`, a query FTS5
+                cannot parse is retried as literal terms rather than raising —
+                seeding a subgraph is a retrieval task, and callers reach it
+                with user-typed questions.
             k: Number of seed hits.
             search_labels: Restrict the text search to these labels.
             expand, direction, rel_types, exclude_rel_types, labels, max_nodes,
                 include_edges: As in :meth:`subgraph`.
 
         Returns:
             A :class:`~grafito.subgraph.Subgraph`.
+
+        Raises:
+            DatabaseError: If the query holds no terms to quote, ``k`` is not
+                positive, or FTS5 is unavailable. A query with no matches is an
+                empty subgraph, not an error.
         """
-        hits = self.text_search(query, k=k, labels=search_labels)
+        try:
+            hits = self.text_search(query, k=k, labels=search_labels)
+        except DatabaseError:
+            # The same retry hybrid_search performs. Unlike there, a second
+            # failure propagates: with no vector half to fall back on, returning
+            # an empty subgraph would report "no matches" for a broken search.
+            literals = _as_fts_literals(query)
+            if not literals:
+                raise
+            hits = self.text_search(literals, k=k, labels=search_labels)
         seeds = [
             {"node": hit["entity"], "score": hit["score"]}
             for hit in hits
@@ -2409,7 +2464,17 @@ def text_search(
         labels: list[str] | None = None,
         rel_types: list[str] | None = None,
     ) -> list[dict[str, Any]]:
-        """Search nodes/relationships using FTS5 BM25."""
+        """Search nodes/relationships using FTS5 BM25.
+
+        ``query`` is FTS5 query syntax, not free text: operators like ``AND``,
+        ``OR``, ``NEAR/10`` and ``term*`` work, and punctuation FTS5 reserves
+        raises rather than matching literally. Pass user-typed questions to
+        :meth:`hybrid_search`, which retries unparseable queries as terms.
+
+        Raises:
+            DatabaseError: If the query is empty, ``k`` is not positive, the
+                filters are malformed, or the query is not valid FTS5 syntax.
+        """
         if not query or not query.strip():
             raise DatabaseError("Query cannot be empty")
         if k is None:
@@ -2476,9 +2541,14 @@ def build_query(entity: str, label_filter: list[str] | None) -> tuple[str, list[
             sql += " ORDER BY score ASC LIMIT ?"
             params.append(k)
 
-        cursor = self.conn.execute(sql, params)
+        try:
+            cursor = self.conn.execute(sql, params)
+            rows = cursor.fetchall()
+        except sqlite3.Error as exc:
+            raise DatabaseError(f"Failed to search text index: {exc}", exc) from exc
+
         results = []
-        for row in cursor.fetchall():
+        for row in rows:
             entity_type = row["entity_type"]
             entity_id = int(row["entity_id"])
             score = float(row["score"])

grafito/text_index/sqlite_fts.py

@@ -5,6 +5,7 @@
 import sqlite3
 from typing import Any
 
+from ..exceptions import DatabaseError
 from .base import TextIndex
 
 
@@ -78,22 +79,29 @@ def search(self, query: str, k: int) -> list[tuple[int, float]]:
             
         Returns:
             List of (id, score) tuples, ordered by relevance.
+
+        Raises:
+            DatabaseError: If the query is not valid FTS5 syntax.
         """
         if not query or not query.strip():
             return []
-        
-        cursor = self.conn.execute(
-            """
-            SELECT entity_id, bm25(fts_index) AS score
-            FROM fts_index
-            WHERE label_type = ? AND fts_index MATCH ?
-            ORDER BY score ASC
-            LIMIT ?
-            """,
-            (self.name, query, k),
-        )
-        
-        return [(int(row[0]), float(row[1])) for row in cursor.fetchall()]
+
+        try:
+            cursor = self.conn.execute(
+                """
+                SELECT entity_id, bm25(fts_index) AS score
+                FROM fts_index
+                WHERE label_type = ? AND fts_index MATCH ?
+                ORDER BY score ASC
+                LIMIT ?
+                """,
+                (self.name, query, k),
+            )
+            rows = cursor.fetchall()
+        except sqlite3.Error as exc:
+            raise DatabaseError(f"Failed to search text index: {exc}", exc) from exc
+
+        return [(int(row[0]), float(row[1])) for row in rows]
 
     def save(self, path: str) -> None:
         """Save is a no-op for SQLite FTS (data is in the database)."""

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "grafitodb"
-version = "0.8.1"
+version = "0.8.2"
 description = "SQLite-based Property Graph Database"
 readme = "README.md"
 requires-python = ">=3.11"

tests/test_hybrid_search.py

@@ -168,6 +168,41 @@ def test_without_a_vector_index_the_text_side_carries_it():
     database.close()
 
 
+@pytest.mark.parametrize(
+    "punctuated,plain",
+    [
+        ("graph databases?", "graph databases"),
+        ("graph & databases", "graph databases"),
+        ("what are graph databases, exactly?", "what are graph databases exactly"),
+    ],
+)
+def test_punctuation_does_not_change_the_ranking(db, punctuated, plain):
+    """FTS5 parses '?' and ',' as syntax; a question must rank like a bare query."""
+    assert _ids(db.hybrid_search(punctuated, k=3)) == _ids(db.hybrid_search(plain, k=3))
+
+
+def test_the_lexical_side_survives_punctuation():
+    """With no vector index there is nowhere else for these hits to come from."""
+    database = GrafitoDatabase(':memory:')
+    database.create_text_index("node", "Doc", ["text"])
+    for row in DOCS:
+        database.create_node(labels=["Doc"], properties=row)
+
+    assert "d1" in _ids(database.hybrid_search("graph databases?", k=2))
+    database.close()
+
+
+def test_a_query_of_pure_punctuation_finds_nothing():
+    """Nothing to quote as a term, and no vector index to fall back on."""
+    database = GrafitoDatabase(':memory:')
+    database.create_text_index("node", "Doc", ["text"])
+    for row in DOCS:
+        database.create_node(labels=["Doc"], properties=row)
+
+    assert database.hybrid_search("?!", k=2) == []
+    database.close()
+
+
 def test_with_neither_index_it_returns_nothing():
     database = GrafitoDatabase(':memory:')
     database.create_node(labels=["Doc"], properties={"id": "d1", "text": "graph"})

tests/test_subgraph.py

@@ -217,6 +217,23 @@ def test_text_subgraph_seeds_from_fts(chain):
     assert sub.hops[chain.node_ids["C"]] == 0
 
 
+@pytest.mark.parametrize("query", ["gamma?", "gamma, retrieval", "gamma & retrieval"])
+def test_text_subgraph_tolerates_natural_language_punctuation(chain, query):
+    """FTS5 would reject these outright; seeding retries them as literal terms."""
+    sub = chain.text_subgraph(query, k=2, expand=0)
+    assert "C" in _names(sub)
+
+
+def test_text_subgraph_still_raises_when_the_retry_cannot_help(chain):
+    """Tolerating punctuation must not turn real failures into empty graphs."""
+    # Nothing to quote as a term: the FTS syntax error stands.
+    with pytest.raises(DatabaseError, match="Failed to search text index"):
+        chain.text_subgraph("?!", k=2)
+    # And validation errors survive the retry rather than being swallowed.
+    with pytest.raises(DatabaseError, match="k must be"):
+        chain.text_subgraph("gamma?", k=0)
+
+
 def test_empty_search_yields_an_empty_subgraph(chain):
     sub = chain.subgraph([], expand=2)
     assert sub.is_empty()

tests/test_text_search.py

@@ -3,6 +3,8 @@
 import pytest
 
 from grafito import GrafitoDatabase
+from grafito.exceptions import DatabaseError
+from grafito.text_index.sqlite_fts import SQLiteFTSIndex
 
 
 def _fts5_available() -> bool:
@@ -77,3 +79,34 @@ def test_text_search_rebuild_index(db):
     results = db.text_search("Graph", labels=["Doc"])
     assert len(results) == 1
     assert results[0]["entity"].id == node.id
+
+
+@pytest.mark.parametrize("query", ["graph databases?", "graph, databases", '"unbalanced'])
+def test_text_search_reports_invalid_fts_syntax_as_a_database_error(db, query):
+    """text_search takes FTS5 syntax, so bad syntax fails — but as our error type.
+
+    Free-text callers want :meth:`hybrid_search`, which retries these as literals.
+    """
+    db.create_text_index("node", "Doc", ["title"])
+    db.create_node(labels=["Doc"], properties={"title": "Graph Databases"})
+
+    with pytest.raises(DatabaseError, match="Failed to search text index"):
+        db.text_search(query, labels=["Doc"])
+
+
+def test_text_search_still_honours_fts_operators(db):
+    """The wrapping must not swallow the query language the docs promise."""
+    db.create_text_index("node", "Doc", ["title"])
+    db.create_node(labels=["Doc"], properties={"title": "Graph Databases"})
+
+    assert len(db.text_search("graph AND databases", labels=["Doc"])) == 1
+    assert len(db.text_search("graph OR nothing", labels=["Doc"])) == 1
+    assert len(db.text_search("dat*", labels=["Doc"])) == 1
+
+
+def test_custom_fts_backend_reports_invalid_syntax_as_a_database_error(db):
+    index = SQLiteFTSIndex(db.conn, "custom")
+    index.add([1], ["graph databases"])
+
+    with pytest.raises(DatabaseError, match="Failed to search text index"):
+        index.search("graph databases?", k=2)