diff --git a/skills/pinecone-assistant/scripts/__pycache__/chat.cpython-314.pyc b/skills/pinecone-assistant/scripts/__pycache__/chat.cpython-314.pyc deleted file mode 100644 index d82299e..0000000 Binary files a/skills/pinecone-assistant/scripts/__pycache__/chat.cpython-314.pyc and /dev/null differ diff --git a/skills/pinecone-assistant/scripts/__pycache__/context.cpython-314.pyc b/skills/pinecone-assistant/scripts/__pycache__/context.cpython-314.pyc deleted file mode 100644 index d988591..0000000 Binary files a/skills/pinecone-assistant/scripts/__pycache__/context.cpython-314.pyc and /dev/null differ diff --git a/skills/pinecone-assistant/scripts/__pycache__/create.cpython-314.pyc b/skills/pinecone-assistant/scripts/__pycache__/create.cpython-314.pyc deleted file mode 100644 index 0bdb750..0000000 Binary files a/skills/pinecone-assistant/scripts/__pycache__/create.cpython-314.pyc and /dev/null differ diff --git a/skills/pinecone-assistant/scripts/__pycache__/list.cpython-314.pyc b/skills/pinecone-assistant/scripts/__pycache__/list.cpython-314.pyc deleted file mode 100644 index a0962dd..0000000 Binary files a/skills/pinecone-assistant/scripts/__pycache__/list.cpython-314.pyc and /dev/null differ diff --git a/skills/pinecone-assistant/scripts/__pycache__/sync.cpython-314.pyc b/skills/pinecone-assistant/scripts/__pycache__/sync.cpython-314.pyc deleted file mode 100644 index 5a5bd27..0000000 Binary files a/skills/pinecone-assistant/scripts/__pycache__/sync.cpython-314.pyc and /dev/null differ diff --git a/skills/pinecone-assistant/scripts/__pycache__/upload.cpython-314.pyc b/skills/pinecone-assistant/scripts/__pycache__/upload.cpython-314.pyc deleted file mode 100644 index 7725433..0000000 Binary files a/skills/pinecone-assistant/scripts/__pycache__/upload.cpython-314.pyc and /dev/null differ diff --git a/skills/pinecone-full-text-search/SKILL.md b/skills/pinecone-full-text-search/SKILL.md index 3a26296..af69f7d 100644 --- a/skills/pinecone-full-text-search/SKILL.md +++ b/skills/pinecone-full-text-search/SKILL.md @@ -1,29 +1,26 @@ --- name: pinecone-full-text-search -description: Create, ingest into, and query a Pinecone full-text-search (FTS) index using the preview API (2026-01.alpha, public preview). Use when the user or agent asks to build a text search index on Pinecone, add dense or sparse vector fields, ingest documents, construct score_by clauses (text / query_string / dense_vector / sparse_vector), or compose with text-match filters ($match_phrase / $match_all / $match_any). Ships `scripts/ingest.py` for safe bulk ingestion (batch_upsert + error inspection + readiness polling); query construction is documented inline in this skill — write `documents.search(...)` calls directly, validated against `pc.preview.indexes.describe(...)` output. +description: Create, ingest into, and query a Pinecone full-text-search (FTS) document index using the graduated document-schema API (Python SDK 10.0.0, API version 2026-07). Use when the user or agent asks to build a text search index on Pinecone, add dense or sparse vector fields, ingest documents, construct score_by clauses (text / query_string / dense_vector / sparse_vector), or compose with text-match filters ($match_phrase / $match_all / $match_any). Ships `scripts/ingest.py` for safe bulk ingestion (batch_upsert + error inspection + readiness polling); query construction is documented inline in this skill — write `documents.search(...)` calls directly, validated against `pc.indexes.describe(...)` output. --- # Pinecone Full-Text Search -> **Requires `pinecone` Python SDK ≥ 9.0** (`pip install pinecone>=9.0`). The FTS document-schema API lives under `pinecone.preview` and is incomplete or absent in earlier SDK builds. The packaged helper scripts pin `pinecone==9.1.0` via PEP 723 inline metadata; if you're writing your own code against this skill, **pin the exact version** — -`pinecone.preview` is explicitly outside SemVer, so signatures can change in any -minor release. `documents.fetch` and `documents.delete` both lost their `filter` -parameter between 9.0.0 and 9.1.0. The wire API version is `2026-01.alpha`. +> **Requires `pinecone` Python SDK ≥ 10.0.0** (`pip install pinecone>=10.0.0`). The document-schema API graduated out of `pinecone.preview` in 10.0.0 — it is now a first-class, SemVer-covered part of the SDK, reachable directly off `pc` (`pc.indexes`, `pc.index(...)`). If you land on this skill from an older habit of importing `pinecone.preview`, stop: that package is deleted outright in 10.0.0 (`ModuleNotFoundError`, no shim). The packaged helper script pins `pinecone==10.0.0` via PEP 723 inline metadata; if you're writing your own code against this skill, pin at least that version. The wire API version is `2026-07`. -> **Authoritative reference (last resort).** If you hit a question this skill and its `references/*.md` files don't answer, the official Pinecone FTS docs are at . Prefer this skill's content for anything covered here — the docs may describe surfaces (e.g. classic vector API) that don't apply to the document-schema FTS path. Consult the link only when you're genuinely stuck. +> **Authoritative reference (last resort).** If you hit a question this skill and its `references/*.md` files don't answer, the official Pinecone FTS docs are at . Prefer this skill's content for anything covered here — the docs may describe surfaces (e.g. classic vector API, or the older `pinecone.preview` shape) that don't apply to the graduated document-schema path. Consult the link only when you're genuinely stuck. > **Tell the user up front:** "This skill ships a helper at `scripts/ingest.py` that handles bulk ingestion safely (batched upsert, error inspection, readiness polling). When we get to the ingest step, I'll use it." Surface this at the start of the conversation so the user knows the helper exists. Query construction is hand-written `documents.search(...)` per the **Querying** section below — there is no query helper. -A workflow skill for building a Pinecone full-text-search index with the preview API (`pinecone.preview`, API version `2026-01.alpha`, public preview as of April 2026). Covers schema design (text, dense vector, sparse vector, filterable metadata), ingestion (including async indexing and polling), and query construction (`text` / `query_string` / `dense_vector` / `sparse_vector` scoring; `$match_phrase` / `$match_all` / `$match_any` text-match filters; `$eq` / `$in` / `$gte` / `$exists` / `$and` / `$or` / `$not` metadata filters). +A workflow skill for building a Pinecone full-text-search index with the graduated document-schema API (`pc.indexes`, `pc.index(name)`, API version `2026-07`). Covers schema design (text, dense vector, sparse vector, filterable metadata), ingestion (including async indexing and polling), and query construction (`text` / `query_string` / `dense_vector` / `sparse_vector` scoring; `$match_phrase` / `$match_all` / `$match_any` text-match filters; `$eq` / `$in` / `$gte` / `$exists` / `$and` / `$or` / `$not` metadata filters). ## Scope — this skill is for the document-schema FTS API only -This skill covers `pc.preview.indexes.create(..., schema=...)`, `pc.preview.index(name)`, `idx.documents.upsert(...)` / `idx.documents.batch_upsert(...)` / `idx.documents.search(...)`. If you find yourself reaching for any of the following, **stop** — those are different Pinecone APIs and this skill's guidance and helpers won't apply: +This skill covers `pc.indexes.create(..., schema=...)`, `pc.index(name)`, `idx.documents.upsert(...)` / `idx.documents.batch_upsert(...)` / `idx.documents.search(...)`. If you find yourself reaching for any of the following, **stop** — those are different Pinecone APIs and this skill's guidance and helpers won't apply: -- **Classic vector / records API**: `pc.Index(name)`, `index.upsert(vectors=[...])` / `index.upsert_records(...)`, `index.query(vector=..., sparse_vector=...)`, `index.search_records(...)`, `pc.create_index(...)` with `ServerlessSpec`, the legacy `pinecone_text.sparse.BM25Encoder` for sparse-dense hybrid. For indexes WITHOUT a schema (raw vectors). -- **Integrated-embedding indexes**: `pc.create_index_for_model(...)` with `embed={...}`. Pinecone vectorizes text server-side. Different upsert/search shapes. Cannot be combined with `full_text_search` fields in the same index. +- **Classic vector / records API**: `pc.Index(name)`, `index.upsert(vectors=[...])`, `index.query(vector=..., sparse_vector=...)`, `pc.create_index(dimension=..., metric=..., spec=ServerlessSpec(...))`. This is the *deprecated sugar* path in 10.0.0 — it still runs, but it creates a schemaless index served by the vector data plane, addressing the vector by the reserved `_values` field. It cannot hold `full_text_search` fields. +- **Integrated-embedding / records indexes**: `pc.create_index_for_model(...)` / `pc.indexes.create_for_model(...)` with `embed={...}`. Pinecone vectorizes text server-side, and the resulting `semantic_text` field is served by the **records** API (`upsert_records` / `search_records`), not the documents API. Different upsert/search shapes. A `semantic_text` field cannot be combined with `full_text_search` fields in the same index. -If the user already has a non-document-schema index, they can stand up a separate document-schema index alongside it — the two are independent — but you can't add FTS fields to a classic index after the fact. +If the user already has a non-document-schema index, they can stand up a separate document-schema index alongside it — the two are independent — but you can't add FTS fields to a classic or integrated-embedding index after the fact, and a document-schema index only ever serves reads and writes through `index.documents.*` — never `index.upsert` / `index.query` / `index.upsert_records` (those calls are refused with "This index has a document schema, so writes must go through the documents API"). ## Querying — construct `documents.search(...)` calls @@ -31,9 +28,9 @@ For any task that asks you to query an FTS index, you write a `documents.search( **Workflow:** -1. **Discover the schema.** Call `pc.preview.indexes.describe()` and read the `schema.fields` dict. Each field's class indicates its type (`PreviewStringField`, `PreviewIntegerField`, `PreviewDenseVectorField`, etc.); attributes tell you whether it's FTS-enabled (`full_text_search`), filterable, or carries a `dimension`. Skip this step only if you've already seen the schema in this conversation. +1. **Discover the schema.** Call `pc.indexes.describe()` and read the `schema.fields` dict. Each field's class indicates its type (`StringField`, `FloatField`, `DenseVectorField`, etc.); attributes tell you whether it's FTS-enabled (`full_text_search`), filterable, or carries a `dimension`. Skip this step only if you've already seen the schema in this conversation. 2. **Construct the call** matching the rules below — one scoring type per request, hard requirements in `filter`, ranking signals in `score_by`, `include_fields` explicit on every call. -3. **Execute** with `idx = pc.preview.index(name=); resp = idx.documents.search(...)` and read `resp.matches`. +3. **Execute** with `idx = pc.index(name=); resp = idx.documents.search(...)` and read `resp.matches`. **Canonical shapes:** @@ -60,8 +57,8 @@ resp = idx.documents.search( **Key rules** (the server enforces these; following them locally keeps the agent loop tight): - `score_by` is a list of clauses, but **exactly one scoring type per request** (server rejects mixed types). Multi-field BM25 is the one exception: multiple `text` clauses, or one `query_string` with `fields: [...]`. To combine BM25 + dense signals, restrict the dense search with a text-match filter (`$match_all` / `$match_phrase` / `$match_any`); do NOT mix scoring types in `score_by`. -- `filter` keys are field names (must exist in schema and be filterable) OR logical operators (`$and`, `$or`, `$not`). Field values are operator dicts (`{"$gt": 5}`, NOT bare values). -- `include_fields` is required on every call. Pass `["*"]` for all stored fields, `[]` for ids+score only, or a list of names. Some SDK builds 400/422 if it's omitted. +- `filter` keys are field names (must exist in schema, or be an auto-indexed metadata field from upserted documents — see **Filterable metadata isn't declared in the schema** below) OR logical operators (`$and`, `$or`, `$not`). Field values are operator dicts (`{"$gt": 5}`, NOT bare values). +- `include_fields` is required on every call. Pass `["*"]` for all stored fields, `[]` for ids+score only, or a list of names. Omitting it on some SDK/backend builds 400s. **Clause shapes** (for `score_by`): @@ -79,10 +76,10 @@ resp = idx.documents.search( | Field type | Legal operators | |---|---| | `string` with FTS | `$match_phrase`, `$match_all`, `$match_any` | -| `string` filterable | `$eq`, `$ne`, `$in`, `$nin`, `$exists` | -| `string_list` filterable | `$in`, `$nin`, `$exists` | -| `float` filterable | `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$exists` | -| `boolean` filterable | `$eq`, `$exists` | +| filterable metadata (string / auto-indexed) | `$eq`, `$ne`, `$in`, `$nin`, `$exists` | +| `string_list` filterable (auto-indexed, not schema-declared) | `$in`, `$nin`, `$exists` | +| `float` filterable (auto-indexed, not schema-declared) | `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$exists` | +| `boolean` filterable (auto-indexed, not schema-declared) | `$eq`, `$exists` | | logical wrappers | `$and: [filters]`, `$or: [filters]`, `$not: filter` | **Match shape on response:** @@ -90,7 +87,7 @@ resp = idx.documents.search( ```python for m in resp.matches: m._id # document id - m._score # match score (NOT `score`); some older SDK builds may also surface `score` + m._score # match score (NOT `score`) m.to_dict() # full doc payload (when include_fields includes the field) ``` @@ -125,7 +122,8 @@ uv run --script scripts/ingest.py \ | `--index` | `-i` | yes | Pinecone index name (must already exist) | | `--sentinel-field` | `-f` | yes | An FTS-enabled field on the index, used for the readiness-poll query. Pick the longest free-text field on your schema. | | `--namespace` | `-n` | no | Default `__default__` | -| `--batch-size` | `-b` | no | Default 100. **Reduce for large dense vectors.** A 50-doc batch with 3072-dim float vectors lands ~5-10 MB and can be rejected; drop to `--batch-size 50` (or lower) at high dimensions. | +| `--batch-size` | `-b` | no | Default 50 (matches the SDK's own `batch_upsert` default). **Reduce for large dense vectors.** A 50-doc batch with 3072-dim float vectors lands ~5-10 MB and can be rejected; drop to `--batch-size 25` (or lower) at high dimensions. | +| `--max-concurrency` | — | no | Default 4. Parallel HTTP connections used to upload batches. | | `--poll-deadline` | — | no | Default 300 (seconds). Time to wait for documents to become searchable before giving up. | | `--sentinel` | `-s` | no | Token used for the readiness-poll query. Default: first whitespace-separated token of `doc[0][sentinel-field]`. | @@ -136,9 +134,9 @@ Loading processed.jsonl ... Loaded 5000 document(s). Sentinel: body='The' -Upserting in batches of 100 ... - batch @ 0: 100 docs in 0.42s (total: 100/5000) - batch @ 100: 100 docs in 0.39s (total: 200/5000) +Upserting in batches of 50 ... + batch @ 0: 50 docs in 0.31s (total: 50/5000) + batch @ 50: 50 docs in 0.29s (total: 100/5000) ... Upsert complete: 5000 doc(s) in 21.4s. @@ -153,7 +151,7 @@ If a batch fails, the script prints every error message and exits non-zero. If t **When you should NOT use the script:** -- The user is doing per-doc patch updates (single-doc `documents.upsert` calls with selective fields). The script is for bulk loads, not per-record operations. +- The user is doing per-doc patch updates. Use `documents.update(...)` for partial field updates (see **Updating documents** in `references/ingestion.md`) — the script is for bulk loads, not per-record operations. - The user is ingesting from a non-JSONL source (CSV, Parquet, Postgres dump). Convert to JSONL first; the script doesn't parse other formats. - The user explicitly asks you to write the ingestion code from scratch (teaching context). Honor the request and follow the canonical pattern: `documents.batch_upsert` + `result.has_errors` inspection + `documents.search` polling with sentinel and deadline. @@ -173,11 +171,11 @@ If the user already gave you a clean JSONL + a schema spec, follow the abbreviat **Steps (when data is already prepared and the schema is decided):** 1. Inspect the corpus shape — text fields, structured metadata, do you also need a vector? Match it to one of the canonical shapes in `references/schema-design.md` (articles, products, tickets, image library, code). -2. Pick analyzer settings on each text field — `language`, `stemming`, `stop_words`. Stemming on for long prose, off for proper nouns / identifiers. -3. Assemble the schema with `SchemaBuilder` and **confirm it with the user before calling `indexes.create`** — schemas are immutable in `2026-01.alpha`, so a wrong call costs a re-ingest. -4. Create the index, poll `describe()` until `status.ready: true`. +2. Pick analyzer settings on each text field — `language`, `stemming`, `stop_words`. Stemming on for long prose, off for proper nouns / identifiers. **Decide which fields are FTS and which are filterable-only metadata** — see **Filterable metadata isn't declared in the schema** below; filterable-only metadata is a documents-side decision, not a schema field. +3. Assemble the schema with `SchemaBuilder` and **confirm it with the user before calling `indexes.create`** — schemas are immutable in `2026-07`, so a wrong call costs a re-ingest. +4. Create the index. `pc.indexes.create(...)` polls until the index is ready by default — no separate wait loop needed unless you passed `timeout=-1`. 5. **Run `scripts/ingest.py --data --index --sentinel-field `** — see the **Ingesting — use the packaged helper** section above. The script handles `batch_upsert` + per-batch error inspection + post-upsert readiness polling in one invocation. Don't hand-write the loop unless the user explicitly asks you to. -6. (The script polls automatically — by the time it exits cleanly, the index is searchable. If you skip the script and roll your own, you must poll `documents.search` with a sentinel query and a deadline; `batch_upsert` returning ≠ searchable.) +6. (The script polls automatically — by the time it exits cleanly, the index is searchable. If you skip the script and roll your own, you must poll `documents.search` with a sentinel query and a deadline; `batch_upsert` returning ≠ searchable — this is a *document*-indexing wait, separate from and in addition to the index-creation wait in step 4.) 7. Validate with one or two probe queries against fields you know contain the sentinel content. **Result.** A working `documents.search` call against the user's data, returning ranked matches. @@ -188,9 +186,9 @@ If the user already gave you a clean JSONL + a schema spec, follow the abbreviat **Steps.** 1. Confirm the new signal represents a **modality or signal text can't express** — image / audio / external score, *or* a different corpus than the existing FTS field. Re-encoding the same text into a dense field is an anti-pattern (`references/schema-design.md` → "When to add a dense field at all"). -2. Because schemas are immutable, **plan a new index, not a migration**. Get user confirmation before recreating. +2. Because schemas are immutable, **plan a new index, not a migration**. Get user confirmation before recreating. A hybrid index must declare its `sparse_vector` field explicitly at creation — there is no way to add one later. 3. Pick an embedding provider and pin its output dimension at schema time. Beware payload-size pitfalls at native dimensions — Gemini-3072 etc. need truncation (`references/ingestion.md` → "Dense-vector payload size"). -4. Schema → create → wait Ready → ingest with embeddings inline or pre-cached. +4. Schema → create (blocks until ready by default) → ingest with embeddings inline or pre-cached. 5. Validate with a **hybrid query**: `dense_vector` score_by + text-match filter (`$match_phrase` / `$match_all`). That's the supported single-call cross-modal shape. **Result.** One index, two retrieval shapes — pure text *and* dense+filter hybrid — both runnable without further setup. @@ -200,7 +198,7 @@ If the user already gave you a clean JSONL + a schema spec, follow the abbreviat **Trigger.** Agent receives a user prompt like "find articles about machine learning that mention TensorFlow and were published after 2024" or "documents about climate policy ranked by similarity to this paragraph." The index already exists. **Steps.** -1. **(Optional) Discover the schema** by calling `pc.preview.indexes.describe()` and reading `schema.fields`. Skip if you already know the field types from earlier in the conversation. +1. **(Optional) Discover the schema** by calling `pc.indexes.describe()` and reading `schema.fields`. Skip if you already know the field types from earlier in the conversation. 2. **Decompose the user's prompt** into `score_by` / `filter` shapes using the agent-mode decomposition table below. (Hard requirements → `filter`. Ranking signals → `score_by`. Always include `include_fields` explicitly.) 3. **Construct the `documents.search(...)` call** following the rules in the Querying section above — one scoring type per request, operator/field-type matching, `include_fields` always set. 4. **Execute** the call. The response carries `resp.matches`; iterate to get `m._id`, `m._score`, and field values via `m.to_dict()`. Use the matches in whatever shape the user asked for. @@ -212,7 +210,7 @@ If the user already gave you a clean JSONL + a schema spec, follow the abbreviat - Mixing scoring types in `score_by` (server rejects). Put hard requirements in `filter`; rank by one signal in `score_by`. - Putting hard requirements in `score_by` as BM25 terms instead of in `filter` as `$match_all` / `$match_phrase` (returns ranked results that don't *guarantee* the term is present). - Operator/field-type mismatches (e.g. `$match_all` on a float field, `$gt` on a string field). Consult the operator table in the Querying section. -- Omitting `include_fields` (some SDK builds 400/422). Always pass it explicitly. +- Omitting `include_fields` (some SDK/backend builds 400). Always pass it explicitly. ## Agent-mode query decomposition @@ -229,7 +227,7 @@ Map user prompt cues to API shapes. Read top-down — identify the cue, copy the | Boolean / boost / slop / phrase-prefix ("weight 'eagle' 3x", "within N words") | `score_by=[{"type": "query_string", "query": ''}]` — only Lucene supports these | | Cross-field boolean ("title or body contains X") | `score_by=[{"type": "query_string", "query": 'title:(X) OR body:(X)'}]` | | Numeric / date / range / boolean metadata ("after 2024", "rating > 4", "in stock") | `filter={"": {"$gt": ..., "$gte": ..., "$eq": ..., "$exists": true}}` | -| Category / tag / list membership ("category = fiction", "tagged X") | `filter={"": {"$in": [...]}}` (works on `string` and `string_list` filterable fields) | +| Category / tag / list membership ("category = fiction", "tagged X") | `filter={"": {"$in": [...]}}` (works on plain filterable metadata and `string_list` filterable fields) | | Semantic similarity / mood / topic ("articles about ML", "documents that feel sombre") | `score_by=[{"type": "dense_vector", "field": "", "values": embed()}]` — requires a `dense_vector` field | | Visual appearance / cross-modal text query against an image corpus | Same dense_vector shape, with the embedding model that produced the stored image vectors. Multimodal embedders (Gemini-2 etc.) map a text query into the image space. | | Hybrid: lexical requirement + semantic ranking ("articles about ML that mention TensorFlow") | Lexical → `filter` (`$match_all` / `$match_phrase`); semantic → `score_by` (`dense_vector`). Single call. | @@ -239,12 +237,50 @@ Map user prompt cues to API shapes. Read top-down — identify the cue, copy the - **One scoring type per request.** `score_by` accepts `text` / `query_string` / `dense_vector` / `sparse_vector`, but a request ranks by *one*. Don't mix dense + text in `score_by` — the server rejects it. Multi-field BM25 is the only "list" pattern that's allowed (multiple `text` clauses, or one cross-field `query_string`). - **Hybrid = filter + score_by, not two `score_by` clauses.** When a prompt has both a lexical requirement and a semantic ranking signal, lexical goes in `filter` (via `$match_*` operators) and semantic goes in `score_by`. If both signals genuinely need to drive *ranking*, run two searches and merge IDs client-side. +## Filterable metadata isn't declared in the schema at all + +This is the single biggest shape change from the old `pinecone.preview` API, and it's easy to get only half right. + +On a **managed** index (the deployment type every example in this skill uses — `{"deployment_type": "managed", "cloud": ..., "region": ...}`, which is also the default when `deployment=` is omitted), the schema may **only** declare fields that participate in **search**: `dense_vector`, `sparse_vector`, and `string` fields with `full_text_search` enabled. **Every other field type — `string` (filterable, no FTS), `string_list`, `float`, `boolean` — is rejected at create time with a 400 if it appears in the schema.** This is confirmed live, not just documented: the server's own error names all four types explicitly — *"The schema only accepts fields used for search (field types `dense_vector`, `sparse_vector`, and `string` with `full_text_search` configuration). To use field '<name>' for filtering (field types `boolean`, `float`, `string`, or `string_list`), omit it from the schema and include it in documents."* (That restriction is specific to managed/BYOC deployments — schema-declared filterable metadata is only legal on **pod** deployments, which this skill doesn't cover.) The `SchemaBuilder` methods `add_float_field`, `add_boolean_field`, and `add_string_list_field` still exist and still work correctly for a pod deployment; for the managed deployments this skill always uses, don't call any of them. + +Instead: **don't declare any filterable-only field in the schema, of any type.** Just include the field in the documents you upsert — Pinecone indexes whatever's present on an upserted document for filtering automatically (exact-match on strings and numbers/booleans, membership on lists), whether or not it appears in the schema, with no configuration needed. + +```python +# WRONG on a managed index — the server 400s on EVERY one of these, not just category: +schema = ( + SchemaBuilder() + .add_string_field("body", full_text_search={"language": "en"}) + .add_string_field("category", filterable=True) # <-- rejected + .add_float_field("year", filterable=True) # <-- also rejected + .add_string_list_field("tags", filterable=True) # <-- also rejected + .build() +) + +# RIGHT — the schema declares only search fields. category/year/tags are +# simply included on upserted documents and auto-index for filtering. +schema = ( + SchemaBuilder() + .add_string_field("body", full_text_search={"language": "en"}) + .build() +) + +idx.documents.upsert(namespace=NAMESPACE, documents=[{ + "_id": "doc-1", + "body": "...", + "year": 2025.0, # not in the schema — still filterable via $gt / $gte / $eq + "tags": ["classic"], # not in the schema — still filterable via $in / $nin + "category": "fiction", # not in the schema — still filterable via $eq / $in / $exists +}]) +``` + +The **forward-looking note** in `references/schema-design.md` from the old preview docs — "declare metadata fields today to be future-proof" — no longer applies; declaring any filterable-only field is now actively wrong, not just unnecessary. + ## Workflow at a glance Three phases. Each has its own reference file — consult it before writing code for that phase. -1. **Design the schema.** Decide which string fields are full-text-searchable, which are filterable metadata, whether you need a `dense_vector` field (and whether it earns its place), whether you also need a `sparse_vector` field, and which numeric / boolean / array filters to declare. Schemas are **fixed at index creation** in `2026-01.alpha` — plan carefully. → `references/schema-design.md` -2. **Ingest documents.** For bulk loads from a prepared JSONL, run the bundled `scripts/ingest.py` helper (it does `batch_upsert` + error inspection + readiness polling correctly by construction — see the **Ingesting — use the packaged helper** section above). For per-doc patch updates, hand-call `documents.upsert`. Either way, documents are indexed asynchronously after the HTTP call returns; `batch_upsert` returning 202 ≠ searchable. → `references/ingestion.md` for the canonical pattern in detail. +1. **Design the schema.** Decide which string fields are full-text-searchable (declared in the schema), whether you need a `dense_vector` field (and whether it earns its place), and whether you also need a `sparse_vector` field — those are the *only* things that ever go in the schema. Every filterable metadata field — string, string_list, float, boolean alike — is NOT declared; it's just included on upserted documents. Schemas are **fixed at index creation** in `2026-07` — plan carefully. → `references/schema-design.md` +2. **Ingest documents.** For bulk loads from a prepared JSONL, run the bundled `scripts/ingest.py` helper (it does `batch_upsert` + error inspection + readiness polling correctly by construction — see the **Ingesting — use the packaged helper** section above). For per-doc patch updates, use `documents.update(...)`. Either way, documents are indexed asynchronously after the HTTP call returns; `batch_upsert` returning 202 ≠ searchable. → `references/ingestion.md` for the canonical pattern in detail. 3. **Query the index.** A single search request ranks by **one** scoring type — pass exactly one of `text`, `query_string`, `dense_vector`, or `sparse_vector` in `score_by` (multi-field BM25 is supported via multiple `text` clauses or a cross-field `query_string`). Layer `filter={...}` for text-match (`$match_phrase` / `$match_all` / `$match_any`) and metadata filters (`$eq` / `$in` / `$gte` / `$exists` / `$and` / `$or` / `$not`). Control the response payload with `include_fields`. → `references/querying.md` ## Quick template @@ -253,40 +289,46 @@ End-to-end skeleton for a minimal text + filterable-metadata index. Copy it and ```python import time -from pinecone import Pinecone -from pinecone.preview import SchemaBuilder +from pinecone import Pinecone, SchemaBuilder -INDEX_NAME = "my-fts-index" # TODO: name your index (lowercase alphanumeric + hyphens, ≤45 chars) +INDEX_NAME = "my-fts-index" # TODO: name your index (lowercase alphanumeric + hyphens, 1-45 chars) NAMESPACE = "__default__" # TODO: pick a namespace; auto-created on first upsert pc = Pinecone() # reads PINECONE_API_KEY # TODO: preprod backends require an x-environment header on the client: # pc = Pinecone(additional_headers={"x-environment": "preprod-aws-0"}) -# 1. Schema — one FTS string field, one filterable string, one filterable float. -# Field names must NOT start with `_` (reserved for `_id` / `_score`) or `$` -# (reserved for filter operators), and are limited to 64 bytes. +# 1. Schema — one FTS string field. That's the only kind of field that goes +# here: `category` and `year` below are deliberately NOT in the schema — +# see "Filterable metadata isn't declared in the schema at all" in +# SKILL.md. Field names must NOT start with `_` (reserved for `_id` / +# `_score`) or `$` (reserved for filter operators), and are limited to 64 +# bytes. schema = ( SchemaBuilder() .add_string_field("body", full_text_search={"language": "en"}) # TODO: rename for your content - .add_string_field("category", filterable=True) # TODO: any exact-match metadata - .add_float_field("year", filterable=True) # TODO: any numeric filter — `float` is the only numeric wire type .build() ) -# 2. Create the index. read_capacity defaults to {"mode": "OnDemand"}; pass +# 2. Create the index. Polls until ready by default (pass timeout=-1 to +# return immediately instead). Deployment defaults to managed/aws/us-east-1 +# when omitted; pass `deployment=` explicitly to pick a different region. +# read_capacity defaults to {"mode": "OnDemand"}; pass # {"mode": "Dedicated", ...} only if you specifically want provisioned reads. -if not pc.preview.indexes.exists(INDEX_NAME): - pc.preview.indexes.create(name=INDEX_NAME, schema=schema) - -# 3. Wait for the index itself to become Ready. -while not pc.preview.indexes.describe(INDEX_NAME).status.ready: - time.sleep(5) +if not pc.indexes.exists(INDEX_NAME): + pc.indexes.create( + name=INDEX_NAME, + schema=schema, + deployment={"deployment_type": "managed", "cloud": "aws", "region": "us-east-1"}, + ) -idx = pc.preview.index(name=INDEX_NAME) +idx = pc.index(name=INDEX_NAME) -# 4. Upsert a single document. `_id` is required, every other field is optional. -# upsert REPLACES the document on conflict — there is no per-field merge in 2026-01.alpha. +# 3. Upsert a single document. `_id` is required, every other field is optional. +# `category` and `year` aren't in the schema but are still filterable — +# see above. +# upsert REPLACES the document on conflict; use documents.update(...) for +# per-field patches (references/ingestion.md). idx.documents.upsert( namespace=NAMESPACE, documents=[{ @@ -297,7 +339,7 @@ idx.documents.upsert( }], ) -# 5. Poll until the FTS side is searchable (upsert returns BEFORE docs are indexed). +# 4. Poll until the FTS side is searchable (upsert returns BEFORE docs are indexed). deadline = time.time() + 300 while time.time() < deadline: resp = idx.documents.search( @@ -309,7 +351,7 @@ while time.time() < deadline: break time.sleep(5) -# 6. Search — text scoring composed with metadata filter. +# 5. Search — text scoring composed with metadata filter. resp = idx.documents.search( namespace=NAMESPACE, top_k=5, @@ -318,34 +360,38 @@ resp = idx.documents.search( include_fields=["*"], # "*" = all stored fields; [] = `_id` + `_score` only ) for m in resp.matches: - print(m._id, getattr(m, "_score", getattr(m, "score", None)), m.to_dict()) + print(m._id, m._score, m.to_dict()) ``` ## Common gotchas +- **No filterable metadata field goes in the schema on managed indexes — string, string_list, float, and boolean alike.** Only `dense_vector`, `sparse_vector`, and FTS-enabled `string` fields are legal in `schema=`; every other field type is rejected with `400`. Confirmed live against the real API, not just documented. Omit it and let it auto-index from upserted documents instead. See **Filterable metadata isn't declared in the schema at all** above — this is the change most likely to break code carried over from the old `pinecone.preview` API, where such fields were declarable. - **One scoring type per search request.** `score_by` accepts `text`, `query_string`, `dense_vector`, or `sparse_vector` — but a request ranks by *one* type. Multi-field BM25 is fine (pass several `text` clauses, or a single cross-field `query_string`). To combine BM25 ranking with a `dense_vector` (or `sparse_vector`) signal, restrict the dense search with a text-match `filter` operator (`$match_phrase` / `$match_all` / `$match_any`) on the lexical field, *not* by mixing types in `score_by`. The "blend a dense vector and a text clause in `score_by`" pattern is rejected by the server. - **Text-match filter operators are the cross-modal hinge.** `$match_phrase` (exact phrase), `$match_all` (every token, any order), `$match_any` (at least one token) are filter-side operators on `full_text_search` fields. Each takes a single string (max 128 tokens). They reuse the field's tokenizer / stemmer, compose under `$and` / `$or` / `$not`, and are the supported way to compose lexical pre-filtering with dense or sparse ranking. **Phrase slop (`"…"~N`), term boost (`^N`), and phrase prefix (`"… word"*`) are scoring-only — they live in `query_string`, not in `filter`.** - **Preprod backends need `additional_headers={"x-environment": "..."}` on the `Pinecone()` client.** Missing the header lands you on prod and you'll see "index not found" / empty-result symptoms that look like code bugs but aren't. -- **`include_fields` is required on every `documents.search(...)` call.** When omitted, the key is left off the request and the server returns **all** stored fields. Pass `["*"]` for all stored fields or a list of names to project. Omitting it on some SDK builds yields `400` / `422` instead of the documented default; always pass it explicitly to avoid surprises. -- **Match score is `_score`; doc id is `_id`.** Public-preview docs return the system match score on the `_score` field so a user metadata field literally named `score` can coexist. Always prefer `_score` on read; some older SDK builds may still surface plain `score`, so for defensive code use `getattr(m, "_score", getattr(m, "score", None))`. +- **`include_fields` is required on every `documents.search(...)` call.** Pass `["*"]` for all stored fields or a list of names to project. Omitting it on some SDK/backend builds yields `400` instead of a sane default; always pass it explicitly to avoid surprises. +- **Match score is `_score`; doc id is `_id`.** The system match score is always on the `_score` field so a user metadata field literally named `score` can coexist. Always read `m._score`, never `m.score`. - **Reserved field names: leading `_` and `$`, max 64 bytes.** `_` is for system fields (`_id`, `_score`); `$` is for filter operators. Schema validation rejects names that violate either rule. Length cap is bytes, not characters — be careful with non-ASCII names. -- **Vector-field cardinality: at most one `dense_vector` and at most one `sparse_vector` per index** in `2026-01.alpha`. Multiple text fields are fine. +- **Vector-field cardinality: at most one `dense_vector` and at most one `sparse_vector` per index** in `2026-07`. Multiple text fields are fine. +- **A hybrid index must declare its `sparse_vector` field at create time — there's no adding one later.** `metric="dotproduct"` on the dense field is NOT a hybrid declaration by itself in `2026-07` (it was in the old preview API). The create call succeeds either way; if the `sparse_vector` field is missing, only the *sparse writes* are refused, later, often from a different part of the codebase. If you're porting an old preview schema, audit every `metric="dotproduct"` dense field for a missing sparse field before recreating it. - **`batch_upsert` failures are silent by default.** The return value carries `has_errors`, `failed_batch_count`, and a list of `BatchError` objects with `error_message`. If you don't inspect them, you'll see "Uploaded 0 / N" and an indefinite "not yet indexed" poll — with the real cause (payload-too-large, schema mismatch, reserved field name) hidden. Always print `result.errors[*].error_message` before downstream steps. -- **Dense-vector payload size matters at batch time.** A 50-doc batch with 3072-dim float vectors lands around 5–10 MB and can be rejected by the preview backend. If every batch fails, try reducing the embedding dimension via your provider's truncation knob (e.g. Gemini's `output_dimensionality=768`) before debugging schema. -- **Async indexing: `batch_upsert` returning ≠ searchable.** The server builds inverted indexes in the background after the HTTP call returns. If you query immediately you'll see empty result sets. Always poll `documents.search` with a sentinel query and a deadline (pattern in `references/ingestion.md`). -- **String FTS field shape is `full_text_search={...}` (dict).** Pass `{}` to enable with all server defaults. **User-settable sub-fields:** `language`, `stemming`, `stop_words`. **Server-applied** (visible in `describe()` responses but NOT settable at index creation): `lowercase` (default `true`) and `max_token_length` (default `40`). Stemming is opt-in (default `false`); `stop_words` is opt-in (default `false`, opposite of pre-public-preview docs). The earlier SDK shape `full_text_searchable=True, language="en"` is legacy and should be avoided. -- **Schemas are fixed at index creation in `2026-01.alpha`.** Adding, removing, or retyping fields after creation is not supported. Changing dimension or metric on an existing vector field requires a new index. Plan the schema once. -- **No partial / per-field updates.** `documents.upsert` always replaces the entire document for a given `_id`. To update one field, fetch the doc, modify in client code, and upsert the full doc back under the same `_id`. -- **Document operations: search supports `filter`, fetch and delete do not.** Fetch is **ID-only** (`POST /documents/fetch` with `ids: [...]`); delete accepts only `ids` or `delete_all: true`. To act on a metadata expression, search first to collect IDs, then fetch or delete those IDs. -- **Namespaces auto-create on first upsert.** Pass any namespace string to `documents.upsert` / `batch_upsert` and the namespace is created on the fly; documents from different namespaces are fully isolated. Use `"__default__"` if you don't need partitioning. **Caveat:** the namespace management endpoints (`POST /namespaces`, `GET /namespaces`, `DELETE /namespaces/{namespace}`) and `describe_index_stats` are NOT yet supported on indexes with document schemas — you can write to a namespace, you just can't list / delete them via the API yet. -- **Document and request size limits** (preview): per-document max **2 MB**; per-request max **2 MB and 1000 documents**; per FTS-enabled `string` field max **100 KB and 10,000 tokens** (tokens > 256 bytes are truncated by the analyzer); per-document filterable metadata (everything *not* in an FTS field) max **40 KB**. A schema can declare up to **100 FTS string fields**. For long-prose corpora, chunk before ingest — see `references/ingestion.md`. +- **Dense-vector payload size matters at batch time.** A 50-doc batch with 3072-dim float vectors lands around 5–10 MB and can be rejected. If every batch fails, try reducing the embedding dimension via your provider's truncation knob (e.g. Gemini's `output_dimensionality=768`) before debugging schema. +- **Async indexing: `batch_upsert` returning ≠ searchable.** The server builds inverted indexes in the background after the HTTP call returns. If you query immediately you'll see empty result sets. Always poll `documents.search` with a sentinel query and a deadline (pattern in `references/ingestion.md`). This is separate from — and in addition to — `pc.indexes.create()`'s own default polling for the *index* becoming ready. +- **String FTS field shape is `full_text_search={...}` (dict), or `True` for server defaults.** **User-settable sub-fields:** `language`, `stemming`, `stop_words`, `ngram`. **Server-applied** (visible in `describe()` responses but NOT settable at index creation): `lowercase` (default `true`) and `max_term_len`. Stemming is opt-in (default `false`) and required if `stop_words=True` is set. A string field is *either* FTS-enabled *or* filterable, never both on a managed index — passing `filterable=True` alongside `full_text_search` makes the server silently keep the filter and drop the search config. +- **Schemas are fixed at index creation in `2026-07`.** Adding, removing, or retyping fields after creation is not supported. Changing dimension or metric on an existing vector field requires a new index. Plan the schema once. +- **Per-field updates are supported: `documents.update(...)`.** Pass `documents=[{"_id": ..., "set_fields": {...}}]`-style records, or `filter=` + `set_fields=`/`remove_fields=` to patch many documents at once by metadata match. `documents.upsert` still fully replaces a document on conflicting `_id` if that's what you want instead. See `references/ingestion.md` → "Updating documents". +- **Document operations: search, fetch, and delete all support `filter` now.** `documents.fetch` and `documents.delete` both gained a `filter` parameter — pass exactly one of `ids`, `filter`, or (for delete) `delete_all`. A filtered `fetch` is paginated (up to 10,000 docs per page via `pagination_token`); an ID-based fetch is never paginated. `documents.delete` returns a response object with `matched_records` (a point-in-time count for filtered deletes; `None` for ID-list or `delete_all` deletes — the delete itself is applied asynchronously). +- **`documents.list(...)` enumerates document IDs in a namespace, with no equivalent in the old preview API.** Lazily-paginated, sorted by ID, optionally filtered by `prefix`. See `references/querying.md` → "`documents.list` — enumerate document IDs". +- **Namespaces auto-create on first upsert.** Pass any namespace string to `documents.upsert` / `batch_upsert` and the namespace is created on the fly; documents from different namespaces are fully isolated. Use `"__default__"` if you don't need partitioning. +- **Namespace management and `describe_index_stats` now work on document-schema indexes.** `idx.create_namespace(name=...)`, `idx.list_namespaces()`, `idx.describe_namespace(name=...)`, `idx.delete_namespace(name=...)`, and `idx.describe_index_stats()` are all available and confirmed working — see `references/ingestion.md` → "Namespace management" for signatures and examples. +- **Document and request size limits**: per-document max **2 MB**; per-request max **2 MB and 1000 documents**; per FTS-enabled `string` field max **100 KB and 10,000 tokens** (tokens > 256 bytes are truncated by the analyzer); per-document filterable metadata (everything *not* in an FTS field) max **40 KB**. A schema can declare up to **100 FTS string fields**. For long-prose corpora, chunk before ingest — see `references/ingestion.md`. - **`score_by` clause shape — singular `field` is canonical for `text`/`dense_vector`/`sparse_vector`; only `query_string` takes a `fields` array.** - `text`: `{"type":"text", "field":"", "query":""}`. - `query_string`: `{"type":"query_string", "query":"", "fields":["",""]}` (the optional `fields` array; `query_string` also accepts a bare `"fields":"body"` string and the legacy `"field":"body"` as an alias). - `dense_vector`: `{"type":"dense_vector", "field":"", "values":[/*floats*/]}`. - `sparse_vector`: `{"type":"sparse_vector", "field":"", "sparse_values":{"indices":[...],"values":[...]}}` — note `sparse_values` (NOT `values`) for sparse clauses. - **Single-term prefix wildcards aren't supported.** `auto*` doesn't work in `query_string`; use phrase prefix (`"machine lea"*` — phrase must contain at least two terms, last term is matched as prefix). -- **Indexes can't be created in CMEK-enabled projects, no backup/restore, no fuzzy or regex search, no S3 bulk import** for document-shaped indexes in `2026-01.alpha`. If any of these are hard requirements, the public-preview FTS surface isn't yet ready. +- **Indexes can't be created in CMEK-enabled projects alongside any `full_text_search` field, no backup/restore, no fuzzy or regex search, no S3 bulk import** for document-shaped indexes in `2026-07`. If any of these are hard requirements, the document-schema FTS surface isn't yet ready. ## Extension points @@ -354,4 +400,3 @@ Currently shipped under `scripts/`: - `scripts/ingest.py` — bulk-ingest a prepared JSONL into an existing FTS index. Handles `batch_upsert` in safe-sized chunks, inspects every batch's `result.errors` and aborts loudly on failure, then polls `documents.search` with a sentinel + deadline until docs are searchable. Schema-agnostic: takes only `--data`, `--index`, `--sentinel-field`. Usage in **Ingesting — use the packaged helper** section above. Query construction does NOT have a packaged helper — write `documents.search(...)` calls directly per the **Querying** section above. - diff --git a/skills/pinecone-full-text-search/references/ingestion.md b/skills/pinecone-full-text-search/references/ingestion.md index 56e3502..cdb8a08 100644 --- a/skills/pinecone-full-text-search/references/ingestion.md +++ b/skills/pinecone-full-text-search/references/ingestion.md @@ -1,11 +1,11 @@ # Ingestion -Writing documents into a Pinecone preview document index uses two methods. Pick based on volume, then handle the *async indexing* gotcha on the other side. +Writing documents into a Pinecone document index uses two methods. Pick based on volume, then handle the *async indexing* gotcha on the other side. ## `documents.upsert` — small writes / patches ```python -idx = pc.preview.index(name=INDEX_NAME) +idx = pc.index(name=INDEX_NAME) upsert_resp = idx.documents.upsert( namespace=NAMESPACE, @@ -14,10 +14,10 @@ upsert_resp = idx.documents.upsert( "_id": "doc-1", "title": "A landmark work that every reader should experience.", "body": "Lorem ipsum...", - "category": "fiction", + "category": "fiction", # not declared in schema — auto-indexed for filtering anyway "year": 2024.0, }, - # ... up to ~1000 documents per call (per public-preview docs) + # ... up to ~1000 documents per call ], ) print(upsert_resp.upserted_count) @@ -26,10 +26,10 @@ print(upsert_resp.upserted_count) Use `upsert` when: - You're writing a single document (e.g. a sentinel doc to verify end-to-end before a bulk load). -- You're "patching" a doc after a correction. *Note*: `2026-01.alpha` has **no per-field merge** — every upsert replaces the entire document on conflicting `_id`. To update a single field, fetch the doc, modify in client code, and upsert the full doc back under the same `_id`. - You're streaming writes from user actions and each request fits in a single batch. +- You want to fully replace a document by `_id` (upsert always replaces the whole document on conflict — for a true per-field patch, use `documents.update` below instead). -Each document is a dict keyed by field name. `_id` is required and must be a non-empty unique string within the namespace. Values must match the declared schema types (FTS strings → `str`, filterable `float` → `int|float`, dense vectors → `list[float]`, sparse → `{"indices": [...], "values": [...]}`). Field names that start with `_` or `$` are rejected; field names are limited to 64 bytes. +Each document is a dict keyed by field name (or a `DocumentRecord`/`UpdateDocumentRecord` instance — both accept a plain dict positionally or fields as keyword arguments). `_id` is required and must be a non-empty unique string within the namespace. Values must match the declared schema types for the few fields actually declared in the schema (FTS strings → `str`, dense vectors → `list[float]`, sparse → `{"indices": [...], "values": [...]}`); every other field on the document — strings, numbers, booleans, string lists — is auto-indexed metadata with nothing declared for it. Field names that start with `_` or `$` are rejected; field names are limited to 64 bytes. The endpoint returns `202 Accepted` (async) and the body's `upserted_count` is the number of items accepted, not the number that have finished indexing. @@ -39,8 +39,8 @@ The endpoint returns `202 Accepted` (async) and the body's `upserted_count` is t result = idx.documents.batch_upsert( namespace=NAMESPACE, documents=documents, # list of dicts, any length - batch_size=50, - max_workers=2, + batch_size=50, # SDK default + max_concurrency=4, # SDK default show_progress=True, ) print(f"{result.successful_item_count:,} / {result.total_item_count:,} succeeded") @@ -54,17 +54,17 @@ if result.has_errors: f"first _id={sample!r}): {err.error_message}") ``` -The SDK splits `documents` into `batch_size`-sized chunks and uploads them over `max_workers` parallel HTTP connections. `show_progress=True` prints a tqdm-style bar. +The SDK splits `documents` into `batch_size`-sized chunks and uploads them over `max_concurrency` parallel HTTP connections. `show_progress=True` prints a tqdm-style bar. `max_concurrency` must be a plain `int` (default `4`) — it no longer accepts `None`. -### Tuning `batch_size` and `max_workers` +### Tuning `batch_size` and `max_concurrency` -- **`batch_size=50`** is the sweet spot — comfortably below the per-request cap and small enough that transient failures cost less to redo. -- **`max_workers=2`** is a safe default. Bump to `4` for large (thousands-of-docs) loads where you're not simultaneously embedding. Ramp cautiously above 4 — you'll hit Pinecone or upstream embedding-provider rate limits first. -- If you're embedding on the fly (computing vectors inside the upsert loop), keep `max_workers` low so embedding latency dominates rather than index write latency. +- **`batch_size=50`** (the SDK's own default) is the sweet spot — comfortably below the per-request cap and small enough that transient failures cost less to redo. +- **`max_concurrency=4`** (the SDK's own default) is a safe default for large (thousands-of-docs) loads where you're not simultaneously embedding. Ramp cautiously above 4 — you'll hit Pinecone or upstream embedding-provider rate limits first. +- If you're embedding on the fly (computing vectors inside the upsert loop), keep `max_concurrency` low so embedding latency dominates rather than index write latency. ### Document and request size caps -**Hard limits in `2026-01.alpha`:** +**Hard limits in `2026-07`:** - **Per document**: max **2 MB** (serialized JSON, all stored fields combined). - **Per `full_text_search` string field**: max **100 KB** AND max **10,000 tokens**. Tokens longer than 256 bytes are silently truncated by the analyzer. @@ -76,7 +76,7 @@ If any one of these is exceeded, the batch fails as a whole. The most common lim ### Dense-vector payload size -A high-dimensional dense field can silently turn a 50-doc batch into a 5–10 MB request, which the preview backend will reject wholesale. If every batch fails and the error message is opaque, the first thing to try is dropping the embedding dimension before debugging schema: +A high-dimensional dense field can silently turn a 50-doc batch into a 5–10 MB request, which the backend will reject wholesale. If every batch fails and the error message is opaque, the first thing to try is dropping the embedding dimension before debugging schema: - **Gemini**: pass `config=types.EmbedContentConfig(output_dimensionality=768)`. The model uses Matryoshka representations, so smaller dimensions are valid truncations of the native output. 768 is usually a 4× payload reduction vs. the native 3072 and costs very little quality. - **OpenAI `text-embedding-3-*`**: pass `dimensions=768` (or similar) to `embeddings.create`. @@ -86,6 +86,8 @@ A high-dimensional dense field can silently turn a 50-doc batch into a 5–10 MB After `batch_upsert` returns, **your documents are written but not yet searchable.** The server builds inverted indexes for FTS fields and ANN graphs for vector fields in the background. A search query issued immediately will return empty matches. Schemas with multiple indexed fields (e.g. text + dense + sparse) may take slightly longer. +This is distinct from — and in addition to — `pc.indexes.create(...)`'s own polling for the *index itself* to become ready (which it does by default before returning). Even a fully-ready index needs this separate poll after each ingest. + **Always poll with a deadline** before trusting the index: ```python @@ -111,7 +113,7 @@ Pick a sentinel query likely to hit at least one document. For a typical corpus, ## Chunking oversized text -Per the public-preview docs (above), the per-FTS-field hard limits are 100 KB and 10,000 tokens. In practice, plan for the *token* limit kicking in first on natural prose (~5,000 English words at ~2 tokens each is the rough ceiling). Probe before ingesting at scale — chunk anything that approaches either bound, with safety margin. +Per the size caps above, the per-FTS-field hard limits are 100 KB and 10,000 tokens. In practice, plan for the *token* limit kicking in first on natural prose (~5,000 English words at ~2 tokens each is the rough ceiling). Probe before ingesting at scale — chunk anything that approaches either bound, with safety margin. **Strategy: probe first, then chunk if needed.** @@ -158,40 +160,80 @@ Conventions: ## Updating documents -There is **no per-field update or merge** in `2026-01.alpha`. `documents.upsert` always replaces the entire document for a given `_id`. To update one field: +`2026-07` has real per-field updates via `documents.update(...)` — this replaces the fetch → modify → re-upsert workaround from the old preview API. Two shapes: + +**By `_id`, patching specific fields:** ```python -fetched = idx.documents.fetch( +idx.documents.update( namespace=NAMESPACE, - ids=["doc-42"], - include_fields=["*"], # need the full doc to round-trip it + documents=[{"_id": "doc-42", "set_fields": {"category": "biography"}}], ) -doc = fetched.documents["doc-42"].to_dict() -doc["category"] = "biography" # patch in client code +``` -idx.documents.upsert(namespace=NAMESPACE, documents=[doc]) +**By filter, patching every matching document at once:** + +```python +resp = idx.documents.update( + namespace=NAMESPACE, + filter={"category": {"$eq": "fiction"}}, + set_fields={"featured": True}, + # remove_fields=["stale_field"], # drop a field entirely, by name +) +print(resp.matched_records) # point-in-time count, same semantics as documents.delete ``` -If the document includes a dense vector, you re-upsert that vector verbatim. If it changes, embed the new content first. +`documents.update` accepts `documents=` (a list of per-record patches) *or* `filter=` + `set_fields=`/`remove_fields=` (a bulk patch by metadata match) — not both. `set_fields` merges into the existing document; fields you don't mention are left untouched. `remove_fields` deletes named fields outright. For a filtered update, `matched_records` on the response is a point-in-time count when the server accepted the request, the same caveat as `documents.delete`'s `matched_records` — the update itself applies asynchronously. + +If you need to fully replace a document (including its vector fields) rather than patch it, `documents.upsert` with the complete document under the same `_id` is still the right tool — `upsert` replaces, `update` patches. ## Deletes -`documents.delete` accepts either `ids: [...]` (1–1000 items) or `delete_all: true`. There is **no delete-by-filter** — to delete documents matching a metadata expression, search first to collect IDs, then pass them in: +`documents.delete` accepts exactly one of `ids: [...]` (1–1000 items), `filter: {...}`, or `delete_all: true`: ```python -ids_to_kill = [ - m._id for m in idx.documents.search( - namespace=NAMESPACE, top_k=1000, - score_by=[{"type": "text", "field": "body", "query": "deprecated"}], - filter={"category": {"$eq": "archive"}}, - include_fields=[], - ).matches -] -idx.documents.delete(namespace=NAMESPACE, ids=ids_to_kill) +# Delete-by-filter — no need to search for IDs first anymore. +resp = idx.documents.delete(namespace=NAMESPACE, filter={"category": {"$eq": "archive"}}) +print(resp.matched_records) ``` `delete_all=True` wipes the entire namespace. Use carefully. +## Namespace management + +Confirmed working against document-schema indexes in `2026-07` — this was **not** supported in the old preview API, which could write to a namespace but not list, describe, create, or delete one via the API. + +```python +# Create explicitly (namespaces otherwise auto-create on first upsert — see below). +ns = idx.create_namespace(name="movies-en") +print(ns.name, ns.record_count, ns.size_bytes) + +# List every namespace on the index in one call. +for page in idx.list_namespaces(): + for ns in page.namespaces: + print(ns.name, ns.record_count) + +# Describe one namespace. Prefer list_namespaces() over repeated describe_namespace() +# calls — describe_namespace is rate-limited per index; list_namespaces isn't. +ns = idx.describe_namespace(name="movies-en") + +# Delete a namespace and everything in it. +idx.delete_namespace(name="movies-en") +``` + +`create_namespace`'s optional `schema` parameter (`{"fields": {"": {"filterable": True}}}`) controls *which metadata fields get indexed for filtering in that namespace specifically* — omitting it means the namespace inherits the index's own metadata-indexing configuration, which for the managed indexes this skill covers is "index everything" by default (see "Filterable metadata isn't declared in the schema at all" in SKILL.md). There's rarely a reason to pass it explicitly unless you're deliberately restricting which fields are filterable in one namespace. + +`__default__` is reserved — it always exists and can't be created or deleted; every namespace-taking call already defaults to it when `namespace` is omitted, but it's worth knowing when you see it show up unbidden in a `list_namespaces()` result. + +### `describe_index_stats` — also now works + +```python +stats = idx.describe_index_stats() +print(stats.total_vector_count, stats.namespaces) # total record count, namespace count +``` + +`describe_index_stats(filter=...)` is documented as rejected on every index type (there's no operation that returns a filtered count) — call it with no arguments. + ## Integrating embedding providers If your index has a dense or sparse vector field, you need embeddings. Three common paths: @@ -256,8 +298,7 @@ This adapter also gives you a single chokepoint for retries, rate-limit backoff, ## Limits to be aware of -- **No bulk import (S3 import job)** for document-shaped indexes in `2026-01.alpha`. Load through `documents.upsert` / `documents.batch_upsert`. +- **No bulk import (S3 import job)** for document-shaped indexes in `2026-07`. Load through `documents.upsert` / `documents.batch_upsert`. - **No backup/restore.** If you need recoverability, snapshot your source data, not the index. -- **No CMEK projects** — indexes can't be created in CMEK-enabled projects. +- **No CMEK projects alongside any `full_text_search` field** — such indexes can't be created in CMEK-enabled projects. - **Indexing latency**: documents become searchable in ≲1 minute typically; multi-field schemas can take slightly longer. - diff --git a/skills/pinecone-full-text-search/references/onboarding-walkthrough.md b/skills/pinecone-full-text-search/references/onboarding-walkthrough.md index c45f047..0f1969c 100644 --- a/skills/pinecone-full-text-search/references/onboarding-walkthrough.md +++ b/skills/pinecone-full-text-search/references/onboarding-walkthrough.md @@ -48,7 +48,7 @@ If any text field exceeds **100 KB** (or roughly **10,000 tokens** ≈ ~5,000 En If types don't match what an FTS schema would want: - **Numbers stored as strings**: "Your `year` field is a string like `"2024"`. The schema needs a number. I'll coerce — but if any value can't be parsed, I'll abort and show you the offending row." - **Booleans as strings**: same. "Convert `"true"` / `"false"` to booleans?" -- **Comma-separated tags**: "`tags` is a string `'classic,american'`. The schema would index tags as a list. I'll split on `,` — speak up if your data uses a different separator." +- **Comma-separated tags**: "`tags` is a string `'classic,american'`. I'll split it into a list at ingest time — speak up if your data uses a different separator." (Same as `category`: the resulting list isn't declared in the schema either — it's just included on each document and auto-indexes for `$in`/`$nin` filtering — see Stage 3.) - **Dates / timestamps**: "Pinecone has no date type. We'll either store as ISO-8601 strings (filterable for exact match), or convert to epoch milliseconds (filterable as numeric). Which do you want?" ### Missing fields @@ -81,22 +81,19 @@ If any field starts with `_` or `$`: > # Searchable text — long prose, stemming on so "running" matches "ran" > .add_string_field("body", full_text_search={"language":"en", "stemming":True}) > # Searchable text — short titles, stemming off (proper nouns shouldn't over-match) -> .add_string_field("title", full_text_search={}) -> # Filter only — exact-match category like "fiction" -> .add_string_field("category", filterable=True) -> # Numeric range filter (e.g. year > 2024) -> .add_float_field("year", filterable=True) -> # Tag filter — list membership ($in) -> .add_string_list_field("tags", filterable=True) +> .add_string_field("title", full_text_search=True) > .build() > ``` > +> Notice `category`, `year`, and `tags` are ALL absent from this schema — on a managed index (this one), the schema can only hold fields that participate in *search*: FTS strings and vectors. Every filterable-metadata type, not just plain strings, gets rejected with a 400 if declared here. +> > A few notes: +> - **`category`, `year`, and `tags` all just get included on every upserted document instead** — Pinecone indexes each one for filtering automatically (exact-match for `category`, numeric range for `year`, list-membership `$in`/`$nin` for `tags`), with nothing to declare up front for any of them. > - **No dense_vector field** — you said you don't have embeddings yet. We can add one later, but it requires creating a *new* index because schemas are immutable. Want to add a placeholder now and keep the door open? -> - **`year` uses `add_float_field`** — `float` is the only numeric wire type; there is no integer helper. -> - **`tags`** will become `["a","b","c"]` after the comma-split we discussed. +> - **`year` upserts as a Python `float`** (e.g. `2024.0`) — there's no integer wire type, schema-declared or not. +> - **`tags`** will become `["a","b","c"]` after the comma-split we discussed — a plain Python list on the document, no schema field needed. > -> Schemas are immutable in `2026-01.alpha` — once we create this, changing it means re-creating the index and re-ingesting all the data. +> Schemas are immutable in `2026-07` — once we create this, changing it means re-creating the index and re-ingesting all the data. **ASK** (one question this time): "Look right? Want to adjust anything before I create the index?" @@ -106,10 +103,10 @@ If any field starts with `_` or `$`: Once approved: 1. Write the Python (`create.py` or inline) using the approved schema. -2. Run it. Poll until `pc.preview.indexes.describe(name).status.ready: True`. +2. Run it: `pc.indexes.create(name=..., schema=schema, deployment={"deployment_type": "managed", "cloud": ..., "region": ...})`. This polls internally and returns only once the index is ready — no separate wait loop needed. 3. Tell the user when it's ready: "Index `` created and ready. Now ingesting your data." -If creation fails for a reason you didn't anticipate (e.g. name conflict, region mismatch, CMEK restriction), tell the user the specific error and how to fix — don't auto-retry under a different name without asking. +If creation fails for a reason you didn't anticipate (e.g. name conflict, region mismatch, CMEK restriction, or a `400` on a filterable metadata field — string, string_list, float, or boolean — that snuck back into the schema), tell the user the specific error and how to fix — don't auto-retry under a different name without asking. ## Stage 5 — Process and ingest @@ -140,8 +137,9 @@ Tell them: 1. **The index name and schema** in one line. 2. **How to query** — give them a copy-pasteable `idx.documents.search(...)` snippet shaped to their schema (one `score_by` clause + the `include_fields` they care about). Refer them to the **Querying** section in SKILL.md or `references/querying.md` for variations. 3. **How to ingest more** — `scripts/ingest.py` with the same `--sentinel-field` they should use. -4. **How to delete** — `pc.preview.indexes.delete("")` when done. -5. **What's in the way of changing the schema** — recreate + re-ingest, no schema migration. +4. **How to update or delete records** — `idx.documents.update(...)` for per-field patches (by `_id` or by `filter`), `idx.documents.delete(...)` for removal (by `ids`, `filter`, or `delete_all`). See `references/ingestion.md`. +5. **How to delete the whole index** — `pc.indexes.delete("")` when done. +6. **What's in the way of changing the schema** — recreate + re-ingest, no schema migration. Optionally save a small `README.md` in the working directory with the same info, so they have it when they come back. @@ -149,8 +147,9 @@ Optionally save a small `README.md` in the working directory with the same info, - **Don't decide silently.** Every decision in Stage 2 should be surfaced. If you assume a separator, a coercion, a dedup policy, you'll be wrong sometimes and the user won't know to push back. - **Don't call `indexes.create()` without explicit approval** — schemas are immutable. +- **Don't declare any filterable metadata field in the schema** — string, string_list, float, or boolean alike. On a managed index all of them are rejected at create time. Include them on documents instead — see Stage 3. - **Don't write a giant pre-flight script** that does Stages 1-2 in code without ever showing the user. The point is the conversation, not the automation. -- **Don't skip Stage 6.** Polling says the index is "ready"; only a real query confirms the documents are there in the shape you expected. +- **Don't skip Stage 6.** `pc.indexes.create()` returning means the *index* is ready — it says nothing about whether your just-ingested *documents* are searchable yet. Only a real query confirms the documents are there in the shape you expected. - **Don't add a dense_vector field "just in case."** It commits the user to a specific embedding dimension forever — and if they don't have embeddings to ingest, the field is useless. - **Don't promise reversibility.** Whenever you say "we can change this later," follow up with: "...by creating a new index and re-ingesting. There's no schema migration." diff --git a/skills/pinecone-full-text-search/references/querying.md b/skills/pinecone-full-text-search/references/querying.md index 86da8b5..34aeab3 100644 --- a/skills/pinecone-full-text-search/references/querying.md +++ b/skills/pinecone-full-text-search/references/querying.md @@ -1,6 +1,6 @@ # Querying -All reads on a Pinecone preview document index go through `idx.documents.search(...)` (ranked) or `idx.documents.fetch(...)` (direct, ID-only). The interesting shape is the **single** `score_by` clause and the `filter={...}` predicate — everything else is plumbing. +All reads on a Pinecone document index go through `idx.documents.search(...)` (ranked), `idx.documents.fetch(...)` (direct, by ID or filter), or `idx.documents.list(...)` (enumerate IDs). The interesting shape is the **single** `score_by` clause and the `filter={...}` predicate — everything else is plumbing. ## The one-scoring-type rule @@ -12,7 +12,7 @@ A single `documents.search` request ranks by **one** scoring type. `score_by` ac - `sparse_vector` clauses must appear alone. - You **cannot** blend types — no `text` + `query_string`, no `text` + `dense_vector`, no cross-type mix. The server rejects it. -To compose lexical and dense / sparse signals, put the lexical signal in `filter` via the text-match operators (`$match_phrase` / `$match_all` / `$match_any`) and let the vector clause in `score_by` do the ranking. That's the supported hybrid pattern in `2026-01.alpha`. +To compose lexical and dense / sparse signals, put the lexical signal in `filter` via the text-match operators (`$match_phrase` / `$match_all` / `$match_any`) and let the vector clause in `score_by` do the ranking. That's the supported hybrid pattern in `2026-07`. ## `score_by` signal types @@ -45,7 +45,7 @@ resp = idx.documents.search( ) ``` -Supported operators (full table in the public-preview docs, summarized here): +Supported operators (full table in the public docs, summarized here): | Operator | Syntax | Example | |----------------|---------------------|-----------------------------------| @@ -100,7 +100,7 @@ resp = idx.documents.search( ) ``` -Stored and queried as `{"indices": [...], "values": [...]}`. Hosted sparse models (e.g. `pinecone-sparse-english-v0`) return embeddings with `.sparse_indices` and `.sparse_values` ready to drop in. Must appear alone in `score_by`. +Stored and queried as `{"indices": [...], "values": [...]}`. Hosted sparse models (e.g. `pinecone-sparse-english-v0`) return embeddings with `.sparse_indices` and `.sparse_values` ready to drop in. Must appear alone in `score_by`. The `sparse_vector` field this scores against must have been declared explicitly at schema-creation time — see `references/schema-design.md`. ## Multi-field BM25 @@ -125,7 +125,7 @@ score_by=[{ }] ``` -Both reward documents that match in multiple fields. **`2026-01.alpha` weights every contributing field equally** — there is no per-clause weight parameter. To approximate weighting, use Option B with `^N` term boosts inside the query string (`title:({q})^3 OR body:({q})`). +Both reward documents that match in multiple fields. **`2026-07` weights every contributing field equally** — there is no per-clause weight parameter. To approximate weighting, use Option B with `^N` term boosts inside the query string (`title:({q})^3 OR body:({q})`). ## Filtering @@ -151,12 +151,12 @@ These are the supported way to compose lexical pre-filtering with `dense_vector` > **Scoring-only operators don't go in `filter`.** Phrase slop (`"…"~N`), term boost (`^N`), and phrase prefix (`"… word"*`) influence ranking, so they're available in `query_string` `score_by` but not in `filter`. -### Metadata filters (on `filterable: true` fields) +### Metadata filters (on filterable fields) -Standard comparison and membership operators — work on `string`, `string_list`, `float`, and `boolean` filterable fields. +Standard comparison and membership operators. On a managed index, **none** of `string`, `string_list`, `float`, or `boolean` metadata fields are declared in the schema — any field present on an upserted document is auto-indexed for filtering regardless of type (see `references/schema-design.md` → "Filterable metadata — never schema-declared on managed indexes"). | Operator | Example | Semantics | -|----------|-------------------------------------------------------------|--------------------------------------| +|----------|---------------------------------------------------------------|--------------------------------------| | `$eq` | `{"category": {"$eq": "tech"}}` | Equals | | `$ne` | `{"category": {"$ne": "archive"}}` | Not equals | | `$gt` | `{"year": {"$gt": 2023}}` | Greater than | @@ -175,8 +175,8 @@ Multiple keys at the top level of a filter object are implicitly AND-ed. Use `$a filter={ "$and": [ {"body": {"$match_all": "federal reserve"}}, # text-match operator - {"category": {"$eq": "finance"}}, # metadata operator - {"year": {"$gte": 2024}}, + {"category": {"$eq": "finance"}}, # metadata operator (auto-indexed, not schema-declared) + {"year": {"$gte": 2024}}, # metadata operator (also auto-indexed, not schema-declared) {"$not": {"tags": {"$in": ["opinion"]}}}, ], } @@ -205,7 +205,7 @@ Read it top-down: only docs whose `body` contains the exact phrase `"beautifully When to use which text-match operator inside a hybrid query: | Use `$match_phrase` when… | Use `$match_all` when… | Use `$match_any` when… | -|---------------------------|----------------------------------------------|----------------------------------------| +|----------------------------|------------------------------------------------|------------------------------------------| | Adjacency matters (named events, idioms, multi-word concepts where order is the signal). | All tokens are required but order is not (geography + topic, e.g. `"illinois cardinal"`). | At least one token is enough (broader recall — useful as a soft filter). | ## `include_fields` modes @@ -213,13 +213,13 @@ When to use which text-match operator inside a hybrid query: `include_fields` controls what each match object carries back in the response. | Value | Behaviour | -|------------------------------|----------------------------------------------------------------| -| *(omitted, or `null`)* | Defaults to `[]` — `_id` and `_score` only. | +|-------------------------------|------------------------------------------------------------------| +| *(omitted, or `null`)* | `_id` and `_score` only, on most builds — but some backend builds `400` on omission. | | `[]` | `_id` and `_score` only (lightest payload). | | `["*"]` | All stored fields (including fields not declared in the schema).| | `["field1", "field2"]` | Only the listed fields (projection). | -**Always pass `include_fields` explicitly** on `documents.search`. Some SDK builds default to `[]`; some return `400` / `422` if it's missing. Being explicit avoids surprises and makes the call's intent obvious. +**Always pass `include_fields` explicitly** on `documents.search`. Being explicit avoids surprises and makes the call's intent obvious. User metadata fields literally named `score` are returned alongside the system-owned `_score` match score — the leading underscore prevents collisions. @@ -231,13 +231,14 @@ Match objects carry: - `_score` (float) — system match score; **higher is better**. - The fields requested via `include_fields`. -The `score` field name is reserved for **user metadata**; the system match score is always `_score`. Older SDK / backend builds may still emit unprefixed `score`; reading via `getattr(match, "_score", getattr(match, "score", None))` covers both. +The `score` field name is reserved for **user metadata**; the system match score is always `_score`. Read `m._score`, not `m.score`. -## `documents.fetch` — direct retrieval, ID-only +## `documents.fetch` — direct retrieval by ID or filter -Fetch is **ID-only** in `2026-01.alpha`. It does **not** accept a `filter`. To retrieve documents matching a metadata expression, search first to get IDs, then fetch: +`documents.fetch` accepts exactly one of `ids` or `filter`: ```python +# By ID — never paginated. fetched = idx.documents.fetch( namespace=NAMESPACE, ids=["doc-1", "doc-2", "does-not-exist"], @@ -249,23 +250,63 @@ for doc_id, doc in fetched.documents.items(): Missing IDs are silently omitted from the response (no error). `ids` accepts 1–1000 entries per call. -## `documents.delete` — by ID or `delete_all` +A `filter`-based fetch is **paginated** — up to 10,000 documents per page: + +```python +page = idx.documents.fetch(namespace=NAMESPACE, filter={"views": {"$gt": 100}}, include_fields=["*"]) +while True: + for doc_id, doc in page.documents.items(): + print(doc_id, doc.title) + if page.pagination is None: + break + page = idx.documents.fetch( + namespace=NAMESPACE, + filter={"views": {"$gt": 100}}, + include_fields=["*"], + pagination_token=page.pagination.next, + ) +``` + +An ID-based fetch never sets `response.pagination` (always `None`), so existing single-page `fetch(ids=...)` call sites need no loop. + +## `documents.delete` — by ID, filter, or `delete_all` ```python # By IDs (1–1000 per call). Non-existent IDs are silently ignored. -idx.documents.delete( - namespace=NAMESPACE, - ids=["doc-1", "doc-2"], -) +resp = idx.documents.delete(namespace=NAMESPACE, ids=["doc-1", "doc-2"]) + +# By filter — deletes every document matching the metadata expression. +resp = idx.documents.delete(namespace=NAMESPACE, filter={"views": {"$lt": 5}}) +print(resp.matched_records) # point-in-time count when the server accepted the request # Wipe the entire namespace. -idx.documents.delete( - namespace=NAMESPACE, - delete_all=True, -) +idx.documents.delete(namespace=NAMESPACE, delete_all=True) +``` + +Exactly one of `ids`, `filter`, or `delete_all` must be given. `documents.delete` returns a `DeleteDocumentsResponse` (it used to return `None`). `matched_records` is populated only for a filtered delete — it's a point-in-time count when the server accepted the request, not a promise about how many documents ultimately disappear (deletes apply asynchronously). It's `None` for an ID-list or `delete_all` delete. Deletes are permanent within the namespace. + +## `documents.list` — enumerate document IDs + +New in the graduated API — no equivalent existed under `pinecone.preview`. Lazily paginated, sorted by `_id`, and returns IDs only (no other fields): + +```python +# Iterate every document ID in a namespace. +for doc in idx.documents.list(namespace=NAMESPACE): + print(doc.id) + +# Restrict to IDs starting with a prefix — handy for the chunk-ID convention +# in references/ingestion.md ("doc-42", "doc-42#p2", "doc-42#p3", ...). +for doc in idx.documents.list(namespace=NAMESPACE, prefix="doc-42"): + print(doc.id) + +# Page manually instead of letting the iterator follow every page. +for page in idx.documents.list(namespace=NAMESPACE, limit=20).pages(): + print(len(page.items), "ids, next token:", page.pagination_token) ``` -Delete does **not** accept a `filter`. To delete documents matching a metadata expression, search first to collect IDs, then pass them to `delete`. Deletes are permanent within the namespace. +`namespace` is required. `limit` (1–100) tunes page size only — the default iterator form above still walks every page; use `.pages()` (or `itertools.islice`) if you want to stop early. `prefix` is ASCII-only, ≤512 characters. There's no `filter` — for anything beyond an ID prefix, use `documents.search` or `documents.fetch(filter=...)` instead, both of which return actual field data. + +Useful for confirming what's in a namespace before a `delete_all`, or auditing chunk coverage for a parent document by prefix. ## Worked cross-modal example — "pick your signal" pattern diff --git a/skills/pinecone-full-text-search/references/schema-design.md b/skills/pinecone-full-text-search/references/schema-design.md index 618054c..8012df0 100644 --- a/skills/pinecone-full-text-search/references/schema-design.md +++ b/skills/pinecone-full-text-search/references/schema-design.md @@ -1,39 +1,41 @@ # Schema design -Everything a Pinecone preview document index needs is declared up-front via `SchemaBuilder`. The schema pins which fields are searchable, which are filterable metadata, which hold vectors, and what their dimensions / metrics are. **Schemas are fixed at index creation in `2026-01.alpha`** — adding, removing, or retyping fields afterwards is not supported. Plan carefully. +Everything a Pinecone document index needs is declared up-front via `SchemaBuilder`. The schema pins which fields are searchable — full-text or vector — and what their dimensions / metrics are. **Filterable-only metadata is never declared in the schema on a managed index, no matter its type** (see "Filterable metadata — never schema-declared on managed indexes" below). **Schemas are fixed at index creation in `2026-07`** — adding, removing, or retyping fields afterwards is not supported. Plan carefully. ## `SchemaBuilder` overview ```python -from pinecone.preview import SchemaBuilder +from pinecone import SchemaBuilder schema = ( SchemaBuilder() .add_string_field("title", full_text_search={"language": "en"}) .add_string_field("body", full_text_search={"language": "en", "stemming": True}) - .add_string_field("category", filterable=True) - .add_float_field("year", filterable=True) # `float` is the only numeric wire type .add_dense_vector_field("embedding", dimension=1024, metric="cosine") - .add_sparse_vector_field("sparse_embedding", metric="dotproduct") + .add_sparse_vector_field("sparse_embedding") # no `metric` — sparse scoring isn't configurable .build() # terminal: returns the schema object you pass to indexes.create ) ``` -`.build()` is the terminal call — every chain ends with it. The resulting schema is passed to `pc.preview.indexes.create(name=..., schema=schema, read_capacity=...)`. `read_capacity` defaults to `{"mode": "OnDemand"}` (auto-scaled shared reads); pass `{"mode": "Dedicated", "dedicated": {...}}` only if you specifically want provisioned read nodes. +Notice there's no `category`, `year`, or `tags` field here even though the corpus this schema serves might have all three — on a managed index the server rejects **any** schema-declared filterable-only field, string or otherwise, with `400`. See "Filterable metadata — never schema-declared on managed indexes" below. + +`.build()` is the terminal call — every chain ends with it. The resulting schema is passed to `pc.indexes.create(name=..., schema=schema, deployment=..., read_capacity=...)`. `deployment` defaults to a managed index on AWS `us-east-1` when omitted; pass `{"deployment_type": "managed", "cloud": ..., "region": ...}` to pick a different region. `read_capacity` defaults to `{"mode": "OnDemand"}` (auto-scaled shared reads); pass `{"mode": "Dedicated", "dedicated": {...}}` only if you specifically want provisioned read nodes. `pc.indexes.create(...)` polls until the index is ready by default — pass `timeout=-1` to return immediately instead, or a positive number of seconds for a bounded wait (raises `PineconeTimeoutError` past the deadline). ## Field types at a glance -| Type | Purpose | Required options | How it's queried | -|-----------------|--------------------------------------------|-------------------------------------------|-------------------------------------------| -| `string` (text) | Full-text search (BM25 / Lucene) | `full_text_search: {...}` (dict, may be `{}`) | `score_by` `text` or `query_string`; filter via `$match_phrase` / `$match_all` / `$match_any` | -| `string` (metadata) | Exact-match metadata filtering | `filterable: true` | `filter` with `$eq` / `$in` / `$ne` / `$nin` / `$exists` | -| `string_list` | Array-valued metadata filtering | `filterable: true` | `filter` with `$in` / `$nin` (membership) | -| `float` | Numeric metadata filtering | `filterable: true` | `filter` with `$eq` / `$gt` / `$gte` / `$lt` / `$lte` / `$in` / `$nin` | -| `boolean` | Boolean metadata filtering | `filterable: true` | `filter` with `$eq` / `$exists` | -| `dense_vector` | ANN similarity search | `dimension`, `metric` (`cosine` / `dotproduct` / `euclidean`) | `score_by` `dense_vector` | -| `sparse_vector` | Sparse-vector lexical / hybrid scoring | `metric` (typically `dotproduct`) | `score_by` `sparse_vector` | +| Type | Purpose | Declared in schema (managed index)? | Required options | How it's queried | +|-----------------|---------------------------------------------|---------------------|--------------------------------------------|--------------------------------------------| +| `string` (FTS) | Full-text search (BM25 / Lucene) | Yes | `full_text_search: {...}` (or `True` for defaults) | `score_by` `text` or `query_string`; filter via `$match_phrase` / `$match_all` / `$match_any` | +| `string` (metadata) | Exact-match metadata filtering | **No** | — just include the field on upserted documents | `filter` with `$eq` / `$in` / `$ne` / `$nin` / `$exists` | +| `string_list` | Array-valued metadata filtering | **No** | — just include the field on upserted documents | `filter` with `$in` / `$nin` (membership) | +| `float` | Numeric metadata filtering | **No** | — just include the field on upserted documents | `filter` with `$eq` / `$gt` / `$gte` / `$lt` / `$lte` / `$in` / `$nin` | +| `boolean` | Boolean metadata filtering | **No** | — just include the field on upserted documents | `filter` with `$eq` / `$exists` | +| `dense_vector` | ANN similarity search | Yes | `dimension`, `metric` (`cosine` / `dotproduct` / `euclidean`) | `score_by` `dense_vector` | +| `sparse_vector` | Sparse-vector lexical / hybrid scoring | Yes | none required (no `metric`, no `dimension`) | `score_by` `sparse_vector` | + +The `add_float_field`, `add_boolean_field`, and `add_string_list_field` `SchemaBuilder` methods still exist and still work for a **pod** deployment (not covered by this skill); on a managed deployment, don't call any of them — declaring the field in the schema is exactly what gets it rejected. -Every field can also include an optional `description` string — surfaced by `DescribeIndex` and useful for agentic workflows where an LLM inspects the schema to decide how to query. +Every schema-declared field can also include an optional `description` string — surfaced by `DescribeIndex` and useful for agentic workflows where an LLM inspects the schema to decide how to query. ## Reserved field names @@ -45,9 +47,31 @@ Field names must be unique, non-empty strings. Two hard rules: `_id` is required on every document. `_score` is the system match-score field name returned by `documents.search`. A user metadata field literally named `score` is allowed and won't collide with `_score`. -## String fields — text vs. metadata +## Filterable metadata — never schema-declared on managed indexes + +This is the single biggest behavioral change from the earlier `pinecone.preview` API, where filterable-only fields of any type were a normal schema declaration. + +On a **managed or BYOC index** (the deployment type this skill always uses), the schema may declare **only** search-participating fields: FTS `string` fields, `dense_vector`, and `sparse_vector`. Every filterable-metadata shape — a plain `string` with no `full_text_search`, `string_list`, `float`, `boolean` — is rejected at create time with `400` if it's declared in the schema, confirmed against the live API: + +> *"The schema only accepts fields used for search (field types `dense_vector`, `sparse_vector`, and `string` with `full_text_search` configuration). To use field '<name>' for filtering (field types `boolean`, `float`, `string`, or `string_list`), omit it from the schema and include it in documents. It will be indexed automatically."* + +(All four of those schema-declared filterable shapes — including `add_float_field`/`add_boolean_field`/`add_string_list_field` — are legal only on **pod** deployments, which this skill doesn't cover.) + +There's nothing to configure for the managed case: any field present on an upserted document — of any type, whether or not it's in the schema — is automatically indexed for filtering. + +```python +# WRONG on a managed index — the server 400s on ALL THREE of these: +.add_string_field("category", filterable=True) +.add_float_field("year", filterable=True) +.add_string_list_field("tags", filterable=True) + +# RIGHT — omit them from the schema entirely, and just include them on upserted docs: +idx.documents.upsert(namespace=NS, documents=[{ + "_id": "doc-1", "category": "fiction", "year": 2024.0, "tags": ["classic"], ... +}]) +``` -A single `string` field is *either* full-text-search (BM25 / Lucene scoring + text-match filters) **or** filterable metadata (exact-match), never both. If you need both surfaces for the same logical content, duplicate it into two differently-configured fields. +Confirmed live, end to end: `{"year": {"$gte": 2020}}`, `{"featured": {"$eq": True}}`, and `{"tags": {"$in": ["classic"]}}` all filter correctly against documents that never had those fields declared in any schema. ### Full-text-searchable string @@ -55,13 +79,16 @@ A single `string` field is *either* full-text-search (BM25 / Lucene scoring + te .add_string_field("body", full_text_search={"language": "en", "stemming": True}) ``` -`full_text_search` takes a dict. Pass `{}` for all server defaults; populate it with any of: +`full_text_search` takes `True` (server defaults) or a dict; populate the dict with any of: - `language` (string, default `"en"`) — selects the analyzer (tokenizer + stemmer + stopword set). Supported short codes: `ar`, `da`, `de`, `el`, `en`, `es`, `fi`, `fr`, `hu`, `it`, `nl`, `no`, `pt`, `ro`, `ru`, `sv`, `ta`, `tr`. Full names are also accepted (e.g. `"english"`, `"french"`, `"arabic"`). Stop-word lists are available for most languages but a few are tokenize/stem only (no stop_word filtering even when `stop_words: true` is set) — `ar`, `da`, `de` are notable cases; `en`, `es`, `fr` etc. have full stop-word support. -- `stemming` (boolean, default `false`) — if `true`, applies the language's stemmer so `running` matches `runs`. -- `stop_words` (boolean, default `false`) — if `true`, the analyzer's stopword set is filtered out at index and query time. -- `lowercase` (boolean, default `true`, server-applied) — case-insensitive matching. -- `max_token_length` (int, default `40`, server-applied) — discards excessively long tokens. +- `stemming` (boolean, default `false`) — if `true`, applies the language's stemmer so `running` matches `runs`. Required when `stop_words=True` is also set. +- `stop_words` (boolean, default `false`) — if `true`, the analyzer's stopword set is filtered out at index and query time. Requires `stemming=True`. +- `ngram` (dict, e.g. `{"min_gram": 2, "max_gram": 4}`) — character n-gram tokenization, for substring/autocomplete-style matching. Cannot be combined with `stemming` or `stop_words`. +- `lowercase` (boolean, default `true`, server-applied) — case-insensitive matching, not configurable via the SDK. +- `max_term_len` (server-applied) — discards excessively long tokens, not configurable via the SDK. + +`SchemaBuilder.add_string_field` also accepts `language`, `stemming`, and `stop_words` as direct keyword arguments instead of nesting them in a `full_text_search=` dict — e.g. `add_string_field("title", full_text_search=True, language="en", stemming=True)`. When both forms are given for the same key, the keyword argument wins. Heuristic on stemming: turn it on for long prose fields where morphological variants of a root should match (`running` ~ `runs` ~ `ran`); leave off for short / identifier fields like titles, tags, or proper nouns where stemming would over-match (a book titled `Running` probably shouldn't also match the query `ran`). Typical pattern: stemming on for `body`, off for `title` / proper-noun fields. @@ -70,32 +97,17 @@ Enables, on `field_name`: - Lucene scoring with `score_by=[{"type": "query_string", "query": "field_name:(a AND (b OR c)) NOT field_name:d"}]`. - Phrase / token filters: `filter={"field_name": {"$match_phrase": "..."}}`, `{"$match_all": "..."}`, `{"$match_any": "..."}`. -### Filterable-only string - -```python -.add_string_field("category", filterable=True) -``` - -Stored verbatim, not tokenized, not text-scored. Enables exact-match filtering: `{"category": {"$eq": "fiction"}}`, `{"category": {"$in": ["fiction", "biography"]}}`, `{"category": {"$exists": true}}`. +## Filter operators by metadata type -## Numeric, boolean, and array metadata +None of these types are schema-declared on a managed index (see above) — they're just whatever value type you put on the field when you upsert: -```python -.add_float_field("year", filterable=True) # `float` is the only numeric wire type -.add_boolean_field("featured", filterable=True) -.add_string_list_field("tags", filterable=True) -``` - -- **`float`** is the only numeric wire type — there is no separate integer type, and **there is no `add_integer_field` helper**. Use `add_float_field`, which emits `{"type": "float", "filterable": ...}`. Note `describe()` may still report the field class as `PreviewIntegerField`. Supports `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`, `$exists`. -- **`boolean`** uses `add_boolean_field("name", filterable=True)`. Supports `$eq` and `$exists`. -- **`string_list`** supports `$in` / `$nin` membership semantics — handy for tag-style metadata. +- **Plain string** (e.g. `category`): `$eq`, `$ne`, `$in`, `$nin`, `$exists`. +- **Numeric** (e.g. `year`): there is no separate integer wire type and no `add_integer_field` helper even on a pod deployment — always upsert as a Python `float` (`2024.0`, not `2024`). Supports `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`, `$exists`. +- **Boolean** (e.g. `featured: True`): `$eq`, `$exists`. +- **List of strings** (e.g. `tags: ["classic", "american"]`): `$in`, `$nin`, `$exists` — membership semantics, handy for tag-style metadata. All filter operators compose under `$and`, `$or`, `$not`. Multiple keys at the top level of `filter` are combined with implicit AND. -> **SchemaBuilder numeric-type pitfall**: there is no `add_integer_field()` helper — calling it raises `AttributeError`. Use `add_float_field()` for any numeric metadata; `float` is the only numeric wire type. `describe()` responses may still name the class `PreviewIntegerField`, but the wire/server type is `"float"`. - -> **Forward-looking note.** In the public preview, metadata fields you send at upsert time are auto-indexed for filtering even if not declared in the schema. In a future release, only schema-declared fields with `filterable: true` will be indexed. Declare your metadata fields in the schema today to be future-proof. - > **Metadata size limit.** Filterable metadata on a single document is capped at **40 KB** combined (everything that's not in an FTS-enabled `string` field). FTS-enabled `string` fields don't count toward this — they have their own per-field limit (100 KB / 10,000 tokens, see `references/ingestion.md`). ## Dense vector fields @@ -104,23 +116,24 @@ All filter operators compose under `$and`, `$or`, `$not`. Multiple keys at the t .add_dense_vector_field("embedding", dimension=1024, metric="cosine") ``` +- `dimension` and `metric` are both **required** on `add_dense_vector_field` (no longer `None`-defaulted). - `dimension` must match whatever embedding model you'll store. If the model is chosen at runtime, query its default dimension first (e.g. `pc.inference.get_model(model="multilingual-e5-large").default_dimension`) and pass that in. - `metric` is one of `"cosine"`, `"dotproduct"`, `"euclidean"`. Pick the metric the embedding provider recommends — most text embedders use cosine. - Scored at query time with `score_by=[{"type": "dense_vector", "field": "embedding", "values": [...]}]`. -**At most one `dense_vector` field per index** in `2026-01.alpha`. If you need two semantically distinct dense signals, you need two indexes. +**At most one `dense_vector` field per index** in `2026-07`. If you need two semantically distinct dense signals, you need two indexes. ## Sparse vector fields ```python -.add_sparse_vector_field("sparse_embedding", metric="dotproduct") +.add_sparse_vector_field("sparse_embedding") ``` -- No `dimension` — sparse vectors are variable-length. -- `metric="dotproduct"` is the standard choice for learned sparse embeddings (e.g. `pinecone-sparse-english-v0`). +- No `dimension`, no `metric` — sparse vectors are variable-length and sparse scoring isn't configurable. Passing either raises `PineconeValueError` at schema-build time (the earlier preview API accepted and silently discarded `metric="dotproduct"` here; `2026-07` refuses it instead). - Stored and queried as `{"indices": [...], "values": [...]}`; query side: `score_by=[{"type": "sparse_vector", "field": "sparse_embedding", "sparse_values": {"indices": [...], "values": [...]}}]`. +- **A hybrid index must declare its `sparse_vector` field at create time.** In the old preview API, `metric="dotproduct"` on the dense field alone was enough to accept sparse writes. That's no longer true: the create call succeeds either way, but without a declared `sparse_vector` field, sparse *writes* are refused later — often surfacing far from the original create call. There's no way to add the field afterward; an index created without one has to be recreated. -**At most one `sparse_vector` field per index** in `2026-01.alpha`. +**At most one `sparse_vector` field per index** in `2026-07`. ## When to add a dense field at all @@ -138,7 +151,7 @@ Anti-pattern: **re-encoding text that already lives in an FTS field on the same When a document has a natural hierarchy (title → intro → body, or summary → transcript, or headline → lede → article), splitting across FTS fields enables two things you can't get from one blob: 1. **Per-field scoring.** A match on `title` is almost always a stronger signal than a match on `body`. With separate fields you can search just the title, just the body, or blend them at query time by listing each as its own `score_by` entry (see `references/querying.md` — multi-field BM25). -2. **Multi-field blended relevance.** Passing `score_by=[{text, title, q}, {text, intro, q}, {text, body, q}]` rewards documents that match in multiple fields. (`2026-01.alpha` weights every contributing field equally — no per-clause weight parameter.) +2. **Multi-field blended relevance.** Passing `score_by=[{text, title, q}, {text, intro, q}, {text, body, q}]` rewards documents that match in multiple fields. (`2026-07` weights every contributing field equally — no per-clause weight parameter.) Keep it a single field when: @@ -148,17 +161,18 @@ Keep it a single field when: ## Schemas are fixed at creation -`2026-01.alpha` does **not** support schema migration. You cannot: +`2026-07` does **not** support schema migration. You cannot: - Add a new field after creation. - Remove an existing field. - Change a field's type or sub-config (e.g. flip a filterable string to FTS, toggle stemming, change dense vector dimension). +- Add a `sparse_vector` field to an index that didn't declare one at creation. -The supported workaround is to create a new index with the desired schema and reindex documents (the document set is small enough at preview-launch scale that this is usually painless). Existing pre-public-preview indexes from earlier API versions cannot be backfilled with a 2026-01.alpha schema. +The supported workaround is to create a new index with the desired schema and reindex documents. Indexes from earlier API versions (including pre-graduation `pinecone.preview` indexes) cannot be backfilled with a `2026-07` schema — treat porting an old preview index as "design a new schema, create a new index, reingest," not an in-place upgrade. ## `description` for agentic / LLM-driven workflows -Each field accepts an optional `description` string: +Each schema-declared field accepts an optional `description` string: ```python .add_string_field( diff --git a/skills/pinecone-full-text-search/scripts/__pycache__/ingest.cpython-314.pyc b/skills/pinecone-full-text-search/scripts/__pycache__/ingest.cpython-314.pyc deleted file mode 100644 index b73fde2..0000000 Binary files a/skills/pinecone-full-text-search/scripts/__pycache__/ingest.cpython-314.pyc and /dev/null differ diff --git a/skills/pinecone-full-text-search/scripts/ingest.py b/skills/pinecone-full-text-search/scripts/ingest.py index 210c3a1..90cbe1a 100644 --- a/skills/pinecone-full-text-search/scripts/ingest.py +++ b/skills/pinecone-full-text-search/scripts/ingest.py @@ -3,7 +3,7 @@ # requires-python = ">=3.10" # dependencies = [ # "typer>=0.12", -# "pinecone==9.1.0", +# "pinecone==10.0.0", # ] # /// """Ingest a JSONL file into a Pinecone FTS index — safely. @@ -157,15 +157,13 @@ def poll_until_searchable( def resolve_index_with_retry(pc, name: str, *, deadline_s: int = 60): - """Resolve `pc.preview.index(name=...)`, retrying briefly during data-plane warmup. - - """ + """Resolve `pc.index(name=...)`, retrying briefly during data-plane warmup.""" deadline = time.time() + deadline_s delay = 2.0 last_exc = None while time.time() < deadline: try: - return pc.preview.index(name=name) + return pc.index(name=name) except Exception as exc: last_exc = exc time.sleep(delay) @@ -211,9 +209,9 @@ def main( help="Index namespace.", ), batch_size: int = typer.Option( - 100, "--batch-size", "-b", min=1, max=200, + 50, "--batch-size", "-b", min=1, max=200, help="Documents per batch_upsert call. Reduce if your dense vectors are large " - "(e.g. 50 for dim=3072) and you hit payload-size errors.", + "(e.g. 25 for dim=3072) and you hit payload-size errors.", ), poll_deadline: int = typer.Option( 300, "--poll-deadline", min=10, max=3600, diff --git a/skills/pinecone-help/SKILL.md b/skills/pinecone-help/SKILL.md index 280eaf4..ff64f91 100644 --- a/skills/pinecone-help/SKILL.md +++ b/skills/pinecone-help/SKILL.md @@ -43,7 +43,7 @@ Invoke any skill from Cursor Agent chat with `/pinecone-` — for ex | `pinecone-cli` | Use the Pinecone CLI (`pc`) for terminal-based index and vector management | | `pinecone-assistant` | Create, manage, and chat with Pinecone Assistants for document Q&A with citations | | `pinecone-mcp` | Reference for all Pinecone MCP server tools and their parameters | -| `pinecone-full-text-search` | Build a full-text-search index — schema design, safe bulk ingestion, and query construction (`text` / `query_string` / dense / sparse scoring with text-match and metadata filters). **Preview API (`2026-01.alpha`); requires `pinecone` Python SDK ≥ 9.0.** | +| `pinecone-full-text-search` | Build a full-text-search index — schema design, safe bulk ingestion, and query construction (`text` / `query_string` / dense / sparse scoring with text-match and metadata filters). **Document-schema API (`2026-07`); requires `pinecone` Python SDK ≥ 10.0.0.** | | `pinecone-docs` | Curated links to official Pinecone documentation, organized by topic | | `pinecone-n8n` | Build n8n workflows with the Pinecone Assistant node or Pinecone Vector Store node, including best practices and full workflow JSON generation | @@ -59,7 +59,7 @@ Invoke any skill from Cursor Agent chat with `/pinecone-` — for ex **Working with documents and Q&A?** → `pinecone-assistant` -**Building a full-text search index (BM25-style keyword/phrase matching, optionally combined with dense or sparse vectors)?** → `pinecone-full-text-search` (preview API, needs `pinecone` Python SDK ≥ 9.0) +**Building a full-text search index (BM25-style keyword/phrase matching, optionally combined with dense or sparse vectors)?** → `pinecone-full-text-search` (document-schema API, needs `pinecone` Python SDK ≥ 10.0.0) **Building an n8n workflow with Pinecone (RAG pipeline, chat with docs)?** → `pinecone-n8n` diff --git a/skills/pinecone-quickstart/scripts/__pycache__/quickstart_complete.cpython-314.pyc b/skills/pinecone-quickstart/scripts/__pycache__/quickstart_complete.cpython-314.pyc deleted file mode 100644 index c6d4cf0..0000000 Binary files a/skills/pinecone-quickstart/scripts/__pycache__/quickstart_complete.cpython-314.pyc and /dev/null differ diff --git a/skills/pinecone-quickstart/scripts/__pycache__/upsert.cpython-314.pyc b/skills/pinecone-quickstart/scripts/__pycache__/upsert.cpython-314.pyc deleted file mode 100644 index 3a05d5e..0000000 Binary files a/skills/pinecone-quickstart/scripts/__pycache__/upsert.cpython-314.pyc and /dev/null differ