Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 56 additions & 32 deletions skills/autorag-setup/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
---
name: autorag-setup
description: Configure AutoRAG for first use or repair its single-agent model, approved document roots, retrieval indexes, datasource skills, and health checks without exposing credentials.
description: Install and configure AutoRAG, or repair its single search model, approved document roots, retrieval indexes, datasource skills, and health checks without exposing credentials. Use when autorag is missing, init/refresh/health fails, indexes are stale, or the user wants to add folders or datasources.
license: MIT
---

# AutoRAG setup

Use this skill when AutoRAG is unconfigured, model resolution fails, indexes are
missing or stale, or the user wants to change the document collection.
Use this skill when AutoRAG is unconfigured, the `autorag` CLI is missing, model
resolution fails, indexes are missing or stale, or the user wants to change the
document collection or datasources.

## Safety

Expand All @@ -19,15 +21,26 @@ missing or stale, or the user wants to change the document collection.
`node_modules`, `.git`, `dist`, `build`, `target`, `.cache`, `.autorag`, or
`.jikji`.

## Install the CLI if needed

The CLI is `@autorag/librarian` (`autorag`). Runtime is Node.js ≥ 24 or Bun.

```bash
command -v autorag >/dev/null || bun install -g @autorag/librarian
autorag --help
```

If Bun is unavailable, `npm install -g @autorag/librarian` is acceptable.

## Inspect existing configuration

Check `~/.autorag/config.json`, an explicit `--config` path, or
`AUTORAG_CONFIG`. Relevant fields are:
Check `--config`, `AUTORAG_CONFIG`, `$AUTORAG_HOME/config.json`, or
`~/.autorag/config.json`. Relevant fields are:

- `searchPaths`, `workspacePath`, and `memoryPath`
- `model.provider`, `model.id`, `model.api`, `model.baseUrl`, `model.apiKeyEnv`
- `bm25`, `minSync`, and `jikji`
- `datasources` and `datasourceAccess`
- `datasources`, `datasourceAccess`, and `ui`

Preserve explicit user choices and a working config unless the user asks to
replace them or health checks fail.
Expand Down Expand Up @@ -73,9 +86,10 @@ approval before indexing:
| Linux | XDG Documents/Downloads/Desktop or their `~/` defaults | `~/Notes`, `~/Sync`, Nextcloud/Syncthing roots |
| Windows | Documents, Downloads, Desktop shell folders | OneDrive document roots, user-named project docs |

Supported index formats include `md`, `markdown`, `txt`, `text`, `pdf`, `docx`,
`pptx`, `xlsx`, `hwpx`, and `eml`; OCR image formats are optional. Do not
present legacy `doc`, `xls`, or `hwp` as fully supported parsed formats.
Supported parsed formats are `md`, `markdown`, `txt`, `text`, `pdf`, `docx`,
`pptx`, `xlsx`, `hwp`, `hwpx`, and `eml`. OCR for `jpg`, `jpeg`, `png`, `bmp`,
and `tiff` is optional (`parserOptions.ocr.enabled`). Do not present legacy
`.doc` or `.xls` as supported parsed formats; `.xls` is rejected.

Keep the first-run set small, usually one to three roots. Present a concrete
proposal and require `yes`, a narrowed keep-list, a custom list, or `skip`
Expand Down Expand Up @@ -114,9 +128,11 @@ Use `--force` only when intentionally replacing an existing config. Legacy cwd
### Retrieval defaults

BM25, MinSync, and Jikji are enabled by default. Leave them enabled unless the
user explicitly asks otherwise. MinSync auto-install is off by default and its
binary must already be installed or on `PATH`; Jikji can install `jikji-cli`
through cargo when enabled and allowed.
user explicitly asks otherwise. MinSync auto-installs a verified GitHub release
into `<workspace>/.autorag/bin` on first use (`minSync.autoInstall` defaults to
true). Set `"autoInstall": false` only when managing the binary yourself. Jikji
auto-installs `jikji-cli` through cargo when enabled (`jikji.autoInstall`
defaults to true; requires the Rust toolchain).

Exact duplicate exclusion is enabled by default. AutoRAG invokes the external
`dupey` CLI before parsed-mirror indexing, keeps the newest filesystem copy for
Expand All @@ -140,10 +156,18 @@ size must be positive integers.

## Configure datasource skills when requested

Datasource skills belong in trusted config and remain default-deny. Common
entries include WhatsApp, Telegram, Slack, Discord, Notion, GitHub, Google
Drive, a generic cloud-drive/rclone source, Gmail, local mail exports,
Mailcrawl, Obsidian, RSS/news, and Spotlight.
Prefer `autorag ui --no-open` to connect datasources. It writes the same trusted
`datasources` / `datasourceAccess` fields as a hand-edited config, stores
env-var names rather than secrets, and prints a loopback URL (`127.0.0.1`).
Do not bind non-loopback hosts unless the user explicitly set `ui.allowRemote`.

Datasource skills belong in trusted config and remain default-deny. Builtin
template names are `kakao`, `whatsapp`, `telegram`, `slack`, `discord`,
`clawgallery`, `notion`, `github`, `cloud-drive`, `gmail`, `mail-export`,
`mailcrawl`, `obsidian`, `rss`, and `spotlight`. Config keys may be connection
aliases with `"type": "<template>"`. Unknown names are skipped with an
`unknown-datasource-skill` warning; they do not fail config resolution.
`scope` and tags can narrow trusted access but cannot grant it.

```jsonc
{
Expand All @@ -164,9 +188,7 @@ Mailcrawl, Obsidian, RSS/news, and Spotlight.
```

Tokens are environment-variable names, not raw secrets. CLI-backed connectors
keep authentication in their external tool configuration. Unknown skill names
fail config resolution. `scope` and tags can narrow trusted access but cannot
grant it.
keep authentication in their external tool configuration.

Mailcrawl must be installed separately (`@nomadamas/mailcrawl@0.1.4` or newer)
and configured through its own Himalaya account. AutoRAG runs its local `sync`
Expand All @@ -179,10 +201,10 @@ fails with `text array must be non-empty`.
Configuration alone is not a successful setup:

```bash
autorag status
autorag health
autorag refresh
autorag search "summarize the collection" --top-k 3 --json
autorag status --json
autorag health --json
autorag refresh --json
autorag search "summarize the collection" --top-k 3 --json --debug
```

- `status` is model-free and path-opaque.
Expand All @@ -191,21 +213,23 @@ autorag search "summarize the collection" --top-k 3 --json
- `health --skip-probes` is only for intentionally offline validation and does
not prove live provider access.
- `refresh` syncs parsed mirrors, BM25, MinSync, Jikji, and authorized
datasources. `--method <csv>` may deliberately narrow it.
datasources. `--method <csv>` may deliberately narrow it
(`parsed,bm25,minsync,datasources,jikji,all`).
- Use `refresh --force` for a full resync only when incremental refresh is not
enough. Keep destructive reset/rebuild operations scoped to workspace
`.autorag` indexes, never source documents.

## Keep indexes fresh

For continuous freshness, create or verify an OS-appropriate scheduled
`autorag watch --once` job, typically every 30 minutes. Prefer cron or launchd
on macOS, cron or a user systemd timer on Linux, and Task Scheduler on Windows.
Use the same config as search, avoid overlapping runs, and keep logs outside
source trees.
`autorag watch --once` job, typically every 15–30 minutes. Prefer cron or
launchd on macOS, cron or a user systemd timer on Linux, and Task Scheduler on
Windows. Use the same config as search, avoid overlapping runs, and keep logs
outside source trees.

## Environment overrides

- `AUTORAG_HOME`
- `AUTORAG_CONFIG`
- `AUTORAG_SEARCH_PATHS`
- `AUTORAG_WORKSPACE`
Expand All @@ -215,7 +239,7 @@ source trees.

## Completion condition

Setup is complete only when roots are approved, a non-secret single-model
config is written, `status` is acceptable, live `health` passes, `refresh`
builds the requested indexes, one real structured search succeeds, and any
requested ongoing schedule is installed or verified.
Setup is complete only when the CLI is installed, roots are approved, a
non-secret single-model config is written, `status` is acceptable, live `health`
passes, `refresh` builds the requested indexes, one real structured search
succeeds, and any requested ongoing schedule is installed or verified.
88 changes: 48 additions & 40 deletions skills/autorag/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
---
name: autorag
description: Use an already configured AutoRAG librarian agent to search, summarize, compare, and answer questions from local document collections. Use autorag-setup for configuration or indexing changes.
description: Search, summarize, compare, and answer questions from an already configured AutoRAG librarian over local documents and authorized datasources. Use when the user asks AutoRAG to search PDFs, wikis, notes, or a knowledge base. Use autorag-setup for install, model, roots, indexing, or datasource changes.
license: MIT
---

# AutoRAG Librarian Skill

Use this skill when AutoRAG is already configured and the user asks to search,
summarize, compare, or answer questions from local PDFs, wikis, notes, research
papers, or knowledge bases.
papers, knowledge bases, or authorized datasources.

AutoRAG is the specialized librarian agent. One configured model plans the
search, calls BM25, MinSync, Jikji, datasource, and filesystem tools, reads
Expand All @@ -20,14 +21,14 @@ move, rename, edit, or delete source files.

## Preflight

Confirm `~/.autorag/config.json`, an explicit `--config`, or `AUTORAG_CONFIG`
exists. Inspect only non-secret `searchPaths` and `model` metadata, then run:
Confirm a config exists at `--config`, `AUTORAG_CONFIG`,
`$AUTORAG_HOME/config.json`, or `~/.autorag/config.json`. Inspect only
non-secret `searchPaths` and `model` metadata, then run:

```bash
autorag duplicates # read-only duplicate-family review
autorag duplicates --json # machine-readable cleanup planning input
autorag status
autorag health
autorag duplicates --json
autorag status --json
autorag health --json
```

### Duplicate-file review
Expand All @@ -45,59 +46,65 @@ checks credential presence, and normally probes one live completion. If the
model, authentication, configuration, or indexes are unhealthy, use
`autorag-setup` rather than guessing private provider details.

BM25, MinSync, and Jikji should normally be healthy. If they are missing or
stale, run a full `autorag refresh` or return to setup rather than silently
degrading to lexical-only search.
BM25, MinSync, and Jikji should normally be healthy. MinSync and Jikji
auto-install on first use by default. If they are missing or stale, run a full
`autorag refresh` or return to setup rather than silently degrading to
lexical-only search.

## Search

Prefer `--json --debug` when another agent will consume the result or record
feedback. `--json` alone omits `sessionId`. `--debug` adds session/diagnostics
fields and does not print filesystem paths.

```bash
autorag search "what were the key findings in the Q3 report" --top-k 5 --json
autorag search "what were the key findings in the Q3 report" --top-k 5 --json --debug
```

The response contains a `sessionId`, numbered curated results, source mapping,
and an answer grounded in those results.
`--json --debug` includes `answer`, numbered `results` (`number`, `title`,
`summary`, optional `source`), and `sessionId`. Use that `sessionId` for
feedback. `--json` without `--debug` is only `answer` plus `results`.

- `--scope` narrows to a configured virtual sub-path.
- `--tags` narrows already-authorized datasource results and never grants new
access.
- `--json` is preferred for programmatic consumption.
- `--debug` is for diagnostics only.
- `--scope` narrows datasource retrieval to a requested sub-path; it cannot
grant access.
- `--tags` further narrows already-authorized datasource results and never
grants new access.
- `--json` is required for programmatic consumption.
- `--debug` is required for `sessionId`, evidence, and diagnostics.

Do not bypass the librarian with ad hoc raw search when the user requested
AutoRAG. The search loop can use Jikji, BM25, MinSync, datasource retrieval, and
direct source reading as appropriate. If search fails because of model,
provider, auth, or timeout problems, diagnose with `autorag health`.
provider, auth, or timeout problems, diagnose with `autorag health --json`.

Record feedback so retrieval memory can learn:
Record feedback so retrieval memory can learn. Numbers refer to the returned
knowledge units. Supply at least one feedback list:

```bash
autorag feedback <sessionId> --useful 1,3 --not-useful 2
autorag feedback <sessionId> --useful 1,3 --not-useful 2 --json
```

Supply at least one feedback list. Numbers refer to the returned knowledge
units.

## Maintenance

```bash
autorag status
autorag health
autorag refresh
autorag refresh --method bm25,minsync,jikji
autorag watch --once
autorag status --json
autorag health --json
autorag refresh --json
autorag refresh --method bm25,minsync,jikji --json
autorag watch --once --json
autorag watch
autorag refresh --force
autorag index rebuild --yes
autorag index reset --method bm25 --yes
autorag memory inspect
autorag refresh --force --json
autorag index rebuild --yes --json
autorag index reset --method bm25 --yes --json
autorag memory inspect --json
```

Prefer a full refresh so parsed mirrors, BM25, MinSync, Jikji, and configured
datasources stay aligned. Use `--method` only for deliberate narrowing.
Scheduled maintenance should use non-daemon `autorag watch --once`, typically
every 15–30 minutes, with the same config used by search and no overlapping
runs.
datasources stay aligned. `--method` accepts
`parsed,bm25,minsync,datasources,jikji,all`. Use it only for deliberate
narrowing. Scheduled maintenance should use non-daemon `autorag watch --once`,
typically every 15–30 minutes, with the same config used by search and no
overlapping runs.

Reset and rebuild commands remove only selected workspace `.autorag` indexes.
Never target source documents. `memory inspect` is read-only and path-opaque.
Expand All @@ -106,7 +113,8 @@ Never target source documents. `memory inspect` is read-only and path-opaque.

- Use only configured and approved search paths.
- Never expose provider credentials or authentication payloads.
- Never invent or reveal private provider names or model ids.
- Never invent provider identities or model ids.
- A Pi-usable subscription is valid; a subscription Pi cannot invoke is not.
- Preserve real source mapping and numbered feedback identifiers.
- Prefer `--json` when another agent consumes the response.
- Prefer `--json --debug` when another agent consumes search output or will
call `autorag feedback`.
Loading
Loading