Skip to content

highlights-ref-sync.md

Latest commit

 

History

History
1200 lines (961 loc) · 52.5 KB

highlights-ref-sync.md

File metadata and controls

1200 lines (961 loc) · 52.5 KB
Raw

Highlights Reference Note Sync

bob highlights creates Highlights-ready PDFs from Markdown and turns Highlights app PDF annotations into Obsidian reference notes in the Bob vault.

Typical order on a new PDF: create writes into xlib/, scan --dry-run previews the intake move and note plan, then scan (without --write-pdfs) moves the PDF into lib/ and writes the reference note. Marker write-back to the PDF is a separate opt-in (sync --write-pdf or scan --write-pdfs).

Code lives in this bob-cli repository. On the MacBook, use a checkout at ~/projects/bob-cli; do not install from ad hoc scripts outside that checkout.

Contents

What it does

bob highlights implements marker/frontmatter synchronization, Markdown/TextBundle sidecar parsing, generated note rendering, TextBundle image selection asset copying, recursive library scan with xlib intake, prerequisite checks, output collision detection, dirty-target refusal, and atomic note writes.

sync <pdf> loads one PDF with native Rust code, treats the first standalone /Text annotation on page 1 as the marker note, parses the marker list, creates or updates ref/<ref_type>/<pdf-basename>.md for PDFs under lib/<ref_type>/, and rewrites only the managed Highlights body region from the sidecar when one is present. Top-level library PDFs and explicit out-of-library syncs keep the legacy ref/<pdf-basename>.md target. marker <pdf> inspects the same marker without writing.

scan runs the configured pre-scan hook on writing runs, then moves pending PDFs from the configured intake directory into the mirrored library path, recursively finds PDFs under the configured library directory, and processes them in stable path order. Per-PDF validation or write failures are reported without stopping unrelated PDFs; the final command status is still non-zero when any PDF fails. It refuses intake destinations that already exist and duplicate output paths such as two PDFs that would both write the same ref/<ref_type>/<basename>.md target.

doctor checks vault paths, library/ref/xlib directories, pending intake, the configured pre-scan command, sidecar presence, marker readability, Git worktree status, and optional ob availability. It never writes files.

Available commands:

bob highlights create <md-file> [-b|--bob-dir PATH] [-d|--dry-run] [-f|--force] [-i|--include-id] [-l|--lib-dir PATH] [-o|--output PDF] [-P|--parent NOTE] [-r|--ref-dir PATH] [-s|--status STATUS] [-t|--ref-type DIR] [-x|--xlib-dir PATH]
bob highlights doctor [-b|--bob-dir PATH] [-l|--lib-dir PATH] [-r|--ref-dir PATH] [-x|--xlib-dir PATH]
bob highlights marker <pdf> [-b|--bob-dir PATH] [-l|--lib-dir PATH] [-r|--ref-dir PATH] [-x|--xlib-dir PATH]
bob highlights scan [-b|--bob-dir PATH] [-d|--dry-run] [-j|--jobs N] [-l|--lib-dir PATH] [-r|--ref-dir PATH] [-v|--verbose] [-w|--write-pdfs] [-x|--xlib-dir PATH]
bob highlights sync <pdf> [-b|--bob-dir PATH] [-d|--dry-run] [-l|--lib-dir PATH] [-p|--prefer marker|frontmatter] [-r|--ref-dir PATH] [-w|--write-pdf] [-x|--xlib-dir PATH]

create accepts an existing .md file and writes <xlib-dir>/<ref-type>/<basename>.pdf (by default ~/bob/xlib/chat/<basename>.pdf) unless -o, --output <PDF> selects the complete output path. --output keeps the supplied filename and requires a nonempty name with a case-insensitive .pdf extension. Relative output values are resolved from the current working directory; a leading ~ is expanded the same way as other Bob path options. --output cannot be combined with an explicit --ref-type, because --ref-type only participates in default target derivation.

When --output is omitted, the next scan moves the default intake PDF to <lib-dir>/<ref-type>/<basename>.pdf, refusing to create when that archived library PDF or sidecar already exists. An exact path inside the configured intake directory keeps the same intake workflow: the mirrored library destination is <lib-dir>/<intake-relative-path>, including any nested directories, and that occupied library PDF or Markdown/TextBundle sidecar is still refused even with --force. An exact path already inside the configured library treats the selected PDF as the library destination: an existing file requires --force, and there is no second mirrored-destination check. An exact path outside both managed directories does not invent or check an unrelated library destination.

It uses the frontmatter title, first H1, or file stem in that order, asks pandoc/xelatex for a hyperlinked three-level table of contents and PDF outline bookmarks, and embeds a page-1 /Text annotation containing:

- status: ready
- parent: obsidian_ref
- title: <document title>

The marker parent is deliberately bare; generated Obsidian frontmatter turns it into a wikilink later. -P, --parent, -s, --status, and -t, --ref-type override those defaults. -d, --dry-run prints the resolved paths and marker with writes: none. An existing target PDF requires -f, --force; a same-stem Markdown file beside the target PDF is always refused because Highlights would interpret it as a sidecar. Dry-run and success output show a mirrored library_destination only for intake targets. Intake and direct library targets recommend bob highlights scan; a PDF outside both managed directories states that recursive scan will not discover it and recommends bob highlights sync <PDF>. BOB_PANDOC_COMMAND overrides the pandoc executable, and a render failure includes pandoc's diagnostic output. -i, --include-id opts into a stable marker ID derived from the Markdown filename. Bob canonicalizes the Markdown source, removes only the final .md extension from the filename, requires the resulting stem to be nonempty valid UTF-8, and embeds it as id:

- status: ready
- parent: obsidian_ref
- title: <document title>
- id: xprompt_role_binding

The option is intended for research hook execution, for example:

bob highlights create --include-id 202608/xprompt_role_binding/xprompt_role_binding.md

If the filename stem is invalid for marker output, the command fails before running pandoc or creating output directories. Without --include-id, manual create calls keep the previous marker shape and do not add id.

Path configuration options are -b, --bob-dir <PATH>, -l, --lib-dir <PATH>, -r, --ref-dir <PATH>, and -x, --xlib-dir <PATH>. scan also accepts -j, --jobs <N> and -v, --verbose (the detailed per-PDF plan report; the default scan report is concise).

sync <pdf> --dry-run prints the resolved configuration and planned note/PDF actions without modifying either side. Without --dry-run, the command writes the reference note when frontmatter changes. It only writes the PDF marker when the selected projection needs marker write-back and --write-pdf is supplied. For recursive scans, --write-pdfs is the bulk opt-in for marker write-back; scan --dry-run --write-pdfs remains read-only and previews the same marker updates.

MacBook validation notes

Linux-side tests cover synthetic PDF markers, sidecar fixtures, conflict detection, generated highlight rendering, recursive scan preflights, and dirty target refusal. Validate the following against real Highlights-authored files on the MacBook before enabling PDF marker write-back:

  • Real Highlights sidecar Markdown may vary from the fixture-backed parser contract. Keep the documented Highlights Note Format settings fixed while testing.
  • PDF marker write-back is implemented with native PDF annotation writes, but real Highlights-authored files may expose annotation shapes not covered by Linux fixtures. Use --write-pdf or --write-pdfs only after backing up the target PDFs.
  • Scheduled scan should stay dry-run or note-only unless bulk PDF marker writes are intentionally wanted and backed up.

Default Paths

The Bob vault root is BOB_DIR, defaulting to ~/bob.

PDFs are discovered under the Highlights library directory:

BOB_HIGHLIGHTS_LIB_DIR=lib

Freshly created PDFs are written under the Highlights intake directory:

BOB_HIGHLIGHTS_XLIB_DIR=xlib

Reference notes are written under:

BOB_HIGHLIGHTS_REF_DIR=ref

Writing scans can run a pre-scan command before xlib intake:

highlights:
  # Runs from the vault root before `bob highlights scan` inspects the library.
  # Use it to deliver PDFs that do not travel on the git sync channel.
  pre_scan_command: bob_xlib_pull

BOB_HIGHLIGHTS_PRE_SCAN_COMMAND overrides the config file value. Set it to an empty value to disable a configured hook. The command runs as sh -c <command> from BOB_DIR and inherits stdout/stderr, so scheduled logs include its output. If it exits non-zero, scan reports the exit code and aborts before intake. scan --dry-run reports the command it would run without executing it.

Relative BOB_HIGHLIGHTS_LIB_DIR, BOB_HIGHLIGHTS_XLIB_DIR, and BOB_HIGHLIGHTS_REF_DIR values are resolved under BOB_DIR. Absolute paths and ~/... paths are used directly after tilde expansion. lib and xlib must be distinct, non-nested directories so intake cannot move PDFs inside the tree being scanned.

During scan, every pending intake PDF maps from xlib/<rel> to lib/<rel>. For example:

~/bob/xlib/chat/report.pdf
~/bob/lib/chat/report.pdf
~/bob/ref/chat/report.md

For ~/bob/lib/books/systems-performance.pdf, the default output note is:

~/bob/ref/books/systems-performance.md

The first path component below the library directory is the reference type. For the example above, generated frontmatter includes:

ref_type: books

Deeper paths preserve the full library-relative path in ref/ while keeping the first component as ref_type:

~/bob/lib/books/performance/systems-performance.pdf
~/bob/ref/books/performance/systems-performance.md

Top-level library PDFs remain supported for existing files and do not derive a ref_type:

~/bob/lib/systems-performance.pdf
~/bob/ref/systems-performance.md

Marker Note Grammar

The marker note is the first standalone /Text PDF note annotation on page 1. It is not identified by a sentinel token, and standalone notes on later pages are not marker candidates.

The marker note is an unordered list of key: value pairs:

- status: ready
- parent: obsidian
- title: Systems Performance
- id: systems-performance
- aliases: ["Systems Performance", "Brendan Gregg systems performance"]
- topics: [linux, performance]
- source_url: https://example.com/book

Allowed list markers:

- key: value
* key: value

status and parent are required marker keys. Marker parent must be a bare note target such as obsidian or Systems Performance; Obsidian wikilinks such as [[obsidian]], aliases, embeds, and block links are rejected in the PDF marker. Generated reference notes still render parent frontmatter as an Obsidian wikilink, for example parent: "[[obsidian]]".

status must be a scalar string and must exactly match one of:

  • ready
  • next
  • wip
  • read
  • abandoned
  • legacy

Existing status: unread and status: done marker/frontmatter/base inputs are accepted only as deprecated aliases and are normalized to ready and read, respectively, during sync.

Values should be parsed as a YAML-compatible subset when possible:

  • strings
  • numbers
  • booleans
  • nulls
  • inline lists

Invalid values fall back to strings when that is unambiguous. Duplicate marker keys, invalid marker list items, pipeline-owned marker keys, command-managed marker keys such as type or ref_type, and missing or empty status or parent produce clear errors. Unsupported status values are also rejected.

Required Parent, Type, and Ref Type

New Markdown notes under ~/bob require parent frontmatter. The PDF marker is authoritative for parent, so the marker must include it.

Generated reference notes also always include command-managed frontmatter:

type: "[[ref]]"

Nested library PDFs also include path-derived frontmatter:

ref_type: books

type and ref_type are replaced on each successful sync, excluded from marker/frontmatter round-trip hashes, and rejected if either appears in the PDF marker note. ref_type is omitted for top-level library PDFs and explicit out-of-library syncs because there is no library category to derive.

Synced Properties

The PDF marker note and the reference note frontmatter share a synced user-property projection.

Pipeline/provenance fields are excluded from marker sync:

source_pdf
source_pdf_sha256
highlights_sidecar
highlights_count
highlights_synced_at
highlights_marker_base
highlights_marker_hash
highlights_marker_fields
pipeline_version

The command-managed type and ref_type fields are also excluded from marker sync. type is always rendered as [[ref]]; ref_type is rendered only when the PDF path is under lib/<ref_type>/.

Standard synced user fields are status, parent, title, id, research, aliases, topics, source_url, author, and published. id is ordinary user frontmatter for the basename-derived source identifier and does not require highlights_marker_fields. Existing research values remain legacy standard fields for repository-relative Markdown source paths; they round-trip without being renamed automatically to id.

Unknown marker keys should round-trip into frontmatter. New frontmatter keys should sync back to the marker only when they are standard supported fields or explicitly listed in highlights_marker_fields.

legacy_status is ordinary frontmatter used to preserve a migrated note's old status value. It is preserved when rendering notes, but it is not a standard marker-synced field and will not be written back to the PDF marker unless a marker explicitly opts into it as an unknown synced field.

Implemented conflict policy:

  • If only the marker changed, update frontmatter when any status edit agrees with the visible lifecycle task.
  • If only frontmatter changed, update the PDF marker note when PDF writes are enabled and any status edit agrees with the visible lifecycle task.
  • Treat the generated PDF task line as the reference-reading status: [ ] means ready, [*] means next, [/] means wip, [x]/[X] means read, and [-] means abandoned.
  • If that task status differs from the selected marker/frontmatter projection, apply it as a note-side status signal. If marker or frontmatter also changed status to a different value from the stored base, report a conflict instead of guessing.
  • If marker and frontmatter changed different fields from the stored base, auto-merge them. PDF marker writes are still opt-in with targeted sync --write-pdf or bulk scan --write-pdfs.
  • If both changed the same field differently, fail without modifying either side unless --prefer marker or --prefer frontmatter is supplied.

The last synced user-property projection is stored as highlights_marker_hash and as compact JSON in highlights_marker_base. The hash keeps old-note compatibility; the base snapshot lets the command prove safe field-level merges. --prefer frontmatter, any generated-task lifecycle change, and any auto-merge that includes frontmatter changes require --write-pdf for targeted sync, or --write-pdfs for recursive scan, whenever the PDF marker must be updated.

PDF marker writes are performed by saving a temporary PDF next to the original and renaming it over the target. Before first PDF writes, commit or otherwise back up the PDF library. Keep Highlights and Obsidian idle while testing PDF writes so the apps do not race the CLI.

Scan, Safety, and Git/ob Behavior

scan --dry-run reports every discovered PDF, including PDFs that are still pending under xlib. If a pre-scan hook is configured, the dry run reports that it would run and the exact command, but does not execute it. The default report is concise. -v, --verbose prints the detailed per-PDF plan, including each planned xlib/<rel> -> lib/<rel> intake move. Dry runs do not move anything. Valid PDFs show their target reference note, sidecar path if present, selected sync source, and note/PDF marker action. Invalid PDFs show a plan_error. Scan output also reports write_pdfs: true|false so bulk marker-write runs are auditable. Dry runs do not create directories, move intake files, write notes, write PDFs, or run the pre-scan command, even when combined with --write-pdfs.

Before planning notes, writing scans run the optional pre-scan command and then preflight intake. Intake moves each PDF under xlib to the mirrored lib path and moves same-stem Markdown sidecars and TextBundle directories with the PDF, so annotation text and image assets are not orphaned. If any destination PDF or sidecar already exists, the whole scan aborts before moving or writing anything. Intake runs before the dirty-target Git check because a newly rendered intake PDF is normally untracked, and the move never overwrites an existing path; a tracked synced PDF behaves as an ordinary Git rename.

After intake, the command rejects duplicate output paths, builds per-PDF plans, and checks Git status for existing vault files that successfully planned PDFs would modify. If a target ref note or PDF marker target is dirty, it fails before any note/PDF write, except for a tracked target ref note whose only body change is the exact generated ^ref checkbox toggle, optionally combined with frontmatter edits. There is no force mode for scan; commit, stash, or clean unrelated dirty files before rerunning.

Image assets copied from TextBundle sidecars are treated as note-side writes. They are planned during dry runs, checked by the same dirty-target preflight when the destination already exists, and written only during non-dry-run syncs. Scan output includes image totals when a planned PDF contains image selections.

Planning failures for one PDF do not stop valid plans from writing. Write-time failures such as a changed note/PDF or temporary save failure are reported as write_failure entries, and later valid PDFs continue. The scan summary reports plan_failures, write_failures, and scan_failures; any non-zero failure count makes the command exit non-zero even when other PDFs were processed.

Reference note writes and image asset copies are atomic temporary-file renames and are skipped when the rendered note or content-addressed asset is byte-identical to the existing file.

bob highlights does not reconcile the vault before or after writes. bob nightly runs vault-sync before and after the maintenance commands. The source_pdf field is rewritten on every sync, so a direct bob highlights sync ~/bob/xlib/... dry run or one-off sync self-heals to the library-relative path on the next scan after intake.

Generated Body Contract

Highlights, image selections, comments on annotations, and standalone non-marker notes sync one way from the PDF/sidecar into the reference note.

For foo.pdf, sidecar discovery looks for foo.md first. If there is no Markdown sidecar, it recognizes foo.textbundle only when the bundle contains text.md or text.markdown; other TextBundle contents fail with an explicit unsupported-sidecar error. Image selections require the referenced asset file to exist relative to the sidecar text file's directory. This makes foo.textbundle/text.md resolve ![](assets/figure.png) to foo.textbundle/assets/figure.png. Plain Markdown sidecars can still reference local image files, but Highlights' plain Markdown export does not include image assets; re-export as TextBundle when area selections should sync. If no sidecar exists, sync still performs marker/frontmatter sync. For a new note it creates an empty managed region; for an existing note it preserves the existing managed region.

Known Highlights TextBundle Creation Bug

For PDFs that contain image or area (rectangle) annotations, the Highlights app sometimes fails to create the TextBundle sidecar in the first place, even with autosave enabled. This is a Highlights app limitation, not a bob highlights behavior: Bob never creates or mutates sidecars, it only reads the PDF and sidecar content the app owns. Because Highlights can still update a TextBundle that already exists, the workaround is a one-time manual export per affected PDF. After the initial export, Highlights keeps the bundle updated, so this step is only needed once per PDF whose image annotations should sync.

Create or export the bundle beside the PDF, using the exact PDF basename, so that example.pdf is paired with example.textbundle/. For ~/bob/lib/<ref_type>/example.pdf, the expected layout is:

~/bob/lib/<ref_type>/example.pdf
~/bob/lib/<ref_type>/example.textbundle/text.md
~/bob/lib/<ref_type>/example.textbundle/assets/figure.png

The sidecar text must live at ~/bob/lib/<ref_type>/example.textbundle/text.md (or text.markdown), and every referenced image must live under ~/bob/lib/<ref_type>/example.textbundle/assets/.

After creating the bundle, run bob highlights scan --dry-run and confirm the affected PDF reports the expected sidecar: path and a nonzero images: count. The user-visible symptoms of a missing or incomplete bundle are:

  • A missing .textbundle looks like no sidecar, so bob highlights scan performs marker/frontmatter planning but generates no image annotations.
  • A .textbundle without text.md or text.markdown surfaces as unsupported textbundle sidecar.
  • A bundle whose text.md references images it does not actually contain under assets/ surfaces as image asset not found.

Two Markdown sidecar shapes are supported. The simple shape is:

  • Page labels are Markdown headings such as ## Page 12 or ## p. 12.
  • Annotation spacers are horizontal rules such as ---.
  • A highlight starts with one or more blockquote lines.
  • Non-heading text after a highlight is rendered as a highlight comment. A leading Comment: or Note: label is stripped.
  • A standalone note is non-heading text outside a blockquote. A leading Note: label is stripped.
  • A Markdown image line such as ![alt](assets/figure.png) outside a blockquote is an image selection when the target has a supported image extension. Text after the image is treated as its comment. Multiple images in one chunk render as separate image annotations. Non-image targets such as PDFs remain ordinary standalone note text.
  • The first standalone note in sidecar order is treated as the PDF marker mirror and is excluded from generated content.

Simple sidecar fragment:

## Page 12

Note: marker note mirrored from the PDF

---

> Latency is not throughput.

Comment: Compare this with SLO notes.

---

Note: Keep this standalone observation.

Image selection sidecar fragment:

## Page 12

![Latency figure](assets/figure.png)

Comment: Compare this figure with the latency table on p.14.

The linked-page Highlights export shape is also supported:

  • Page labels may be Markdown-link headings such as #### [Page 1](highlights://book#page=1). The link label, Page 1, is used as the rendered page heading.
  • Non-page headings such as the document title and dated metadata headings are ignored by the annotation parser.
  • Annotation spacers may use any Markdown horizontal rule style, including ***.
  • Highlights may be hard-wrapped so that only the first physical line starts with >. Immediate nonblank continuation lines are kept as highlight text, but marker-list fields and explicit Comment:/Note: labels start the comment side instead.
  • User comments exported as Markdown list items after a highlight are rendered without the list marker.
  • A blockquoted marker mirror title followed by marker-list fields such as status and parent is excluded from generated content.

Linked sidecar fragment:

# Highlights Reference Note Sync

#### [Page 1](highlights://highlights-ref-sync#page=1)

##### 2026-06-03:

> Highlights Reference Note Sync

- status: ready
- parent: obsidian

***

#### [Page 2](highlights://highlights-ref-sync#page=2)

##### 2026-06-03:

> It only writes the PDF marker when marker write-back is needed
and the matching write opt-in is supplied.

- Support sase tool call replay?

The generated region is the only body region the tool owns:

<!-- highlights:begin -->

<!-- highlights:end -->

Manual content outside those markers must be preserved. User edits inside the generated region may be overwritten.

New generated notes include a title, a PDF wikilink Obsidian task line with the #hide tag and the stable ^ref block ID, and ## Highlights. ## Tasks is not part of that skeleton; the section is created later, directly below the ^ref task, when the first annotation task is imported. Existing notes must already contain the managed begin/end markers; otherwise sync fails instead of guessing where generated content belongs.

Generated annotations render as Obsidian callouts. Highlight text uses a [!quote] callout; a highlight or image comment is nested inside that quote as a [!note] Comment callout so the annotation's trailing ^h-... block ID still covers the quote/image and the comment together. Image selections use [!quote] Image with a vault-relative Obsidian embed. Standalone non-marker notes render as [!note] callouts. Removed annotation tombstones render under ### Removed highlights as [!warning] Removed highlight callouts.

Example generated body:

### Page 12

> [!quote] Latency is not throughput.
>
> > [!note] Comment Compare this with SLO notes.

^h-2b91f0a4c7de

> [!note] Keep this standalone observation.

^h-8f42a61a90cc

Image selections copy their asset into a per-note assets directory beside the reference note. For ref/books/example.md, the destination is ref/books/example.assets/h-<id>.<ext>, and the generated block embeds it with a vault-relative wikilink:

### Page 12

> [!quote] Image ![[ref/books/example.assets/h-2b91f0a4c7de.png]]
>
> > [!note] Comment Compare this figure with the latency table on p.14.

^h-2b91f0a4c7de

Asset filenames and image block IDs are content-addressed from the source PDF path and image bytes, so re-exported or renamed TextBundle assets keep the same ^h-... block and stored filename when the image bytes are unchanged. Existing matching assets are skipped; an existing destination with different bytes is a write failure instead of an overwrite. When an image selection is removed from the sidecar, the generated block is tombstoned like other removed annotations, but the copied asset file is left in place. Asset garbage collection is future work.

Annotation text is beautified only while rendering the generated region:

  • Consecutive prose lines are reflowed into one line. Blank lines remain paragraph breaks.
  • Markdown unordered list lines (- , * , + , including task checkboxes) start their own logical lines, and wrapped continuation text joins onto that list item.
  • Hyphenated PDF line endings are healed at join points: lowercase next words drop the hyphen (through- + put -> throughput), while uppercase or digit next fragments keep it (Marie- + Curie -> Marie-Curie).
  • Ligature glyphs, non-breaking spaces, soft hyphens, zero-width characters, doubled spaces, and tab runs are normalized away.

The cleanup is render-only. Block IDs, annotation-task source links, and processed [h:: ...] markers continue to use the raw normalized sidecar text, so unchanged sidecar annotations keep their existing ^h-... IDs when a note is regenerated with the callout format.

The generated task line is the reference-reading lifecycle:

- [ ] #task #ref [[lib/books/example.pdf]] #hide ^ref

The #ref tag immediately after #task marks this as the machine-managed reference lifecycle task so Obsidian task views can tell it apart from ordinary annotation-derived follow-up tasks. It is additive: legacy generated lines without #ref are still recognized.

Checkbox Obsidian Tasks status Reference status Meaning
[ ] Todo ready In the reading queue, not started
[*] Next next Queued for action
[/] In Progress wip Actively reading
[x] / [X] Done read Finished
[-] Cancelled abandoned Dropped

The mapping is bidirectional. On read, the checkbox is authoritative and moves frontmatter and the PDF marker to its status. On generation or write-back, the final status selects the matching checkbox. legacy has no lifecycle checkbox and falls back to [ ]; legacy notes without a generated ^ref task remain untouched. unread is accepted only as a deprecated alias of ready, while done remains a deprecated alias of read.

Changing [x] or [-] back to [ ] therefore reopens a reference to ready; use [/] to resume it directly as wip. Metadata such as [cancelled:: 2026-06-04] is preserved when only the checkbox mark changes. If the marker or frontmatter also moved status to a different value from the stored base, the task signal reports a conflict rather than silently choosing a side. When editing status directly, move the visible ^ref task to the same state.

If a lifecycle change would update the PDF marker, sync --dry-run previews pdf_marker_action: would-update, plain sync refuses before writes, and targeted sync --write-pdf writes the marker. scan --dry-run previews this work. A writing scan keeps the default note-only refusal unless --write-pdfs is supplied.

Highlight comments and standalone non-marker notes can also create actionable Obsidian tasks when the marker/frontmatter-selected PDF status is wip, before the generated PDF ^ref checkbox contributes another lifecycle status. The final run that moves a wip PDF to read or abandoned still imports newly added annotation tasks in that same run, and moving a reference to [/] imports them too. Runs whose selected and final status are both non-wip skip task intake. Any unordered Markdown bullet line whose item text contains #task as a whitespace-delimited token is copied to an unchecked task:

- #task Compare this claim with the appendix.
- [x] #task Email the citation to Alex.
- #task Send the quote to Alice @alice

Created tasks go into the note's ## Tasks section. If the note has no Tasks heading, one is created as ## Tasks directly below the generated PDF ^ref task, separated from it by a blank line. An existing Tasks heading at any level is reused. New tasks are appended after the tasks already in the section:

## Tasks

- [ ] #task Compare this claim with the appendix. [[#^h-2b91f0a4c7de|🔖]] [h:: 4c0a13d2...] [created::2026-06-07]
- [ ] #task Email the citation to Alex. [[#^h-2b91f0a4c7de|🔖]] [h:: 910f6ce7...] [created::2026-06-07]

If the final whitespace-delimited token is a strict @name route suffix, the suffix is removed from the created task text and the task is appended to the end of existing root-level note ~/bob/name.md instead. Routed tasks keep that append-at-end behavior; this command does not create a ## Tasks section on the routed note. Route names must match @([A-Za-z0-9][A-Za-z0-9_-]*); tokens with punctuation, dots, slashes, empty names, or path traversal content are ordinary task text. Routed target notes must already exist and must not be directories, because new root notes need explicit frontmatter that this command cannot infer. For example, @alice creates this line in ~/bob/alice.md:

- [ ] #task Send the quote to Alice [[ref/books/example#^h-2b91f0a4c7de|🔖]] [h:: 9b31c4a0...] [created::2026-06-07]

Each created task carries a block backlink to the annotation-level generated source block, rendered as an aliased 🔖 link between the task prose and processed/created properties. Same-note tasks use compact same-file targets ([[#^h-...]]). Routed tasks use full vault-relative note targets ([[ref/books/example#^h-...]]) so the source note remains recoverable after the task moves to another note or done/. The managed Highlights region keeps the annotation-level ^h-... block and does not attach task-specific ^ht-... block IDs to source task lines:

> [!quote] Highlighted claim.
>
> > [!note] Comment #task Compare this claim with the appendix.
> > #task Email the citation to Alex.

^h-2b91f0a4c7de

The [h:: ...] processed ID is computed from the source reference note path, source annotation block, and normalized task identity. Duplicate identical #task bullets in the same annotation still create only one task. The property is the durable processed marker that moves with the task when it is completed, cancelled, edited, or archived by bob move-done-tasks, so later syncs do not recreate it and do not write any processed state back into the PDF or sidecar. Older tasks that already have [highlight_task:: ...] are still recognized for compatibility, and tasks created by the old ^ht implementation are recognized by their #^ht-... backlinks. New tasks do not write [highlight_task:: ...] or ^ht source anchors. If both processed properties and old source backlinks are removed from a moved or heavily edited task, the legacy normalized text fallback only covers unchanged text.

Annotation-created tasks are independent tasks, not subtasks of the PDF reading-status task. Later syncs preserve existing checkbox state and task properties such as [completion::], [cancelled::], [due::], or edited priority fields. Completing or cancelling these annotation-created tasks does not update the PDF marker or reference-note reading status.

Generated blocks use Obsidian block IDs beginning with ^h-. Text highlight and standalone-note IDs are deterministic content hashes over source PDF path, page label, annotation kind, sidecar order on the page, and quote/note text. Highlight comments are not part of the hash, so editing only a comment updates the block without changing its ID. Image IDs use the source PDF path plus image bytes instead, so asset renames and nearby annotation order changes do not churn the image block. If a previously generated block disappears from the sidecar, the command keeps the old block ID under ### Removed highlights with a tombstone message. Editing text highlight content itself mints a new block ID; a task created earlier keeps its original link, which then targets the tombstoned block under ### Removed highlights — still a valid, resolvable jump.

MacBook Setup Guide

Run these steps on the MacBook. The intended checkout is ~/projects/bob-cli, and the intended vault paths are ~/bob/xlib for PDF intake, ~/bob/lib for archived PDFs, and ~/bob/ref for generated reference notes.

Some vaults also have a lit/ folder from an older layout. That is not the production library path. The defaults remain ~/bob/xlib (intake), ~/bob/lib (archived PDFs), and ~/bob/ref (generated notes). For a one-off test against lit, pass --lib-dir lit explicitly.

Install prerequisites if needed:

xcode-select --install
command -v cargo >/dev/null || curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"

Clone or update bob-cli, then install the local checkout:

mkdir -p ~/projects
if [ -d ~/projects/bob-cli/.git ]; then
  git -C ~/projects/bob-cli pull --ff-only
else
  git clone git@github.com:bobs-org/bob-cli.git ~/projects/bob-cli
fi
cargo install --path ~/projects/bob-cli --locked --force
bob highlights --help

Create or confirm the vault layout:

mkdir -p ~/bob/xlib/chat ~/bob/lib/books ~/bob/ref
git -C ~/bob status --short

In Highlights Pro on the MacBook:

  • Fresh PDFs created by bob highlights create land under ~/bob/xlib/<ref_type>/; the next scan moves them to ~/bob/lib/<ref_type>/.
  • Keep existing PDFs that should sync under ~/bob/lib/<ref_type>/, such as ~/bob/lib/books.
  • Enable autosaved sidecars next to each PDF. Plain Markdown creates ~/bob/lib/books/example.md; TextBundle creates ~/bob/lib/books/example.textbundle/text.md.
  • Use TextBundle export for PDFs where Highlights area/rectangle selections should sync, because plain Markdown export does not include image assets.
  • For PDFs with image or area annotations, Highlights may fail to create the TextBundle the first time even with autosave on. Manually export/create example.textbundle/ beside example.pdf once, then confirm it with bob highlights scan --dry-run. See Known Highlights TextBundle Creation Bug.
  • Lock the Highlights Note Format to the sidecar contract above: page headings, --- annotation separators, highlights as blockquote lines, highlight comments as plain paragraphs, standalone notes as plain paragraphs, and image selections as Markdown image links.
  • Add exactly one marker note as the first standalone /Text PDF note annotation on page 1.
  • Put at least status and parent in that page-1 standalone note.

Use this marker as a starting point:

- status: ready
- parent: obsidian
- title: Example Title
- topics: [example]

Run the initial checks:

bob highlights doctor
bob highlights scan --dry-run
bob highlights sync ~/bob/lib/books/example.pdf --dry-run
bob highlights marker ~/bob/lib/books/example.pdf

MacBook validation checklist:

  • cargo install --path ~/projects/bob-cli --locked --force installs the local checkout.
  • bob highlights doctor reports valid vault/library/ref paths, the xlib intake path and pending count, marker readability, Git status, and optional ob availability. A missing ~/bob/xlib is warning-only because create creates it on demand.
  • bob highlights scan --dry-run lists the expected PDFs under ~/bob/lib and any pending ~/bob/xlib intake moves, reports the intended ~/bob/ref/<ref_type>/*.md targets, and prints writes: none. If scan_failures is non-zero, inspect the per-PDF plan_error lines while noting that valid PDFs were still reported.
  • bob highlights sync ~/bob/lib/books/example.pdf --dry-run shows the expected marker page/note, sync source, sidecar path, note action, and no writes.
  • The first real note write creates or updates ~/bob/ref/books/example.md with parent, type: "[[ref]]", ref_type: books, pipeline metadata, manual sections, and the managed Highlights region.
  • For a TextBundle image selection, the first real write also creates ~/bob/ref/books/example.assets/h-<id>.<ext> and embeds it from the generated callout with a vault-relative wikilink.
  • A second run with unchanged inputs reports writes: none.
  • If frontmatter edits or a generated lifecycle-task change require PDF marker write-back, a dry run reports pdf_marker_action: would-update before any targeted --write-pdf or bulk --write-pdfs run.
  • git -C ~/bob status --short is reviewed before and after each write pass.

The sync model is deliberately asymmetric:

  • Marker/frontmatter fields are 2-way and can conflict.
  • Highlights, highlight comments, and standalone non-marker notes are PDF or sidecar to reference note only.
  • Annotation-created #task bullets are created in the reference note or an explicitly routed existing root note only, and do not sync completion state back to the PDF marker.
  • Edits inside the managed <!-- highlights:begin --> region in Obsidian may be overwritten.

Enable note writes only after reviewing the dry-run output:

git -C ~/bob status --short
bob highlights sync ~/bob/lib/books/example.pdf
bob highlights scan

scan does not enable PDF marker write-back by default. If a dry run reports pdf_markers_would_update, review those PDFs and back up the library before bulk write-back:

bob highlights scan --dry-run --write-pdfs
bob highlights scan --write-pdfs

For a single PDF, keep using the targeted singular flag:

bob highlights sync ~/bob/lib/books/example.pdf --dry-run
bob highlights sync ~/bob/lib/books/example.pdf --write-pdf

The intended frontmatter edit workflow is:

$EDITOR ~/bob/ref/books/example.md
bob highlights sync ~/bob/lib/books/example.pdf --dry-run
bob highlights sync ~/bob/lib/books/example.pdf --write-pdf

If the ref note is tracked in Git, the write-back command may update that dirty note only when the dirty changes are unstaged frontmatter edits and/or the exact generated ^ref checkbox toggle, and the file still matches what the command planned from. Other body edits, managed-region edits, staged changes, untracked notes, and dirty PDFs are still refused.

Keep Highlights and Obsidian idle while testing PDF marker writes so the apps do not race the CLI.

Scheduled Scan

The intended automation is a scheduled scan, not a live recursive watcher. Start with a dry-run schedule. On the MacBook account, create this LaunchAgent:

mkdir -p ~/Library/LaunchAgents ~/Library/Logs/bob
cat > ~/Library/LaunchAgents/com.bryan.bob-highlights-scan.plist <<'PLIST'
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>com.bryan.bob-highlights-scan</string>
  <key>ProgramArguments</key>
  <array>
    <string>/bin/zsh</string>
    <string>-lc</string>
    <string>/Users/bryan/.cargo/bin/bob highlights scan --dry-run</string>
  </array>
  <key>EnvironmentVariables</key>
  <dict>
    <key>BOB_DIR</key>
    <string>/Users/bryan/bob</string>
    <key>BOB_HIGHLIGHTS_LIB_DIR</key>
    <string>lib</string>
    <key>BOB_HIGHLIGHTS_REF_DIR</key>
    <string>ref</string>
    <key>BOB_HIGHLIGHTS_XLIB_DIR</key>
    <string>xlib</string>
  </dict>
  <key>StartInterval</key>
  <integer>3600</integer>
  <key>StandardOutPath</key>
  <string>/Users/bryan/Library/Logs/bob/highlights-scan.out</string>
  <key>StandardErrorPath</key>
  <string>/Users/bryan/Library/Logs/bob/highlights-scan.err</string>
</dict>
</plist>
PLIST
plutil -lint ~/Library/LaunchAgents/com.bryan.bob-highlights-scan.plist
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/com.bryan.bob-highlights-scan.plist 2>/dev/null || true
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.bryan.bob-highlights-scan.plist
launchctl kickstart -k gui/$(id -u)/com.bryan.bob-highlights-scan
tail -n 80 ~/Library/Logs/bob/highlights-scan.out
tail -n 80 ~/Library/Logs/bob/highlights-scan.err

After several clean dry-run cycles, remove --dry-run from the ProgramArguments command and reload the LaunchAgent with the same launchctl bootout, bootstrap, and kickstart commands. Keep PDF marker write-back manual; do not schedule sync --write-pdf or scan --write-pdfs unless bulk PDF writes are deliberately desired and backed up.

A cron fallback is also acceptable:

SHELL=/bin/zsh
PATH=/Users/bryan/.cargo/bin:/opt/homebrew/bin:/usr/local/bin:/bin:/usr/bin
BOB_DIR=/Users/bryan/bob
BOB_HIGHLIGHTS_LIB_DIR=lib
BOB_HIGHLIGHTS_REF_DIR=ref
BOB_HIGHLIGHTS_XLIB_DIR=xlib
0 * * * * /Users/bryan/.cargo/bin/bob highlights scan --dry-run >> /Users/bryan/Library/Logs/bob/highlights-scan.log 2>&1

Disable the LaunchAgent with:

launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/com.bryan.bob-highlights-scan.plist

Disable One PDF

There is no marker key such as sync: false. scan processes every .pdf under the configured library directory.

To disable one PDF, move that PDF and its sidecar out of ~/bob/lib:

mkdir -p ~/bob/lib-disabled
mv ~/bob/lib/books/example.pdf ~/bob/lib-disabled/
mv ~/bob/lib/books/example.md ~/bob/lib-disabled/ 2>/dev/null || true
mv ~/bob/lib/books/example.textbundle ~/bob/lib-disabled/ 2>/dev/null || true
git -C ~/bob status --short

The existing ~/bob/ref/books/example.md is no longer touched once the PDF is outside the scanned library. Leave it in place for archival use, or move it explicitly:

mkdir -p ~/bob/ref-disabled
mv ~/bob/ref/books/example.md ~/bob/ref-disabled/

Do not remove the managed Highlights markers as a disable mechanism. If a PDF is still scanned and its existing reference note lacks the managed markers, the command fails before guessing where generated content belongs.

Backup and Rollback

Before first writes, make the vault clean or intentionally checkpointed:

git -C ~/bob status --short
git -C ~/bob stash push -u -m "pre-highlights"
git -C ~/bob status --short

If the current vault changes should be kept as the baseline, commit them instead of stashing:

git -C ~/bob add ref lib
git -C ~/bob commit -m "Checkpoint before highlights sync"

Before enabling --write-pdf or --write-pdfs, keep a PDF backup outside the write path:

backup_dir=~/bob/backups/highlights/$(date +%Y%m%d-%H%M%S)
mkdir -p "$backup_dir/lib/books"
cp -p ~/bob/lib/books/example.pdf "$backup_dir/lib/books/"
cp -p ~/bob/lib/books/example.md "$backup_dir/lib/books/" 2>/dev/null || true

For a full library backup before broader testing:

backup_dir=~/bob/backups/highlights/$(date +%Y%m%d-%H%M%S)
mkdir -p "$backup_dir"
rsync -a --include='*/' --include='*.pdf' --include='*.md' --include='*.textbundle/***' --exclude='*' ~/bob/lib/ "$backup_dir/lib/"

Inspect note writes:

git -C ~/bob diff -- ref/books/example.md
git -C ~/bob status --short

Rollback a generated reference note:

git -C ~/bob restore -- ref/books/example.md

Rollback a PDF marker write from the backup:

cp -p "$backup_dir/lib/books/example.pdf" ~/bob/lib/books/example.pdf

Conflict Resolution

When the marker and frontmatter both changed since the last synced projection, the command compares both sides to highlights_marker_base. Non-overlapping field edits auto-merge and dry runs report sync_source: auto-merge. Same-field conflicts still fail and write nothing. Inspect both sides:

bob highlights marker ~/bob/lib/books/example.pdf
sed -n '1,120p' ~/bob/ref/books/example.md
bob highlights sync ~/bob/lib/books/example.pdf --dry-run

Choose the PDF marker as the source of truth:

bob highlights sync ~/bob/lib/books/example.pdf --prefer marker

Choose the Obsidian frontmatter as the source of truth and write it back to the PDF marker:

bob highlights sync ~/bob/lib/books/example.pdf --prefer frontmatter --write-pdf

If the only change is frontmatter or a generated lifecycle-task mark, or a dry-run auto-merge reports pdf_marker_action: would-update, review the marker first, back up the PDF, then run the targeted write:

bob highlights sync ~/bob/lib/books/example.pdf --write-pdf

For reviewed bulk scan write-back, preview the library and then opt in:

bob highlights scan --dry-run --write-pdfs
bob highlights scan --write-pdfs

Expected Failures

Common failure snippets and fixes:

During scan, marker validation, sidecar validation, managed-region validation, --write-pdf refusals, and per-PDF write races are reported for the affected PDF while unrelated valid PDFs may still be planned or written. The summary shows result: partial-failure and the command exits non-zero when any PDF fails. Library discovery errors, output path collisions, and dirty target preflight failures remain hard global failures before writes.

Message snippet Meaning Fix
library directory does not exist or is not a directory ~/bob/lib is missing or --lib-dir points at the wrong path. Create ~/bob/lib or pass the intended --lib-dir.
no standalone /Text note annotations found on page 1 The PDF has no page-1 standalone marker note. Add the first standalone PDF note on page 1 in Highlights.
missing required marker key: status The marker list lacks status. Add - status: ready to the marker.
unsupported status status is not one of ready, next, wip, read, abandoned, or legacy. Change the marker or frontmatter status to a supported value. unread is accepted only as a deprecated alias of ready.
missing required marker key: parent The marker/frontmatter projection lacks parent. Add a bare marker parent such as - parent: obsidian, or add frontmatter parent such as parent: "[[obsidian]]".
wikilinks are not supported The PDF marker parent uses Obsidian link syntax. Remove the brackets in the PDF marker, e.g. use - parent: obsidian.
'type' is command-managed The marker tries to set the generated note type. Remove type from the marker; generated notes get type: "[[ref]]" automatically.
'ref_type' is command-managed The marker tries to set the path-derived reference type. Remove ref_type from the marker; nested library paths derive it automatically.
invalid marker item on line A marker line is not - key: value or * key: value. Rewrite the marker as a flat list.
duplicate marker key on line The marker repeats a normalized key. Keep only one value for that key.
xlib intake collision(s) detected before writes A PDF or sidecar under ~/bob/xlib would move over an existing ~/bob/lib destination. Remove, rename, or archive one side deliberately before scanning.
output path collision(s) detected before writes Multiple PDFs would write the same reference note path, such as ref/books/example.md, or the same planned image asset destination. Rename or move one PDF before scanning.
image asset not found The sidecar references an image file that is not present relative to the sidecar text file. This usually means the PDF was exported as plain Markdown, or Highlights failed to create the TextBundle for a PDF with image/area annotations. Manually create/export the TextBundle once, beside the PDF and with the exact PDF basename, so assets/... files are included, then rerun. See Known Highlights TextBundle Creation Bug.
image asset destination exists with different bytes A content-addressed ref/.../*.assets/h-<id>.<ext> destination already exists but its bytes do not match the source asset. Inspect the existing asset, then remove or restore it before rerunning.
refusing to modify dirty vault files Git reports dirty touched paths outside the allowed frontmatter and generated-task checkbox write-back case. Commit, stash, or clean those paths.
reference note changed but --write-pdf was not supplied Frontmatter or the generated task contributes a lifecycle status to the selected projection, so the PDF marker needs an opt-in write. Back up the PDF, then run targeted sync --write-pdf or reviewed bulk scan --write-pdfs.
checked PDF task conflicts The generated task says status: read, but marker or frontmatter changed status to another value from the stored base. Uncheck the task or set the marker/frontmatter status to read.
cancelled PDF task conflicts The generated task says status: abandoned, but marker or frontmatter changed status to another value from the stored base. Uncancel the task or set the marker/frontmatter status to abandoned.
ready, next, or in-progress PDF task conflicts The generated task selects ready, next, or wip, but marker or frontmatter changed status to a different value from the stored base. Move the task to the intended state or set marker/frontmatter status to match it.
marker/frontmatter conflict Marker and frontmatter changed the same field differently, or the note has no stored base snapshot for a safe merge. Inspect both sides, then rerun with --prefer marker or --prefer frontmatter --write-pdf.
changed during sync; rerun The note or PDF changed after planning and before writing. Rerun after closing or pausing apps that may touch the file.
scan completed with ... per-PDF failure(s) A recursive scan finished reporting or writing valid PDFs, but at least one PDF had a plan_error or write_failure. Fix the named PDFs and rerun; review successful writes before assuming the scan wrote nothing.
existing reference note is missing the managed Highlights region An existing ref note lacks <!-- highlights:begin --> and <!-- highlights:end -->. Add both markers around the generated section or move the note aside and regenerate.
unsupported textbundle sidecar The .textbundle has no text.md or text.markdown. Verify the bundle contains text.md or text.markdown; manually re-export/create a valid TextBundle beside the PDF if it does not. See Known Highlights TextBundle Creation Bug.