Skip to content

feat(observability): add SQLite + JSONL observability foundation - #38

Merged
DrAbdulmalek merged 2 commits into
mainfrom
feat/observability-foundation
Oct 7, 2026
Merged

DrAbdulmalek merged 2 commits into
mainfrom
feat/observability-foundation

Conversation

@DrAbdulmalek

Copy link
Copy Markdown
Owner

What changed

  • New src/db/observability.py (689 lines) — ObservabilityStore + RunContext + redact() helper. SQLite-backed, thread-safe writes, JSONL mirrors for tailing.
  • New tests/test_observability.py (185 lines) — 12 unit tests covering all five behaviors + privacy defaults + JSONL mirror.
  • New docs/OBSERVABILITY.md (137 lines) — API reference, what it is NOT (no telemetry, no remote upload, no full issue tracker), how Omni can adopt the same shape later.

Why it matters

Per ecosystem policy (Phase 3), the active product repos must have a practical local observability + notes foundation. This PR implements it for IntelliFile (Omni can adopt the same pattern in a follow-up PR).

Satisfies the five required behaviors:

  1. System self-profile (version, env, feature flags, dependency health, model/service availability)
  2. Structured run logging (run_id, session_id, component, action, start/end, status, duration, input/output summaries, error summary, redaction level)
  3. Artifact/result registry (what, where, by what, from what, success/failure)
  4. Manual operator/user notes (attached to runs/files/folders/OCR jobs/actions, with tags + severity)
  5. Improvement queue (repeated failures + warn-or-worse notes auto-bumped, small and practical)

Plus privacy-aware defaults:

  • Sensitive field names (token, password, api_key, api_id, api_hash, session, auth, credential, private_key) auto-redacted to <REDACTED>.
  • Raw payloads only stored when debug=True.
  • No network, no telemetry, no remote upload.

Reference: ECOSYSTEM_STATE.md in repo-sync-toolkit (PR #1).

Risks / intentionally not changed

  • Backend/storage only. UI surfacing (Activity / Notes / Diagnostics panels) deferred to a separate PR per the user instruction: If UI exposure is too large for one PR, first implement the backend/storage structure and minimal surfacing, then iterate.
  • Does NOT touch src/core/smart_tagger.py (medical scope tracked separately).
  • Does NOT touch mobile/ (quarantined in PR chore(deps): bump actions/checkout from 4 to 7 #3).
  • Does NOT integrate into existing components yet — adoption is a follow-up PR per component. This PR only adds the foundation + tests + docs.
  • Does NOT add OpenTelemetry / Prometheus / remote exporters — intentionally local-first.
  • Does NOT create cross-repo coupling with Omni — the same shape can be copied later if a shared library is extracted, but for now each repo has its own copy.

Verification performed

  • python -m pytest tests/test_observability.py -v — 12/12 PASS.
  • Smoke test: started a run, recorded an artifact, added a note, captured system profile, listed all four entity types, queue auto-bumped on warn note.
  • Redaction verified: api_key, api_id, api_hash, session, password, auth all redacted in stored input_summary; non-sensitive keys preserved; debug=True bypasses.
  • JSONL mirror verified: runs.jsonl appended correctly, last line parses as JSON.
  • 3 files, +1011 lines, 0 deletions.

Blockers / manual follow-up

None. Future PRs should:

  1. Wire RunContext into existing components (search, tagger, extractor, etc.) one component at a time.
  2. Add UI surfaces (Activity panel, Notes panel, Health/Diagnostics panel) in the PySide6 desktop GUI.
  3. Mirror the same shape into omni-medical-suite/src/observability/ in a separate PR.

Next PR

PR #7 — repo-sync-toolkit-safety-hardening.

Z User and others added 2 commits July 31, 2026 02:06
Adds src/db/observability.py — a lightweight, local-first observability
+ notes foundation for IntelliFile. Satisfies the five required
behaviors (system self-profile, structured run logging, artifact
registry, manual notes, improvement queue) plus privacy-aware defaults.

Components:
- ObservabilityStore: SQLite-backed store with thread-safe writes
  and JSONL mirrors (runs.jsonl, notes.jsonl) for tailing.
- RunContext: context manager that auto-finishes runs (success on
  clean exit, failed with error_summary on exception).
- redact(): regex-based redaction of sensitive field names (token,
  password, api_key, api_id, api_hash, session, auth, credential,
  private_key) — case-insensitive. Bypassed only when debug=True.
- Repeated failures and warn-or-worse notes auto-bump the
  improvement_queue table (matched by source+title to keep the queue
  small).

Storage layout:
  $XDG_DATA_HOME/intellifile/observability/
    observability.db     (source of truth)
    runs.jsonl           (append-only mirror)
    notes.jsonl          (append-only mirror)

12 unit tests in tests/test_observability.py cover all five behaviors
+ privacy defaults + JSONL mirror. All tests pass.

Also adds docs/OBSERVABILITY.md explaining the API, what it is NOT
(no telemetry, no remote upload, no full issue tracker), and how Omni
could adopt the same shape in a separate PR.

UI surfacing (Activity / Notes / Diagnostics panels) intentionally
deferred to a separate PR per the user's instruction: 'If UI exposure
is too large for one PR, first implement the backend/storage structure
and minimal surfacing, then iterate.'

Reference: ECOSYSTEM_STATE.md in repo-sync-toolkit (PR #1).
@DrAbdulmalek
DrAbdulmalek merged commit 3f9c85f into main Oct 7, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant