Commit b644bc0
authored
docs: comprehensive documentation overhaul for analyst onboarding (#30)
* ci: add dev branch to CI triggers with advisory mode
- Push to dev: all checks run as advisory (continue-on-error)
- Push to main / PR to main: pytest and ruff check remain blocking
- Supports two-branch workflow (dev → PR → main)
* feat(opensearch-web): add shareable URL generation for PISCES views
Users can now copy a permanent PISCES link or a shortened OpenSearch
Dashboards Discover link directly from the toolbar share dropdown.
The share system is also exposed as a `build_share_urls` MCP tool.
- `src/utils/share_url.py` — standalone `ShareContext` dataclass, minimal
Rison encoder, KQL builder, `build_pisces_url()`, `build_dashboards_path()`,
and `shorten_dashboards_url()` (falls back to long URL when Malcolm's
shorten API is unavailable)
- `tests/test_share_url.py` — full unit coverage of the above
- `GET /api/share` endpoint resolves the current search context to absolute
timestamps and returns both PISCES and Dashboards URLs as JSON
- `build_search_params_from_request()` now accepts `from`/`to` query params
so shared links land on the exact same time window
- `build_base_query()` and `run_query()` propagate `time_from`/`time_to`
through to the OpenSearch query, overriding relative ranges
- Toolbar gains a share dropdown (Copy PISCES link, Copy Dashboards link,
Open in Dashboards) and a compact/full timestamp toggle with click-to-copy
- Page headers display a human-readable time-window badge
- `mcp/opensearch/server.py` exposes `build_share_urls` for AI assistant use
* chore(filters): untrack suricata false_positives.yaml and extend gitignore
filters/suricata/false_positives.yaml was committed earlier but contains
environment-specific suppression rules that may include sensitive IPs.
Remove it from tracking via git rm --cached and extend the gitignore
pattern to cover filters/suricata/*.yaml going forward.
Also adds the missing !filters/notices/example-*.yaml exception that was
absent from the gitignore (matching the convention already in place for
ips/, signatures/, composite/, and ports/).
Adds filters/suricata/.gitkeep and filters/suricata/example-filters.yaml
so the directory structure is preserved in the repo without leaking
analyst-specific filter content.
* refactor: remove risk score fields and add class-level protocol metadata
Remove event.risk_score and event.risk_score_norm from all 24 Zeek
modules: SOURCE_FIELDS, parse_hit() return dicts, DETAIL_FIELDS, and
WEB_COLUMNS.
In base.py, drop the min_risk_score parameter from build_base_query()
and the risk-score carry-forward logic in deduplicate_zeek().
Add WEB_ICON, EXTRA_PARAMS, SUMMARY_FIELD, and SUMMARY_PARAM class
attributes to ZeekModule so protocol metadata lives on the module
classes instead of parallel dicts in the web app.
Remove capture_loss and dpd modules and the diagnostic category.
* refactor: unify browse modal and remove risk score from web UI
Replace separate notice and suricata summary modals with a single
generic browse modal driven by SUMMARY_FIELD/SUMMARY_PARAM class
attributes. Merge /api/notice/summary and /api/suricata/summary into
/api/summary/<log_type>.
Replace the hardcoded proto_icons dict and MODULE_PARAM_KEYS with
reads from ZeekModule.WEB_ICON and ZeekModule.EXTRA_PARAMS.
Remove the min_risk_score filter input, risk score table columns,
risk bar CSS, and the diagnostic category from the sidebar.
Delete macros.html, notice_summary.html, suricata_summary.html.
Add summary_modal.html as the unified replacement.
* test: update tests for risk score removal
Remove risk_score and risk_score_norm from test fixtures and drop
test_risk_score and test_dedup_carries_highest_risk_score tests.
* fix(weird): update field mappings for Malcolm ECS remapping
Malcolm's Logstash pipeline remaps Zeek weird.log fields to ECS:
zeek.weird.name is moved to rule.name (as an array), and
zeek.weird.peer is removed in favour of zeek.weird.source.
- Use _first() on rule.name array for parse_hit()
- Update FETCH_FIELDS, build_extra_must(), and DETAIL_FIELDS
- Add SUMMARY_TYPE = "flat" class attribute to ZeekModule base class
to distinguish flat vs grouped aggregation strategies
* feat(opensearch-web): add inline summary mode and grouped aggregation type
Log views for summary-capable modules now land on an aggregated view
by default instead of running the full raw query upfront. Users click
a row to drill into raw hits, and a "Back to summary" button restores
the aggregation view — reducing unnecessary query load for noisy log
types.
Extends /api/summary with:
- "grouped" SUMMARY_TYPE: scripted prefix bucketing (for Suricata-style
rule names) with nested severity breakdown and top-rules sub-agg
- inline=1 flag: renders summary_inline.html or summary_grouped.html
directly into the page rather than the browse modal
New templates:
- partials/summary_inline.html — flat clickable bucket list for page embed
- partials/summary_grouped.html — prefix-grouped view with severity dots,
pills, and expandable nested rule rows
* style(opensearch-web): fix pagination overflow and add grouped summary styles
Three CSS changes:
- Make #table-container a flex column so it fills remaining viewport
height, keeping the pagination bar visible without scrolling off-screen
- Add #sp-info to the pager-info min-width rule (summary pager label)
- Add severity dot, pill, and bar styles for the grouped summary view
(sev-dot, sev-pill, sev-bar-* classes keyed on severity 1/2/3)
* feat(mantis): add student activity reporting script
Adds student_activity.py to report per-student ticket creation and
note counts from the Mantis offline index. Supports offline and live
modes, --student filtering with interactive disambiguation, a detailed
single-student view showing ticket titles and links, and a --graph flag
that renders a terminal line graph of submissions over time.
Adds plotext as a dependency to power the terminal graph output.
* test(security): replace real sensor names and IPs in test fixtures
Real sensor hostnames (hedgehog-bonney-lake, hedgehog-puyallup) and
public IP addresses sourced from actual traffic and ticket data were
present in test fixtures, creating a potential disclosure risk given
the repository is public. Replaced with generic placeholder names and
RFC 5737 TEST-NET addresses (198.51.100.x, 203.0.113.x) that are
reserved for documentation and will never route to real hosts.
* chore(deps): upgrade five packages to resolve pip-audit CVEs
Bumped minimum version pins to clear vulnerabilities flagged by
pip-audit and OSSF Scorecard:
- cryptography 46.0.5 → 46.0.7 (patches 2 CVEs)
- pygments 2.19.2 → 2.20.0
- pytest 9.0.2 → 9.0.3
- python-multipart 0.0.22 → 0.0.26
- requests 2.32.5 → 2.33.1
All 450 tests pass after the upgrade.
* ci(dependabot): auto-merge patch-level dependency updates
Adds a GitHub Actions workflow that enables squash auto-merge for
Dependabot PRs when the update type is semver-patch. Minor and major
updates are left for manual review. Uses the official
dependabot/fetch-metadata action to inspect the update type before
acting.
* docs: rewrite README and flesh out CONTRIBUTING.md
README is restructured around analyst-facing use cases rather than
internal architecture. Removes the project-structure tree and
verbose feature descriptions in favour of a grouped docs table,
screenshots, and direct links to the setup and workflow guides.
CONTRIBUTING.md gains the full branching model, commit convention
table, PR expectations, CI requirements, and a section on AI
tooling so external contributors know what is expected.
* docs: add five new analyst-facing guides
- getting-started.md — install, configure, and launch the toolkit
on Ubuntu, replacing the install section that was in README
- getting-started-mcp.md — MCP server setup with per-client configs
for Claude Code, kiro-cli, gemini-cli, and codex-cli
- vm-setup.md — Ubuntu VM creation and OpenVPN cert transfer via
python http.server for cyber range connectivity
- cli-workflow.md — terminal querier walkthrough split out from
workflow.md, which now covers the web UI triage path only
- mantis-threat-model.md — documents mantis_threat_model.py flags,
output files, and conflict resolution, filling a gap left by
the trimmed mantis.md
Updates .secrets.baseline to register placeholder credential strings
in the new MCP config examples as known false positives.
* docs: correct and trim five existing reference docs
- workflow.md — replaced the CLI querier walkthrough with a web UI
triage walkthrough accurate to the actual templates and routes;
CLI content moved to the new cli-workflow.md
- advanced-usage.md — fixed mantis_index.py flag reference;
--classify-stats moved to mantis_threat_model.py, updated
command examples to reflect the split
- filter-schema.md — added analyst callout pointing to the
interactive fp_manager as the preferred authoring approach
- mantis.md — trimmed developer-only classification algorithm
content that belonged in mantis-threat-model.md
- mcp-servers.md — replaced pip with uv, removed macOS/Windows
paths, added pointer to getting-started-mcp.md for setup1 parent dd0a0a1 commit b644bc0
22 files changed
Lines changed: 1711 additions & 865 deletions
File tree
- .github/workflows
- docs
- src/mantis
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
4 | 91 | | |
5 | 92 | | |
6 | 93 | | |
7 | 94 | | |
8 | 95 | | |
9 | 96 | | |
10 | | - | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
11 | 107 | | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
| 108 | + | |
16 | 109 | | |
17 | 110 | | |
18 | 111 | | |
19 | 112 | | |
20 | | - | |
| 113 | + | |
21 | 114 | | |
0 commit comments