Skip to content

Commit b644bc0

Browse files
authored
docs: comprehensive documentation overhaul for analyst onboarding (#30)
* ci: add dev branch to CI triggers with advisory mode - Push to dev: all checks run as advisory (continue-on-error) - Push to main / PR to main: pytest and ruff check remain blocking - Supports two-branch workflow (dev → PR → main) * feat(opensearch-web): add shareable URL generation for PISCES views Users can now copy a permanent PISCES link or a shortened OpenSearch Dashboards Discover link directly from the toolbar share dropdown. The share system is also exposed as a `build_share_urls` MCP tool. - `src/utils/share_url.py` — standalone `ShareContext` dataclass, minimal Rison encoder, KQL builder, `build_pisces_url()`, `build_dashboards_path()`, and `shorten_dashboards_url()` (falls back to long URL when Malcolm's shorten API is unavailable) - `tests/test_share_url.py` — full unit coverage of the above - `GET /api/share` endpoint resolves the current search context to absolute timestamps and returns both PISCES and Dashboards URLs as JSON - `build_search_params_from_request()` now accepts `from`/`to` query params so shared links land on the exact same time window - `build_base_query()` and `run_query()` propagate `time_from`/`time_to` through to the OpenSearch query, overriding relative ranges - Toolbar gains a share dropdown (Copy PISCES link, Copy Dashboards link, Open in Dashboards) and a compact/full timestamp toggle with click-to-copy - Page headers display a human-readable time-window badge - `mcp/opensearch/server.py` exposes `build_share_urls` for AI assistant use * chore(filters): untrack suricata false_positives.yaml and extend gitignore filters/suricata/false_positives.yaml was committed earlier but contains environment-specific suppression rules that may include sensitive IPs. Remove it from tracking via git rm --cached and extend the gitignore pattern to cover filters/suricata/*.yaml going forward. Also adds the missing !filters/notices/example-*.yaml exception that was absent from the gitignore (matching the convention already in place for ips/, signatures/, composite/, and ports/). Adds filters/suricata/.gitkeep and filters/suricata/example-filters.yaml so the directory structure is preserved in the repo without leaking analyst-specific filter content. * refactor: remove risk score fields and add class-level protocol metadata Remove event.risk_score and event.risk_score_norm from all 24 Zeek modules: SOURCE_FIELDS, parse_hit() return dicts, DETAIL_FIELDS, and WEB_COLUMNS. In base.py, drop the min_risk_score parameter from build_base_query() and the risk-score carry-forward logic in deduplicate_zeek(). Add WEB_ICON, EXTRA_PARAMS, SUMMARY_FIELD, and SUMMARY_PARAM class attributes to ZeekModule so protocol metadata lives on the module classes instead of parallel dicts in the web app. Remove capture_loss and dpd modules and the diagnostic category. * refactor: unify browse modal and remove risk score from web UI Replace separate notice and suricata summary modals with a single generic browse modal driven by SUMMARY_FIELD/SUMMARY_PARAM class attributes. Merge /api/notice/summary and /api/suricata/summary into /api/summary/<log_type>. Replace the hardcoded proto_icons dict and MODULE_PARAM_KEYS with reads from ZeekModule.WEB_ICON and ZeekModule.EXTRA_PARAMS. Remove the min_risk_score filter input, risk score table columns, risk bar CSS, and the diagnostic category from the sidebar. Delete macros.html, notice_summary.html, suricata_summary.html. Add summary_modal.html as the unified replacement. * test: update tests for risk score removal Remove risk_score and risk_score_norm from test fixtures and drop test_risk_score and test_dedup_carries_highest_risk_score tests. * fix(weird): update field mappings for Malcolm ECS remapping Malcolm's Logstash pipeline remaps Zeek weird.log fields to ECS: zeek.weird.name is moved to rule.name (as an array), and zeek.weird.peer is removed in favour of zeek.weird.source. - Use _first() on rule.name array for parse_hit() - Update FETCH_FIELDS, build_extra_must(), and DETAIL_FIELDS - Add SUMMARY_TYPE = "flat" class attribute to ZeekModule base class to distinguish flat vs grouped aggregation strategies * feat(opensearch-web): add inline summary mode and grouped aggregation type Log views for summary-capable modules now land on an aggregated view by default instead of running the full raw query upfront. Users click a row to drill into raw hits, and a "Back to summary" button restores the aggregation view — reducing unnecessary query load for noisy log types. Extends /api/summary with: - "grouped" SUMMARY_TYPE: scripted prefix bucketing (for Suricata-style rule names) with nested severity breakdown and top-rules sub-agg - inline=1 flag: renders summary_inline.html or summary_grouped.html directly into the page rather than the browse modal New templates: - partials/summary_inline.html — flat clickable bucket list for page embed - partials/summary_grouped.html — prefix-grouped view with severity dots, pills, and expandable nested rule rows * style(opensearch-web): fix pagination overflow and add grouped summary styles Three CSS changes: - Make #table-container a flex column so it fills remaining viewport height, keeping the pagination bar visible without scrolling off-screen - Add #sp-info to the pager-info min-width rule (summary pager label) - Add severity dot, pill, and bar styles for the grouped summary view (sev-dot, sev-pill, sev-bar-* classes keyed on severity 1/2/3) * feat(mantis): add student activity reporting script Adds student_activity.py to report per-student ticket creation and note counts from the Mantis offline index. Supports offline and live modes, --student filtering with interactive disambiguation, a detailed single-student view showing ticket titles and links, and a --graph flag that renders a terminal line graph of submissions over time. Adds plotext as a dependency to power the terminal graph output. * test(security): replace real sensor names and IPs in test fixtures Real sensor hostnames (hedgehog-bonney-lake, hedgehog-puyallup) and public IP addresses sourced from actual traffic and ticket data were present in test fixtures, creating a potential disclosure risk given the repository is public. Replaced with generic placeholder names and RFC 5737 TEST-NET addresses (198.51.100.x, 203.0.113.x) that are reserved for documentation and will never route to real hosts. * chore(deps): upgrade five packages to resolve pip-audit CVEs Bumped minimum version pins to clear vulnerabilities flagged by pip-audit and OSSF Scorecard: - cryptography 46.0.5 → 46.0.7 (patches 2 CVEs) - pygments 2.19.2 → 2.20.0 - pytest 9.0.2 → 9.0.3 - python-multipart 0.0.22 → 0.0.26 - requests 2.32.5 → 2.33.1 All 450 tests pass after the upgrade. * ci(dependabot): auto-merge patch-level dependency updates Adds a GitHub Actions workflow that enables squash auto-merge for Dependabot PRs when the update type is semver-patch. Minor and major updates are left for manual review. Uses the official dependabot/fetch-metadata action to inspect the update type before acting. * docs: rewrite README and flesh out CONTRIBUTING.md README is restructured around analyst-facing use cases rather than internal architecture. Removes the project-structure tree and verbose feature descriptions in favour of a grouped docs table, screenshots, and direct links to the setup and workflow guides. CONTRIBUTING.md gains the full branching model, commit convention table, PR expectations, CI requirements, and a section on AI tooling so external contributors know what is expected. * docs: add five new analyst-facing guides - getting-started.md — install, configure, and launch the toolkit on Ubuntu, replacing the install section that was in README - getting-started-mcp.md — MCP server setup with per-client configs for Claude Code, kiro-cli, gemini-cli, and codex-cli - vm-setup.md — Ubuntu VM creation and OpenVPN cert transfer via python http.server for cyber range connectivity - cli-workflow.md — terminal querier walkthrough split out from workflow.md, which now covers the web UI triage path only - mantis-threat-model.md — documents mantis_threat_model.py flags, output files, and conflict resolution, filling a gap left by the trimmed mantis.md Updates .secrets.baseline to register placeholder credential strings in the new MCP config examples as known false positives. * docs: correct and trim five existing reference docs - workflow.md — replaced the CLI querier walkthrough with a web UI triage walkthrough accurate to the actual templates and routes; CLI content moved to the new cli-workflow.md - advanced-usage.md — fixed mantis_index.py flag reference; --classify-stats moved to mantis_threat_model.py, updated command examples to reflect the split - filter-schema.md — added analyst callout pointing to the interactive fp_manager as the preferred authoring approach - mantis.md — trimmed developer-only classification algorithm content that belonged in mantis-threat-model.md - mcp-servers.md — replaced pip with uv, removed macOS/Windows paths, added pointer to getting-started-mcp.md for setup
1 parent dd0a0a1 commit b644bc0

22 files changed

Lines changed: 1711 additions & 865 deletions
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
name: Dependabot Auto-Merge
2+
3+
on: pull_request
4+
5+
permissions:
6+
contents: write
7+
pull-requests: write
8+
9+
jobs:
10+
auto-merge:
11+
runs-on: ubuntu-latest
12+
if: github.actor == 'dependabot[bot]'
13+
steps:
14+
- name: Fetch Dependabot metadata
15+
id: meta
16+
uses: dependabot/fetch-metadata@v2
17+
18+
- name: Enable auto-merge for patch updates
19+
if: steps.meta.outputs.update-type == 'version-update:semver-patch'
20+
run: gh pr merge --auto --squash "$PR_URL"
21+
env:
22+
PR_URL: ${{ github.event.pull_request.html_url }}
23+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.secrets.baseline

Lines changed: 23 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

CONTRIBUTING.md

Lines changed: 100 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,114 @@
11
# Contributing
22

3-
## Development Setup
3+
## Branching model
4+
5+
This project uses a three-tier branching model:
6+
7+
- **`main`** — stable release branch. Direct pushes are blocked; all changes arrive via PR from `dev`.
8+
- **`dev`** — integration branch. Feature branches are merged here first. CI runs as advisory.
9+
- **`feat/`, `fix/`, `chore/` branches** — short-lived branches for individual tasks, branched off `dev` and merged back via PR.
10+
11+
### Day-to-day workflow
12+
13+
```bash
14+
# Start from up-to-date dev
15+
git checkout dev && git pull origin dev
16+
17+
# Create a branch for your task
18+
git checkout -b feat/my-feature
19+
20+
# Work, commit, push
21+
git add <files>
22+
git commit -m "feat: description of change"
23+
git push origin feat/my-feature
24+
25+
# Open a PR into dev
26+
gh pr create --base dev --title "feat: description" --body "..."
27+
```
28+
29+
Once reviewed and merged into `dev`, changes are promoted to `main` via a separate PR when `dev` is in a stable state.
30+
31+
---
32+
33+
## Commit messages
34+
35+
This project uses [Conventional Commits](https://www.conventionalcommits.org/). Use one of these types:
36+
37+
| Type | When to use |
38+
|---|---|
39+
| `feat` | New feature or capability |
40+
| `fix` | Bug fix |
41+
| `chore` | Maintenance, dependency updates, config changes |
42+
| `docs` | Documentation only |
43+
| `test` | Adding or updating tests |
44+
| `refactor` | Code restructuring with no behaviour change |
45+
| `style` | Formatting changes only |
46+
| `ci` | CI/CD pipeline changes |
47+
48+
For changes tied to a user report or GitHub issue, add trailers:
49+
50+
```bash
51+
git commit -m "fix: description" \
52+
--trailer "Reported-by:Name" \
53+
--trailer "Github-Issue:#42"
54+
```
55+
56+
Keep commits focused — one logical change per commit makes review and revert easier.
57+
58+
---
59+
60+
## Pull requests
61+
62+
- **Target `dev`**, not `main`, for all feature and fix PRs
63+
- Keep PRs small and focused — a PR that does one thing is easier to review than one that does five
64+
- Write a description that explains *why* the change is needed, not just what changed
65+
- Add `liamadale` as a reviewer on all PRs
66+
- Do not merge your own PR without review unless it is a trivial docs or config fix
67+
68+
---
69+
70+
## CI and code quality
71+
72+
Before opening a PR, make sure these pass locally:
73+
74+
```bash
75+
uv run pytest tests/ -v # all tests pass
76+
uv run ruff check src/ apps/ mcp/ tests/ *.py # no lint errors
77+
uv run ruff format src/ apps/ mcp/ tests/ *.py # code formatted
78+
uv run pre-commit run --all-files # all hooks pass
79+
```
80+
81+
CI runs automatically on PR open and push. Checks that block merging to `main`:
82+
83+
- `pytest` — all tests must pass
84+
- `ruff check` — no lint errors
85+
86+
Format, pip-audit, and bandit run as advisory. Fix them if you can, but they will not block merge to `dev`.
87+
88+
---
89+
90+
## Development setup
491

592
1. Install [uv](https://github.com/astral-sh/uv)
693
2. `uv sync --extra mcp --group dev` (or `uv sync --extra all --group dev` for everything)
794
3. `uv run pre-commit install`
895
4. `cp .env.example .env` and fill in credentials
996

10-
## Before Submitting a PR
97+
---
98+
99+
## Using AI tooling
100+
101+
AI coding assistants are encouraged — this project was built with them. A few expectations:
102+
103+
- **Review all AI-generated code before committing.** You are responsible for what goes into the PR, regardless of how it was written.
104+
- **Do not commit AI tooling config files** (`CLAUDE.md`, `AGENT.md`, `.kiro/`, etc.). These are gitignored intentionally — they may contain personal paths or sensitive context.
105+
- **Test AI-generated changes.** Run the relevant tests and check the output manually. AI tools can produce plausible-looking code that doesn't actually work.
106+
- **Keep commit messages accurate.** Describe what the code does, not what the AI was asked to do.
11107

12-
- `uv run pytest tests/ -v` — all tests pass
13-
- `uv run ruff check src/ apps/ mcp/ tests/ *.py` — no lint errors
14-
- `uv run ruff format src/ apps/ mcp/ tests/ *.py` — code formatted
15-
- `uv run pre-commit run --all-files` — all hooks pass
108+
---
16109

17110
## Security
18111

19112
- Never commit credentials, API keys, or tokens
20-
- Use environment variables for all secrets
113+
- Use environment variables for all secrets (see `.env.example`)
21114
- See [SECURITY.md](SECURITY.md) for vulnerability reporting

0 commit comments

Comments
 (0)