Skip to content

feat: shareable URLs, inline summary mode, risk score removal, and MCP expansion - #29

Merged
liamadale merged 9 commits into
mainfrom
dev
Apr 23, 2026
Merged

liamadale merged 9 commits into
mainfrom
dev

Conversation

@liamadale

Copy link
Copy Markdown
Owner

Summary

This PR promotes the current dev state to main, covering several discrete improvements developed since the last release:

  • Shareable URL generation — users can now generate a URL that encodes the current view state (filters, time range, protocol, sensor, etc.) for sharing or bookmarking. Backed by src/utils/share_url.py with full test coverage.
  • Inline summary mode and grouped aggregation — the OpenSearch web UI gains a new inline summary display and a grouped aggregation type, giving analysts a faster at-a-glance view without opening a modal.
  • Risk score removed — risk score fields have been stripped from all Zeek protocol modules, the web UI browse modal, and the dashboard aggregations. The unified summary_modal.html replaces the previous per-protocol modal templates.
  • MCP OpenSearch server expanded — additional tools added to the OpenSearch MCP server.
  • Weird module field mapping fix — field names corrected for Malcolm ECS remapping.
  • CI: dev branch advisory mode — CI now runs on dev pushes with all checks in advisory (non-blocking) mode.
  • Suricata filters untrackedfalse_positives.yaml is no longer committed; an example-filters.yaml is provided instead and .gitignore updated accordingly.

Test plan

  • Shareable URL round-trips correctly for all filter combinations (tests/test_share_url.py)
  • Inline summary and grouped aggregation render correctly in the OpenSearch web UI
  • No risk score fields appear in any module output, browse modal, or dashboard
  • Weird log entries parse correctly against the updated ECS field mappings
  • CI passes (ruff, pytest, bandit, pip-audit)

🤖 Generated with Claude Code

- Push to dev: all checks run as advisory (continue-on-error)
- Push to main / PR to main: pytest and ruff check remain blocking
- Supports two-branch workflow (dev → PR → main)
Users can now copy a permanent PISCES link or a shortened OpenSearch
Dashboards Discover link directly from the toolbar share dropdown.
The share system is also exposed as a `build_share_urls` MCP tool.

- `src/utils/share_url.py` — standalone `ShareContext` dataclass, minimal
  Rison encoder, KQL builder, `build_pisces_url()`, `build_dashboards_path()`,
  and `shorten_dashboards_url()` (falls back to long URL when Malcolm's
  shorten API is unavailable)
- `tests/test_share_url.py` — full unit coverage of the above
- `GET /api/share` endpoint resolves the current search context to absolute
  timestamps and returns both PISCES and Dashboards URLs as JSON
- `build_search_params_from_request()` now accepts `from`/`to` query params
  so shared links land on the exact same time window
- `build_base_query()` and `run_query()` propagate `time_from`/`time_to`
  through to the OpenSearch query, overriding relative ranges
- Toolbar gains a share dropdown (Copy PISCES link, Copy Dashboards link,
  Open in Dashboards) and a compact/full timestamp toggle with click-to-copy
- Page headers display a human-readable time-window badge
- `mcp/opensearch/server.py` exposes `build_share_urls` for AI assistant use
…gnore

filters/suricata/false_positives.yaml was committed earlier but contains
environment-specific suppression rules that may include sensitive IPs.
Remove it from tracking via git rm --cached and extend the gitignore
pattern to cover filters/suricata/*.yaml going forward.

Also adds the missing !filters/notices/example-*.yaml exception that was
absent from the gitignore (matching the convention already in place for
ips/, signatures/, composite/, and ports/).

Adds filters/suricata/.gitkeep and filters/suricata/example-filters.yaml
so the directory structure is preserved in the repo without leaking
analyst-specific filter content.
Remove event.risk_score and event.risk_score_norm from all 24 Zeek
modules: SOURCE_FIELDS, parse_hit() return dicts, DETAIL_FIELDS, and
WEB_COLUMNS.

In base.py, drop the min_risk_score parameter from build_base_query()
and the risk-score carry-forward logic in deduplicate_zeek().

Add WEB_ICON, EXTRA_PARAMS, SUMMARY_FIELD, and SUMMARY_PARAM class
attributes to ZeekModule so protocol metadata lives on the module
classes instead of parallel dicts in the web app.

Remove capture_loss and dpd modules and the diagnostic category.
Replace separate notice and suricata summary modals with a single
generic browse modal driven by SUMMARY_FIELD/SUMMARY_PARAM class
attributes. Merge /api/notice/summary and /api/suricata/summary into
/api/summary/<log_type>.

Replace the hardcoded proto_icons dict and MODULE_PARAM_KEYS with
reads from ZeekModule.WEB_ICON and ZeekModule.EXTRA_PARAMS.

Remove the min_risk_score filter input, risk score table columns,
risk bar CSS, and the diagnostic category from the sidebar.

Delete macros.html, notice_summary.html, suricata_summary.html.
Add summary_modal.html as the unified replacement.
Remove risk_score and risk_score_norm from test fixtures and drop
test_risk_score and test_dedup_carries_highest_risk_score tests.
Malcolm's Logstash pipeline remaps Zeek weird.log fields to ECS:
zeek.weird.name is moved to rule.name (as an array), and
zeek.weird.peer is removed in favour of zeek.weird.source.

- Use _first() on rule.name array for parse_hit()
- Update FETCH_FIELDS, build_extra_must(), and DETAIL_FIELDS
- Add SUMMARY_TYPE = "flat" class attribute to ZeekModule base class
  to distinguish flat vs grouped aggregation strategies
… type

Log views for summary-capable modules now land on an aggregated view
by default instead of running the full raw query upfront. Users click
a row to drill into raw hits, and a "Back to summary" button restores
the aggregation view — reducing unnecessary query load for noisy log
types.

Extends /api/summary with:
- "grouped" SUMMARY_TYPE: scripted prefix bucketing (for Suricata-style
  rule names) with nested severity breakdown and top-rules sub-agg
- inline=1 flag: renders summary_inline.html or summary_grouped.html
  directly into the page rather than the browse modal

New templates:
- partials/summary_inline.html — flat clickable bucket list for page embed
- partials/summary_grouped.html — prefix-grouped view with severity dots,
  pills, and expandable nested rule rows
…y styles

Three CSS changes:
- Make #table-container a flex column so it fills remaining viewport
  height, keeping the pagination bar visible without scrolling off-screen
- Add #sp-info to the pager-info min-width rule (summary pager label)
- Add severity dot, pill, and bar styles for the grouped summary view
  (sev-dot, sev-pill, sev-bar-* classes keyed on severity 1/2/3)
@liamadale
liamadale merged commit 16ff761 into main Apr 23, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant