Skip to content

gateway: answer a peer's GET /faults without fanning out again - #721

Merged
mfaferek93 merged 4 commits into
mainfrom
fix/faults-list-honour-no-fan-out
Oct 25, 2026
Merged

mfaferek93 merged 4 commits into
mainfrom
fix/faults-list-honour-no-fan-out

Conversation

@mfaferek93

Copy link
Copy Markdown
Collaborator

Problem

GET /faults (list_all_faults) fans out to peers through AggregationManager::fan_out_get, which marks the request with X-Medkit-No-Fan-Out. The handler on the receiving side never checked that header, so it fanned out again. The other collection handlers already honour it through merge_peer_items, only this path called fan_out_get directly.

With two gateways that aggregate each other (A lists B, B lists A) a single GET /faults bounces between them. Each hop holds a worker while it waits, so both gateways run out of workers. In the test below A then answers 200 with an empty item list, and every peer call logs Failed to connect to peer.

Fix

A request carrying X-Medkit-No-Fan-Out is answered from the local fault manager only. A request without it fans out as before.

Tests

New test_mutual_peer_faults (two gateways on two domains, each the other's static peer, B runs the faulty lidar):

  • test_no_fan_out_request_answers_local_faults_only: A with the header returns only its own faults.
  • test_mutual_peers_answer_without_bouncing: GET /faults on A returns LIDAR_RANGE_INVALID exactly once, not partial, faster than one peer timeout.

Without the fix both cases fail: the header request returns B's faults, and the plain request returns an empty list after the bounce. With the fix: 3 of 3 runs pass. test_cross_ecu_fanout, test_peer_aggregation, test_daisy_chain_aggregation, test_aggregator_fault_stream and test_fault_handlers (38) still pass. clang-format-18 and ament_flake8 clean.

Note: #700 and #680 also touch fault_handlers.cpp. This change is one small hunk in list_all_faults, so a rebase either way should be trivial.

Copilot AI balanced review requested due to automatic review settings October 11, 2026 19:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

OpenAPI and documentation remain inconsistent with the new behavior, and required test conventions are unmet.

2 open findings
What changed in this PR

Prevents recursive global fault fan-out between mutually peered gateways.

Changes:

  • Suppresses peer fan-out when the request carries X-Medkit-No-Fan-Out.
  • Adds a two-gateway regression test.
  • Registers the test with two isolated ROS domains.
File Description
fault_handlers.cpp Adds global fault fan-out suppression.
test_mutual_peer_faults.test.py Tests mutual-peer loop prevention.
CMakeLists.txt Registers the two-domain test.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@bburda bburda left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three places outside this diff still describe the old behaviour:

  • docs/api/rest.rst, the X-Medkit-No-Fan-Out entry, says the global GET /api/v1/faults neither declares nor honours the header.
  • src/ros2_medkit_gateway/design/aggregation.rst, the warning in the fan-out section, says list_all_faults never reads the header and two mutual peers recurse.
  • The comment in fan_out_aware() in src/ros2_medkit_gateway/src/core/openapi/route_registry.cpp says the same.

Could you update them, and add .fan_out_aware() to the GET /faults registration in rest_server.cpp so the OpenAPI document declares the header?

mfaferek93 and others added 4 commits October 25, 2026 19:10
list_all_faults ignored X-Medkit-No-Fan-Out and fanned out on a request
that came from an aggregating peer. Two gateways that aggregate each
other bounced the request until they ran out of workers and answered
empty. A request with the header is now answered from the local store.
A peer request on GET /faults now carries X-Medkit-Fan-Out-Visited with the
instance ids already asked, so B still asks C in A -> B -> C and never asks A
back. Peers learn ids from /health, nested partial answers surface as
<peer>/<its peer>, the query string reaches the peers and count is recounted.
Three gateways, A <-> B -> C, on GatewayTestCase with scaled budgets. Each
fault is told apart per gateway, and the cases fail on main, on the local-only
fix and on a fan-out that still sends the request back round the loop.
@mfaferek93
mfaferek93 force-pushed the fix/faults-list-honour-no-fan-out branch from 819e4d8 to 3f84d60 Compare October 25, 2026 18:42
@mfaferek93
mfaferek93 merged commit 44d577e into main Oct 25, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants