diff --git a/docs/config/action-status-bridge.rst b/docs/config/action-status-bridge.rst index 4a7bf0225..24de93c28 100644 --- a/docs/config/action-status-bridge.rst +++ b/docs/config/action-status-bridge.rst @@ -88,6 +88,8 @@ Parameters The bridge also declares ``fault_reporter.local_filtering.enabled`` (default ``false``) on its own node. See :doc:`fault-manager` for what the reporter-side filter does. +Its reports are also confirmed and sent again when the manager does not answer; see the +``fault_reporter.delivery.*`` parameters in the ``ros2_medkit_fault_reporter`` README. See Also -------- diff --git a/docs/tutorials/plugin-system.rst b/docs/tutorials/plugin-system.rst index 7d6683c99..1fb338158 100644 --- a/docs/tutorials/plugin-system.rst +++ b/docs/tutorials/plugin-system.rst @@ -252,7 +252,7 @@ Plugin Lifecycle 3. ``create_plugin()`` factory function creates the plugin instance 4. Provider interfaces are queried via ``get_update_provider()`` / ``get_introspection_provider()`` / ``get_log_provider()`` / ``get_script_provider()`` / ``get_data_provider()`` / ``get_operation_provider()`` / ``get_fault_provider()`` 5. ``configure()`` is called with per-plugin JSON config -6. ``set_context()`` provides ``PluginContext`` with ROS 2 node, entity cache, faults, and HTTP utilities +6. ``set_context()`` provides ``PluginContext`` with entity cache, faults, locks, and HTTP utilities, and the ROS 2 node when the host has one 7. ``get_routes()`` returns custom REST endpoint definitions as ``vector`` 8. Runtime: subsystem managers call provider methods as needed 9. ``shutdown()`` is called before the plugin is destroyed @@ -263,7 +263,7 @@ PluginContext After ``configure()``, the gateway calls ``set_context()`` with a ``PluginContext`` reference providing access to gateway data and utilities: -- ``node()`` - ROS 2 node pointer for service clients, timers, and parameter access. +- ``node()`` - ROS 2 node pointer for service clients, timers, and parameter access. Null when the host is a program that hosts the core without a ROS 2 node (see ``CorePluginContext`` in the gateway README). A plugin that needs ROS checks it and declines to start; a plugin that needs no ROS never calls it. ``as_ros_plugin_context()`` is valid on every host. .. warning:: diff --git a/scripts/check_cores_without_ros/CMakeLists.txt b/scripts/check_cores_without_ros/CMakeLists.txt index 68c5f26df..107d82dc9 100644 --- a/scripts/check_cores_without_ros/CMakeLists.txt +++ b/scripts/check_cores_without_ros/CMakeLists.txt @@ -24,3 +24,10 @@ include(${CMAKE_CURRENT_LIST_DIR}/../../src/ros2_medkit_gateway/cmake/gateway_co add_executable(check_cores_without_ros main.cpp) target_link_libraries(check_cores_without_ros PRIVATE fault_reporter_core fault_manager_core gateway_core) + +# A plugin with no ROS dependency, loaded by the check through the core PluginManager. +add_library(check_context_plugin MODULE context_plugin.cpp) +target_link_libraries(check_context_plugin PRIVATE gateway_core) +target_compile_definitions(check_cores_without_ros PRIVATE + CHECK_CONTEXT_PLUGIN_PATH="$") +add_dependencies(check_cores_without_ros check_context_plugin) diff --git a/scripts/check_cores_without_ros/Dockerfile b/scripts/check_cores_without_ros/Dockerfile index 86f9b338e..c8ed7410a 100644 --- a/scripts/check_cores_without_ros/Dockerfile +++ b/scripts/check_cores_without_ros/Dockerfile @@ -9,5 +9,5 @@ COPY src/ros2_medkit_gateway src/ros2_medkit_gateway COPY scripts/check_cores_without_ros scripts/check_cores_without_ros RUN CXX=g++-9 cmake -S scripts/check_cores_without_ros -B /tmp/b && cmake --build /tmp/b --parallel "$(nproc)" RUN CXX=g++-9 scripts/check_cores_without_ros/hosts/run.sh -RUN ldd /tmp/b/check_cores_without_ros | tee /tmp/ldd.txt && ! grep -q rclcpp /tmp/ldd.txt +RUN ldd /tmp/b/check_cores_without_ros /tmp/b/libcheck_context_plugin.so | tee /tmp/ldd.txt && ! grep -q rclcpp /tmp/ldd.txt CMD ["sh", "-c", "ldd /tmp/b/check_cores_without_ros; /tmp/b/check_cores_without_ros"] diff --git a/scripts/check_cores_without_ros/README.md b/scripts/check_cores_without_ros/README.md index 6bbe4fba1..512445642 100644 --- a/scripts/check_cores_without_ros/README.md +++ b/scripts/check_cores_without_ros/README.md @@ -31,6 +31,17 @@ not installed, so the PostgreSQL backend is not built. trigger's event stream, the fault SSE stream gets the confirmation and the capture holds a frame for it. A data trigger on a plugin-served item does not fire while the value is below its range, fires once when the value enters it, and the item is read again and again by the core's polling source. A fourth trigger with `max_triggers = 3` answers 503. A limit below 1 or past `INT_MAX` and a polling interval outside 100-60000 ms fall back with a notice. +- Plugin context: `context_plugin.cpp` builds as a shared object without ROS. The core `PluginManager` loads it, a + `make_core_plugin_context()` context goes to it, and its routes and `DataProvider` answer over HTTP. The plugin gets a null + `node()`, reads the host's entities, registers a capability, takes, checks and releases a lock, lists faults, reaches the + notifier and the condition registry, and its entity change notification reaches the host's callback. A route on an unknown + entity answers 404. +- Fault-trigger engine: a host built by `make_fault_trigger_host()` over that plugin. A rule created over HTTP reports nothing + while the plugin's value is below the threshold. When the value crosses it, the fault goes through the host's fault manager + into the host's single event source, and the fault SSE stream and the freeze-frame capture (which holds the value that crossed + the threshold) both get the confirmation. When the value comes back the engine clears the fault. A rule on an unknown app + answers 404, a disabled configuration or a host with no plugin builds no engine, and a polling interval below 50 ms is raised + and one past `INT_MAX` is capped, each with a notice. - Bulk data: a store built by `make_bulk_data_store()` takes an upload over HTTP, lists it, returns it byte-equal and deletes it; an upload past `max_upload_size` is refused; the categories do not contain `rosbags` when the fault manager serves none. - `GET /` capabilities: `bulk_data` and `cyclic_subscriptions` are false for services without a store, rosbags, transport or @@ -59,8 +70,8 @@ not installed, so the PostgreSQL backend is not built. `hosts/` holds two more plain-CMake hosts that build `gateway_core` with their own `httplib::httplib` target and compile their own code as C++14: one with a version 0.26 target whose include directory has no `httplib.h` (the vendored header must be used), one with an unversioned non-global target (CMake 3.16 cannot alias it). `hosts/run.sh` builds both and runs in the Dockerfile and in the CI job. -The Dockerfile also runs `ldd` on the binary and fails when `rclcpp` appears. +The Dockerfile also runs `ldd` on the binary and on the plugin and fails when `rclcpp` appears. The script sends only the needed source directories to `docker build` as the build -context. CI runs the same CMake steps in an `ubuntu:20.04` container (job `cores-without-ros`). +context. The build runs with `-q`; when it fails, the script repeats it with the full log, so the compiler message is printed. CI runs the same CMake steps in an `ubuntu:20.04` container (job `cores-without-ros`). `COLCON_IGNORE` keeps colcon from treating this directory as a package. diff --git a/scripts/check_cores_without_ros/context_plugin.cpp b/scripts/check_cores_without_ros/context_plugin.cpp new file mode 100644 index 000000000..45bbdd588 --- /dev/null +++ b/scripts/check_cores_without_ros/context_plugin.cpp @@ -0,0 +1,131 @@ +// Copyright 2026 bburda +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// A plugin with no ROS dependency. The check loads it as a shared object through the core PluginManager. + +#include +#include +#include + +#include + +#include "ros2_medkit_gateway/core/plugins/gateway_plugin.hpp" +#include "ros2_medkit_gateway/core/plugins/plugin_context.hpp" +#include "ros2_medkit_gateway/core/plugins/plugin_types.hpp" +#include "ros2_medkit_gateway/core/providers/data_provider.hpp" +#include "ros2_medkit_gateway/core/resource_change_notifier.hpp" +#include "ros2_medkit_gateway/plugins/ros_plugin_context.hpp" + +using namespace ros2_medkit_gateway; + +namespace { + +class ContextPlugin : public GatewayPlugin, public DataProvider { + public: + std::string name() const override { + return "check_context_plugin"; + } + + void configure(const nlohmann::json & /*config*/) override { + } + + void set_context(PluginContext & context) override { + ctx_ = &context; + // The cast is valid on every host: a host without a ROS node gives a null node(). + ros_ctx_ = as_ros_plugin_context(context); + ctx_->register_capability(SovdEntityType::APP, "x-check-context"); + } + + std::vector get_routes() override { + return { + {"GET", "x-check/context", + [this](const PluginRequest &, PluginResponse & res) { + res.send_json(describe_context()); + }}, + {"GET", "x-check/set", + [this](const PluginRequest & req, PluginResponse & res) { + value_ = std::stoi(req.query_param("value")); + res.send_json({{"value", value_.load()}}); + }}, + {"GET", R"(apps/([^/]+)/x-check-context)", + [this](const PluginRequest & req, PluginResponse & res) { + auto entity = ctx_->validate_entity_for_route(req, res, req.path_param(1)); + if (!entity) { + return; + } + res.send_json({{"entity_id", entity->id}, {"capabilities", ctx_->get_entity_capabilities(entity->id)}}); + }}, + // The route the core reads the values of a plugin-served entity from. + {"GET", R"(apps/([^/]+)/x-plc-data)", + [this](const PluginRequest &, PluginResponse & res) { + res.send_json({{"connected", true}, {"items", items()}}); + }}, + }; + } + + tl::expected list_data(const std::string & /*entity_id*/) override { + return dto::DataListResult{{{"items", items()}}}; + } + tl::expected read_data(const std::string & /*entity_id*/, + const std::string & resource_name) override { + return dto::DataValue{{{"source", "plugin"}, {"name", resource_name}, {"value", value_.load()}}}; + } + tl::expected write_data(const std::string & /*entity_id*/, + const std::string & /*resource_name*/, + const nlohmann::json & /*value*/) override { + return tl::make_unexpected(DataProviderErrorInfo{DataProviderError::ReadOnly, "read only", 405}); + } + + private: + nlohmann::json items() const { + return nlohmann::json::array({{{"id", "speed"}, {"name", "speed"}, {"value", value_.load()}}}); + } + + // What the plugin can reach through the context, as the plugin sees it. + nlohmann::json describe_context() { + nlohmann::json out; + out["node_is_null"] = ros_ctx_ != nullptr && ros_ctx_->node() == nullptr; + const auto entity = ctx_->get_entity("plugin_app"); + out["entity_found"] = entity.has_value() && entity->type == SovdEntityType::APP; + out["snapshot_apps"] = ctx_->get_entity_snapshot().apps.size(); + out["type_capabilities"] = ctx_->get_type_capabilities(SovdEntityType::APP); + out["notifier"] = ctx_->get_resource_change_notifier() != nullptr; + out["conditions"] = ctx_->get_condition_registry() != nullptr; + const auto lock = ctx_->acquire_lock("plugin_app", "plugin_client", {}, 60); + out["lock_acquired"] = lock.has_value(); + out["lock_denies_other_client"] = !ctx_->check_lock("plugin_app", "other_client", "data").allowed; + out["lock_released"] = ctx_->release_lock("plugin_app", "plugin_client").has_value(); + out["faults"] = ctx_->list_all_faults(); + ctx_->notify_entities_changed(EntityChangeScope::full_refresh()); + return out; + } + + PluginContext * ctx_{nullptr}; + RosPluginContext * ros_ctx_{nullptr}; + std::atomic value_{10}; +}; + +} // namespace + +extern "C" GATEWAY_PLUGIN_EXPORT int plugin_api_version() { + return PLUGIN_API_VERSION; +} + +extern "C" GATEWAY_PLUGIN_EXPORT GatewayPlugin * create_plugin() { + return new ContextPlugin(); +} + +extern "C" GATEWAY_PLUGIN_EXPORT DataProvider * get_data_provider(GatewayPlugin * plugin) { + return static_cast(plugin); +} diff --git a/scripts/check_cores_without_ros/main.cpp b/scripts/check_cores_without_ros/main.cpp index 093e996df..57c325552 100644 --- a/scripts/check_cores_without_ros/main.cpp +++ b/scripts/check_cores_without_ros/main.cpp @@ -25,6 +25,7 @@ #include #include #include +#include #include #include #include @@ -52,6 +53,7 @@ #include "ros2_medkit_gateway/core/config.hpp" #include "ros2_medkit_gateway/core/data/topic_data_provider.hpp" #include "ros2_medkit_gateway/core/discovery/entity_discovery.hpp" +#include "ros2_medkit_gateway/core/fault_trigger_host.hpp" #include "ros2_medkit_gateway/core/faults/fault_event_source.hpp" #include "ros2_medkit_gateway/core/faults/freeze_frame_capture.hpp" #include "ros2_medkit_gateway/core/gateway_services.hpp" @@ -70,6 +72,7 @@ #include "ros2_medkit_gateway/core/managers/subscription_manager.hpp" #include "ros2_medkit_gateway/core/managers/trigger_manager.hpp" #include "ros2_medkit_gateway/core/models/thread_safe_entity_cache.hpp" +#include "ros2_medkit_gateway/core/plugins/core_plugin_context.hpp" #include "ros2_medkit_gateway/core/plugins/gateway_plugin.hpp" #include "ros2_medkit_gateway/core/plugins/plugin_manager.hpp" #include "ros2_medkit_gateway/core/providers/data_provider.hpp" @@ -137,9 +140,19 @@ namespace gw = ros2_medkit_gateway; // Minimal transports: every call answers "not supported". class NoFaultTransport : public gw::FaultServiceTransport { public: - gw::FaultResult report_fault(const std::string & /*fault_code*/, uint8_t /*severity*/, - const std::string & /*description*/, const std::string & /*source_id*/) override { - return none(); + gw::FaultResult report_fault(const std::string & fault_code, uint8_t /*severity*/, + const std::string & /*description*/, const std::string & source_id) override { + std::function hook; + { + std::lock_guard lock(detail_mutex_); + hook = on_report_; + } + if (!hook) { + return none(); + } + ++reports; + hook(fault_code, source_id); + return {true, {}, {}, gw::FaultFailure::Declined}; } // Once set, every list answers one fault whose occurrence_count grows with each call. gw::FaultResult list_faults(const std::string & /*source_id*/, bool /*include_prefailed*/, bool /*include_confirmed*/, @@ -176,7 +189,11 @@ class NoFaultTransport : public gw::FaultServiceTransport { return none(); } gw::FaultResult clear_fault(const std::string & /*fault_code*/, bool /*skip_correlation_auto_clear*/) override { - return none(); + if (!on_report_set()) { + return none(); + } + ++clears; + return {true, {}, {}, gw::FaultFailure::Declined}; } gw::FaultResult get_snapshots(const std::string & /*fault_code*/, const std::string & /*topic*/) override { return none(); @@ -196,11 +213,23 @@ class NoFaultTransport : public gw::FaultServiceTransport { bool serves_rosbags() const override { return rosbags.load(); } + // Once set, a report is accepted and `hook` is called with its fault code and source. + void set_on_report(std::function hook) { + std::lock_guard lock(detail_mutex_); + on_report_ = std::move(hook); + } std::atomic reports_faults{false}; std::atomic rosbags{false}; std::atomic list_calls{0}; + std::atomic reports{0}; + std::atomic clears{0}; private: + bool on_report_set() { + std::lock_guard lock(detail_mutex_); + return static_cast(on_report_); + } + std::function on_report_; std::mutex detail_mutex_; std::map detail_sources_; static gw::FaultResult none() { @@ -561,7 +590,7 @@ class MinimalServices : public gw::GatewayServices { return bulk_store_.get(); } gw::FaultTriggerEngine * fault_trigger_engine() override { - return nullptr; + return fault_trigger_engine_; } gw::FreezeFrameSource * freeze_frames() override { return freeze_frames_; @@ -609,6 +638,14 @@ class MinimalServices : public gw::GatewayServices { void set_trigger_manager(gw::TriggerManager * triggers) { triggers_ = triggers; } + void set_fault_trigger_engine(gw::FaultTriggerEngine * engine) { + fault_trigger_engine_ = engine; + } + // An empty plugin manager, for plugins loaded from shared objects. + gw::PluginManager & make_plugin_manager() { + plugins_ = std::make_unique(); + return *plugins_; + } private: gw::ThreadSafeEntityCache cache_; @@ -622,6 +659,7 @@ class MinimalServices : public gw::GatewayServices { gw::LockManager * locks_{nullptr}; gw::ScriptManager * scripts_{nullptr}; gw::TriggerManager * triggers_{nullptr}; + gw::FaultTriggerEngine * fault_trigger_engine_{nullptr}; gw::FaultManager fault_; gw::ConfigurationManager config_; gw::DataAccessManager data_; @@ -1275,6 +1313,187 @@ bool check_triggers() { return ok; } +// A plugin with no ROS dependency, loaded from a shared object through the core PluginManager, gets the core plugin +// context: its routes and data provider answer over HTTP, it reaches the services the context documents and its node is +// null. The fault-trigger engine of the core host reads the plugin's data, reports through the host's fault manager +// into the host's single event source, and the fault SSE stream and the freeze-frame capture both get the confirmation. +bool check_plugin_context_and_fault_triggers() { + bool ok = true; + MinimalServices services; + services.mutable_cache().update_apps({online_app("plugin_app", "Plugin app")}); + gw::LockManager locks(services.entity_cache(), gw::LockConfig{}); + services.set_lock_manager(&locks); + auto & plugins = services.make_plugin_manager(); + const auto loaded = plugins.load_plugins( + {gw::PluginConfig{"check_context_plugin", CHECK_CONTEXT_PLUGIN_PATH, nlohmann::json::object()}}); + ok &= check(loaded == 1, "the core PluginManager loads a plugin built without ROS"); + plugins.configure_plugins(); + plugins.register_entity_ownership("check_context_plugin", {"plugin_app"}); + gw::ResourceChangeNotifier notifier; + gw::ConditionRegistry conditions; + std::atomic notified{0}; + gw::CorePluginContextWiring wiring; + wiring.notifier = [¬ifier] { + return ¬ifier; + }; + wiring.conditions = [&conditions] { + return &conditions; + }; + wiring.on_entities_changed = [¬ified](const gw::EntityChangeScope &) { + ++notified; + }; + auto context = gw::make_core_plugin_context(services, wiring); + ok &= check(context->node() == nullptr, "the core plugin context has no ROS node"); + plugins.set_context(*context); + + ControlledFaultEvents upstream; + auto capture = gw::make_freeze_frame_capture(gw::FreezeFrameCaptureConfig{}, upstream, services.plugin_manager()); + ok &= check(capture != nullptr, "the core builds a freeze-frame capture over the loaded plugin"); + if (!capture) { + return false; + } + services.set_freeze_frames(capture.get()); + services.set_fault_events(&upstream); + auto & transport = services.fault_transport(); + transport.serve_detail("SPEED_HIGH", "plugin_app"); + // A report accepted by the host's fault manager confirms on the host's single event source. + transport.set_on_report([&upstream](const std::string & code, const std::string & source) { + upstream.confirm(code, source); + }); + + gw::FaultTriggerHostConfig engine_config; + engine_config.poll_interval_ms = 50; + gw::FaultTriggerHostWiring engine_wiring; + engine_wiring.cache = &services.entity_cache(); + engine_wiring.plugins = services.plugin_manager(); + engine_wiring.faults = services.fault_manager(); + { + gw::FaultTriggerHostConfig disabled = engine_config; + disabled.enabled = false; + gw::PluginManager no_plugins; + auto without_plugins = engine_wiring; + without_plugins.plugins = &no_plugins; + ok &= check(gw::make_fault_trigger_host(disabled, engine_wiring) == nullptr, + "a disabled configuration builds no fault-trigger engine"); + ok &= check(gw::make_fault_trigger_host(engine_config, without_plugins) == nullptr, + "a host with no plugin loaded builds no fault-trigger engine"); + } + std::vector notices; + auto noticing = engine_wiring; + noticing.warn = [¬ices](const std::string & message) { + notices.push_back(message); + }; + { + gw::FaultTriggerHostConfig low = engine_config; + low.poll_interval_ms = 5; + auto raised = gw::make_fault_trigger_host(low, noticing); + ok &= check(raised && raised->poll_interval_ms() == 50 && notices.size() == 1, + "a polling interval below 50 ms is raised to 50 ms with a notice"); + low.poll_interval_ms = 9000000000LL; + auto capped = gw::make_fault_trigger_host(low, noticing); + ok &= check(capped && capped->poll_interval_ms() == std::numeric_limits::max() && notices.size() == 2, + "a polling interval past what an int holds is capped at INT_MAX with a notice"); + } + auto engine_host = gw::make_fault_trigger_host(engine_config, engine_wiring); + ok &= check(engine_host != nullptr, "the core builds the fault-trigger engine for a host with a plugin"); + if (!engine_host) { + return false; + } + services.set_fault_trigger_engine(&engine_host->engine()); + + RunningServer running(services); + auto client = make_client(running.port()); + + auto described = client.Get("/api/v1/x-check/context"); + const auto reach = described ? nlohmann::json::parse(described->body, nullptr, false) : nlohmann::json(); + ok &= check(described && described->status == 200 && reach.is_object(), "a plugin route answers over HTTP"); + if (reach.is_object()) { + ok &= check(reach.value("node_is_null", false), "the plugin gets a null node from the context"); + ok &= check(reach.value("entity_found", false) && reach.value("snapshot_apps", 0) == 1, + "the plugin reads the host's entities through the context"); + ok &= check(reach.value("type_capabilities", nlohmann::json::array()) == nlohmann::json::array({"x-check-context"}), + "the capability the plugin registered is listed"); + ok &= check(reach.value("notifier", false) && reach.value("conditions", false), + "the plugin reaches the host's change notifier and condition registry"); + ok &= check(reach.value("lock_acquired", false) && reach.value("lock_denies_other_client", false) && + reach.value("lock_released", false), + "the plugin takes, checks and releases a lock on the host's lock manager"); + ok &= check(reach.contains("faults") && reach["faults"].is_object(), "the plugin lists the host's faults"); + } + ok &= check(notified.load() == 1, "an entity change the plugin reports reaches the host's callback"); + auto entity_route = client.Get("/api/v1/apps/plugin_app/x-check-context"); + ok &= check(entity_route && entity_route->status == 200, "an entity-scoped plugin route validates the entity"); + auto unknown_entity = client.Get("/api/v1/apps/no_such_app/x-check-context"); + ok &= check(unknown_entity && unknown_entity->status == 404, "the same route answers 404 for an unknown entity"); + auto data = client.Get("/api/v1/apps/plugin_app/data"); + ok &= check(data && data->status == 200 && data->body.find("speed") != std::string::npos, + "the plugin's data provider answers over HTTP"); + + const nlohmann::json rule = {{"data_name", "speed"}, + {"operator", ">"}, + {"threshold", 50}, + {"fault_code", "SPEED_HIGH"}, + {"severity", "ERROR"}}; + auto unknown_app = client.Post("/api/v1/apps/no_such_app/fault-triggers", rule.dump(), "application/json"); + ok &= check(unknown_app && unknown_app->status == 404, "a rule on an unknown app is refused with 404"); + auto created = client.Post("/api/v1/apps/plugin_app/fault-triggers", rule.dump(), "application/json"); + ok &= check(created && created->status == 201, "a threshold rule is created over HTTP"); + const auto rule_json = created ? nlohmann::json::parse(created->body, nullptr, false) : nlohmann::json(); + const std::string rule_id = rule_json.is_object() ? rule_json.value("id", "") : ""; + std::this_thread::sleep_for(std::chrono::milliseconds(600)); + ok &= check(transport.reports.load() == 0, "a value below the threshold reports no fault"); + + std::atomic sse_seen{false}; + std::atomic stream_done{false}; + std::thread stream([&] { + auto stream_client = make_client(running.port()); + std::string buffer; + stream_client.Get( + "/api/v1/faults/stream", httplib::Headers{}, + [](const httplib::Response &) { + return true; + }, + [&](const char * chunk, size_t len) { + buffer.append(chunk, len); + if (buffer.find("SPEED_HIGH") != std::string::npos) { + sse_seen = true; + } + return !sse_seen && !stream_done; + }); + }); + // The engine reports on every poll while the value stays past the threshold, so the stream sees one after it + // attaches. + client.Get("/api/v1/x-check/set?value=80"); + ok &= check(wait_until([&] { + return transport.reports.load() >= 1; + }), + "a value past the threshold reports the fault through the host's fault manager"); + ok &= check(wait_until([&] { + return sse_seen.load(); + }), + "the fault SSE stream gets the confirmation from the host's single event source"); + stream_done = true; + stream.join(); + ok &= check(wait_until([&] { + return !capture->frames_for("SPEED_HIGH").empty(); + }), + "the freeze-frame capture gets the same confirmation"); + const auto detail = client.Get("/api/v1/apps/plugin_app/faults/SPEED_HIGH"); + const auto frame = detail ? freeze_frame_of(detail->body, "plugin_app") : nlohmann::json(); + ok &= check(!frame.is_null() && frame["x-medkit"]["full_data"].value("speed", -1) == 80, + "the freeze frame holds the value that crossed the threshold"); + client.Get("/api/v1/x-check/set?value=10"); + ok &= check(wait_until([&] { + return transport.clears.load() >= 1; + }), + "the engine clears the fault when the value comes back"); + if (!rule_id.empty()) { + auto removed = client.Delete("/api/v1/apps/plugin_app/fault-triggers/" + rule_id); + ok &= check(removed && removed->status == 204, "the rule is deleted over HTTP"); + } + return ok; +} + } // namespace static int run() { @@ -1856,6 +2075,7 @@ static int run() { ok &= check_aggregation_host(); ok &= check_lifecycle_without_reader(); ok &= check_triggers(); + ok &= check_plugin_context_and_fault_triggers(); return ok ? 0 : 1; } diff --git a/scripts/check_cores_without_ros/run.sh b/scripts/check_cores_without_ros/run.sh index 8b3ff54a1..7b135bc11 100755 --- a/scripts/check_cores_without_ros/run.sh +++ b/scripts/check_cores_without_ros/run.sh @@ -19,7 +19,12 @@ trap 'exit 130' INT trap 'exit 143' TERM tar -c src/ros2_medkit_fault_detection src/ros2_medkit_fault_reporter src/ros2_medkit_fault_manager src/ros2_medkit_gateway scripts/check_cores_without_ros >"$ctx" || exit 1 -image=$(docker build -q -f scripts/check_cores_without_ros/Dockerfile - <"$ctx") || exit 1 +# -q hides the compiler output, so a failed build is repeated without it. The cached layers make the repeat fast. +if ! image=$(docker build -q -f scripts/check_cores_without_ros/Dockerfile - <"$ctx"); then + image="" + docker build --progress=plain -f scripts/check_cores_without_ros/Dockerfile - <"$ctx" >&2 + exit 1 +fi # Detached run: attached stdout can be cut short when the daemon is remote. container=$(docker run -d "$image") || exit 1 rc=$(docker wait "$container") || rc=1 diff --git a/src/ros2_medkit_action_status_bridge/CMakeLists.txt b/src/ros2_medkit_action_status_bridge/CMakeLists.txt index ba3e2a3d7..9ba3a1a89 100644 --- a/src/ros2_medkit_action_status_bridge/CMakeLists.txt +++ b/src/ros2_medkit_action_status_bridge/CMakeLists.txt @@ -99,6 +99,8 @@ if(BUILD_TESTING) DESTINATION share/${PROJECT_NAME} ) medkit_add_launch_test(test_integration test/test_integration.test.py TIMEOUT 120) + medkit_add_launch_test(test_sigint_exit test/test_sigint_exit.test.py TIMEOUT 120) + medkit_add_launch_test(test_source_resolution test/test_source_resolution.test.py TIMEOUT 180) ros2_medkit_relax_vendor_warnings() endif() diff --git a/src/ros2_medkit_action_status_bridge/README.md b/src/ros2_medkit_action_status_bridge/README.md index 57cd30662..e4b0268c6 100644 --- a/src/ros2_medkit_action_status_bridge/README.md +++ b/src/ros2_medkit_action_status_bridge/README.md @@ -31,8 +31,8 @@ generically, by observing the goal-status topic. - `fault_code` is `__ABORTED` (or `_CANCELED`), e.g. `ACTION_NAVIGATE_TO_POSE_ABORTED`. `source_id` is the action **server's node FQN** (resolved from the status topic's publisher, e.g. `/bt_navigator`), so - the fault associates with that node's SOVD entity; it falls back to the action - name only if no publisher is visible. + the fault associates with that node's SOVD entity. The first report waits up to + 5 s for that name to resolve; it falls back to the action name only after that. ## Per-action state, not per-goal diff --git a/src/ros2_medkit_action_status_bridge/include/ros2_medkit_action_status_bridge/action_status_bridge_node.hpp b/src/ros2_medkit_action_status_bridge/include/ros2_medkit_action_status_bridge/action_status_bridge_node.hpp index 64585aa45..8fe9f89f6 100644 --- a/src/ros2_medkit_action_status_bridge/include/ros2_medkit_action_status_bridge/action_status_bridge_node.hpp +++ b/src/ros2_medkit_action_status_bridge/include/ros2_medkit_action_status_bridge/action_status_bridge_node.hpp @@ -15,6 +15,7 @@ #pragma once #include +#include #include #include #include @@ -108,6 +109,15 @@ class ActionStatusBridgeNode : public rclcpp::Node { void rescan_actions(); void status_callback(const std::string & action_name, const action_msgs::msg::GoalStatusArray::ConstSharedPtr & msg); + /// Store the net state of `msg` as the desired state of the action. Returns + /// that state, or kUnknown when the message has no terminal verdict. + ActionState record_desired(const std::string & action_name, const action_msgs::msg::GoalStatusArray & msg); + + /// True while the action has no reporter yet and its server's node name is not + /// resolved, for at most kSourceResolveGrace after the first check. The first + /// report waits so its source is the server FQN, not the action-name fallback. + bool source_pending(const std::string & action_name); + /// Deliver the pending raise/heal for one action when the desired state /// differs from what the FaultManager was last told AND the report is /// deliverable (a null reporter is the test seam, treated as deliverable; a @@ -170,6 +180,9 @@ class ActionStatusBridgeNode : public rclcpp::Node { // fixed at creation (server FQN if resolved, else the action-name fallback) and // never changed. Guarded by reporters_mutex_. std::map> reporters_; + // First time source_pending() found the action's server unresolved. Guarded by + // reporters_mutex_. + std::map source_wait_since_; std::mutex reporters_mutex_; // Desired (latest observed) action-level state derived from status messages: diff --git a/src/ros2_medkit_action_status_bridge/src/action_status_bridge_node.cpp b/src/ros2_medkit_action_status_bridge/src/action_status_bridge_node.cpp index 0329e9ea7..c3c665686 100644 --- a/src/ros2_medkit_action_status_bridge/src/action_status_bridge_node.cpp +++ b/src/ros2_medkit_action_status_bridge/src/action_status_bridge_node.cpp @@ -41,6 +41,10 @@ constexpr size_t kMaxActionPart = 48; // Longest status suffix that may follow the action part, "_CANCELED". constexpr size_t kMaxStatusSuffix = 9; +// How long a first report waits for the action server's node name to resolve +// before it falls back to the action name as source. +constexpr std::chrono::seconds kSourceResolveGrace{5}; + bool contains_substr(const std::vector & list, const std::string & value) { for (const auto & item : list) { if (!item.empty() && value.find(item) != std::string::npos) { @@ -72,6 +76,22 @@ bool describe_failure_is_cancel(const action_msgs::msg::GoalStatusArray & msg, b return saw_canceled; } +// Runs a node callback unless the context is shut down. A graph or entity call +// racing with shutdown throws; that ends the callback quietly. +template +void run_unless_shutdown(F && fn) { + if (!rclcpp::ok()) { + return; + } + try { + fn(); + } catch (const std::exception &) { + if (rclcpp::ok()) { + throw; + } + } +} + } // namespace ActionStatusBridgeNode::ActionStatusBridgeNode(const rclcpp::NodeOptions & options) @@ -81,13 +101,17 @@ ActionStatusBridgeNode::ActionStatusBridgeNode(const rclcpp::NodeOptions & optio // Fast retry timer, created up front but kept disarmed (cancelled) until a // report is actually deferred, so an idle bridge does no periodic work. retry_timer_ = create_wall_timer(std::chrono::duration(retry_period_sec_), [this]() { - reconcile_pending(); + run_unless_shutdown([this]() { + reconcile_pending(); + }); }); retry_timer_->cancel(); rescan_actions(); rescan_timer_ = create_wall_timer(std::chrono::duration(rescan_period_sec_), [this]() { - rescan_actions(); + run_unless_shutdown([this]() { + rescan_actions(); + }); }); RCLCPP_INFO(get_logger(), "ActionStatusBridge started (prefix=%s, aborted_severity=%u, rescan=%.1fs, retry=%.0fms)", @@ -206,7 +230,9 @@ void ActionStatusBridgeNode::rescan_actions() { auto sub = create_subscription( topic, qos, [this, action_name](const action_msgs::msg::GoalStatusArray::ConstSharedPtr & msg) { - status_callback(action_name, msg); + run_unless_shutdown([&]() { + status_callback(action_name, msg); + }); }, sub_options); subs_.emplace(topic, sub); @@ -263,6 +289,7 @@ void ActionStatusBridgeNode::prune_vanished(const std::map lock(state_mutex_); @@ -294,8 +321,7 @@ ActionStatusBridgeNode::ActionState ActionStatusBridgeNode::derive_state(const a } ActionStatusBridgeNode::ActionState -ActionStatusBridgeNode::apply_message(const std::string & action_name, const action_msgs::msg::GoalStatusArray & msg, - ros2_medkit_fault_reporter::FaultReporter * reporter) { +ActionStatusBridgeNode::record_desired(const std::string & action_name, const action_msgs::msg::GoalStatusArray & msg) { const ActionState net = derive_state(msg, canceled_is_fault_); if (net == ActionState::kUnknown) { return ActionState::kUnknown; // no terminal verdict in this message @@ -315,6 +341,15 @@ ActionStatusBridgeNode::apply_message(const std::string & action_name, const act desired.net = ActionState::kHealthy; } } + return net; +} + +ActionStatusBridgeNode::ActionState +ActionStatusBridgeNode::apply_message(const std::string & action_name, const action_msgs::msg::GoalStatusArray & msg, + ros2_medkit_fault_reporter::FaultReporter * reporter) { + if (record_desired(action_name, msg) == ActionState::kUnknown) { + return ActionState::kUnknown; + } const ActionState delivered = reconcile(action_name, reporter); update_retry_timer(); // arm the fast retry if this could not be delivered yet @@ -400,6 +435,9 @@ void ActionStatusBridgeNode::reconcile_pending() { } } for (const auto & action_name : pending) { + if (source_pending(action_name)) { + continue; // the retry timer stays armed and tries again + } reconcile(action_name, reporter_for(action_name)); } @@ -461,9 +499,30 @@ void ActionStatusBridgeNode::status_callback(const std::string & action_name, // valid for this call - prune_vanished() only erases reporters_ from that same // thread, never concurrently. This invariant must hold if a multi-threaded // executor is ever introduced. + if (source_pending(action_name)) { + record_desired(action_name, *msg); + update_retry_timer(); + return; + } apply_message(action_name, *msg, reporter_for(action_name)); } +bool ActionStatusBridgeNode::source_pending(const std::string & action_name) { + { + std::lock_guard lock(reporters_mutex_); + if (reporters_.find(action_name) != reporters_.end()) { + return false; + } + } + if (!server_fqn_for_action(action_name).empty()) { + return false; + } + const auto now = std::chrono::steady_clock::now(); + std::lock_guard lock(reporters_mutex_); + const auto since = source_wait_since_.emplace(action_name, now).first->second; + return now - since < kSourceResolveGrace; +} + ros2_medkit_fault_reporter::FaultReporter * ActionStatusBridgeNode::reporter_for(const std::string & action_name) { std::lock_guard lock(reporters_mutex_); auto it = reporters_.find(action_name); @@ -486,6 +545,7 @@ ros2_medkit_fault_reporter::FaultReporter * ActionStatusBridgeNode::reporter_for auto reporter = std::make_unique(this->shared_from_this(), source_id); auto * raw = reporter.get(); reporters_[action_name] = std::move(reporter); + source_wait_since_.erase(action_name); return raw; } diff --git a/src/ros2_medkit_action_status_bridge/src/main.cpp b/src/ros2_medkit_action_status_bridge/src/main.cpp index 08fa511a7..33aaf9374 100644 --- a/src/ros2_medkit_action_status_bridge/src/main.cpp +++ b/src/ros2_medkit_action_status_bridge/src/main.cpp @@ -12,13 +12,49 @@ // See the License for the specific language governing permissions and // limitations under the License. +#include + +#include +#include +#include + #include "rclcpp/rclcpp.hpp" #include "ros2_medkit_action_status_bridge/action_status_bridge_node.hpp" int main(int argc, char * argv[]) { - rclcpp::init(argc, argv); + // SIGINT and SIGTERM stay blocked until the node and its executor are built: rclcpp does not + // support a context shut down while either is constructed. Threads started meanwhile inherit + // the mask. + sigset_t shutdown_signals; + sigset_t previous_mask; + sigemptyset(&shutdown_signals); + sigaddset(&shutdown_signals, SIGINT); + sigaddset(&shutdown_signals, SIGTERM); + if (pthread_sigmask(SIG_BLOCK, &shutdown_signals, &previous_mask) != 0) { + fprintf(stderr, "[action_status_bridge] Fatal: could not block SIGINT and SIGTERM\n"); + return 1; + } + + rclcpp::init(argc, argv, rclcpp::InitOptions(), rclcpp::SignalHandlerOptions::None); auto node = std::make_shared(); - rclcpp::spin(node); + auto executor = std::make_unique(); + executor->add_node(node); + + // A signal sent during startup is delivered at the unblock and shuts the context down, so spin + // returns at once. init() with None already counts as installed. + rclcpp::uninstall_signal_handlers(); + const bool handling_signals = + rclcpp::install_signal_handlers() && pthread_sigmask(SIG_SETMASK, &previous_mask, nullptr) == 0; + if (handling_signals) { + executor->spin(); + } else { + fprintf(stderr, "[action_status_bridge] Fatal: could not install the SIGINT and SIGTERM handlers\n"); + } + + // The executor goes before the node it holds. + executor->remove_node(node); + executor.reset(); + node.reset(); rclcpp::shutdown(); - return 0; + return handling_signals ? 0 : 1; } diff --git a/src/ros2_medkit_action_status_bridge/test/test_sigint_exit.test.py b/src/ros2_medkit_action_status_bridge/test/test_sigint_exit.test.py new file mode 100644 index 000000000..4c8549f32 --- /dev/null +++ b/src/ros2_medkit_action_status_bridge/test/test_sigint_exit.test.py @@ -0,0 +1,103 @@ +#!/usr/bin/env python3 +# Copyright 2026 mfaferek93, bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""The bridge exits with code 0 on SIGINT, wherever the signal lands.""" + +import os +import random +import signal +import subprocess +import unittest + +from ament_index_python.packages import get_package_prefix +import launch +import launch_testing.actions + +RUNS = 40 +STARTUP_RUNS = 80 + + +def generate_test_description(): + # The bridge runs are spawned by the test body; this idle process keeps the + # launch alive while they do. + idle = launch.actions.ExecuteProcess(cmd=['sleep', '600']) + return launch.LaunchDescription([idle, launch_testing.actions.ReadyToTest()]) + + +class TestSigintExit(unittest.TestCase): + + def test_sigint_during_rescan_exits_cleanly(self): + """SIGINT lands in a rescan on every run; each run must exit 0.""" + exe = os.path.join( + get_package_prefix('ros2_medkit_action_status_bridge'), + 'lib', 'ros2_medkit_action_status_bridge', 'action_status_bridge_node') + rng = random.Random(7) + failures = [] + for run in range(RUNS): + # A 1 ms rescan period keeps the node inside rescan_actions() or + # reconcile_pending() almost continuously. + proc = subprocess.Popen( + [exe, '--ros-args', '-p', 'rescan_period_sec:=0.001', + '-p', 'retry_period_sec:=0.001'], + stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + try: + proc.wait(timeout=0.6 + rng.random() * 0.4) + except subprocess.TimeoutExpired: + pass + if proc.poll() is None: + proc.send_signal(signal.SIGINT) + try: + out, _ = proc.communicate(timeout=30) + except subprocess.TimeoutExpired: + proc.kill() + out, _ = proc.communicate() + if proc.returncode != 0: + failures.append((run, proc.returncode)) + print(f'--- run {run} exited {proc.returncode} ---\n{out}') + self.assertEqual( + failures, [], f'{len(failures)} of {RUNS} runs did not exit 0 (run, code): {failures}') + + def test_sigint_during_startup_exits_cleanly(self): + """SIGINT within the first milliseconds, while the constructor scans.""" + exe = os.path.join( + get_package_prefix('ros2_medkit_action_status_bridge'), + 'lib', 'ros2_medkit_action_status_bridge', 'action_status_bridge_node') + rng = random.Random(11) + failures = [] + for run in range(STARTUP_RUNS): + proc = subprocess.Popen( + [exe], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + # Spread the signal over the whole startup, from before main() blocks + # it to well after the first scan. + delay = rng.random() * 0.25 + try: + proc.wait(timeout=delay) + except subprocess.TimeoutExpired: + pass + if proc.poll() is None: + proc.send_signal(signal.SIGINT) + try: + out, _ = proc.communicate(timeout=30) + except subprocess.TimeoutExpired: + proc.kill() + out, _ = proc.communicate() + # -2: SIGINT before main() blocked it; the default action ends the + # process and no bridge code runs. + if proc.returncode not in (0, -signal.SIGINT): + failures.append((run, round(delay, 3), proc.returncode)) + print(f'--- run {run}, signal at {delay:.3f} s, exit {proc.returncode} ---\n{out}') + self.assertEqual( + failures, [], + f'{len(failures)} of {STARTUP_RUNS} runs aborted (run, delay, code): {failures}') diff --git a/src/ros2_medkit_action_status_bridge/test/test_source_resolution.test.py b/src/ros2_medkit_action_status_bridge/test/test_source_resolution.test.py new file mode 100644 index 000000000..d16020312 --- /dev/null +++ b/src/ros2_medkit_action_status_bridge/test/test_source_resolution.test.py @@ -0,0 +1,206 @@ +#!/usr/bin/env python3 +# Copyright 2026 mfaferek93, bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""A fault's reporting source is the action server node, not the action name.""" + +import os +import signal +import time +import unittest + +from action_msgs.msg import GoalStatus, GoalStatusArray +from launch import LaunchDescription +import launch_ros.actions +import launch_testing.actions +import rclpy +from rclpy.node import Node +from rclpy.qos import ( + DurabilityPolicy, + HistoryPolicy, + QoSProfile, + ReliabilityPolicy, +) +from ros2_medkit_msgs.srv import ListFaults +from unique_identifier_msgs.msg import UUID + +SERVERS = 6 +ACTIONS_PER_SERVER = 4 +QOS = QoSProfile( + depth=10, + reliability=ReliabilityPolicy.RELIABLE, + durability=DurabilityPolicy.TRANSIENT_LOCAL, + history=HistoryPolicy.KEEP_LAST, +) + + +def generate_test_description(): + fault_manager = launch_ros.actions.Node( + package='ros2_medkit_fault_manager', + executable='fault_manager_node', + name='fault_manager', + output='screen', + parameters=[{ + 'storage_type': 'memory', + 'confirmation_threshold': -1, + 'healing_enabled': True, + 'healing_threshold': 0, + }], + sigterm_timeout='30', + sigkill_timeout='15', + ) + # A long rescan period: the bridge subscribes to a status topic on its first + # tick after the topic appears, and nothing prunes it before the test is done. + bridge = launch_ros.actions.Node( + package='ros2_medkit_action_status_bridge', + executable='action_status_bridge_node', + name='action_status_bridge', + output='screen', + parameters=[{'rescan_period_sec': 3.0}], + sigterm_timeout='30', + sigkill_timeout='15', + ) + return LaunchDescription([fault_manager, bridge, launch_testing.actions.ReadyToTest()]) + + +def _aborted(goal_byte): + gsa = GoalStatusArray() + gs = GoalStatus() + gs.goal_info.goal_id = UUID(uuid=[goal_byte] * 16) + gs.status = int(GoalStatus.STATUS_ABORTED) + gsa.status_list = [gs] + return gsa + + +def _empty(): + return GoalStatusArray() + + +class TestSourceResolution(unittest.TestCase): + + @classmethod + def setUpClass(cls): + rclpy.init() + cls.client_node = Node('source_resolution_client') + cls.list_faults = cls.client_node.create_client( + ListFaults, '/fault_manager/list_faults') + assert cls.list_faults.wait_for_service(timeout_sec=20.0) + + @classmethod + def tearDownClass(cls): + cls.client_node.destroy_node() + rclpy.shutdown() + + @staticmethod + def _bridge_pid(): + """PID of the bridge started for this test (same ROS_DOMAIN_ID).""" + domain = os.environ['ROS_DOMAIN_ID'].encode() + for entry in os.listdir('/proc'): + if not entry.isdigit(): + continue + try: + with open(f'/proc/{entry}/cmdline', 'rb') as f: + cmd = f.read() + with open(f'/proc/{entry}/environ', 'rb') as f: + env = f.read().split(b'\0') + except OSError: + continue + if b'action_status_bridge_node' in cmd and b'ROS_DOMAIN_ID=' + domain in env: + return int(entry) + raise AssertionError('bridge process not found') + + def _faults(self): + req = ListFaults.Request() + req.filter_by_severity = False + req.statuses = [] + fut = self.list_faults.call_async(req) + rclpy.spin_until_future_complete(self.client_node, fut, timeout_sec=5.0) + return fut.result().faults if fut.result() is not None else [] + + def test_source_is_the_server_node_while_the_bridge_is_stalled(self): + """ + The status arrives before the bridge has the server's node name. + + The bridge is stopped while the servers appear and publish, so on resume + the status data and the node names reach it together and in no fixed order. + """ + pid = self._bridge_pid() + # Each action first has a publisher with no verdict, so the bridge + # subscribes to its status topic; that publisher's node then goes away. + probe = Node('source_resolution_probe') + actions = {} + pubs = [] + for s in range(SERVERS): + for a in range(ACTIONS_PER_SERVER): + actions[f'/sr_{s}_{a}'] = f'/server_{s}' + for name in actions: + pubs.append(probe.create_publisher( + GoalStatusArray, name + '/_action/status', QOS)) + deadline = time.time() + 30.0 + while time.time() < deadline: + watching = sum( + 1 for name in actions + if any(i.node_name == 'action_status_bridge' for i in + probe.get_subscriptions_info_by_topic(name + '/_action/status'))) + if watching == len(actions): + break + rclpy.spin_once(probe, timeout_sec=0.2) + self.assertEqual(watching, len(actions), 'bridge did not subscribe to every action') + + os.kill(pid, signal.SIGSTOP) + try: + for p in pubs: + probe.destroy_publisher(p) + probe.destroy_node() + servers = [] + for s in range(SERVERS): + node = Node(f'server_{s}') + held = [] + for name, server in actions.items(): + if server == f'/server_{s}': + pub = node.create_publisher( + GoalStatusArray, name + '/_action/status', QOS) + pub.publish(_aborted(s + 1)) + held.append(pub) + servers.append((node, held)) + # The bridge's reader was announced before the stop, so each new writer can match it. + held_pubs = [p for _, held in servers for p in held] + deadline = time.time() + 30.0 + while time.time() < deadline and any( + p.get_subscription_count() == 0 for p in held_pubs): + time.sleep(0.05) + matched = sum(1 for p in held_pubs if p.get_subscription_count() > 0) + finally: + os.kill(pid, signal.SIGCONT) + self.assertEqual(matched, len(actions), 'a server publisher never matched the bridge') + + want = len(actions) + deadline = time.time() + 30.0 + faults = [] + while time.time() < deadline: + faults = [f for f in self._faults() if f.fault_code.startswith('ACTION_SR_')] + if len(faults) == want: + break + time.sleep(0.5) + self.assertEqual(len(faults), want) + wrong = [] + for f in faults: + s = f.fault_code.split('_')[2] + expected = f'/server_{s}' + if expected not in list(f.reporting_sources): + wrong.append((f.fault_code, list(f.reporting_sources))) + for node, _ in servers: + node.destroy_node() + self.assertEqual( + wrong, [], f'{len(wrong)} of {want} faults lack the server node as source') diff --git a/src/ros2_medkit_fault_reporter/CMakeLists.txt b/src/ros2_medkit_fault_reporter/CMakeLists.txt index c75c7b48d..368542a8c 100644 --- a/src/ros2_medkit_fault_reporter/CMakeLists.txt +++ b/src/ros2_medkit_fault_reporter/CMakeLists.txt @@ -103,6 +103,12 @@ if(BUILD_TESTING) target_link_libraries(test_validate_filter_params fault_reporter_core) medkit_test_needs_no_domain(test_validate_filter_params) + # Test node of the delivery tests. + add_executable(delivery_probe_node test/delivery_probe_node.cpp) + target_link_libraries(delivery_probe_node fault_reporter_lib) + medkit_target_dependencies(delivery_probe_node rclcpp ros2_medkit_msgs) + install(TARGETS delivery_probe_node DESTINATION lib/${PROJECT_NAME}) + # Integration tests install(DIRECTORY test DESTINATION share/${PROJECT_NAME} @@ -110,6 +116,8 @@ if(BUILD_TESTING) medkit_add_launch_test(test_integration test/test_integration.test.py TIMEOUT 60) + medkit_add_launch_test(test_delivery test/test_delivery.test.py TIMEOUT 300) + # A launch test and its children on one domain. It lives here rather than in # ros2_medkit_cmake because that package is declared NONE, and on Humble # launch_testing_ament_cmake reaches the deprecated FindPythonLibs, which diff --git a/src/ros2_medkit_fault_reporter/README.md b/src/ros2_medkit_fault_reporter/README.md index ec91cbfd2..3be48c1e1 100644 --- a/src/ros2_medkit_fault_reporter/README.md +++ b/src/ros2_medkit_fault_reporter/README.md @@ -76,6 +76,7 @@ class MyLifecycleNode : public rclcpp_lifecycle::LifecycleNode { - **Simple API**: Just call `report()` to report a fault - **Lifecycle node support**: Construct from a regular `Node` or an `rclcpp_lifecycle::LifecycleNode` - **Local Filtering** (optional): Suppress repeated faults until threshold is met +- **Confirmed delivery**: each report is kept until the FaultManager answers, and sent again if it does not - **Per-fault tracking**: Each fault_code has independent filtering - **Severity bypass**: High-severity faults bypass local filtering - **ROS-free core**: `ReporterCore` (filtering and report building) builds without ROS, see below @@ -131,6 +132,41 @@ A `default_threshold` above `INT_MAX` or a `bypass_severity` outside `[0, 255]` default and logged as a warning. A `default_threshold` below 1 becomes 1. A `default_window_sec` that is NaN, infinite or not positive becomes 1.0. These last two are not logged. +## Delivery + +`report()` and `report_passed()` return at once. The reporter keeps each request until the +FaultManager answers it, and sends it again when no response comes within `ack_timeout_sec`. After +`max_attempts` requests it logs a warning that names the fault code and drops the report. A client +can see the service as ready before the server has matched it, and a request sent then is lost. This +retry covers that gap. + +- **Order:** per fault code only one request is in flight, and later ones wait behind it. The manager + receives the events of one code in the order they were reported. +- **Cap:** at most `max_pending` reports wait for a response, all codes of one reporter together. A + report past the cap is dropped with a warning and the older ones are kept. +- **Not ready:** a report made while the service is not ready is dropped at once, as before. +- **Duplicates:** if a request arrived and only its response was lost, the manager gets the event + twice. A repeated FAILED counts toward the manager's debounce. +- **Spinning:** responses and retries need the node to be spun. Every constructor puts the retry + timer on the node; the raw interfaces constructor without a timers interface uses the node's + default callback group. +- **Retry while not ready:** a retry is sent whether or not the service is ready, and it counts toward + `max_attempts`. A FaultManager that restarts within `ack_timeout_sec` x `max_attempts` (5 s by + default) gets the pending reports. After a crash the dead server can look ready for some seconds. +- **Destroyed reporter:** destroying a reporter does not cancel its reports. The ones still waiting + are sent until confirmed or dropped after `max_attempts`, as long as the node is spun. +- **Shutdown:** when the ROS context shuts down, each report still waiting is dropped with a warning + that names its fault code. Reports of a destroyed reporter whose node is gone wait until then. + +| Parameter | Type | Default | Range | Why | +|-----------|------|---------|-------|-----| +| `fault_reporter.delivery.ack_timeout_sec` | double | 0.5 | 0.05 - 60 | Well above a local round trip, short enough to recover within a second | +| `fault_reporter.delivery.max_attempts` | int | 10 | 1 - 100 | With the default timeout a report is kept for 5 s | +| `fault_reporter.delivery.max_pending` | int | 1024 | 1 - 100000 | Bounds memory when the manager does not answer | + +A value outside its range (a NaN timeout, an integer beyond the range) is replaced by its default and +logged as a warning. + ## ROS-free core The package has two parts. The core compiles without ROS and without DDS setup. diff --git a/src/ros2_medkit_fault_reporter/design/index.rst b/src/ros2_medkit_fault_reporter/design/index.rst index a551fccaf..a966b8294 100644 --- a/src/ros2_medkit_fault_reporter/design/index.rst +++ b/src/ros2_medkit_fault_reporter/design/index.rst @@ -123,7 +123,7 @@ Main Components - Loads filter configuration from ROS parameters - Provides simple ``report(fault_code, severity, description)`` method - Integrates ``LocalFilter`` to reduce noise from repeated faults, through ``ReporterCore`` - - Fire-and-forget service calls (non-blocking) + - Non-blocking service calls that are confirmed by the response and sent again when it does not come 2. **LocalFilter** - Per-fault-code filtering with threshold and time window @@ -200,6 +200,10 @@ Parameters are loaded from the node's parameter server: default_threshold: 3 default_window_sec: 10.0 bypass_severity: 2 + delivery: + ack_timeout_sec: 0.5 + max_attempts: 10 + max_pending: 1024 Plain-CMake Build ~~~~~~~~~~~~~~~~~ @@ -228,13 +232,29 @@ The default threshold of 3 reports within 10 seconds balances: - **Responsiveness**: Persistent issues are reported within seconds - **Safety**: High-severity faults (ERROR, CRITICAL) bypass filtering entirely -Fire-and-Forget Service Calls -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The ``report()`` method uses asynchronous service calls without waiting for -responses. This ensures fault reporting never blocks the calling node, which -is important for real-time systems. If the FaultManager service is unavailable, -reports are silently dropped (logged at DEBUG level). +Confirmed Delivery +~~~~~~~~~~~~~~~~~~ + +``report()`` never blocks the calling node. It queues the request and returns. A client can see the +service as ready before the server has matched it, and a request sent in that gap is lost, so the +reporter keeps each request until its response arrives. With no response within +``ack_timeout_sec`` it sends the request again, up to ``max_attempts`` requests. It then logs a +warning that names the fault code and drops the report. + +Per fault code the requests form a queue and only the front is in flight, so the manager receives the +events of one code in the order they were reported. At most ``max_pending`` reports wait for a +response, all codes together. A report past that is dropped with a warning. A report made while the +service is not ready is dropped at once (DEBUG log, WARN for a high-severity FAILED). + +Responses and retries are processed when the node is spun. Every constructor puts the retry timer on +the node; without a timers interface it goes to the node's default callback group. A retry is sent +whether or not the service is ready and counts toward ``max_attempts``, so a manager that restarts +within the attempt budget receives the pending reports. If a request arrived and only its response +was lost, the manager gets the event twice. A repeated FAILED counts toward the manager's debounce. + +A destroyed reporter leaves its pending reports to its delivery, which the ROS context holds until +the last one is confirmed or dropped. When the context shuts down, each report still waiting is +dropped with a warning that names its fault code. Thread Safety ~~~~~~~~~~~~~ diff --git a/src/ros2_medkit_fault_reporter/include/ros2_medkit_fault_reporter/core/reporter_core.hpp b/src/ros2_medkit_fault_reporter/include/ros2_medkit_fault_reporter/core/reporter_core.hpp index af762eca1..146867c94 100644 --- a/src/ros2_medkit_fault_reporter/include/ros2_medkit_fault_reporter/core/reporter_core.hpp +++ b/src/ros2_medkit_fault_reporter/include/ros2_medkit_fault_reporter/core/reporter_core.hpp @@ -24,6 +24,23 @@ namespace ros2_medkit_fault_reporter::core { +/// How the reporter confirms delivery of each report. +struct DeliveryConfig { + /// Wait for a response this long before sending a request again. + double ack_timeout_sec{0.5}; + /// Requests sent per report before it is dropped. + int max_attempts{10}; + /// Reports waiting for a response, all fault codes together. A newer report past this is dropped. + int max_pending{1024}; +}; + +inline constexpr double kMinAckTimeoutSec = 0.05; +inline constexpr double kMaxAckTimeoutSec = 60.0; +inline constexpr int kMinMaxAttempts = 1; +inline constexpr int kMaxMaxAttempts = 100; +inline constexpr int kMinMaxPending = 1; +inline constexpr int kMaxMaxPending = 100000; + /// Receives each report that passed the filter. using ReportSink = std::function; @@ -55,4 +72,10 @@ class ReporterCore { FilterConfig validate_filter_params(bool enabled, int64_t threshold, double window_sec, int64_t bypass_severity, std::vector * warnings); +/// Builds a valid delivery config from raw parameter values. A value outside its range (a NaN +/// timeout, an int64 beyond the range) is replaced by its default and a message is appended to warnings +/// (when not null). +DeliveryConfig validate_delivery_params(double ack_timeout_sec, int64_t max_attempts, int64_t max_pending, + std::vector * warnings); + } // namespace ros2_medkit_fault_reporter::core diff --git a/src/ros2_medkit_fault_reporter/include/ros2_medkit_fault_reporter/fault_reporter.hpp b/src/ros2_medkit_fault_reporter/include/ros2_medkit_fault_reporter/fault_reporter.hpp index 837585b3d..6fdad385b 100644 --- a/src/ros2_medkit_fault_reporter/include/ros2_medkit_fault_reporter/fault_reporter.hpp +++ b/src/ros2_medkit_fault_reporter/include/ros2_medkit_fault_reporter/fault_reporter.hpp @@ -14,9 +14,11 @@ #pragma once +#include #include #include +#include "rclcpp/node_interfaces/node_timers_interface.hpp" #include "rclcpp/rclcpp.hpp" #include "rclcpp_lifecycle/lifecycle_node.hpp" #include "ros2_medkit_fault_reporter/core/reporter_core.hpp" @@ -72,6 +74,8 @@ class FaultReporter { /// @param logger ROS2 Node Logger for reporting logs /// @param source_id Identifier for this reporter (typically node's FQN) /// @param service_name Name of the ReportFault service (default: /fault_manager/report_fault) + /// + /// The retry timer goes to the node's default callback group, as with the overloads that take a node. FaultReporter(const rclcpp::node_interfaces::NodeBaseInterface::SharedPtr & node_base, const rclcpp::node_interfaces::NodeGraphInterface::SharedPtr & node_graph, const rclcpp::node_interfaces::NodeServicesInterface::SharedPtr & node_services, @@ -79,6 +83,26 @@ class FaultReporter { const rclcpp::Logger & logger, const std::string & source_id, const std::string & service_name = "/fault_manager/report_fault"); + /// Base Constructor with a timers interface for the timer that retries a request with no response. + /// + /// @param node_timers ROS2 Node Timers Interface for the retry timer; null uses the node's default + /// callback group + FaultReporter(const rclcpp::node_interfaces::NodeBaseInterface::SharedPtr & node_base, + const rclcpp::node_interfaces::NodeGraphInterface::SharedPtr & node_graph, + const rclcpp::node_interfaces::NodeServicesInterface::SharedPtr & node_services, + const rclcpp::node_interfaces::NodeParametersInterface::SharedPtr & node_params, + const rclcpp::node_interfaces::NodeTimersInterface::SharedPtr & node_timers, + const rclcpp::Logger & logger, const std::string & source_id, + const std::string & service_name = "/fault_manager/report_fault"); + + /// Reports still waiting for a response are kept and sent until confirmed or dropped. At context + /// shutdown each one left is dropped with a warning. + ~FaultReporter(); + FaultReporter(const FaultReporter &) = delete; + FaultReporter & operator=(const FaultReporter &) = delete; + FaultReporter(FaultReporter &&) = delete; + FaultReporter & operator=(FaultReporter &&) = delete; + /// Construct a FaultReporter via Node Shared Ptr (maintain backwards compatibility) /// /// @param node The ROS 2 node to use for service client and parameters @@ -135,6 +159,14 @@ class FaultReporter { /// Check if the FaultManager service is available bool is_service_ready() const; + /// Reports sent and still waiting for a response, all fault codes together. + size_t pending_reports() const; + + /// Delivery settings in effect, after validation. + const core::DeliveryConfig & delivery_config() const { + return delivery_config_; + } + /// Get read-only access to the local filter (for testing) const LocalFilter & filter() const { return core_.filter(); @@ -145,12 +177,20 @@ class FaultReporter { static FilterConfig load_parameters(const rclcpp::node_interfaces::NodeParametersInterface::SharedPtr & node_params, const rclcpp::Logger & logger); - /// Send the fault report to FaultManager (async, fire-and-forget) + static core::DeliveryConfig + load_delivery_config(const rclcpp::node_interfaces::NodeParametersInterface::SharedPtr & node_params, + const rclcpp::Logger & logger); + + /// Hand the report to the delivery queue: it is sent, and sent again until a response arrives. void send_report(const ros2_medkit::fault_detection::FaultReport & report); + struct Delivery; + std::string source_id_; rclcpp::Client::SharedPtr client_; rclcpp::Logger logger_; + core::DeliveryConfig delivery_config_; + std::shared_ptr delivery_; core::ReporterCore core_; }; diff --git a/src/ros2_medkit_fault_reporter/src/core/reporter_core.cpp b/src/ros2_medkit_fault_reporter/src/core/reporter_core.cpp index 407b8546f..bbfa8347c 100644 --- a/src/ros2_medkit_fault_reporter/src/core/reporter_core.cpp +++ b/src/ros2_medkit_fault_reporter/src/core/reporter_core.cpp @@ -94,4 +94,36 @@ FilterConfig validate_filter_params(bool enabled, int64_t threshold, double wind return config; } +DeliveryConfig validate_delivery_params(double ack_timeout_sec, int64_t max_attempts, int64_t max_pending, + std::vector * warnings) { + const DeliveryConfig defaults; + DeliveryConfig config; + auto refuse = [warnings](const std::string & name, const std::string & value, const std::string & fallback) { + if (warnings != nullptr) { + warnings->push_back("fault_reporter.delivery." + name + " " + value + " is out of range, using " + fallback); + } + }; + + // Positive form: NaN fails every comparison, so a below-or-above test would let it through. + if (std::isfinite(ack_timeout_sec) && ack_timeout_sec >= kMinAckTimeoutSec && ack_timeout_sec <= kMaxAckTimeoutSec) { + config.ack_timeout_sec = ack_timeout_sec; + } else { + config.ack_timeout_sec = defaults.ack_timeout_sec; + refuse("ack_timeout_sec", std::to_string(ack_timeout_sec), std::to_string(defaults.ack_timeout_sec)); + } + if (max_attempts >= kMinMaxAttempts && max_attempts <= kMaxMaxAttempts) { + config.max_attempts = static_cast(max_attempts); + } else { + config.max_attempts = defaults.max_attempts; + refuse("max_attempts", std::to_string(max_attempts), std::to_string(defaults.max_attempts)); + } + if (max_pending >= kMinMaxPending && max_pending <= kMaxMaxPending) { + config.max_pending = static_cast(max_pending); + } else { + config.max_pending = defaults.max_pending; + refuse("max_pending", std::to_string(max_pending), std::to_string(defaults.max_pending)); + } + return config; +} + } // namespace ros2_medkit_fault_reporter::core diff --git a/src/ros2_medkit_fault_reporter/src/fault_reporter.cpp b/src/ros2_medkit_fault_reporter/src/fault_reporter.cpp index a5ccf6c34..59d21b6d2 100644 --- a/src/ros2_medkit_fault_reporter/src/fault_reporter.cpp +++ b/src/ros2_medkit_fault_reporter/src/fault_reporter.cpp @@ -14,19 +14,289 @@ #include "ros2_medkit_fault_reporter/fault_reporter.hpp" +#include +#include +#include +#include +#include +#include + +#include "rclcpp/node_interfaces/node_timers.hpp" #include "rclcpp/version.h" #include "rclcpp_lifecycle/lifecycle_node.hpp" namespace ros2_medkit_fault_reporter { +namespace { +using Clock = std::chrono::steady_clock; +using ReportFault = ros2_medkit_msgs::srv::ReportFault; +} // namespace + +// Confirms delivery. Per fault code the reports form a queue and only the front is in flight, so the +// manager receives the events of one code in the order they were reported. The front is sent again +// when no response arrives within the ack timeout, up to max_attempts requests. +struct FaultReporter::Delivery : std::enable_shared_from_this { + struct Entry { + std::shared_ptr request; + int attempts{0}; + Clock::time_point sent_at; + int64_t request_id{-1}; + }; + + // Deliveries whose reporter is destroyed while they hold reports. One per context, owned by it; + // the context's shutdown drops what is left, with a warning per report. + struct Orphans : std::enable_shared_from_this { + void watch(rclcpp::Context & owner) { + std::call_once(watched, [this, &owner]() { + std::weak_ptr weak = weak_from_this(); + owner.add_on_shutdown_callback([weak]() { + if (auto self = weak.lock()) { + self->drop_all(); + } + }); + }); + } + + void adopt(const std::shared_ptr & delivery) { + std::lock_guard lock(mutex); + if (delivery->pending_count() > 0) { + held[delivery.get()] = delivery; + } + } + + void forget(const Delivery * delivery) { + std::shared_ptr last; // destroyed after the lock is released + std::lock_guard lock(mutex); + auto it = held.find(delivery); + if (it != held.end()) { + last = std::move(it->second); + held.erase(it); + } + } + + void drop_all() { + std::map> taken; + { + std::lock_guard lock(mutex); + taken.swap(held); + } + for (auto & entry : taken) { + entry.second->drop_pending(); + } + } + + std::mutex mutex; + std::map> held; + std::once_flag watched; + }; + + Delivery(rclcpp::Client::SharedPtr c, const rclcpp::Logger & l, const core::DeliveryConfig & cfg, + const rclcpp::Context::SharedPtr & ctx) + : client(std::move(c)), logger(l), config(cfg), context(ctx) { + } + + void enqueue(const std::shared_ptr & request) { + std::lock_guard lock(mutex); + if (pending >= static_cast(config.max_pending)) { + RCLCPP_WARN(logger, "%zu reports wait for the FaultManager, dropping the report for '%s'", pending, + request->fault_code.c_str()); + return; + } + auto & queue = queues[request->fault_code]; + Entry entry; + entry.request = request; + queue.push_back(std::move(entry)); + ++pending; + if (queue.size() == 1) { + send_front_locked(request->fault_code); + } + arm_timer_locked(); + } + + size_t pending_count() { + std::lock_guard lock(mutex); + return pending; + } + + // The reporter is destroyed. Pending reports stay here, held by the context, and are sent until + // confirmed or dropped like any other. + void release() { + { + std::lock_guard lock(mutex); + if (pending == 0) { + stop_timer_locked(); + return; + } + } + auto ctx = context.lock(); + if (!ctx || !ctx->is_valid()) { + drop_pending(); + return; + } + auto registry = ctx->get_sub_context(); + registry->watch(*ctx); + { + std::lock_guard lock(mutex); + orphans = registry; + } + registry->adopt(shared_from_this()); + if (!ctx->is_valid()) { + registry->drop_all(); // the shutdown ran before the adoption + } + } + + // Drops every pending report with a warning that names its fault code. + void drop_pending() { + std::lock_guard lock(mutex); + for (const auto & [code, queue] : queues) { + if (!queue.empty() && queue.front().request_id >= 0) { + client->remove_pending_request(queue.front().request_id); + } + for (size_t i = 0; i < queue.size(); ++i) { + RCLCPP_WARN(logger, "Shut down before the FaultManager answered, dropping the report for '%s'", code.c_str()); + } + } + queues.clear(); + pending = 0; + stop_timer_locked(); + } + + void on_response(const std::string & code, const std::shared_ptr & request) { + std::shared_ptr done; + { + std::lock_guard lock(mutex); + auto it = queues.find(code); + if (it == queues.end() || it->second.empty() || it->second.front().request != request) { + return; // the answer to an attempt that was already replaced + } + pop_front_locked(code); + done = drained_orphans_locked(); + } + if (done) { + done->forget(this); + } + } + + void on_timer() { + std::shared_ptr done; + { + std::lock_guard lock(mutex); + retry_due_locked(); + if (pending == 0 && timer) { + timer->cancel(); + } + done = drained_orphans_locked(); + } + if (done) { + done->forget(this); + } + } + + std::mutex mutex; + std::map> queues; + size_t pending{0}; + rclcpp::TimerBase::SharedPtr timer; + + private: + void arm_timer_locked() { + if (timer && timer->is_canceled()) { + timer->reset(); + } + } + + void stop_timer_locked() { + if (timer) { + timer->cancel(); + timer.reset(); + } + } + + // The registry that holds this delivery once it has nothing left to send, else null. + std::shared_ptr drained_orphans_locked() { + return pending == 0 ? orphans.lock() : nullptr; + } + + void send_front_locked(const std::string & code) { + Entry & entry = queues[code].front(); + ++entry.attempts; + entry.sent_at = Clock::now(); + std::weak_ptr weak = weak_from_this(); + auto request = entry.request; + entry.request_id = + client + ->async_send_request( + request, + [weak, code, + request](rclcpp::Client::SharedFuture) { // NOLINT(performance-unnecessary-value-param) + if (auto self = weak.lock()) { + self->on_response(code, request); + } + }) + .request_id; + } + + void pop_front_locked(const std::string & code) { + auto & queue = queues[code]; + queue.pop_front(); + --pending; + if (queue.empty()) { + queues.erase(code); + } else { + send_front_locked(code); + } + } + + void retry_due_locked() { + const auto now = Clock::now(); + const auto timeout = + std::chrono::duration_cast(std::chrono::duration(config.ack_timeout_sec)); + std::vector codes; + for (const auto & [code, queue] : queues) { + if (!queue.empty() && now - queue.front().sent_at >= timeout) { + codes.push_back(code); + } + } + for (const auto & code : codes) { + Entry & entry = queues[code].front(); + client->remove_pending_request(entry.request_id); + if (entry.attempts >= config.max_attempts) { + RCLCPP_WARN(logger, "No response from the FaultManager after %d attempts, dropping the report for '%s'", + entry.attempts, code.c_str()); + pop_front_locked(code); + } else { + RCLCPP_DEBUG(logger, "No response for the report for '%s', sending attempt %d of %d (service %s)", code.c_str(), + entry.attempts + 1, config.max_attempts, client->service_is_ready() ? "ready" : "not ready"); + send_front_locked(code); + } + } + } + + rclcpp::Client::SharedPtr client; + rclcpp::Logger logger; + core::DeliveryConfig config; + std::weak_ptr context; + std::weak_ptr orphans; +}; + +FaultReporter::FaultReporter(const rclcpp::node_interfaces::NodeBaseInterface::SharedPtr & node_base, + const rclcpp::node_interfaces::NodeGraphInterface::SharedPtr & node_graph, + const rclcpp::node_interfaces::NodeServicesInterface::SharedPtr & node_services, + const rclcpp::node_interfaces::NodeParametersInterface::SharedPtr & node_params, + const rclcpp::Logger & logger, const std::string & source_id, + const std::string & service_name) + : FaultReporter(node_base, node_graph, node_services, node_params, nullptr, logger, source_id, service_name) { +} + FaultReporter::FaultReporter(const rclcpp::node_interfaces::NodeBaseInterface::SharedPtr & node_base, const rclcpp::node_interfaces::NodeGraphInterface::SharedPtr & node_graph, const rclcpp::node_interfaces::NodeServicesInterface::SharedPtr & node_services, const rclcpp::node_interfaces::NodeParametersInterface::SharedPtr & node_params, + const rclcpp::node_interfaces::NodeTimersInterface::SharedPtr & node_timers, const rclcpp::Logger & logger, const std::string & source_id, const std::string & service_name) : source_id_(source_id) , logger_(logger) + , delivery_config_(load_delivery_config(node_params, logger)) , core_(source_id, load_parameters(node_params, logger), [this](const ros2_medkit::fault_detection::FaultReport & report) { send_report(report); @@ -47,32 +317,66 @@ FaultReporter::FaultReporter(const rclcpp::node_interfaces::NodeBaseInterface::S node_base, node_graph, node_services, service_name, rmw_qos_profile_services_default, nullptr); #endif + delivery_ = std::make_shared(client_, logger_, delivery_config_, node_base->get_context()); + std::weak_ptr weak = delivery_; + const auto period = std::chrono::duration_cast( + std::chrono::duration(delivery_config_.ack_timeout_sec / 2.0)); + delivery_->timer = std::make_shared>>( + period, + [weak]() { + if (auto self = weak.lock()) { + self->on_timer(); + } + }, + node_base->get_context()); + // Without a timers interface the timer goes to the node's default callback group all the same. + if (node_timers) { + node_timers->add_timer(delivery_->timer, nullptr); + } else { + rclcpp::node_interfaces::NodeTimers(node_base.get()).add_timer(delivery_->timer, nullptr); + } + delivery_->timer->cancel(); + RCLCPP_DEBUG(logger_, "FaultReporter initialized for source: %s", source_id_.c_str()); } +FaultReporter::~FaultReporter() { + try { + delivery_->release(); + } catch (const std::exception & e) { + RCLCPP_ERROR(logger_, "Could not keep the pending reports of a destroyed reporter: %s", e.what()); + } +} + +size_t FaultReporter::pending_reports() const { + return delivery_->pending_count(); +} + FaultReporter::FaultReporter(const rclcpp::Node::SharedPtr & node, const std::string & source_id, const std::string & service_name) : FaultReporter(node->get_node_base_interface(), node->get_node_graph_interface(), - node->get_node_services_interface(), node->get_node_parameters_interface(), node->get_logger(), - source_id, service_name) { + node->get_node_services_interface(), node->get_node_parameters_interface(), + node->get_node_timers_interface(), node->get_logger(), source_id, service_name) { } FaultReporter::FaultReporter(rclcpp::Node & node, const std::string & source_id, const std::string & service_name) : FaultReporter(node.get_node_base_interface(), node.get_node_graph_interface(), node.get_node_services_interface(), - node.get_node_parameters_interface(), node.get_logger(), source_id, service_name) { + node.get_node_parameters_interface(), node.get_node_timers_interface(), node.get_logger(), source_id, + service_name) { } FaultReporter::FaultReporter(rclcpp_lifecycle::LifecycleNode & node, const std::string & source_id, const std::string & service_name) : FaultReporter(node.get_node_base_interface(), node.get_node_graph_interface(), node.get_node_services_interface(), - node.get_node_parameters_interface(), node.get_logger(), source_id, service_name) { + node.get_node_parameters_interface(), node.get_node_timers_interface(), node.get_logger(), source_id, + service_name) { } FaultReporter::FaultReporter(const rclcpp_lifecycle::LifecycleNode::SharedPtr & node, const std::string & source_id, const std::string & service_name) : FaultReporter(node->get_node_base_interface(), node->get_node_graph_interface(), - node->get_node_services_interface(), node->get_node_parameters_interface(), node->get_logger(), - source_id, service_name) { + node->get_node_services_interface(), node->get_node_parameters_interface(), + node->get_node_timers_interface(), node->get_logger(), source_id, service_name) { } FilterConfig @@ -113,6 +417,35 @@ FaultReporter::load_parameters(const rclcpp::node_interfaces::NodeParametersInte return config; } +core::DeliveryConfig +FaultReporter::load_delivery_config(const rclcpp::node_interfaces::NodeParametersInterface::SharedPtr & node_params, + const rclcpp::Logger & logger) { + const core::DeliveryConfig defaults; + if (!node_params->has_parameter("fault_reporter.delivery.ack_timeout_sec")) { + node_params->declare_parameter("fault_reporter.delivery.ack_timeout_sec", + rclcpp::ParameterValue(defaults.ack_timeout_sec)); + } + if (!node_params->has_parameter("fault_reporter.delivery.max_attempts")) { + node_params->declare_parameter("fault_reporter.delivery.max_attempts", + rclcpp::ParameterValue(static_cast(defaults.max_attempts))); + } + if (!node_params->has_parameter("fault_reporter.delivery.max_pending")) { + node_params->declare_parameter("fault_reporter.delivery.max_pending", + rclcpp::ParameterValue(static_cast(defaults.max_pending))); + } + + // Parameters are int64: the range is checked before narrowing. + std::vector warnings; + const auto config = core::validate_delivery_params( + node_params->get_parameter("fault_reporter.delivery.ack_timeout_sec").as_double(), + node_params->get_parameter("fault_reporter.delivery.max_attempts").as_int(), + node_params->get_parameter("fault_reporter.delivery.max_pending").as_int(), &warnings); + for (const auto & warning : warnings) { + RCLCPP_WARN(logger, "%s", warning.c_str()); + } + return config; +} + void FaultReporter::report(const std::string & fault_code, uint8_t severity, const std::string & description) { // Validate fault_code if (fault_code.empty()) { @@ -166,8 +499,7 @@ void FaultReporter::send_report(const ros2_medkit::fault_detection::FaultReport request->description = report.description; request->source_id = source_id_; - // Fire and forget - don't block on response - client_->async_send_request(request); + delivery_->enqueue(request); if (failed) { RCLCPP_DEBUG(logger_, "Reported FAILED: %s (severity=%d)", report.fault_code.c_str(), report.severity); diff --git a/src/ros2_medkit_fault_reporter/test/delivery_probe_node.cpp b/src/ros2_medkit_fault_reporter/test/delivery_probe_node.cpp new file mode 100644 index 000000000..fc3e9764e --- /dev/null +++ b/src/ros2_medkit_fault_reporter/test/delivery_probe_node.cpp @@ -0,0 +1,185 @@ +// Copyright 2026 bburda +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Test node for the delivery tests: reports faults through FaultReporter in the pattern named by +// its `mode` parameter and logs what it did, so a launch test can compare it with the FaultManager. + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "rclcpp/rclcpp.hpp" +#include "ros2_medkit_fault_reporter/fault_reporter.hpp" +#include "ros2_medkit_msgs/msg/fault.hpp" + +using ros2_medkit_fault_reporter::FaultReporter; +using Clock = std::chrono::steady_clock; + +namespace { + +void poll_until(const rclcpp::Node::SharedPtr & node, std::chrono::seconds limit, const std::function & done) { + const auto deadline = Clock::now() + limit; + while (rclcpp::ok() && !done() && Clock::now() < deadline) { + rclcpp::spin_some(node); + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } +} + +// `ctor` "raw" builds the reporters from node interfaces without a timers interface. +std::vector> +fresh_reporters(const rclcpp::Node::SharedPtr & node, size_t count, const std::string & ctor = "node", + const std::string & service = "/fault_manager/report_fault") { + std::vector> reporters; + for (size_t i = 0; i < count; ++i) { + if (ctor == "raw") { + reporters.push_back(std::make_unique( + node->get_node_base_interface(), node->get_node_graph_interface(), node->get_node_services_interface(), + node->get_node_parameters_interface(), node->get_logger(), "/delivery_probe", service)); + } else { + reporters.push_back(std::make_unique(node, "/delivery_probe", service)); + } + } + return reporters; +} + +bool all_ready(const std::vector> & reporters) { + for (const auto & r : reporters) { + if (!r->is_service_ready()) { + return false; + } + } + return true; +} + +size_t total_pending(const std::vector> & reporters) { + size_t n = 0; + for (const auto & r : reporters) { + n += r->pending_reports(); + } + return n; +} + +} // namespace + +int run(int argc, char * argv[]) { + rclcpp::init(argc, argv); + auto node = std::make_shared("delivery_probe"); + const auto mode = node->declare_parameter("mode", "burst"); + const auto count = static_cast(node->declare_parameter("count", 24)); + const int rounds = static_cast(node->declare_parameter("rounds", 1)); + const auto prefix = node->declare_parameter("prefix", "P"); + constexpr auto kLimit = std::chrono::seconds(60); + const auto severity = ros2_medkit_msgs::msg::Fault::SEVERITY_ERROR; + + if (mode == "config") { + FaultReporter reporter(node, "/delivery_probe"); + const auto & c = reporter.delivery_config(); + std::printf("CONFIG ack=%g attempts=%d pending=%d\n", c.ack_timeout_sec, c.max_attempts, c.max_pending); + } else if (mode == "burst") { + // Fresh reporters, each reporting the moment its client sees the service. + for (int r = 0; r < rounds; ++r) { + auto reporters = fresh_reporters(node, count); + std::vector sent(count, false); + poll_until(node, kLimit, [&]() { + bool all = true; + for (size_t i = 0; i < count; ++i) { + if (!sent[i] && reporters[i]->is_service_ready()) { + reporters[i]->report(prefix + "_" + std::to_string(r) + "_" + std::to_string(i), severity, "burst"); + sent[i] = true; + } + all = all && sent[i]; + } + return all; + }); + poll_until(node, kLimit, [&]() { + return total_pending(reporters) == 0; + }); + std::printf("ROUND %d pending=%zu\n", r, total_pending(reporters)); + } + std::printf("DONE\n"); + } else if (mode == "order") { + // FAILED then PASSED at once, on a fresh reporter per code. + auto reporters = fresh_reporters(node, count); + poll_until(node, kLimit, [&]() { + return all_ready(reporters); + }); + for (size_t i = 0; i < count; ++i) { + const auto code = prefix + "_" + std::to_string(i); + reporters[i]->report(code, severity, "order"); + reporters[i]->report_passed(code); + } + poll_until(node, kLimit, [&]() { + return total_pending(reporters) == 0; + }); + std::printf("DONE pending=%zu\n", total_pending(reporters)); + } else if (mode == "stalled") { + // The manager is stopped by the test: report and then log the pending count as it changes. + // Code i goes to reporter i % reporters. With `destroy` the reporters are destroyed after reporting. + const auto reporter_count = + static_cast(node->declare_parameter("reporters", static_cast(count))); + const auto ctor = node->declare_parameter("ctor", "node"); + const auto service = node->declare_parameter("service", "/fault_manager/report_fault"); + const bool destroy = node->declare_parameter("destroy", false); + auto reporters = fresh_reporters(node, reporter_count, ctor, service); + const auto go_file = node->declare_parameter("go_file", ""); + poll_until(node, kLimit, [&]() { + return all_ready(reporters); + }); + // The test stops the manager after READY and then creates go_file. + std::printf("READY\n"); + std::fflush(stdout); + poll_until(node, kLimit, [&]() { + return std::ifstream(go_file).good(); + }); + for (size_t i = 0; i < count; ++i) { + reporters[i % reporter_count]->report(prefix + "_" + std::to_string(i), severity, "stalled"); + } + std::printf("REPORTED pending=%zu\n", total_pending(reporters)); + std::fflush(stdout); + size_t last = total_pending(reporters); + if (destroy) { + reporters.clear(); + last = 0; + std::printf("DESTROYED\n"); + std::fflush(stdout); + } + while (rclcpp::ok()) { + rclcpp::spin_some(node); + const size_t now = total_pending(reporters); + if (now != last) { + last = now; + std::printf("PENDING %zu\n", now); + std::fflush(stdout); + } + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } + } + std::fflush(stdout); + rclcpp::shutdown(); + return 0; +} + +int main(int argc, char * argv[]) { + try { + return run(argc, argv); + } catch (const std::exception & e) { + std::fprintf(stderr, "delivery_probe_node: %s\n", e.what()); + return 1; + } +} diff --git a/src/ros2_medkit_fault_reporter/test/test_delivery.test.py b/src/ros2_medkit_fault_reporter/test/test_delivery.test.py new file mode 100644 index 000000000..20e9b776e --- /dev/null +++ b/src/ros2_medkit_fault_reporter/test/test_delivery.test.py @@ -0,0 +1,412 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""FaultReporter delivery against a real fault_manager_node.""" + +import os +import queue +import signal +import subprocess +import sys +import threading +import time +import unittest + +from ament_index_python.packages import get_package_prefix +from launch import LaunchDescription +import launch_ros.actions +import launch_testing.actions +import rclpy +from rclpy.node import Node +from ros2_medkit_msgs.msg import Fault +from ros2_medkit_msgs.srv import ListFaults + +ALL_STATUSES = [Fault.STATUS_CONFIRMED, Fault.STATUS_HEALED] +MANAGER_PARAMS = { + 'storage_type': 'memory', + 'confirmation_threshold': -1, + 'healing_enabled': True, + 'healing_threshold': 0, +} + + +def generate_test_description(): + fault_manager = launch_ros.actions.Node( + package='ros2_medkit_fault_manager', + executable='fault_manager_node', + name='fault_manager', + output='screen', + parameters=[MANAGER_PARAMS], + ) + return LaunchDescription([fault_manager, launch_testing.actions.ReadyToTest()]) + + +def probe_path(): + return os.path.join(get_package_prefix('ros2_medkit_fault_reporter'), + 'lib', 'ros2_medkit_fault_reporter', 'delivery_probe_node') + + +class Manager: + """A fault_manager_node process named name, so its services are //...""" + + def __init__(self, name): + args = [os.path.join(get_package_prefix('ros2_medkit_fault_manager'), + 'lib', 'ros2_medkit_fault_manager', 'fault_manager_node'), + '--ros-args', '-r', f'__node:={name}'] + for key, value in MANAGER_PARAMS.items(): + args += ['-p', f'{key}:={value}'] + self.proc = subprocess.Popen(args) + + def kill(self): + """SIGKILL, which also ends a stopped process: requests it has not read are lost.""" + if self.proc.poll() is None: + self.proc.kill() + self.proc.wait() + + def stop(self): + if self.proc.poll() is None: + self.proc.send_signal(signal.SIGINT) + try: + self.proc.wait(timeout=30) + except subprocess.TimeoutExpired: + self.kill() + + +def find_pid(name): + """PID of the process of this test's ROS domain whose command line holds name.""" + domain = b'ROS_DOMAIN_ID=' + os.environ['ROS_DOMAIN_ID'].encode() + for entry in os.listdir('/proc'): + if not entry.isdigit(): + continue + try: + with open(f'/proc/{entry}/cmdline', 'rb') as f: + cmd = f.read() + with open(f'/proc/{entry}/environ', 'rb') as f: + env = f.read().split(b'\0') + except OSError: + continue + if name.encode() in cmd and domain in env: + return int(entry) + raise AssertionError(f'{name} not found') + + +class Probe: + """A delivery_probe_node process with its stdout read on a thread.""" + + def __init__(self, **params): + args = [probe_path(), '--ros-args'] + for key, value in params.items(): + args += ['-p', f'{key.replace("__", ".")}:={value}'] + self.proc = subprocess.Popen( + args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + self.lines = queue.Queue() + self.seen = [] + threading.Thread(target=self._read, daemon=True).start() + + def _read(self): + for line in self.proc.stdout: + self.lines.put(line.rstrip()) + + def wait_for(self, predicate, timeout=60.0): + """Return the first output line (new or already seen) for which predicate is true.""" + deadline = time.time() + timeout + index = 0 + while time.time() < deadline: + while index < len(self.seen): + if predicate(self.seen[index]): + return self.seen[index] + index += 1 + try: + self.seen.append(self.lines.get(timeout=0.2)) + except queue.Empty: + pass + raise AssertionError(f'no matching output; saw {self.seen[-10:]}') + + def count(self, text): + while True: + try: + self.seen.append(self.lines.get_nowait()) + except queue.Empty: + break + return sum(1 for line in self.seen if text in line) + + def stop(self, exits_alone=False): + if exits_alone: + try: + self.proc.wait(timeout=30) + except subprocess.TimeoutExpired: + pass + if self.proc.poll() is None: + self.proc.send_signal(signal.SIGINT) + try: + self.proc.wait(timeout=30) + except subprocess.TimeoutExpired: + self.proc.kill() + self.proc.wait() + return self.proc.returncode + + +def stalled_probe(manager_pid, go_file, **params): + """Start a probe, stop the manager once the reporters are ready, then let the probe report.""" + if os.path.exists(go_file): + os.remove(go_file) + probe = Probe(mode='stalled', go_file=go_file, **params) + probe.wait_for(lambda line: line == 'READY') + os.kill(manager_pid, signal.SIGSTOP) + open(go_file, 'w').close() + return probe + + +class TestDelivery(unittest.TestCase): + + @classmethod + def setUpClass(cls): + rclpy.init() + cls.node = Node('delivery_test_client') + cls.lister = cls.node.create_client(ListFaults, '/fault_manager/list_faults') + assert cls.lister.wait_for_service(timeout_sec=20.0) + cls.manager_pid = find_pid('fault_manager_node') + + @classmethod + def tearDownClass(cls): + cls.node.destroy_node() + rclpy.shutdown() + + def faults(self, prefix, lister=None): + lister = lister or self.lister + req = ListFaults.Request() + req.filter_by_severity = False + req.statuses = ALL_STATUSES + fut = lister.call_async(req) + rclpy.spin_until_future_complete(self.node, fut, timeout_sec=5.0) + result = fut.result() + found = result.faults if result is not None else [] + return {f.fault_code: f for f in found if f.fault_code.startswith(prefix)} + + def wait_faults(self, prefix, predicate, timeout=30.0, lister=None): + deadline = time.time() + timeout + found = {} + while time.time() < deadline: + found = self.faults(prefix, lister) + if predicate(found): + break + time.sleep(0.2) + return found + + def wait_clients_gone(self, service, timeout=30.0): + """Poll the graph until the probe node has no client of service; return what is left.""" + deadline = time.time() + timeout + left = [] + while time.time() < deadline: + clients = self.node.get_client_names_and_types_by_node('delivery_probe', '/') + left = [n for n, _ in clients if n == service] + if not left: + break + time.sleep(0.2) + return left + + def restart_case(self, prefix, name, count, **params): + """ + Report to a stopped manager, kill it, start a new one: every report must reach the new one. + + The killed manager never read the requests, so each one arrives only if it is sent again. + """ + lister = self.node.create_client(ListFaults, f'/{name}/list_faults') + first = Manager(name) + second = None + probe = None + try: + self.assertTrue(lister.wait_for_service(timeout_sec=20.0)) + # A budget of 30 s, well past the start of a new manager. + probe = stalled_probe( + first.proc.pid, f'/tmp/delivery_go_{name}', count=count, prefix=prefix, + service=f'/{name}/report_fault', + fault_reporter__delivery__ack_timeout_sec=0.3, + fault_reporter__delivery__max_attempts=100, **params) + probe.wait_for(lambda line: line == f'REPORTED pending={count}') + if params.get('destroy') == 'true': + probe.wait_for(lambda line: line == 'DESTROYED') + first.kill() + second = Manager(name) + found = self.wait_faults(prefix + '_', lambda f: len(f) == count, + timeout=60.0, lister=lister) + self.assertEqual(sorted(found), sorted(f'{prefix}_{i}' for i in range(count))) + if params.get('destroy') == 'true': + # Once confirmed, the reports of destroyed reporters free their clients. + self.assertEqual(self.wait_clients_gone(f'/{name}/report_fault'), []) + else: + probe.wait_for(lambda line: line == 'PENDING 0') + self.assertEqual(probe.count('dropping the report'), 0, + [line for line in probe.seen if 'dropping' in line]) + finally: + if probe is not None: + probe.stop() + first.kill() + if second is not None: + second.stop() + self.node.destroy_client(lister) + + def test_reports_reach_a_restarted_manager(self): + """Reports pending when the manager dies reach the manager started after it.""" + self.restart_case('RESTART', 'fm_restart', count=6) + + def test_raw_interfaces_reporter_sends_a_lone_report_again(self): + """A reporter built from node interfaces without timers sends its one report again.""" + self.restart_case('RAW', 'fm_raw', count=1, reporters=1, ctor='raw') + + def test_destroyed_reporters_deliver_their_pending_reports(self): + """The pending reports of destroyed reporters are sent until the manager confirms them.""" + self.restart_case('GONE', 'fm_gone', count=6, destroy='true') + + def test_reports_pending_at_shutdown_are_dropped_with_a_warning(self): + """At shutdown each report still waiting for the manager is dropped with a warning.""" + count = 3 + for destroy in ('false', 'true'): + prefix = 'DOWN_GONE' if destroy == 'true' else 'DOWN_LIVE' + with self.subTest(destroy=destroy): + probe = None + try: + probe = stalled_probe( + self.manager_pid, f'/tmp/delivery_go_down_{destroy}', count=count, + prefix=prefix, destroy=destroy, + fault_reporter__delivery__ack_timeout_sec=60.0) + probe.wait_for(lambda line: line == f'REPORTED pending={count}') + if destroy == 'true': + probe.wait_for(lambda line: line == 'DESTROYED') + probe.proc.send_signal(signal.SIGINT) + for i in range(count): + probe.wait_for( + lambda line, i=i: f"dropping the report for '{prefix}_{i}'" in line) + self.assertEqual(probe.stop(exits_alone=True), 0) + finally: + os.kill(self.manager_pid, signal.SIGCONT) + if probe is not None: + probe.stop() + + def test_burst_of_fresh_reporters_loses_nothing(self): + """Fresh reporters that report the moment their client is ready lose no report.""" + rounds, count = 12, 24 + # Busy processes crowd the host so a request can overtake the server's match. + load = [subprocess.Popen([sys.executable, '-c', 'while True: pass']) + for _ in range((os.cpu_count() or 4) * 2)] + try: + probe = Probe(mode='burst', count=count, rounds=rounds, prefix='BURST') + probe.wait_for(lambda line: line == 'DONE', timeout=200) + self.assertEqual(probe.stop(exits_alone=True), 0) + finally: + for p in load: + p.kill() + p.wait() + found = self.faults('BURST_') + self.assertEqual( + len(found), rounds * count, + f'{rounds * count - len(found)} of {rounds * count} reports never reached the manager') + + def test_events_of_one_code_arrive_in_order(self): + """FAILED then PASSED reported at once on a fresh reporter ends healed for every code.""" + count = 48 + probe = Probe(mode='order', count=count, prefix='ORDER', + fault_reporter__local_filtering__enabled='false') + probe.wait_for(lambda line: line.startswith('DONE')) + probe.stop(exits_alone=True) + found = self.wait_faults( + 'ORDER_', + lambda f: len(f) == count and all( + x.status == Fault.STATUS_HEALED for x in f.values())) + self.assertEqual(len(found), count) + wrong = sorted(c for c, f in found.items() if f.status != Fault.STATUS_HEALED) + self.assertEqual(wrong, [], f'{len(wrong)} codes not healed: {wrong[:5]}') + + def test_reports_wait_for_an_unresponsive_manager(self): + """Reports made while the manager does not answer arrive once it does.""" + count = 6 + probe = None + try: + probe = stalled_probe( + self.manager_pid, '/tmp/delivery_go_wait', count=count, + fault_reporter__delivery__ack_timeout_sec=0.2, + fault_reporter__delivery__max_attempts=100, prefix='WAIT') + probe.wait_for(lambda line: line.startswith('REPORTED')) + finally: + os.kill(self.manager_pid, signal.SIGCONT) + found = self.wait_faults('WAIT_', lambda f: len(f) == count) + probe.wait_for(lambda line: line == 'PENDING 0') + probe.stop() + self.assertEqual(len(found), count) + + def test_report_is_dropped_after_the_attempt_budget(self): + """With no response, a report is sent max_attempts times, then dropped with a warning.""" + count = 4 + probe = None + try: + probe = stalled_probe( + self.manager_pid, '/tmp/delivery_go_budget', count=count, + fault_reporter__delivery__ack_timeout_sec=0.1, + fault_reporter__delivery__max_attempts=3, prefix='BUDGET') + probe.wait_for(lambda line: line.startswith('REPORTED')) + probe.wait_for(lambda line: line == 'PENDING 0') + for i in range(count): + probe.wait_for(lambda line, i=i: f"dropping the report for 'BUDGET_{i}'" in line + and 'after 3 attempts' in line) + finally: + os.kill(self.manager_pid, signal.SIGCONT) + if probe is not None: + probe.stop() + + def test_pending_reports_are_capped(self): + """Past max_pending a new report is dropped with a warning; memory stays bounded.""" + cap, count = 8, 20 + probe = None + try: + probe = stalled_probe( + self.manager_pid, '/tmp/delivery_go_cap', count=count, reporters=1, + fault_reporter__delivery__max_pending=cap, + fault_reporter__delivery__ack_timeout_sec=60.0, prefix='CAP') + probe.wait_for(lambda line: line == f'REPORTED pending={cap}') + finally: + os.kill(self.manager_pid, signal.SIGCONT) + found = self.wait_faults('CAP_', lambda f: len(f) >= cap) + probe.wait_for(lambda line: line == 'PENDING 0') + self.assertEqual(probe.count('dropping the report for'), count - cap) + # The reports kept are the first `cap`; none after them gets in. + self.assertEqual(sorted(found), sorted(f'CAP_{i}' for i in range(cap))) + probe.stop() + + def test_delivery_parameters(self): + """Each parameter is accepted at both ends of its range; a bad value gets the default.""" + default = 'CONFIG ack=0.5 attempts=10 pending=1024' + cases = [ + ({'ack_timeout_sec': 0.05, 'max_attempts': 1, 'max_pending': 1}, + 'CONFIG ack=0.05 attempts=1 pending=1'), + ({'ack_timeout_sec': 60.0, 'max_attempts': 100, 'max_pending': 100000}, + 'CONFIG ack=60 attempts=100 pending=100000'), + ({'ack_timeout_sec': '.nan'}, default), + ({'ack_timeout_sec': -1.0}, default), + ({'ack_timeout_sec': 0.04}, default), + ({'ack_timeout_sec': 60.5}, default), + ({'max_attempts': 0}, default), + ({'max_attempts': 101}, default), + ({'max_attempts': 4294967306}, default), + ({'max_pending': -5}, default), + ({'max_pending': 100001}, default), + ({'max_pending': 4294967297}, default), + ] + for params, expected in cases: + with self.subTest(params=params): + probe = Probe(mode='config', **{ + f'fault_reporter__delivery__{k}': v for k, v in params.items()}) + line = probe.wait_for(lambda line: line.startswith('CONFIG'), timeout=30) + self.assertEqual(probe.stop(exits_alone=True), 0) + self.assertEqual(line, expected) diff --git a/src/ros2_medkit_gateway/README.md b/src/ros2_medkit_gateway/README.md index 4f4c108e4..230d1415f 100644 --- a/src/ros2_medkit_gateway/README.md +++ b/src/ros2_medkit_gateway/README.md @@ -1750,7 +1750,7 @@ curl --cacert ./certs/ca.crt https://localhost:8443/api/v1 The package compiles into two layered static libraries with a strict dependency direction: - **`gateway_core`** - the ROS-free layer, defined in `cmake/gateway_core.cmake`. Sources: `src/core/`, the HTTP server and REST server, `src/http/handlers/`, `src/openapi/`, the plugin manager and loader, aggregation, and the script manager. Headers live under `include/ros2_medkit_gateway/core/` and the neighbouring `http/`, `plugins/` and `aggregation/` directories. Links only header-only and C-level externals (cpp-httplib, nlohmann/json, yaml-cpp, tl::expected, jwt-cpp, OpenSSL, SQLite, dl). Carries no rclcpp / rcl_interfaces / message-package dependency. -- **`gateway_ros2`** - the ROS adapters. Compiles the remaining sources under `src/` and links `gateway_core` publicly via `medkit_target_dependencies`. Hosts `GatewayNode` (the `rclcpp::Node` entry point), the ROS transports and providers, runtime discovery, the native topic sampler, and the plugin context. +- **`gateway_ros2`** - the ROS adapters. Compiles the remaining sources under `src/` and links `gateway_core` publicly via `medkit_target_dependencies`. Hosts `GatewayNode` (the `rclcpp::Node` entry point), the ROS transports and providers, runtime discovery, the native topic sampler, and the plugin context of the node (a `CorePluginContext` whose `node()` is the gateway node). The core compiles and is tested without ROS and without DDS setup. `cmake/gateway_core.cmake` is plain CMake. The ament build includes it by path, and so does `scripts/check_cores_without_ros`, which links `gateway_core`: @@ -1770,6 +1770,8 @@ A program that hosts the core gets the cyclic-subscription samplers and the bulk - **Scripts.** `make_script_host(config, plugin_backend)` (`core/managers/script_manager.hpp`) returns a `ScriptManager` and, when no plugin backend is given and `ScriptsConfig::scripts_dir` is not empty, the `DefaultScriptProvider` (fork/execvp) behind it. `ScriptsConfig` (`core/script_types.hpp`) has the defaults of the `scripts.*` parameters, and `entries` carries predefined scripts. The host returns `host.manager` from `GatewayServices::script_manager()` and keeps the `ScriptHost` alive as long as the services. With neither a backend nor a directory the script routes answer 501. - **Aggregation.** `AggregationHost(config, listen_port, mdns_name, logger)` (`core/aggregation/aggregation_host.hpp`) builds the `AggregationManager` from an `AggregationConfig` (the defaults of the `aggregation.*` parameters; the host range-checks values it reads from its own configuration) and, when `announce` or `discover` is set, the mDNS discovery that adds and removes peers and ignores this gateway's own announcements. The host passes `host.manager()` to `RESTServer::set_aggregation_manager()` and calls `host.merge_peers(areas, components, apps, functions)` on every entity refresh, before it fills the entity cache. The call merges the peers' entities into the four vectors and publishes the routing table, the leaf warnings and the per-entity peer contributors. It does nothing while there is no peer. Call `host.shutdown()` before `RESTServer::stop()`. - **Triggers.** `make_trigger_host(config, notifier, conditions, wiring)` (`core/trigger_host.hpp`) builds the `SqliteTriggerStore` and the `TriggerManager`, wires the entity hierarchy, entity existence and topic lookups to `wiring.cache`, and restores persistent triggers. `TriggerHostConfig` has the names and defaults of the `triggers.*` parameters (`enabled`, `max_triggers`, `on_restart_behavior`, `storage_path`), and `data_poll_interval_ms` (default 1000, clamped to 100-60000 ms), which the node does not read. `register_builtin_conditions()` fills the `ConditionRegistry`. The host passes the manager to `RESTServer::set_trigger_handlers()` and returns it from `GatewayServices::trigger_manager()`. A trigger past `max_triggers` answers 503. Where its events come from is chosen by the wiring: `fault_events` (a `FaultEventSource`; it serves several subscribers, so this can be the source the host returns from `GatewayServices::fault_events()`) starts a `TriggerFaultFeed` that subscribes to it and turns each fault event into a `faults` notification, and a fault trigger fires on it. The feed's subscription ends when the host is destroyed. A data trigger reads from `topic_transport` (a `TopicSubscriptionTransport`, the ROS 2 node's choice) when its topic is known, else from `polled_plugins`: a `PollingDataItemSource` that reads the item through `read_plugin_data_item`, as `GET /{entity}/data/{item}` does once per `data_poll_interval_ms`, starting at once. The item name is the trigger `resource` path percent-decoded as the GET path is. It delivers the `data` field of the body (the whole body when there is none), so a trigger `path` is relative to the value, as it is for a topic. A host without topics sets only `polled_plugins`; a host with neither gets data triggers that never fire. The trigger's condition runs on every sample, so `EnterRange` fires once at the crossing and `OnChange` at each difference. An item that cannot be read in a round delivers nothing in that round. Keep the host alive as long as the services. The ROS 2 node uses the same builder with its topic transport and feeds fault triggers through `TriggerFaultSubscriber`, a `Ros2FaultEventSource` under a `TriggerFaultFeed`. +- **Plugin context.** `make_core_plugin_context(services, wiring)` (`core/plugins/core_plugin_context.hpp`) builds the context for plugins from `GatewayServices`: entities and the entity snapshot from `entity_cache()`, locks from `lock_manager()`, faults from `fault_manager()`, samplers from `sampler_registry()`. The host passes it to `PluginManager::set_context()` and keeps it alive as long as the plugins. `CorePluginContextWiring` carries what `GatewayServices` does not: getters for the change notifier and the condition registry (a null getter or a null result gives the plugin a null), a getter for the `AggregationManager` (the fault listings add the peers' faults), a callback for `notify_entities_changed()` (none: the call is ignored) and a sink for notices. The context is a `RosPluginContext`, so `as_ros_plugin_context()` is valid for every plugin, and its `node()` returns null: a plugin that needs ROS checks for it and declines to start. Plugins that use no ROS (a provider, a route, an introspection source) run unchanged. `PluginManager::load_plugins()` loads them from shared objects as on the node. The ROS 2 node derives its context from the same class and overrides only `node()`. +- **Fault-trigger engine.** `make_fault_trigger_host(config, wiring)` (`core/fault_trigger_host.hpp`) builds the threshold-rule engine behind `/apps/{id}/fault-triggers` and the thread that evaluates its rules. `FaultTriggerHostConfig` has the names and defaults of the `fault_triggers.*` parameters (`enabled`, `poll_interval_ms`, raised to 50 ms and capped at `INT_MAX`, `storage_path`). The wiring has the entity cache, the `PluginManager` the rules read their values from (through the plugin's `x-plc-data` route), and the `FaultManager` a crossed rule reports to and clears through. The factory returns null when the configuration is disabled or no plugin is loaded. The host returns the engine from `GatewayServices::fault_trigger_engine()`; without it the routes answer 501. The engine consumes no fault events. The fault it reports confirms in the fault manager and reaches the fault SSE stream, the freeze-frame capture and fault triggers through the host's event source. Destroy the host before the plugins and the fault manager go. The ROS 2 node uses the same builder. - **Lifecycle.** `RESTServer` takes a `LifecycleStateReader` or null. A host without one passes null. `GET /{entity}/status` then reports `ready` for an online app and `notReady` for an offline one, also when the app exposes the lifecycle services, because graph presence is the only signal left. A component is ready unless every app it hosts is offline. `PUT /{entity}/status/{action}` answers 501 unless a plugin `LifecycleProvider` serves the entity. The `gateway_node` executable and existing test targets link `gateway_ros2`, so they transitively get both layers from a single dependency. The neutral contract is enforced by two CTest checks: diff --git a/src/ros2_medkit_gateway/design/index.rst b/src/ros2_medkit_gateway/design/index.rst index 2305fcee5..29543915c 100644 --- a/src/ros2_medkit_gateway/design/index.rst +++ b/src/ros2_medkit_gateway/design/index.rst @@ -101,6 +101,22 @@ store from it: data a plugin serves needs no topics. The ROS 2 node passes the topic transport and feeds fault triggers through ``TriggerFaultSubscriber``, a ROS 2 event source under the same feed. +* ``make_core_plugin_context`` (``core/plugins/core_plugin_context.hpp``) builds + the plugin context from ``GatewayServices`` and a ``CorePluginContextWiring`` + (notifier, condition registry, aggregation, entity-change callback, notices). + It is a ``RosPluginContext`` whose ``node()`` is null, so + ``as_ros_plugin_context()`` is valid for every plugin and a plugin that needs + ROS checks the pointer. The host gives it to ``PluginManager::set_context()``. + The ROS 2 node derives its context from the same class and overrides only + ``node()``. +* ``make_fault_trigger_host`` (``core/fault_trigger_host.hpp``) builds the + threshold-rule ``FaultTriggerEngine`` and the thread that evaluates it from a + ``FaultTriggerHostConfig`` (the ``fault_triggers.*`` names and defaults). The + rules read plugin data through the ``PluginManager`` and report and clear + faults through the ``FaultManager``. It returns null when disabled or when no + plugin is loaded. The host returns the engine from + ``GatewayServices::fault_trigger_engine()``. The ROS 2 node uses the same + builder. * ``RESTServer`` takes a null ``LifecycleStateReader`` for a host without lifecycle nodes. ``GET /{entity}/status`` then derives readiness from presence and ``PUT /{entity}/status/{action}`` answers 501 without a diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/fault_trigger_host.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/fault_trigger_host.hpp new file mode 100644 index 000000000..80afefd92 --- /dev/null +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/fault_trigger_host.hpp @@ -0,0 +1,91 @@ +// Copyright 2026 bburda +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#pragma once + +#include +#include +#include +#include +#include +#include + +#include "ros2_medkit_gateway/core/fault_trigger_engine.hpp" + +namespace ros2_medkit_gateway { + +class FaultManager; +class PluginManager; +class ThreadSafeEntityCache; + +/// Settings of the fault-trigger (threshold-rule) engine. Names and defaults are those of the `fault_triggers.*` +/// parameters. +struct FaultTriggerHostConfig { + bool enabled{true}; ///< `fault_triggers.enabled` + int64_t poll_interval_ms{1000}; ///< `fault_triggers.poll_interval_ms`; raised to 50 ms, capped at INT_MAX + std::string storage_path; ///< `fault_triggers.storage.path`; empty keeps the rules in memory +}; + +/// What the host supplies around the engine. Every pointer must outlive the host. +struct FaultTriggerHostWiring { + /// Entity existence for rule creation. Required. + const ThreadSafeEntityCache * cache{nullptr}; + /// The rules read the values of their data points through the plugins' data routes. Required. + PluginManager * plugins{nullptr}; + /// A rule that crosses its threshold reports its fault here, and clears it here. Null: nothing is reported. + FaultManager * faults{nullptr}; + /// Notices: the engine's log lines and its evaluation errors. + std::function log; + /// Operator-facing notices (a clamped setting, an evaluation error). + std::function warn; +}; + +/// The fault-trigger engine and its evaluation loop. +/// +/// The loop runs on a thread of its own: reading a value and reporting a fault must not run on the host's executor. +/// The destructor joins it, so destroy the host before the plugins and the fault manager go. +class FaultTriggerHost { + public: + /// Starts the loop. Use `make_fault_trigger_host()`. + FaultTriggerHost(const FaultTriggerHostConfig & config, int poll_interval_ms, const FaultTriggerHostWiring & wiring); + ~FaultTriggerHost(); + FaultTriggerHost(const FaultTriggerHost &) = delete; + FaultTriggerHost & operator=(const FaultTriggerHost &) = delete; + FaultTriggerHost(FaultTriggerHost &&) = delete; + FaultTriggerHost & operator=(FaultTriggerHost &&) = delete; + + FaultTriggerEngine & engine() { + return *engine_; + } + + /// The interval in use. + int poll_interval_ms() const { + return poll_interval_ms_; + } + + private: + std::function warn_; + int poll_interval_ms_; + std::unique_ptr engine_; + std::atomic stop_{false}; + std::thread thread_; +}; + +/// Builds the engine and starts its loop. Null when the configuration is disabled or `wiring.plugins` has no plugin +/// loaded: there is nothing to read the values from. Throws `std::invalid_argument` when `wiring.cache` or +/// `wiring.plugins` is null. +std::unique_ptr make_fault_trigger_host(const FaultTriggerHostConfig & config, + const FaultTriggerHostWiring & wiring); + +} // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/core_plugin_context.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/core_plugin_context.hpp new file mode 100644 index 000000000..f306bf5de --- /dev/null +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/core_plugin_context.hpp @@ -0,0 +1,107 @@ +// Copyright 2026 bburda +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#pragma once + +#include +#include +#include +#include +#include +#include + +#include "ros2_medkit_gateway/core/gateway_services.hpp" +#include "ros2_medkit_gateway/plugins/ros_plugin_context.hpp" + +namespace ros2_medkit_gateway { + +class AggregationManager; +class ConditionRegistry; +class FaultManager; +class ResourceChangeNotifier; +class ResourceSamplerRegistry; +struct EntityChangeScope; + +/// What a host gives the plugin context besides its `GatewayServices`. +struct CorePluginContextWiring { + /// Fault source of `list_entity_faults()` and `list_all_faults()`. Null: those answer nothing from this host. + FaultManager * fault_manager{nullptr}; + /// Where `register_sampler()` registers. Null: the call is ignored. + ResourceSamplerRegistry * sampler_registry{nullptr}; + /// Read at each call, so the object may be built after the context. A null result gives the plugin a null. + std::function notifier; + std::function conditions; + /// Peer aggregation for the fault listings. Null or a null result: no peers. + std::function aggregation; + /// Runs `notify_entities_changed()`. Null: the notification is ignored. + std::function on_entities_changed; + /// Operator-facing notices (a partial peer fan-out). + std::function warn; +}; + +/// Plugin context over `GatewayServices`, with no ROS node. +/// +/// A program that hosts the core builds it with `make_core_plugin_context()` and gives it to +/// `PluginManager::set_context()`. It is a `RosPluginContext` so that `as_ros_plugin_context()` stays valid for every +/// plugin, and its `node()` is null: a plugin that needs ROS checks for it. The ROS 2 node derives from this class and +/// overrides only `node()`. +class CorePluginContext : public RosPluginContext { + public: + /// `services` may be null: entity queries then answer nothing, locks answer "disabled", and capabilities and + /// samplers still work. + CorePluginContext(GatewayServices * services, CorePluginContextWiring wiring); + + /// Null: this host has no ROS node. + rclcpp::Node * node() const override; + + std::optional get_entity(const std::string & id) const override; + std::vector get_child_apps(const std::string & component_id) const override; + nlohmann::json list_entity_faults(const std::string & entity_id) const override; + std::optional validate_entity_for_route(const PluginRequest & req, PluginResponse & res, + const std::string & entity_id) const override; + void register_capability(SovdEntityType entity_type, const std::string & capability_name) override; + void register_entity_capability(const std::string & entity_id, const std::string & capability_name) override; + std::vector get_type_capabilities(SovdEntityType entity_type) const override; + std::vector get_entity_capabilities(const std::string & entity_id) const override; + LockAccessResult check_lock(const std::string & entity_id, const std::string & client_id, + const std::string & collection) const override; + tl::expected acquire_lock(const std::string & entity_id, const std::string & client_id, + const std::vector & scopes, + int expiration_seconds) override; + tl::expected release_lock(const std::string & entity_id, const std::string & client_id) override; + IntrospectionInput get_entity_snapshot() const override; + nlohmann::json list_all_faults() const override; + void register_sampler( + const std::string & collection, + const std::function(const std::string &, const std::string &)> & fn) + override; + ResourceChangeNotifier * get_resource_change_notifier() override; + ConditionRegistry * get_condition_registry() override; + void notify_entities_changed(const EntityChangeScope & scope) override; + + private: + GatewayServices * services_; + CorePluginContextWiring wiring_; + + mutable std::mutex capabilities_mutex_; + std::unordered_map> type_capabilities_; + std::unordered_map> entity_capabilities_; +}; + +/// Builds the context of a host without a ROS node. The fault manager and the sampler registry of `services` are used +/// unless `wiring` names others. `services` must outlive the context. +std::unique_ptr make_core_plugin_context(GatewayServices & services, + CorePluginContextWiring wiring = {}); + +} // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/plugin_context.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/plugin_context.hpp index 0212eaceb..d5c75a224 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/plugin_context.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/plugins/plugin_context.hpp @@ -197,16 +197,16 @@ class PluginContext { /** * @brief Get the ResourceChangeNotifier for publishing or subscribing to resource changes. * - * Always returns a valid pointer - ResourceChangeNotifier is created unconditionally - * in GatewayNode regardless of trigger configuration. + * Valid on the ROS 2 gateway node regardless of trigger configuration. A host + * that gives its context no notifier answers null. */ virtual ResourceChangeNotifier * get_resource_change_notifier() = 0; /** * @brief Get the ConditionRegistry for registering custom trigger condition evaluators. * - * Always returns a valid pointer - ConditionRegistry is created unconditionally - * in GatewayNode regardless of trigger configuration. + * Valid on the ROS 2 gateway node regardless of trigger configuration. A host + * that gives its context no registry answers null. */ virtual ConditionRegistry * get_condition_registry() = 0; diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp index 679d27569..fd2a31dee 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/gateway_node.hpp @@ -33,6 +33,7 @@ #include "ros2_medkit_gateway/core/data/topic_data_provider.hpp" #include "ros2_medkit_gateway/core/default_script_provider.hpp" #include "ros2_medkit_gateway/core/fault_trigger_engine.hpp" +#include "ros2_medkit_gateway/core/fault_trigger_host.hpp" #include "ros2_medkit_gateway/core/gateway_services.hpp" #include "ros2_medkit_gateway/core/http/rate_limiter.hpp" #include "ros2_medkit_gateway/core/http/rest_server.hpp" @@ -504,11 +505,9 @@ class GatewayNode : public rclcpp::Node, public GatewayServices { std::unique_ptr entity_freeze_frame_capture_; // Destroyed after rest_server_, whose SSE fault handler holds a subscription to it. std::unique_ptr fault_event_source_; - // Config-less threshold-rule engine + its dedicated evaluation loop (issue - // #235). The thread is joined in ~GatewayNode BEFORE plugin/fault shutdown. - std::unique_ptr fault_trigger_engine_; - std::thread fault_trigger_thread_; - std::atomic fault_trigger_stop_{false}; + // Threshold-rule engine and its evaluation loop. Reset in ~GatewayNode + // BEFORE plugin/fault shutdown. + std::unique_ptr fault_trigger_host_; std::unique_ptr trigger_topic_subscriber_; // Adapter routing manager-side subscribe() calls onto trigger_topic_subscriber_. std::shared_ptr trigger_topic_transport_; diff --git a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/plugins/ros_plugin_context.hpp b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/plugins/ros_plugin_context.hpp index 783dc3103..f6ff80199 100644 --- a/src/ros2_medkit_gateway/include/ros2_medkit_gateway/plugins/ros_plugin_context.hpp +++ b/src/ros2_medkit_gateway/include/ros2_medkit_gateway/plugins/ros_plugin_context.hpp @@ -32,16 +32,20 @@ class ResourceSamplerRegistry; * @brief ROS-aware plugin context. * * Extends the middleware-neutral PluginContext with the rclcpp::Node * - * accessor used by ROS-coupled plugins. Lives in gateway_ros2 so that - * gateway_core stays free of any rclcpp coupling, even at the level of - * forward declarations or virtual signatures. Plugins that need ROS - * primitives (subscriptions, service clients, parameter access) take a - * RosPluginContext * member; plugins that only consume the neutral - * surface can keep a PluginContext * and never see rclcpp. + * accessor used by ROS-coupled plugins. The header only forward-declares + * rclcpp::Node, so gateway_core includes it without any ROS include path. + * Plugins that need ROS primitives (subscriptions, service clients, + * parameter access) take a RosPluginContext * member; plugins that only + * consume the neutral surface can keep a PluginContext * and never see rclcpp. + * + * Every context the gateway hands out is a RosPluginContext. On a host + * without a ROS node (see CorePluginContext) node() returns nullptr, so a + * plugin that needs ROS checks it and declines to start. */ class RosPluginContext : public PluginContext { public: /// Get the ROS 2 node pointer for subscriptions, service clients, etc. + /// Null on a host without a ROS node. virtual rclcpp::Node * node() const = 0; }; @@ -50,18 +54,17 @@ class RosPluginContext : public PluginContext { * * The gateway always passes a RosPluginContext through * GatewayPlugin::set_context, so this static_cast is safe for any - * in-process plugin loaded by PluginManager. Out-of-process or - * test-only PluginContext implementations that do not derive from - * RosPluginContext must not be passed through this helper. + * in-process plugin loaded by PluginManager, on the ROS 2 node and on a + * host without one. Out-of-process or test-only PluginContext + * implementations that do not derive from RosPluginContext must not be + * passed through this helper. */ inline RosPluginContext * as_ros_plugin_context(PluginContext & ctx) { return static_cast(&ctx); } -/// Factory for the concrete gateway plugin context. Always returns a -/// RosPluginContext because the gateway is itself a ROS 2 node; callers -/// that only consume the neutral PluginContext API can hold the result -/// as a unique_ptr. +/// Factory for the plugin context of the ROS 2 gateway node. Its node() is the +/// gateway node. A host without a node uses make_core_plugin_context(). std::unique_ptr make_gateway_plugin_context(GatewayNode * node, FaultManager * fault_manager, ResourceSamplerRegistry * sampler_registry = nullptr); diff --git a/src/ros2_medkit_gateway/src/core/fault_trigger_host.cpp b/src/ros2_medkit_gateway/src/core/fault_trigger_host.cpp new file mode 100644 index 000000000..1a6c04f6e --- /dev/null +++ b/src/ros2_medkit_gateway/src/core/fault_trigger_host.cpp @@ -0,0 +1,195 @@ +// Copyright 2026 bburda +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#include "ros2_medkit_gateway/core/fault_trigger_host.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ros2_medkit_gateway/core/faults/freeze_frame_source.hpp" +#include "ros2_medkit_gateway/core/managers/fault_manager.hpp" +#include "ros2_medkit_gateway/core/models/thread_safe_entity_cache.hpp" +#include "ros2_medkit_gateway/core/plugins/plugin_manager.hpp" + +namespace ros2_medkit_gateway { + +namespace { + +constexpr int64_t kMinPollIntervalMs = 50; + +// Coerce a discovered data point's JSON value to a double for threshold +// comparison. Handles the numeric, boolean and numeric-string shapes the PLC +// bridges emit; returns nullopt for anything non-numeric. +std::optional value_to_double(const nlohmann::json & v) { + if (v.is_number()) { + return v.get(); + } + if (v.is_boolean()) { + return v.get() ? 1.0 : 0.0; + } + if (v.is_string()) { + try { + size_t pos = 0; + const double d = std::stod(v.get(), &pos); + if (pos > 0) { + return d; + } + } catch (const std::exception &) { + return std::nullopt; + } + } + return std::nullopt; +} + +uint8_t severity_level(const std::string & severity) { + if (severity == "INFO") { + return 0; + } + if (severity == "ERROR") { + return 2; + } + if (severity == "CRITICAL") { + return 3; + } + return 1; // WARNING +} + +} // namespace + +FaultTriggerHost::FaultTriggerHost(const FaultTriggerHostConfig & config, int poll_interval_ms, + const FaultTriggerHostWiring & wiring) + : warn_(wiring.warn), poll_interval_ms_(poll_interval_ms) { + PluginManager * plugins = wiring.plugins; + FaultManager * faults = wiring.faults; + const ThreadSafeEntityCache * cache = wiring.cache; + + // Same in-process x-plc-data route the freeze-frame capture uses, so a rule sees exactly the value the /data + // endpoint would serve. + auto fetcher = [plugins](const std::string & app_id, const std::string & data_name) -> std::optional { + const auto content = plugins->fetch_entity_data_via_route(app_id); + // A down link serves frozen last-known values. Returning nullopt holds the rule state instead of firing on a stale + // number for the whole outage. + if (!content || !FreezeFrameSource::content_has_live_data(*content) || + FreezeFrameSource::content_reports_disconnected(*content)) { + return std::nullopt; + } + const auto values = FreezeFrameSource::values_from_list_content(*content); + if (!values.is_object() || !values.contains(data_name)) { + return std::nullopt; + } + return value_to_double(values[data_name]); + }; + + auto report = [faults](const std::string & app_id, const std::string & fault_code, const std::string & severity, + const std::string & description) { + if (faults) { + faults->report_fault(fault_code, severity_level(severity), description, app_id); + } + }; + + // A rule's auto-clear is local to the rule: skip the correlation cascade, so clearing one rule's fault never wipes + // correlated faults owned by anything else. fault_code is unique across rules, so the app scope is in the code. + auto clear = [faults](const std::string & /*app_id*/, const std::string & fault_code) { + if (faults) { + faults->clear_fault(fault_code, /*skip_correlation_auto_clear=*/true); + } + }; + + // Same route as the fetcher: "exists" means exactly "the fetcher could ever read it". + auto data_point_names = [plugins](const std::string & app_id) -> std::optional> { + const auto content = plugins->fetch_entity_data_via_route(app_id); + if (!content || !FreezeFrameSource::content_has_live_data(*content)) { + return std::nullopt; + } + const auto values = FreezeFrameSource::values_from_list_content(*content); + if (!values.is_object()) { + return std::nullopt; + } + std::vector names; + names.reserve(values.size()); + for (auto it = values.begin(); it != values.end(); ++it) { + names.push_back(it.key()); + } + return names; + }; + + // Distinct from the data route, which returns nullopt for both an unknown app and a transient outage. + auto entity_exists = [cache](const std::string & app_id) -> bool { + return cache->find_entity(app_id).has_value(); + }; + + engine_ = + std::make_unique(config.storage_path, std::move(fetcher), std::move(report), std::move(clear), + wiring.log, std::move(data_point_names), std::move(entity_exists)); + + const auto interval = std::chrono::milliseconds(poll_interval_ms_); + thread_ = std::thread([this, interval]() { + while (!stop_.load(std::memory_order_relaxed)) { + try { + engine_->evaluate_once(); + } catch (const std::exception & e) { + if (warn_) { + warn_(std::string("fault-trigger evaluation error: ") + e.what()); + } + } + // Short slices, so shutdown does not wait a whole interval. + auto slept = std::chrono::milliseconds(0); + while (slept < interval && !stop_.load(std::memory_order_relaxed)) { + std::this_thread::sleep_for(std::chrono::milliseconds(25)); + slept += std::chrono::milliseconds(25); + } + } + }); +} + +FaultTriggerHost::~FaultTriggerHost() { + stop_.store(true, std::memory_order_relaxed); + if (thread_.joinable()) { + thread_.join(); + } +} + +std::unique_ptr make_fault_trigger_host(const FaultTriggerHostConfig & config, + const FaultTriggerHostWiring & wiring) { + if (!wiring.cache || !wiring.plugins) { + throw std::invalid_argument("make_fault_trigger_host needs an entity cache and a plugin manager"); + } + if (!config.enabled || !wiring.plugins->has_plugins()) { + return nullptr; + } + // Floor so the loop can never busy-spin. The value is int64 and is checked before it is narrowed, or one above + // INT_MAX would wrap past the floor. + const int64_t used = std::clamp(config.poll_interval_ms, kMinPollIntervalMs, + static_cast(std::numeric_limits::max())); + if (used != config.poll_interval_ms && wiring.warn) { + wiring.warn("fault_triggers.poll_interval_ms " + std::to_string(config.poll_interval_ms) + " clamped to " + + std::to_string(used)); + } + auto host = std::make_unique(config, static_cast(used), wiring); + if (wiring.log) { + wiring.log("fault-trigger engine active (poll " + std::to_string(used) + " ms, store '" + + (config.storage_path.empty() ? std::string("in-memory") : config.storage_path) + "')"); + } + return host; +} + +} // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/src/core/plugins/core_plugin_context.cpp b/src/ros2_medkit_gateway/src/core/plugins/core_plugin_context.cpp new file mode 100644 index 000000000..a318fc821 --- /dev/null +++ b/src/ros2_medkit_gateway/src/core/plugins/core_plugin_context.cpp @@ -0,0 +1,348 @@ +// Copyright 2026 bburda +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#include "ros2_medkit_gateway/core/plugins/core_plugin_context.hpp" + +#include +#include + +#include "ros2_medkit_gateway/aggregation/aggregation_manager.hpp" +#include "ros2_medkit_gateway/core/condition_evaluator.hpp" +#include "ros2_medkit_gateway/core/entity_validation.hpp" +#include "ros2_medkit_gateway/core/faults/fault_scope.hpp" +#include "ros2_medkit_gateway/core/http/error_codes.hpp" +#include "ros2_medkit_gateway/core/http/http_utils.hpp" +#include "ros2_medkit_gateway/core/managers/fault_manager.hpp" +#include "ros2_medkit_gateway/core/managers/lock_manager.hpp" +#include "ros2_medkit_gateway/core/models/thread_safe_entity_cache.hpp" +#include "ros2_medkit_gateway/core/resource_change_notifier.hpp" +#include "ros2_medkit_gateway/core/resource_sampler.hpp" + +namespace ros2_medkit_gateway { + +namespace { + +/// Map a SovdEntityType to the plural URL path segment used in the REST API. +/// Returns empty string for SERVER/UNKNOWN (no faults collection on those). +std::string entity_type_to_url_segment(SovdEntityType type) { + switch (type) { + case SovdEntityType::APP: + return "apps"; + case SovdEntityType::COMPONENT: + return "components"; + case SovdEntityType::AREA: + return "areas"; + case SovdEntityType::FUNCTION: + return "functions"; + case SovdEntityType::SERVER: + case SovdEntityType::UNKNOWN: + return ""; + } + return ""; +} + +} // namespace + +CorePluginContext::CorePluginContext(GatewayServices * services, CorePluginContextWiring wiring) + : services_(services), wiring_(std::move(wiring)) { +} + +rclcpp::Node * CorePluginContext::node() const { + return nullptr; +} + +std::optional CorePluginContext::get_entity(const std::string & id) const { + if (!services_) { + return std::nullopt; + } + const auto & cache = services_->entity_cache(); + + if (auto comp = cache.get_component(id)) { + return PluginEntityInfo{SovdEntityType::COMPONENT, id, comp->namespace_path, comp->fqn}; + } + if (auto app = cache.get_app(id)) { + return PluginEntityInfo{SovdEntityType::APP, id, {}, app->effective_fqn()}; + } + if (auto area = cache.get_area(id)) { + return PluginEntityInfo{SovdEntityType::AREA, id, area->namespace_path, {}}; + } + if (cache.get_function(id)) { + return PluginEntityInfo{SovdEntityType::FUNCTION, id, {}, {}}; + } + return std::nullopt; +} + +std::vector CorePluginContext::get_child_apps(const std::string & component_id) const { + std::vector result; + if (!services_) { + return result; + } + const auto & cache = services_->entity_cache(); + auto app_ids = cache.get_apps_for_component(component_id); + for (const auto & app_id : app_ids) { + if (auto app = cache.get_app(app_id)) { + result.push_back(PluginEntityInfo{SovdEntityType::APP, app_id, + "", // namespace_path not needed for Apps + app->effective_fqn()}); + } + } + return result; +} + +nlohmann::json CorePluginContext::list_entity_faults(const std::string & entity_id) const { + // Determine entity type (needed for both local query and peer path construction). + auto entity = get_entity(entity_id); + if (!entity) { + return nlohmann::json::array(); + } + + nlohmann::json local_faults = nlohmann::json::array(); + + if (wiring_.fault_manager && wiring_.fault_manager->is_available()) { + // Mirror the HTTP per-entity fault handler: fetch all faults, then scope + // them to the entity's resolved App-FQN set (APP -> itself, COMPONENT -> + // hosted apps, AREA -> descendant apps, FUNCTION -> host apps) via the + // shared core helper. The fault manager returns a {"faults": [...]} + // object; this method's contract is a bare array, so extract and filter + // the inner array. + auto result = wiring_.fault_manager->list_faults(""); + if (result.success && result.data.contains("faults") && result.data["faults"].is_array()) { + const auto & cache = services_->entity_cache(); + const auto source_fqns = faults::resolve_entity_source_fqns(cache, entity->type, entity_id); + local_faults = faults::filter_faults_by_sources(result.data["faults"], source_fqns); + } + } + + // Fan-out to peer gateways: fetch faults for this entity from each peer. + // The peer path mirrors the HTTP REST route: /api/v1///faults. + // Auth: the ROS 2 service path has no inbound HTTP Authorization header to + // forward, so peers are queried with an empty credential - fan-out over this + // path only surfaces faults from peers that do not gate their /faults route. + auto * agg = wiring_.aggregation ? wiring_.aggregation() : nullptr; + if (agg) { + const std::string type_segment = entity_type_to_url_segment(entity->type); + if (!type_segment.empty()) { + const std::string peer_path = "/api/v1/" + type_segment + "/" + entity_id + "/faults"; + auto fan_result = agg->fan_out_get(peer_path, ""); + if (fan_result.merged_items.is_array()) { + for (const auto & item : fan_result.merged_items) { + local_faults.push_back(item); + } + } + if (fan_result.is_partial) { + if (wiring_.warn) { + wiring_.warn("list_entity_faults('" + entity_id + "'): partial peer fan-out, " + + std::to_string(fan_result.failed_peers.size()) + " peer(s) failed"); + } + } + } + } + + return local_faults; +} + +std::optional CorePluginContext::validate_entity_for_route(const PluginRequest & req, + PluginResponse & res, + const std::string & entity_id) const { + // Validate entity ID format (shared validation with HandlerContext) + if (auto err = validate_entity_id(entity_id); !err) { + res.send_error(400, ERR_INVALID_PARAMETER, err.error()); + return std::nullopt; + } + + // Determine expected type from route path (segment-boundary-aware matching) + auto expected_type = extract_entity_type_from_path(req.path()); + + auto entity = get_entity(entity_id); + if (!entity) { + res.send_error(404, ERR_ENTITY_NOT_FOUND, to_string(expected_type) + " not found: " + entity_id); + return std::nullopt; + } + + // Check type matches route + if (expected_type != SovdEntityType::UNKNOWN && entity->type != expected_type) { + res.send_error(400, ERR_INVALID_PARAMETER, + "Entity '" + entity_id + "' is a " + to_string(entity->type) + ", not a " + + to_string(expected_type)); + return std::nullopt; + } + + return entity; +} + +void CorePluginContext::register_capability(SovdEntityType entity_type, const std::string & capability_name) { + std::lock_guard lock(capabilities_mutex_); + auto & caps = type_capabilities_[entity_type]; + // Avoid duplicates + for (const auto & c : caps) { + if (c == capability_name) { + return; + } + } + caps.push_back(capability_name); +} + +void CorePluginContext::register_entity_capability(const std::string & entity_id, const std::string & capability_name) { + std::lock_guard lock(capabilities_mutex_); + auto & caps = entity_capabilities_[entity_id]; + for (const auto & c : caps) { + if (c == capability_name) { + return; + } + } + caps.push_back(capability_name); +} + +std::vector CorePluginContext::get_type_capabilities(SovdEntityType entity_type) const { + std::lock_guard lock(capabilities_mutex_); + auto it = type_capabilities_.find(entity_type); + if (it != type_capabilities_.end()) { + return it->second; + } + return {}; +} + +std::vector CorePluginContext::get_entity_capabilities(const std::string & entity_id) const { + std::lock_guard lock(capabilities_mutex_); + auto it = entity_capabilities_.find(entity_id); + if (it != entity_capabilities_.end()) { + return it->second; + } + return {}; +} + +LockAccessResult CorePluginContext::check_lock(const std::string & entity_id, const std::string & client_id, + const std::string & collection) const { + auto * lock_mgr = services_ ? services_->lock_manager() : nullptr; + if (!lock_mgr) { + return LockAccessResult{true, "", "", ""}; + } + return lock_mgr->check_access(entity_id, client_id, collection); +} + +tl::expected CorePluginContext::acquire_lock(const std::string & entity_id, + const std::string & client_id, + const std::vector & scopes, + int expiration_seconds) { + auto * lock_mgr = services_ ? services_->lock_manager() : nullptr; + if (!lock_mgr) { + return tl::make_unexpected(LockError{"lock-disabled", "Locking is not enabled", 503, std::nullopt}); + } + return lock_mgr->acquire(entity_id, client_id, scopes, expiration_seconds); +} + +tl::expected CorePluginContext::release_lock(const std::string & entity_id, + const std::string & client_id) { + auto * lock_mgr = services_ ? services_->lock_manager() : nullptr; + if (!lock_mgr) { + return tl::make_unexpected(LockError{"lock-disabled", "Locking is not enabled", 503, std::nullopt}); + } + return lock_mgr->release(entity_id, client_id); +} + +IntrospectionInput CorePluginContext::get_entity_snapshot() const { + IntrospectionInput input; + if (!services_) { + return input; + } + const auto & cache = services_->entity_cache(); + input.areas = cache.get_areas(); + input.components = cache.get_components(); + input.apps = cache.get_apps(); + input.functions = cache.get_functions(); + return input; +} + +nlohmann::json CorePluginContext::list_all_faults() const { + nlohmann::json response = nlohmann::json::object(); + + if (wiring_.fault_manager && wiring_.fault_manager->is_available()) { + auto result = wiring_.fault_manager->list_faults(""); + if (result.success) { + response = result.data; + } + } + + // Fan-out to peer gateways: fetch all faults from /api/v1/faults on each peer. + // Auth: see list_entity_faults - the ROS 2 service path forwards no inbound + // Authorization header, so peers are queried with an empty credential. + // The visited path is what lets a peer ask its own peers, so the list + // reaches past the first hop without coming back here. + auto * agg = wiring_.aggregation ? wiring_.aggregation() : nullptr; + if (agg) { + auto fan_result = agg->fan_out_get("/api/v1/faults", "", nullptr, {{kFanOutVisitedHeader, agg->instance_id()}}); + if (fan_result.merged_items.is_array()) { + // Ensure local response has a "faults" array to append to. + if (!response.contains("faults") || !response["faults"].is_array()) { + response["faults"] = nlohmann::json::array(); + } + for (const auto & item : fan_result.merged_items) { + response["faults"].push_back(item); + } + } + if (fan_result.is_partial) { + if (wiring_.warn) { + wiring_.warn("list_all_faults: partial peer fan-out, " + std::to_string(fan_result.failed_peers.size()) + + " peer(s) failed"); + } + } + } + + // "count" comes from the local-only query; after merging peer faults it + // would undercount, so keep it consistent with the merged array. + if (response.contains("count") && response.contains("faults") && response["faults"].is_array()) { + response["count"] = response["faults"].size(); + } + + return response; +} + +/// A plugin sampler is sampled with no way of saying that it narrows its +/// payload to the resource named by its second argument, so it is registered +/// as streaming its collection whole. A subscription URI naming a single item +/// of that collection is then refused rather than answered with everything. +void CorePluginContext::register_sampler( + const std::string & collection, + const std::function(const std::string &, const std::string &)> & fn) { + if (wiring_.sampler_registry) { + wiring_.sampler_registry->register_sampler(collection, fn, /*is_builtin=*/false, /*honours_resource_path=*/false); + } +} + +ResourceChangeNotifier * CorePluginContext::get_resource_change_notifier() { + return wiring_.notifier ? wiring_.notifier() : nullptr; +} + +ConditionRegistry * CorePluginContext::get_condition_registry() { + return wiring_.conditions ? wiring_.conditions() : nullptr; +} + +void CorePluginContext::notify_entities_changed(const EntityChangeScope & scope) { + if (wiring_.on_entities_changed) { + wiring_.on_entities_changed(scope); + } +} + +std::unique_ptr make_core_plugin_context(GatewayServices & services, + CorePluginContextWiring wiring) { + if (!wiring.fault_manager) { + wiring.fault_manager = services.fault_manager(); + } + if (!wiring.sampler_registry) { + wiring.sampler_registry = services.sampler_registry(); + } + return std::make_unique(&services, std::move(wiring)); +} + +} // namespace ros2_medkit_gateway diff --git a/src/ros2_medkit_gateway/src/gateway_node.cpp b/src/ros2_medkit_gateway/src/gateway_node.cpp index fba5a4e7b..af9c2abae 100644 --- a/src/ros2_medkit_gateway/src/gateway_node.cpp +++ b/src/ros2_medkit_gateway/src/gateway_node.cpp @@ -1382,11 +1382,7 @@ GatewayNode::~GatewayNode() { // 5b. Stop the fault-trigger evaluation loop BEFORE plugin/fault shutdown: // it resolves into plugin routes and the FaultManager, neither of which // may be used once shutdown_all() has run. - fault_trigger_stop_.store(true, std::memory_order_relaxed); - if (fault_trigger_thread_.joinable()) { - fault_trigger_thread_.join(); - } - fault_trigger_engine_.reset(); + fault_trigger_host_.reset(); // 6. Drop the entity freeze-frame capture BEFORE plugin shutdown: its // capture thread (joined here) resolves into plugin DataProviders and // routes, which must not be reachable once shutdown_all() has run. @@ -1589,160 +1585,29 @@ void GatewayNode::shutdown_trigger_subscriber() { } } -namespace { - -// Coerce a discovered data point's JSON value to a double for threshold -// comparison. Handles the numeric, boolean and numeric-string shapes the PLC -// bridges emit; returns nullopt for anything non-numeric. -std::optional value_to_double(const nlohmann::json & v) { - if (v.is_number()) { - return v.get(); - } - if (v.is_boolean()) { - return v.get() ? 1.0 : 0.0; - } - if (v.is_string()) { - try { - size_t pos = 0; - const double d = std::stod(v.get(), &pos); - if (pos > 0) { - return d; - } - } catch (const std::exception &) { - return std::nullopt; - } - } - return std::nullopt; -} - -} // namespace - void GatewayNode::init_fault_trigger_engine() { - if (!get_parameter("fault_triggers.enabled").as_bool() || !plugin_mgr_ || !plugin_mgr_->has_plugins()) { + if (!plugin_mgr_) { return; } - const std::string storage_path = get_parameter("fault_triggers.storage.path").as_string(); - // Floor so the evaluation loop can never busy-spin. Read as int64 and checked - // before narrowing, or a value above INT_MAX would wrap past the floor. - const int64_t poll_ms_raw = get_parameter("fault_triggers.poll_interval_ms").as_int(); - const int64_t poll_ms_used = std::clamp(poll_ms_raw, 50, std::numeric_limits::max()); - if (poll_ms_used != poll_ms_raw) { - RCLCPP_WARN(get_logger(), "fault_triggers.poll_interval_ms %" PRId64 " clamped to %" PRId64, poll_ms_raw, - poll_ms_used); - } - const int poll_ms = static_cast(poll_ms_used); - - // Value fetch: same in-process x-plc-data route the freeze-frame capture uses, - // so a rule always sees exactly the value the /data endpoint would serve. - auto fetcher = [this](const std::string & app_id, const std::string & data_name) -> std::optional { - if (!plugin_mgr_) { - return std::nullopt; - } - const auto content = plugin_mgr_->fetch_entity_data_via_route(app_id); - // A down link serves frozen last-known values; nullopt holds rule state - // (fault_trigger_engine) instead of firing on a stale number all outage. - if (!content || !EntityFreezeFrameCapture::content_has_live_data(*content) || - EntityFreezeFrameCapture::content_reports_disconnected(*content)) { - return std::nullopt; - } - const auto values = EntityFreezeFrameCapture::values_from_list_content(*content); - if (!values.is_object() || !values.contains(data_name)) { - return std::nullopt; - } - return value_to_double(values[data_name]); + FaultTriggerHostConfig config; + config.enabled = get_parameter("fault_triggers.enabled").as_bool(); + config.storage_path = get_parameter("fault_triggers.storage.path").as_string(); + config.poll_interval_ms = get_parameter("fault_triggers.poll_interval_ms").as_int(); + FaultTriggerHostWiring wiring; + wiring.cache = &thread_safe_cache_; + wiring.plugins = plugin_mgr_.get(); + wiring.faults = fault_mgr_.get(); + wiring.log = [this](const std::string & message) { + RCLCPP_INFO(get_logger(), "%s", message.c_str()); }; - - auto report = [this](const std::string & app_id, const std::string & fault_code, const std::string & severity, - const std::string & description) { - if (!fault_mgr_) { - return; - } - uint8_t sev = 1; // WARNING default - if (severity == "INFO") { - sev = 0; - } else if (severity == "WARNING") { - sev = 1; - } else if (severity == "ERROR") { - sev = 2; - } else if (severity == "CRITICAL") { - sev = 3; - } - fault_mgr_->report_fault(fault_code, sev, description, app_id); - }; - - auto clear = [this](const std::string & /*app_id*/, const std::string & fault_code) { - if (fault_mgr_) { - // A trigger-rule auto-clear (falling edge / rule deletion) is a local, - // rule-scoped event: skip the correlation engine's cascade so clearing - // one rule's fault can never wipe correlated faults owned by anything - // else. fault_code is globally unique (create() rejects duplicates), so - // the app scope is already encoded in the code itself. - fault_mgr_->clear_fault(fault_code, /*skip_correlation_auto_clear=*/true); - } - }; - - // Data-point enumeration for create-time validation: same in-process route as - // the fetcher, so "exists" means exactly "the fetcher could ever read it". - auto data_point_names = [this](const std::string & app_id) -> std::optional> { - if (!plugin_mgr_) { - return std::nullopt; - } - const auto content = plugin_mgr_->fetch_entity_data_via_route(app_id); - if (!content || !EntityFreezeFrameCapture::content_has_live_data(*content)) { - return std::nullopt; - } - const auto values = EntityFreezeFrameCapture::values_from_list_content(*content); - if (!values.is_object()) { - return std::nullopt; - } - std::vector names; - names.reserve(values.size()); - for (auto it = values.begin(); it != values.end(); ++it) { - names.push_back(it.key()); - } - return names; - }; - - // Entity existence for create-time validation: an app is real iff the entity - // registry knows its id. Distinct from the data route (which returns nullopt - // for both an unknown app and a transient outage), so create() can 404 a - // bogus app instead of reserving a global fault_code for it. - auto entity_exists = [this](const std::string & app_id) -> bool { - return get_thread_safe_cache().find_entity(app_id).has_value(); + wiring.warn = [this](const std::string & message) { + RCLCPP_WARN(get_logger(), "%s", message.c_str()); }; - - fault_trigger_engine_ = std::make_unique( - storage_path, std::move(fetcher), std::move(report), std::move(clear), - [this](const std::string & m) { - RCLCPP_INFO(get_logger(), "%s", m.c_str()); - }, - std::move(data_point_names), std::move(entity_exists)); - - // Dedicated evaluation loop: the value fetch + synchronous ReportFault call - // must not run on the main ROS executor (mirrors the freeze-frame capture's - // own thread). Joined in ~GatewayNode before plugin/fault shutdown. - const auto interval = std::chrono::milliseconds(poll_ms); - fault_trigger_thread_ = std::thread([this, interval]() { - while (!fault_trigger_stop_.load(std::memory_order_relaxed)) { - try { - fault_trigger_engine_->evaluate_once(); - } catch (const std::exception & e) { - RCLCPP_WARN(get_logger(), "fault-trigger evaluation error: %s", e.what()); - } - // Sleep in short slices so shutdown does not wait a whole interval. - auto slept = std::chrono::milliseconds(0); - while (slept < interval && !fault_trigger_stop_.load(std::memory_order_relaxed)) { - std::this_thread::sleep_for(std::chrono::milliseconds(25)); - slept += std::chrono::milliseconds(25); - } - } - }); - RCLCPP_INFO(get_logger(), "fault-trigger engine active (poll %d ms, store '%s')", poll_ms, - storage_path.empty() ? "in-memory" : storage_path.c_str()); + fault_trigger_host_ = make_fault_trigger_host(config, wiring); } FaultTriggerEngine * GatewayNode::get_fault_trigger_engine() const { - return fault_trigger_engine_.get(); + return fault_trigger_host_ ? &fault_trigger_host_->engine() : nullptr; } ResourceSamplerRegistry * GatewayNode::get_sampler_registry() const { diff --git a/src/ros2_medkit_gateway/src/http/handlers/trigger_handlers.cpp b/src/ros2_medkit_gateway/src/http/handlers/trigger_handlers.cpp index 55da509b9..f1200f4d6 100644 --- a/src/ros2_medkit_gateway/src/http/handlers/trigger_handlers.cpp +++ b/src/ros2_medkit_gateway/src/http/handlers/trigger_handlers.cpp @@ -274,8 +274,8 @@ TriggerHandlers::post_trigger(const http::TypedRequest & req, dto::TriggerCreate } if (resolved_topic_name.empty()) { MEDKIT_LOG_WARN(HandlerContext::logger(), - "Could not resolve resource_path '%s' to a ROS 2 topic for entity '%s'. " - "Topic subscription will be attempted when topic becomes available.", + "Could not resolve resource_path '%s' to a data source for entity '%s'. " + "The trigger resolves it when the entity serves the item.", parsed->resource_path.c_str(), entity_id.c_str()); } } diff --git a/src/ros2_medkit_gateway/src/plugins/plugin_context.cpp b/src/ros2_medkit_gateway/src/plugins/plugin_context.cpp index 16925b17a..67ded011e 100644 --- a/src/ros2_medkit_gateway/src/plugins/plugin_context.cpp +++ b/src/ros2_medkit_gateway/src/plugins/plugin_context.cpp @@ -12,337 +12,63 @@ // See the License for the specific language governing permissions and // limitations under the License. -#include "ros2_medkit_gateway/plugins/ros_plugin_context.hpp" - -#include #include -#include -#include #include "ros2_medkit_gateway/aggregation/aggregation_manager.hpp" -#include "ros2_medkit_gateway/core/entity_validation.hpp" -#include "ros2_medkit_gateway/core/faults/fault_scope.hpp" - -#include "ros2_medkit_gateway/core/condition_evaluator.hpp" -#include "ros2_medkit_gateway/core/http/error_codes.hpp" -#include "ros2_medkit_gateway/core/http/http_utils.hpp" -#include "ros2_medkit_gateway/core/managers/fault_manager.hpp" -#include "ros2_medkit_gateway/core/managers/lock_manager.hpp" -#include "ros2_medkit_gateway/core/resource_change_notifier.hpp" -#include "ros2_medkit_gateway/core/resource_sampler.hpp" +#include "ros2_medkit_gateway/core/plugins/core_plugin_context.hpp" #include "ros2_medkit_gateway/gateway_node.hpp" +#include "ros2_medkit_gateway/plugins/ros_plugin_context.hpp" namespace ros2_medkit_gateway { namespace { -/// Map a SovdEntityType to the plural URL path segment used in the REST API. -/// Returns empty string for SERVER/UNKNOWN (no faults collection on those). -std::string entity_type_to_url_segment(SovdEntityType type) { - switch (type) { - case SovdEntityType::APP: - return "apps"; - case SovdEntityType::COMPONENT: - return "components"; - case SovdEntityType::AREA: - return "areas"; - case SovdEntityType::FUNCTION: - return "functions"; - case SovdEntityType::SERVER: - case SovdEntityType::UNKNOWN: - return ""; - } - return ""; +CorePluginContextWiring node_wiring(GatewayNode * node, FaultManager * fault_manager, + ResourceSamplerRegistry * sampler_registry) { + CorePluginContextWiring wiring; + wiring.fault_manager = fault_manager; + wiring.sampler_registry = sampler_registry; + // A null node is allowed: the context then offers capabilities and samplers only. + if (!node) { + return wiring; + } + wiring.notifier = [node] { + return node->get_resource_change_notifier(); + }; + wiring.conditions = [node] { + return node->get_condition_registry(); + }; + wiring.aggregation = [node] { + return node->get_aggregation_manager(); + }; + // Synchronous: the caller's thread runs the whole discovery refresh. GatewayNode serializes concurrent + // notifications with its own refresh mutex. + wiring.on_entities_changed = [node](const EntityChangeScope & scope) { + node->handle_entity_change_notification(scope); + }; + wiring.warn = [node](const std::string & message) { + RCLCPP_WARN(node->get_logger(), "%s", message.c_str()); + }; + return wiring; } -} // namespace - -// ---- Concrete implementation ---- - -class GatewayPluginContext : public RosPluginContext { +/// The core context with the node of this process. +class GatewayPluginContext : public CorePluginContext { public: GatewayPluginContext(GatewayNode * node, FaultManager * fault_manager, ResourceSamplerRegistry * sampler_registry) - : node_(node), fault_manager_(fault_manager), sampler_registry_(sampler_registry) { + : CorePluginContext(node, node_wiring(node, fault_manager, sampler_registry)), node_(node) { } rclcpp::Node * node() const override { return node_; } - std::optional get_entity(const std::string & id) const override { - const auto & cache = node_->get_thread_safe_cache(); - - if (auto comp = cache.get_component(id)) { - return PluginEntityInfo{SovdEntityType::COMPONENT, id, comp->namespace_path, comp->fqn}; - } - if (auto app = cache.get_app(id)) { - return PluginEntityInfo{SovdEntityType::APP, id, {}, app->effective_fqn()}; - } - if (auto area = cache.get_area(id)) { - return PluginEntityInfo{SovdEntityType::AREA, id, area->namespace_path, {}}; - } - if (cache.get_function(id)) { - return PluginEntityInfo{SovdEntityType::FUNCTION, id, {}, {}}; - } - return std::nullopt; - } - - std::vector get_child_apps(const std::string & component_id) const override { - std::vector result; - const auto & cache = node_->get_thread_safe_cache(); - auto app_ids = cache.get_apps_for_component(component_id); - for (const auto & app_id : app_ids) { - if (auto app = cache.get_app(app_id)) { - result.push_back(PluginEntityInfo{SovdEntityType::APP, app_id, - "", // namespace_path not needed for Apps - app->effective_fqn()}); - } - } - return result; - } - - nlohmann::json list_entity_faults(const std::string & entity_id) const override { - // Determine entity type (needed for both local query and peer path construction). - auto entity = get_entity(entity_id); - if (!entity) { - return nlohmann::json::array(); - } - - nlohmann::json local_faults = nlohmann::json::array(); - - if (fault_manager_ && fault_manager_->is_available()) { - // Mirror the HTTP per-entity fault handler: fetch all faults, then scope - // them to the entity's resolved App-FQN set (APP -> itself, COMPONENT -> - // hosted apps, AREA -> descendant apps, FUNCTION -> host apps) via the - // shared core helper. The fault manager returns a {"faults": [...]} - // object; this method's contract is a bare array, so extract and filter - // the inner array. - auto result = fault_manager_->list_faults(""); - if (result.success && result.data.contains("faults") && result.data["faults"].is_array()) { - const auto & cache = node_->get_thread_safe_cache(); - const auto source_fqns = faults::resolve_entity_source_fqns(cache, entity->type, entity_id); - local_faults = faults::filter_faults_by_sources(result.data["faults"], source_fqns); - } - } - - // Fan-out to peer gateways: fetch faults for this entity from each peer. - // The peer path mirrors the HTTP REST route: /api/v1///faults. - // Auth: the ROS 2 service path has no inbound HTTP Authorization header to - // forward, so peers are queried with an empty credential - fan-out over this - // path only surfaces faults from peers that do not gate their /faults route. - auto * agg = node_->get_aggregation_manager(); - if (agg) { - const std::string type_segment = entity_type_to_url_segment(entity->type); - if (!type_segment.empty()) { - const std::string peer_path = "/api/v1/" + type_segment + "/" + entity_id + "/faults"; - auto fan_result = agg->fan_out_get(peer_path, ""); - if (fan_result.merged_items.is_array()) { - for (const auto & item : fan_result.merged_items) { - local_faults.push_back(item); - } - } - if (fan_result.is_partial) { - RCLCPP_WARN(node_->get_logger(), "list_entity_faults('%s'): partial peer fan-out, %zu peer(s) failed", - entity_id.c_str(), fan_result.failed_peers.size()); - } - } - } - - return local_faults; - } - - std::optional validate_entity_for_route(const PluginRequest & req, PluginResponse & res, - const std::string & entity_id) const override { - // Validate entity ID format (shared validation with HandlerContext) - if (auto err = validate_entity_id(entity_id); !err) { - res.send_error(400, ERR_INVALID_PARAMETER, err.error()); - return std::nullopt; - } - - // Determine expected type from route path (segment-boundary-aware matching) - auto expected_type = extract_entity_type_from_path(req.path()); - - auto entity = get_entity(entity_id); - if (!entity) { - res.send_error(404, ERR_ENTITY_NOT_FOUND, to_string(expected_type) + " not found: " + entity_id); - return std::nullopt; - } - - // Check type matches route - if (expected_type != SovdEntityType::UNKNOWN && entity->type != expected_type) { - res.send_error(400, ERR_INVALID_PARAMETER, - "Entity '" + entity_id + "' is a " + to_string(entity->type) + ", not a " + - to_string(expected_type)); - return std::nullopt; - } - - return entity; - } - - void register_capability(SovdEntityType entity_type, const std::string & capability_name) override { - std::lock_guard lock(capabilities_mutex_); - auto & caps = type_capabilities_[entity_type]; - // Avoid duplicates - for (const auto & c : caps) { - if (c == capability_name) { - return; - } - } - caps.push_back(capability_name); - } - - void register_entity_capability(const std::string & entity_id, const std::string & capability_name) override { - std::lock_guard lock(capabilities_mutex_); - auto & caps = entity_capabilities_[entity_id]; - for (const auto & c : caps) { - if (c == capability_name) { - return; - } - } - caps.push_back(capability_name); - } - - std::vector get_type_capabilities(SovdEntityType entity_type) const override { - std::lock_guard lock(capabilities_mutex_); - auto it = type_capabilities_.find(entity_type); - if (it != type_capabilities_.end()) { - return it->second; - } - return {}; - } - - std::vector get_entity_capabilities(const std::string & entity_id) const override { - std::lock_guard lock(capabilities_mutex_); - auto it = entity_capabilities_.find(entity_id); - if (it != entity_capabilities_.end()) { - return it->second; - } - return {}; - } - - LockAccessResult check_lock(const std::string & entity_id, const std::string & client_id, - const std::string & collection) const override { - auto * lock_mgr = node_->get_lock_manager(); - if (!lock_mgr) { - return LockAccessResult{true, "", "", ""}; - } - return lock_mgr->check_access(entity_id, client_id, collection); - } - - tl::expected acquire_lock(const std::string & entity_id, const std::string & client_id, - const std::vector & scopes, - int expiration_seconds) override { - auto * lock_mgr = node_->get_lock_manager(); - if (!lock_mgr) { - return tl::make_unexpected(LockError{"lock-disabled", "Locking is not enabled", 503, std::nullopt}); - } - return lock_mgr->acquire(entity_id, client_id, scopes, expiration_seconds); - } - - tl::expected release_lock(const std::string & entity_id, const std::string & client_id) override { - auto * lock_mgr = node_->get_lock_manager(); - if (!lock_mgr) { - return tl::make_unexpected(LockError{"lock-disabled", "Locking is not enabled", 503, std::nullopt}); - } - return lock_mgr->release(entity_id, client_id); - } - - IntrospectionInput get_entity_snapshot() const override { - IntrospectionInput input; - const auto & cache = node_->get_thread_safe_cache(); - input.areas = cache.get_areas(); - input.components = cache.get_components(); - input.apps = cache.get_apps(); - input.functions = cache.get_functions(); - return input; - } - - nlohmann::json list_all_faults() const override { - nlohmann::json response = nlohmann::json::object(); - - if (fault_manager_ && fault_manager_->is_available()) { - auto result = fault_manager_->list_faults(""); - if (result.success) { - response = result.data; - } - } - - // Fan-out to peer gateways: fetch all faults from /api/v1/faults on each peer. - // Auth: see list_entity_faults - the ROS 2 service path forwards no inbound - // Authorization header, so peers are queried with an empty credential. - // The visited path is what lets a peer ask its own peers, so the list - // reaches past the first hop without coming back here. - auto * agg = node_->get_aggregation_manager(); - if (agg) { - auto fan_result = agg->fan_out_get("/api/v1/faults", "", nullptr, {{kFanOutVisitedHeader, agg->instance_id()}}); - if (fan_result.merged_items.is_array()) { - // Ensure local response has a "faults" array to append to. - if (!response.contains("faults") || !response["faults"].is_array()) { - response["faults"] = nlohmann::json::array(); - } - for (const auto & item : fan_result.merged_items) { - response["faults"].push_back(item); - } - } - if (fan_result.is_partial) { - RCLCPP_WARN(node_->get_logger(), "list_all_faults: partial peer fan-out, %zu peer(s) failed", - fan_result.failed_peers.size()); - } - } - - // "count" comes from the local-only query; after merging peer faults it - // would undercount, so keep it consistent with the merged array. - if (response.contains("count") && response.contains("faults") && response["faults"].is_array()) { - response["count"] = response["faults"].size(); - } - - return response; - } - - /// A plugin sampler is sampled with no way of saying that it narrows its - /// payload to the resource named by its second argument, so it is registered - /// as streaming its collection whole. A subscription URI naming a single item - /// of that collection is then refused rather than answered with everything. - void register_sampler( - const std::string & collection, - const std::function(const std::string &, const std::string &)> & fn) - override { - if (sampler_registry_) { - sampler_registry_->register_sampler(collection, fn, /*is_builtin=*/false, /*honours_resource_path=*/false); - } - } - - // ---- Trigger infrastructure ---- - - ResourceChangeNotifier * get_resource_change_notifier() override { - return node_->get_resource_change_notifier(); - } - - ConditionRegistry * get_condition_registry() override { - return node_->get_condition_registry(); - } - - // ---- Entity surface notifications (plugin API v7) ---- - - void notify_entities_changed(const EntityChangeScope & scope) override { - // Forward the notification directly to GatewayNode for immediate - // handling. This call is synchronous at this layer - the caller's - // thread runs the full discovery refresh before returning. Any - // serialization across concurrent plugin notifications is performed by - // GatewayNode itself via its internal refresh mutex. - node_->handle_entity_change_notification(scope); - } - private: GatewayNode * node_; - FaultManager * fault_manager_; - ResourceSamplerRegistry * sampler_registry_; - - mutable std::mutex capabilities_mutex_; - std::unordered_map> type_capabilities_; - std::unordered_map> entity_capabilities_; }; +} // namespace + std::unique_ptr make_gateway_plugin_context(GatewayNode * node, FaultManager * fault_manager, ResourceSamplerRegistry * sampler_registry) { return std::make_unique(node, fault_manager, sampler_registry);