Repository navigation
add Finally control node for guaranteed cleanup - #1229
Conversation
a63c668 to
90d4993
Compare
4e840d4 to
2fb0a51
Compare
facontidavide
left a comment
There was a problem hiding this comment.
Thanks for this, it's a carefully done PR. The exception semantics are well thought out, halt() never throwing is the right call, and the tests are thorough. I ran them 30 times, plus 10 more times under 5× parallel load, and they never failed. The full suite passes too.
Before this can go in, there is one design problem to solve, plus a few points to settle.
1. halt() ticks cleanup on whatever thread calls it
TimeoutNode halts its child from its TimerQueue callback (src/decorators/timeout_node.cpp:54), while holding timeout_mutex_. Inside a Timeout, Finally::halt() therefore ticks the cleanup child on the timer thread, and the tick thread is blocked on that mutex until cleanup finishes. I checked it with this test:
TEST(FinallyProbe, HaltedByTimeout_CleanupThread)
{
BT::BehaviorTreeFactory factory;
const auto tick_thread = std::this_thread::get_id();
std::thread::id cleanup_thread;
int cleanup_ticks = 0;
factory.registerSimpleCondition("Forever",
[](BT::TreeNode&) { return BT::NodeStatus::RUNNING; });
factory.registerSimpleCondition("SlowCleanup", [&](BT::TreeNode&) {
cleanup_thread = std::this_thread::get_id();
return ++cleanup_ticks < 30 ? BT::NodeStatus::RUNNING : BT::NodeStatus::SUCCESS;
});
auto tree = factory.createTreeFromText(R"(<root BTCPP_format="4"><BehaviorTree>
<Timeout msec="50"><Finally><Forever/><SlowCleanup/></Finally></Timeout>
</BehaviorTree></root>)");
tree.tickWhileRunning();
EXPECT_EQ(cleanup_thread, tick_thread); // fails: all 30 cleanup ticks ran on the timer thread
}The doc comment says "Call halt() from the thread that ticks the tree", but a built-in decorator doesn't do that. Neither does a common application pattern: calling Tree::haltTree() from another thread, such as a ROS service callback. Cleanup nodes like action clients, or anything else that isn't thread-safe, would then run on a thread they don't expect.
2. A blocking halt() with a 10 s default
When a ReactiveSequence, ReactiveFallback or Parallel halts a branch that contains a Finally, the whole tree stops for up to halt_timeout_msec. During that time no other node is ticked, including conditions that the reactive parent is supposed to keep re-checking. You describe the trade-off clearly. Still, it changes what halt() means in the library, and the 10 s default makes it the normal case rather than an opt-in.
Some possible directions, and I'm open to others:
- on halt, give cleanup a single tick, as
TryCatch catch_on_halt="true"does, and document that a multi-step cleanup will be cut short; - keep the multi-tick loop but make it opt-in, e.g.
halt_timeout_msecdefaulting to 0 meaning "one tick"; - keep the loop, but don't run it when
halt()is called from a thread other than the one that ticks the tree.
Which one would still cover your collision-checking use case?
3. Smaller points
- SKIPPED main: cleanup runs and the node returns SKIPPED, so the parent is told that nothing happened even though cleanup did. A skipped main changed nothing, so not running cleanup seems more consistent. What do you think?
- New built-in ID: registering
Finallybreaks applications that already register their own node with that name (#547 suggests some exist). That's acceptable in a minor release, so this would target 4.11 rather than a patch release. - The halt loop polls every 10 ms and ignores the tree's wake-up signal, so a cleanup that calls
emitWakeUpSignal()doesn't finish any sooner. Minor.
4. Please split out the haltChild fix
Resetting the child before rethrowing in ControlNode::haltChild is a real bug fix, independent of Finally: without it, ~Tree() can halt the same child twice and terminate. Could you open it as a separate small PR with its test? I'd merge that right away, while we settle the design of Finally.
|
I would like to merge this before the end of the week and release 4.11. Do you think you will have some time to look into this? |
Yes, I'll try to look into this over the next few days. Thank you for the in depth review. |
Previously the child stayed RUNNING, so the next halt of the parent halted it again and threw a second time. When that happened inside ~Tree(), the process terminated. Callers still see the exception. This covers haltChild() and haltChildren(), which calls it. ControlNode::resetChildren() and DecoratorNode::resetChild() have the same pattern and are left for a follow-up.
2fb0a51 to
a50c4b5
Compare
|
Thanks for the careful review and the Timeout probe. I've pushed a revision that takes your second and third options together, and split out the 1 and 2. For the collision-checking case, re-enabling collisions is a call to another process that returns RUNNING on its first tick, so a single tick would cut it short. The halts that matter there come from a
On the new built-in ID, 4.11 works for me. The 10 ms poll is unchanged, since the node has no access to the tree's wake-up signal, and it now only applies when the loop is opted in. |
Ports the revision made for BehaviorTree#1229 after review, keeping halt_timeout_msec at 10000 by default because MoveIt Pro 10.2 shipped with it (upstream now defaults to 0, a single cleanup tick). - The halt loop runs only when halt() is called from the thread that last ticked the node. A halt from another thread, such as the Timeout decorator's timer, gives cleanup one tick and prints a note to stderr. A Finally nested inside such a halt also gets one tick. - A main that is SKIPPED on its first tick no longer runs cleanup. A main skipped after it returned RUNNING, for example by _while, still does. - halt_timeout_msec of 0 gives cleanup one tick on halt. - Adds a regression test for the haltChild fix already on this branch. Refs PickNikRobotics/moveit_pro#20169 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
a50c4b5 to
b2df8a7
Compare
facontidavide
left a comment
There was a problem hiding this comment.
Thanks @dvadovsz for the careful revision. I went through it against the code, and every claim in your Oct 29 comment holds. I also ran FinallyTest.* ×30, the timing tests ×20 under full CPU load, TSan ×5 (Finally plus every *Timeout* test) and ASan+UBSan. All clean.
One design change before merging: please drop halt_timeout_msec and the multi-tick loop, and ship Finally with a single cleanup tick on halt, the same as TryCatch catch_on_halt="true". About 90 of the 254 lines (the thread id, the thread-local, the sleep loop, the deadline, two stderr paths) exist only for the loop, and the loop has two properties I don't want in a built-in node:
- it blocks reactive parents for up to
halt_timeout_msec; - it can't be bounded by an enclosing
Timeout: it runs on the tick thread while that Timeout holdstimeout_mutex_, so<Timeout msec="100"> … <Finally halt_timeout_msec="5000">waits 5 s.
If users need multi-tick cleanup on halt, we can add it later as a separate feature.
Small things while you're at it:
finally_node.cpp:88: if the main child threw, cleanup is RUNNING and a halt arrives,main_exception_is cleared silently. Please log it (e.g.printException) before dropping it.- Declare
halt()noexcept, so the "halt never throws" guarantee is checked by the compiler. - In the header and docs, say that the single tick on halt suits synchronous cleanup: an async cleanup child gets
onStart()and thenonHalted()right away. - #1273 is being reworked: the status reset moves into
TreeNode::haltNode()so that every halt path gets it, not onlyhaltChild(). Once that lands, please drop thecontrol_node.cppcommit from this PR and merge master.
Take master's src/control_node.cpp and tests/gtest_halt_child.cpp (BehaviorTree#1273, b6709b6), which supersede this branch's copy of the haltChild fix (fbbcdd1): the status reset now lives in TreeNode::haltNode(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ME1hnSFSzibU5phxboGsjk
- A halt() that arrives while another one runs (e.g. Tree::haltTree() from another thread while the tick thread is in the halt_timeout_msec loop) returns at once, so cleanup is never ticked concurrently. An atomic halting_ flag is set with compare_exchange and cleared by RAII. - halt() is noexcept. The status reset at its end notifies observers that may throw, so it is wrapped and reported like the rest. - A pending main exception dropped by a halt is now printed. - Document that halt_timeout_msec > 10 blocks the halting parent and any enclosing Timeout, adds up across nested Finally nodes, and gives foreign-thread halts a single tick; that the default single tick cuts an asynchronous cleanup short; and "10 or less" means a single tick. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ME1hnSFSzibU5phxboGsjk
facontidavide
left a comment
There was a problem hiding this comment.
Requested changes addressed in 01d987d.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ME1hnSFSzibU5phxboGsjk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ME1hnSFSzibU5phxboGsjk
* Add 'generic' parameter to writeTreeXSD() When generic=true, oneNodeGroup uses xs:any processContents="lax" instead of a closed xs:choice, allowing unknown custom node elements to pass validation. Top-level xs:element declarations are also emitted so that lax processing can still resolve and validate the known built-in node types by name. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add generate_xsd tool Generates the generic XSD schema from the default factory (built-in nodes only) and prints it to stdout, suitable for use as a generic static validator that stays in sync with the built-in node inventory. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix non-deterministic port attribute order in writeTreeXSD() PortsList is an unordered_map so iterating it directly produced XSD attributes in hash-table order, which varies across runs and platforms. Sort ports into a std::map before emitting xs:attribute elements. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add writeTreeSchematron() for cross-reference validation of BT XML files XSD alone cannot express constraints like "every custom node element used in a BehaviorTree body must have a matching TreeNodesModel entry" Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add generate_schematron tool Generates an ISO Schematron schema (btcpp4.sch) complementing the generic XSD Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add generated XSD and Schematron schemas * Add validate_xml.py - XSD and Schematron validation tool Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Only built-ins should go into generic xsd file * Add tests for writeTreeXSD() and writeTreeSchematron() Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix implicit conversion * Also handle inout_port in XSD * Fix reconfiguration after editing src/btcpp4_schematron.sch * Fix writeTreeXSD() signature not to break ABI * Change placeholder replacement logic for Schematron * Add more test cases for InsideXmlComment() * Add xsd and schematron checks to CI (intentionally broken) * Fix clang-format issues * Regenerate schemas to pass CI * Add guard for --no-xsd --no-sch together to validate_xml.py * Remove rule checking explicit notation against TreeNodeModel * Regenerate Schematron file * Add Schematron Rule B: explicit-notation node structure constraints Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Regenerate Schematron file * Drop insideXmlComment(): keep the placeholder out of the template comment insideXmlComment() was added to the public xml_parsing.h only so that writeTreeSchematron() could skip the ##BUILTIN_PIPE## placeholder where the template comment mentions it. Rewording the comment so it no longer contains the placeholder makes a plain replace-all enough, removes the helper from the public API, and also fixes the generated bt4.sch, whose comment still described an unexpanded placeholder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Regenerate tools/generated/bt4.xsd for _onlyIf, _onStart, BTCPP_version Master gained the _onlyIf/_onStart preconditions (#1261) and the BTCPP_version root attribute (#1264); the checked-in XSD was stale and the new 'generated schemas are up to date' CI step would fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ME1hnSFSzibU5phxboGsjk * Regenerate bt4.xsd and bt4.sch for the Finally node (#1229) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ME1hnSFSzibU5phxboGsjk --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Davide Faconti <davide.faconti@gmail.com>
|
@dvadovsz I take back point 1 of my review from yesterday. To get this into 4.11 I pushed 01d987d on top of a merge of master (maintainer edits), and it is now merged as 4f672ea. That commit contains only the small changes:
Locally: full suite, |
Closes #1228.
Adds a
Finallycontrol node with two children, main and cleanup. It ticks main, then ticks cleanup._while, cleanup runs and the node returns SKIPPED.finallyin C++/Java/Python.TryCatchwithcatch_on_halt="true".halt_timeout_msecport above 0 (default 0) lets a multi-step cleanup finish.halt()then keeps ticking cleanup every 10 ms until it finishes, fails or throws, or the timeout runs out and cleanup is halted unfinished. This loop runs only whenhalt()is called from the thread that last ticked the node. A halt from another thread, such as theTimeoutdecorator's timer or aTree::haltTree()from a service callback, gives cleanup the single tick and prints a note to stderr. AFinallynested inside such a halt also gets a single tick.finallyblock does.halt()never throws, because~Tree()callshaltTree(). It prints exceptions from cleanup or from halting a child, and a cleanup FAILURE, to stderr.Finallywithout exactly 2 children, as it does forTryCatch.This PR includes the commit from #1273, which fixes
ControlNode::haltChildleaving a child RUNNING when itshalt()threw. Without it, the test binary aborts instd::terminateon aFinallywhose main throws fromhalt(). Once #1273 merges, I will rebase this PR onto master to drop that commit.Why not an option on
TryCatch? The semantics differ in three places.Finallyruns cleanup on SUCCESS, returns main's status instead of always FAILURE, and lets main's exception through after cleanup. Packing that intoTryCatchwould make one node's return value depend on a flag.Two details worth checking in review:
executeTick()does not store SKIPPED as the node's status, and it doesn't update the status whentick()throws. In both casesFinallywould stay RUNNING, and a later halt would run cleanup a second time. So the node callsresetStatus()before returning SKIPPED or rethrowing.Finally. An application that already registers its ownFinallynode will get an "already registered" error, so this targets 4.11.API/ABI: one new header and one registered node. No signatures change.
Testing:
tests/gtest_finally.cpphas 44 cases. They cover sync and async main and cleanup, a halt in each phase, the single-tick default, the opt-in loop finishing and timing out, halts from another thread and fromTimeout(including a nestedFinally), exceptions from main, from cleanup, and from main's ownhalt(), non-std exceptions, SKIPPED main before and after RUNNING, a restart after a throw,SubTreeandReactiveSequenceparents, and child-count validation. I checked that the key tests fail without their fix. The full suite passes in Release (644 tests), theFinallytests pass 50 times in a row under parallel load, ThreadSanitizer reports no races on them, and ASan/UBSan pass per test.pre-commitpasses. clang-tidy-21 isn't installed on my machine, sorun_clang_tidy.shwill first run in CI.