Skip to content

UCT/BASE: revert log level and token cap #11851

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gleon99 merged 3 commits into from
Sep 1, 2026
+1 −31
Merged

UCT/BASE: revert log level and token cap #11851

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
408e6d6
UCT/BASE: revert log level and token cap
JeynmannZ Aug 28, 2026
11bd528
GTEST/UCT/IB: rm query_token_support
JeynmannZ Aug 31, 2026
fed9f9b
Merge branch 'master' into failover_uct_revert_flags
jeynmann Sep 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/uct/base/uct_iface.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1048,7 +1048,7 @@ ucs_log_level_t uct_base_iface_failure_log_level(uct_base_iface_t *iface,
ucs_status_t err_handler_status,
ucs_status_t status)
{

@svc-nvidia-pr-review svc-nvidia-pr-review Aug 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UCS_INPROGRESS is a documented valid return of uct_error_handler_t ("error handling in progress"), but changing the condition from UCS_STATUS_IS_ERR(err_handler_status) to err_handler_status != UCS_OK now treats it as FATAL. This function is shared by cma, srd, rc_verbs, rc_mlx5, and dc_mlx5 failure paths, so a handler returning UCS_INPROGRESS would now cause a FATAL log (process abort) instead of the normal per-transport failure level. Is escalating UCS_INPROGRESS to FATAL intended? If not, keep UCS_STATUS_IS_ERR(err_handler_status).

@svc-nvidia-pr-review svc-nvidia-pr-review Sep 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The condition change is a behavioral regression for a documented, valid handler return value. err_handler_status != UCS_OK now escalates a UCS_INPROGRESS return to FATAL, but uct_error_handler_t documents UCS_INPROGRESS as a valid non-error return ("The error handling is in progress and the transport should not purge outstanding operations."). Because FATAL at/below log_level_trigger calls ucs_fatal_error_message (i.e. abort), this can turn a legitimate in-progress error-handling flow into a crash. The old UCS_STATUS_IS_ERR(err_handler_status) (status < 0) escalated only genuine errors and let UCS_INPROGRESS (positive) fall through to the normal ERROR/failure_level path. Should this stay UCS_STATUS_IS_ERR(err_handler_status), or is escalating positive statuses intentional?

if (UCS_STATUS_IS_ERR(err_handler_status)) {
if (err_handler_status != UCS_OK) {

@svc-nvidia-pr-review svc-nvidia-pr-review Aug 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

err_handler_status != UCS_OK escalates the documented UCS_INPROGRESS return to FATAL. uct_base_iface_failure_log_level() receives the error handler's return value via uct_iface_handle_ep_err(), i.e. the value returned by uct_error_handler_t. That callback's contract in src/uct/api/uct_def.h explicitly allows three outcomes: UCS_OK (handled), UCS_INPROGRESS ("error handling is in progress and the transport should not purge outstanding operations"), and otherwise (not handled). UCS_STATUS_IS_ERR(x) is x < 0, and UCS_INPROGRESS is positive. So the old code escalated to UCS_LOG_LEVEL_FATAL only for real errors and treated UCS_INPROGRESS as a non-fatal, in-progress case. After the change, UCS_INPROGRESS != UCS_OK is true, so a handler that returns the documented UCS_INPROGRESS now produces a FATAL failure log. This looks like an unintended behavior change rather than a fix. No in-tree error handler currently returns UCS_INPROGRESS (ucp_worker_iface_error_handler returns UCS_OK/UCS_ERR_NO_ELEM; all gtest handlers return UCS_OK), so this is not caught by existing coverage and would only bite an external handler using the documented API contract. Is escalating UCS_INPROGRESS to FATAL intended? If so, update the callback documentation in uct_def.h and add a focused test; if not:

Suggested change
if (err_handler_status != UCS_OK) {
if (UCS_STATUS_IS_ERR(err_handler_status)) {

return UCS_LOG_LEVEL_FATAL;
} else if ((status == UCS_ERR_ENDPOINT_TIMEOUT) ||
(status == UCS_ERR_CONNECTION_RESET)) {
Expand Down
4 changes: 0 additions & 4 deletions src/uct/ib/mlx5/rc/rc_mlx5_iface.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1132,10 +1132,6 @@ uct_rc_mlx5_iface_query_v2(uct_iface_h tl_iface,
}
}

if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_CAP_FLAGS) {
iface_attr->cap.flags |= UCT_IFACE_FLAG_V2_QUERY_TOKEN;
}

if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_TX_TOKEN_LENGTH) {
iface_attr->tx_token_length = sizeof(uct_rc_mlx5_tx_token_t);
}
Expand Down
26 changes: 0 additions & 26 deletions test/gtest/uct/v2/test_uct_query.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,6 @@ extern "C" {
#include <uct/api/uct.h>
#include <uct/api/v2/uct_v2.h>
#include <uct/base/uct_iface.h>
#ifdef HAVE_MLX5_DV
#include <uct/ib/mlx5/ib_mlx5.h>
#endif
}


Expand Down Expand Up @@ -117,29 +114,6 @@ UCS_TEST_P(test_uct_query, query_perf)
}
}

UCS_TEST_P(test_uct_query, query_token_support)
{
uct_iface_attr_v2_t attr = {};
uint64_t iface_cap_flags = 0;

attr.field_mask = UCT_IFACE_ATTR_FIELD_CAP_FLAGS;
attr.cap.flags = UINT64_MAX;

ASSERT_UCS_OK(uct_iface_query_v2(get_iface(), &attr));

#ifdef HAVE_MLX5_DV
if (has_transport("rc_mlx5")) {
uct_ib_mlx5_md_t *md = uct_ib_mlx5_iface_md(
ucs_derived_of(get_iface(), uct_ib_iface_t));
if (md->flags & UCT_IB_MLX5_MD_FLAG_DEVX) {
iface_cap_flags = UCT_IFACE_FLAG_V2_QUERY_TOKEN;
}
}
#endif

EXPECT_EQ(iface_cap_flags, attr.cap.flags & UCT_IFACE_FLAG_V2_QUERY_TOKEN);
}

UCT_INSTANTIATE_TEST_CASE(test_uct_query)

class test_uct_query_ib : public test_uct_query {
Expand Down