Skip to content

Commit 16b50ac

Browse files
committed
fix: Wait out the cancel-token gap from any thread but the action's own
A third review found that a cancel straight after an action started could still be lost under load. The crate reports an action Running, then logs its banner and installs its cancel token; logging needs the GIL and runs every Python logging handler, so the gap is as long as the action thread waits for the GIL, seconds when other Python threads are busy. The previous commit gave up after 2 s everywhere. The binding now records which thread runs the current action, and only a cancel from that thread (a logging handler) gives up after 2 s; from any other thread it waits, with the GIL released, until the token is installed or the session is back, which the action thread does without help. Pinned in a fresh interpreter with a logging handler that takes 2.5 s. Also from the review: - If an action thread ever ended without telling its caller the outcome, the caller would have waited for ever. A guard held by the thread now drops the call's sender, only if the slot still holds that call, so the caller wakes up. - Every refusal test now checks state, action_status and environments_entered, including the busy, ValueError and clear_ending_only cases. The fresh-interpreter tests' timeouts are under pytest's 30 s limit. - The sessions spec's constructor block matches the binding, it documents the binding's own "Cannot cancel: the action has not started yet" and WindowsSessionUser's RuntimeError, and its new paragraphs are rewrapped. The quality report lists the WindowsSessionUser change. Signed-off-by: Mark Wiebe <399551+mwiebe@users.noreply.github.com>
1 parent 8077e87 commit 16b50ac

5 files changed

Lines changed: 180 additions & 70 deletions

File tree

‎reports/sessions-bindings-quality-evaluation-report.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -372,6 +372,7 @@ counted as test-suite failures.
372372
| `run_task` while session not READY | `RuntimeError("Session must be in the READY state to run a task.")` | `SessionError("Session must be in READY state, current: RUNNING")` (the crate's message; a `RuntimeError`) | ⚠ message change |
373373
| `run_task` / `enter_environment` on a `READY_ENDING` session | `RuntimeError("Session must be in the READY state ...")` | `SessionError("Session must be in READY state, current: READY_ENDING")`, a `RuntimeError`; nothing changes | ⚠ message change (the crate's) |
374374
| `Session.clear_ending_only()` | not in reference | takes a `READY_ENDING` session back to `READY`, as the crate's `clear_ending_only` does | ⚠ binding-only API |
375+
| `WindowsSessionUser(...)` when the process user can't be looked up | `get_process_user()` raises | raises `RuntimeError("Could not determine the process user: ...")` (before, treated as "not the process user") | ⚠ class change |
375376
| `cancel_action` when nothing is running | `RuntimeError("No actions are running")` | `SessionError("Session must be in RUNNING state, current: READY")` | ⚠ class+message change (the crate's) |
376377
| `cancel_action` while running | succeeds | succeeds, including straight after the call that started the action | ✓ resolved |
377378
| `Session(session_root_directory=…)` non-existent dir | `RuntimeError("Ensure that the root directory ...")` | likely Rust I/O error → `SessionError(...)` (untested) | ⚠ untested |

‎rust-bindings/src/sessions/session.rs‎

Lines changed: 69 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -198,6 +198,8 @@ enum CallOutcome {
198198
/// The call an action thread is running, until the Python thread that made
199199
/// it has been told the outcome.
200200
struct PendingCall {
201+
/// Tells this call apart from the next one in the same slot.
202+
id: u64,
201203
tx: std::sync::mpsc::SyncSender<CallOutcome>,
202204
/// Set by the session callback: the crate reported an action for this
203205
/// call, so it accepted it.
@@ -206,6 +208,39 @@ struct PendingCall {
206208

207209
type PendingSlot = Arc<Mutex<Option<PendingCall>>>;
208210

211+
static NEXT_CALL_ID: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
212+
213+
/// The thread running a session's current action, so `cancel_action` can
214+
/// tell whether it is called from that thread.
215+
type ActionThread = Arc<Mutex<Option<std::thread::ThreadId>>>;
216+
217+
/// Held by an action thread for its whole life. If the thread ends without
218+
/// having told its caller the outcome (it can't today, but a panic outside
219+
/// `catch_unwind` would), dropping the call's sender wakes the caller instead
220+
/// of leaving it waiting for ever. Also clears the thread's entry in
221+
/// `ActionThread`. Each slot is cleared only while it still names this call
222+
/// or thread, not a later one.
223+
struct ActionThreadGuard {
224+
pending: PendingSlot,
225+
call_id: u64,
226+
action_thread: ActionThread,
227+
}
228+
229+
impl Drop for ActionThreadGuard {
230+
fn drop(&mut self) {
231+
{
232+
let mut slot = lock_recover(&self.pending);
233+
if slot.as_ref().is_some_and(|c| c.id == self.call_id) {
234+
slot.take();
235+
}
236+
}
237+
let mut thread = lock_recover(&self.action_thread);
238+
if *thread == Some(std::thread::current().id()) {
239+
*thread = None;
240+
}
241+
}
242+
}
243+
209244
/// Body of a spawned action thread.
210245
///
211246
/// Builds a lightweight current-thread Tokio runtime, runs the action
@@ -376,6 +411,8 @@ pub(crate) struct PySession {
376411
/// The call whose caller is waiting to learn whether the crate accepted
377412
/// it. Shared with the session callback, which marks it accepted.
378413
pending: PendingSlot,
414+
/// The thread running the current action, if any.
415+
action_thread: ActionThread,
379416
}
380417

381418
impl PySession {
@@ -422,7 +459,9 @@ impl PySession {
422459
return Err(self.busy_error(expected));
423460
};
424461
let (tx, rx) = std::sync::mpsc::sync_channel(1);
462+
let call_id = NEXT_CALL_ID.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
425463
*lock_recover(&self.pending) = Some(PendingCall {
464+
id: call_id,
426465
tx,
427466
accepted: false,
428467
});
@@ -435,9 +474,16 @@ impl PySession {
435474
let session_arc = self.session.clone();
436475
let snapshot = self.snapshot.clone();
437476
let pending = self.pending.clone();
477+
let action_thread = self.action_thread.clone();
438478
let spawned = std::thread::Builder::new()
439479
.name("openjd-session-action".into())
440480
.spawn(move || {
481+
*lock_recover(&action_thread) = Some(std::thread::current().id());
482+
let _guard = ActionThreadGuard {
483+
pending: pending.clone(),
484+
call_id,
485+
action_thread,
486+
};
441487
let Some(session) = lock_recover(&for_thread).take() else {
442488
return;
443489
};
@@ -459,9 +505,10 @@ impl PySession {
459505
accepted: false,
460506
error: Some(e),
461507
}) => Err(session_err_to_py(e)),
462-
// Started, an accepted call that returned (its status says how),
463-
// or an action thread that ended without saying (it panicked
464-
// outside `catch_unwind`; the snapshot reports the failure).
508+
// Started, or an accepted call that returned (its status says
509+
// how). `Err` means the action thread ended without saying, which
510+
// `ActionThreadGuard` turns into a wake-up; nothing does that
511+
// today.
465512
_ => Ok(()),
466513
}
467514
}
@@ -586,6 +633,7 @@ impl PySession {
586633
snapshot,
587634
cancel_handle,
588635
pending,
636+
action_thread: Arc::new(Mutex::new(None)),
589637
})
590638
}
591639

@@ -851,7 +899,10 @@ impl PySession {
851899
///
852900
/// Raises `SessionError` with the crate's message if no action is
853901
/// running, and `ValueError` if `time_limit` is negative, NaN or
854-
/// infinite.
902+
/// infinite. Called from the action's own thread (from a logging
903+
/// handler) before the action has started its subprocess, it waits up
904+
/// to 2 s and then raises `SessionError("Cannot cancel: the action has
905+
/// not started yet")`.
855906
#[pyo3(signature = (time_limit=None, mark_action_failed=None))]
856907
fn cancel_action(
857908
&self,
@@ -866,16 +917,20 @@ impl PySession {
866917
let session_arc = self.session.clone();
867918
let snapshot = self.snapshot.clone();
868919
let handle = self.cancel_handle.clone();
869-
// The crate reports an action `Running` just before it installs the
870-
// action's cancel token, so a cancel straight after the call that
871-
// started it can find no token yet. Keep trying until the token is
872-
// there or the action thread has given the session back; the gap is
873-
// well under a millisecond. Give up after `CANCEL_WAIT`, which is
874-
// only reached when the action thread itself is waiting on this
875-
// call (a logging handler that cancels, for example). The GIL is
876-
// released meanwhile.
920+
// The crate reports an action `Running` just before it logs the
921+
// action's banner and installs its cancel token, so a cancel straight
922+
// after the call that started it can find no token yet. Logging needs
923+
// the GIL, so the gap is as long as the action thread waits for it,
924+
// which under load can be seconds. Keep trying, with the GIL
925+
// released, until the token is there or the action thread has given
926+
// the session back; both happen without help from this thread. On
927+
// the action thread itself (a logging handler that cancels, say),
928+
// neither can happen while this call waits, so give up after
929+
// `CANCEL_WAIT` there.
877930
const CANCEL_WAIT: std::time::Duration = std::time::Duration::from_secs(2);
878-
let deadline = std::time::Instant::now() + CANCEL_WAIT;
931+
let on_action_thread =
932+
*lock_recover(&self.action_thread) == Some(std::thread::current().id());
933+
let deadline = on_action_thread.then(|| std::time::Instant::now() + CANCEL_WAIT);
879934
py.detach(move || loop {
880935
{
881936
let mut guard = lock_recover(&session_arc);
@@ -902,7 +957,7 @@ impl PySession {
902957
return Ok(());
903958
}
904959
}
905-
if std::time::Instant::now() >= deadline {
960+
if deadline.is_some_and(|d| std::time::Instant::now() >= d) {
906961
return Err(session_err_to_py(SessionError::Runtime(
907962
"Cannot cancel: the action has not started yet".into(),
908963
)));

‎specs/python-sessions-interface.md‎

Lines changed: 38 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -100,13 +100,11 @@ class Session:
100100
session_id: str,
101101
job_parameter_values: dict[str, ParameterValue],
102102
path_mapping_rules: Optional[list[PathMappingRule]] = None,
103-
callback: Optional[Callable] = None,
104-
os_env_vars: Optional[dict[str, str]] = None,
105103
retain_working_dir: bool = False,
104+
os_env_vars: Optional[dict[str, str]] = None,
106105
session_root_directory: Optional[Path] = None,
107106
user: Optional[SessionUser] = None,
108-
job_template: Optional[JobTemplate] = None,
109-
environment_templates: Optional[list[EnvironmentTemplate]] = None,
107+
profile: Optional[ModelProfile] = None,
110108
caller_limits: Optional[CallerLimits] = None,
111109
): ...
112110

@@ -116,6 +114,9 @@ class Session:
116114
working_directory: Path
117115
files_directory: Path
118116
action_status: Optional[ActionStatus]
117+
environments_entered: list[str]
118+
119+
def extend_path_mapping_rules(self, additional: list[PathMappingRule]) -> None: ...
119120

120121
# Environment lifecycle (non-blocking — returns once the action is running)
121122
def enter_environment(
@@ -208,44 +209,48 @@ the state, `action_status` (still the previous action's) and
208209
`environments_entered` stay as they were. The crate refuses `run_task`,
209210
`enter_environment` and `run_subprocess` unless the session is `READY`,
210211
`exit_environment` unless it is `READY` or `READY_ENDING`,
211-
`cancel_action` unless an action is running, an unknown, out-of-order
212-
or duplicate environment identifier, a second environment with wrap
213-
hooks, and a blank `command` or zero `timeout`. A call made while
214-
another action runs gets the crate's `InvalidState` message, such as
215-
`Session must be in READY state, current: RUNNING`; before, it was
216-
`RuntimeError("An action is already running")`, and `SessionError` is
217-
a `RuntimeError`. A `timeout` or `time_limit` that is negative, NaN or
212+
`cancel_action` unless an action is running, an unknown, out-of-order or
213+
duplicate environment identifier, a second environment with wrap hooks,
214+
and a blank `command` or zero `timeout`. A call made while another
215+
action runs gets the crate's `InvalidState` message, such as `Session
216+
must be in READY state, current: RUNNING`; before, it was
217+
`RuntimeError("An action is already running")`, and `SessionError` is a
218+
`RuntimeError`. A `timeout` or `time_limit` that is negative, NaN or
218219
infinite raises `ValueError`.
219220

220221
Once the crate accepts a call, the method returns when its action is
221222
`RUNNING` (or, for an environment with no script, has ended), and
222223
`action_status` describes that action. A `cancel_action` made as soon as
223224
it returns reaches the action, unless the action has already ended, as
224225
one that fails before its subprocess starts can: then it raises the
225-
crate's `InvalidState`. A zero `time_limit` terminates at once. Every error after acceptance, such as a
226-
parameter value that can't be coerced, is reported on `action_status`
227-
and doesn't raise, even when it arrives before the call returns. The
228-
crate documents those as failing the action rather than refusing the
229-
call, and the pure-Python reference reports them the same way.
226+
crate's `InvalidState`. A zero `time_limit` terminates at once. Called
227+
from the action's own thread (a logging handler) before the action has
228+
started, `cancel_action` waits up to 2 s and then raises
229+
`SessionError("Cannot cancel: the action has not started yet")`, the
230+
binding's own message. Every error after acceptance, such as a parameter
231+
value that can't be coerced, is reported on `action_status` and doesn't
232+
raise, even when it arrives before the call returns. The crate documents
233+
those as failing the action rather than refusing the call, and the
234+
pure-Python reference reports them the same way.
230235

231236
An action that ends `FAILED`, `CANCELED` or `TIMEOUT` makes the session
232237
ending-only (`READY_ENDING`): `exit_environment` and `cleanup` still
233238
work, and a successful `exit_environment(keep_session_running=True)`
234239
leaves it `READY_ENDING` (openjd-rs#524). `clear_ending_only()` takes it
235240
back to `READY`, keeping the environments entered and the last action's
236-
status; it raises `SessionError` unless the session is `READY_ENDING`. When `run_task`, `enter_environment` or
237-
`exit_environment` fails before its subprocess starts, for example
238-
because a parameter value can't be coerced, the action is `FAILED` with
239-
the error's message, as in the pure-Python reference (openjd-rs#532).
240-
An environment whose enter fails this way still counts as entered, and
241-
one whose exit fails this way counts as exited. When `exit_environment`
242-
can't build its symbol table from `resolved_symtab`, it runs `onExit`
243-
with the table the environment was entered with (openjd-rs#536). White
244-
space around a number in an INT or FLOAT job or task parameter value is
245-
ignored, so `" 7 "` is `7` and `" 1.50 "` is `1.50` (openjd-rs#530).
246-
On Windows, `cleanup` retries removing a session directory that is
247-
still in use, for up to about 31 s, with the GIL released
248-
(openjd-rs#485).
241+
status; it raises `SessionError` unless the session is `READY_ENDING`.
242+
When `run_task`, `enter_environment` or `exit_environment` fails before
243+
its subprocess starts, for example because a parameter value can't be
244+
coerced, the action is `FAILED` with the error's message, as in the
245+
pure-Python reference (openjd-rs#532). An environment whose enter fails
246+
this way still counts as entered, and one whose exit fails this way
247+
counts as exited. When `exit_environment` can't build its symbol table
248+
from `resolved_symtab`, it runs `onExit` with the table the environment
249+
was entered with (openjd-rs#536). White space around a number in an INT
250+
or FLOAT job or task parameter value is ignored, so `" 7 "` is `7` and
251+
`" 1.50 "` is `1.50` (openjd-rs#530). On Windows, `cleanup` retries
252+
removing a session directory that is still in use, for up to about 31 s,
253+
with the GIL released (openjd-rs#485).
249254

250255
### SessionUser (stays in Python, passed to Rust)
251256

@@ -264,6 +269,10 @@ class WindowsSessionUser:
264269
password: Optional[str]
265270
```
266271

272+
`WindowsSessionUser` raises `RuntimeError("Could not determine the
273+
process user: ...")` if it can't look up the process user, as the
274+
pure-Python reference's `get_process_user()` raises.
275+
267276
### Callback interface
268277

269278
The Python callback `Callable[[str, ActionStatus], None]` is invoked from

‎src/openjd/_openjd_rs.pyi‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2303,7 +2303,10 @@ class Session:
23032303
23042304
Raises `SessionError` with the crate's message if no action is
23052305
running, and `ValueError` if `time_limit` is negative, NaN or
2306-
infinite.
2306+
infinite. Called from the action's own thread (from a logging
2307+
handler) before the action has started its subprocess, it waits up
2308+
to 2 s and then raises `SessionError("Cannot cancel: the action has
2309+
not started yet")`.
23072310
"""
23082311

23092312
def cleanup(self) -> None:

0 commit comments

Comments
 (0)