Skip to content

Report every abnormal end of the standalone driver - #5

Merged
SMI-Lab-Inha merged 6 commits into
devfrom
fix/silent-stall
Oct 9, 2026
Merged

SMI-Lab-Inha merged 6 commits into
devfrom
fix/silent-stall

Conversation

@SMI-Lab-Inha

Copy link
Copy Markdown
Owner

When the standalone driver ends abnormally, it now says so.

  • Fatal events. A crash, stack overflow, or Ctrl+C / Ctrl+Break prints one line naming the event and the simulated time reached, then passes the event on, so exit statuses do not change. Example: CableDyn_driver: fatal error: stack overflow (exception 0xC00000FD) after the step at simulated time t = 7534.600 s. The run did not finish …
    • Windows: a console-control handler, a vectored exception handler and an unhandled-exception filter.
    • POSIX: signal handlers on an alternate stack.
    • Only the driver installs these handlers; the shared library does not.
  • Closing line. Every non-zero exit the driver makes itself now ends stderr with CableDyn_driver: ended with exit code <n>. A process ended from outside (for example with taskkill /F) leaves no closing line. That is the only way to tell it apart from a refused input, which also exits 1.
  • Python runner. cabledyn-run and cabledyn-study now separate three cases: a reported failure, an abnormal end, and a process ended from outside. Each states the time the output reached.
  • Docs. New section "When a run ends early" in doc/standalone_driver.rst, with updates to the CLI and troubleshooting pages. The docs note that taskkill /IM ends every run on the machine.

Tests. New CTest cases for overflow, null access and termination reports during and before the march. All existing non-zero-exit checks now require the closing line. The POSIX branch has been desk-checked only, so CI on Linux and macOS is its first compile.

A process that is interrupted or hits a fatal fault now writes one line to
stderr naming the cause and the simulated time of the last committed step,
then lets the event continue to the runtime's own handler, so the exit
status is unchanged. On Windows a console-control handler, a vectored
handler for fatal faults and an unhandled-exception filter (with a stack
guarantee, so a stack overflow is reported) are used; on Linux and macOS
signal handlers on an alternate stack. Only the driver installs them; the
shared library leaves the host process alone.

Every non-zero exit the driver makes itself now ends stderr with the
closing line "CableDyn_driver: ended with exit code <n>". A process ended
from outside (taskkill /F, End task, kill -9) runs none of its own code and
cannot report anything; the missing closing line identifies it, since
taskkill /F leaves exit code 1, the code of a refused input. The Python
runner reports such a run as ended early, with the time its output reached.

Tests: a stack overflow on a 1 MiB stack, a null-pointer write and SIGTERM
each end the process with the report; the march records the time of its
last committed step; the driver's failure checks require the closing line.
@SMI-Lab-Inha

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-09T06:39:57.160260Z 782e63e Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@read-the-docs-community

read-the-docs-community Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Documentation build overview

📚 CableDyn | 🛠️ Build #35033691 | 📁 Comparing 782e63e against latest (c0a88eb)

  🔍 Preview build  

18 files changed · ± 18 modified

± Modified

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 12992a9eff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/cabledyn_fatal.c Resolved
Comment thread src/cabledyn_fatal.c Resolved
… report

Every OpenMP parallel region of the library now starts with
CD_Fatal_Thread_Init, which gives the calling thread room to report its
own stack overflow: a stack guarantee on Windows, an alternate signal
stack (freed when the thread ends) on Linux and macOS. It does nothing
until the driver has installed the report, and costs one thread-local
test after a thread's first call. Previously only the installing thread
had that room, so an overflow on a worker thread could still end the
process without a word.

On Linux and macOS the handlers are now three-argument (SA_SIGINFO)
handlers. A hardware fault returns from the handler under the previous
disposition, so the faulting instruction raises the same fault with its
own address and context; a previous three-argument handler is called
with the original siginfo and context. Only asynchronous signals and
faults sent by kill or raise are sent again.

Tests: a stack overflow on an OpenMP worker with a 1 MiB stack is
reported (it ended silently without the per-thread setup); a forked child
with an earlier SA_SIGINFO handler faults at a known address and that
handler receives the kernel's own siginfo, the child ends by the fault's
signal, and the report is written once; and every OpenMP parallel region
in src/ and app/ is checked to begin with the per-thread call.
@SMI-Lab-Inha

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 367d7b42df

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/cabledyn_fatal.c Resolved
The previous disposition of each signal is now classified by one helper
that compares sa_handler with SIG_IGN, SIG_DFL and SIG_ERR whatever the
SA_SIGINFO flag says (sa_handler and sa_sigaction share storage). The
report is not installed on a signal that was ignored, with or without
SA_SIGINFO, or whose disposition cannot be classified, and it chains only
to a real handler function. Previously an inherited SIG_IGN stored with
SA_SIGINFO set was taken for a handler: the report was installed, and the
ignored signal was then passed to the SIG_IGN sentinel as a function.
If restoring the previous disposition fails, the default action is set
so the handler is never entered again.

Test (POSIX, forked children): SIGTERM set to SIG_IGN with SA_SIGINFO,
and to plain SIG_IGN, leaves the child running to a normal exit with no
report; SIG_DFL is reported once and ends the child by SIGTERM; a
one-argument handler is reported once and then runs.
@SMI-Lab-Inha

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b965006e3e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/cabledyn_fatal.c Resolved
…n a crash

The report no longer calls a previous signal handler itself. It
re-installs the saved action exactly as it was (handler, flags and mask)
and lets the kernel deliver the signal again: a hardware fault returns so
the instruction re-executes, and an asynchronous signal is raised. The
previous handler therefore runs under its own SA_RESETHAND, sa_mask and
self-blocking, as it would have without the report.

Tests (POSIX, forked children): a previous handler with SA_RESETHAND gets
the first SIGTERM and a second SIGTERM ends the process; a previous
handler with SIGUSR1 in its sa_mask runs with SIGUSR1 and SIGTERM
blocked. The fault-context test still sees the kernel's own siginfo.

test_modal and stack_modal ended once on a hosted Windows runner with
"Exception: SegFault" and no output, and passed on a re-run of the same
commit; the toolchain CI resolves passes here. They now install the
abnormal-end report, so a recurrence names the fault (stack overflow or
access violation), and print the OpenBLAS build and kernel the host
selected before any case runs.
@SMI-Lab-Inha

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9d7b17b9dc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread python/cabledyn/driver.py Resolved
Comment thread src/CableDyn_Assemble.f90 Resolved
The driver now writes, after its banner, one line stating that every
failure ends stderr with "CableDyn_driver: ended with exit code <n>".
The Python runner infers an outside end (no closing line) only when that
line is present; with a driver that does not write it (0.1.0 and
earlier), a failure is reported with the driver's own text and the time
its output reached, as before.

Every OpenMP region is now a PARALLEL region whose first statement is
CD_Fatal_Thread_Init, followed by the worksharing DO, so a thread is
prepared once per region instead of once per element. The source guard
now refuses combined PARALLEL DO constructs and a per-iteration call.

Hardening: a thread checks the process-wide "installed" flag before its
thread-local one, so a host that never installs the report never touches
it; the POSIX report flag is set with an atomic exchange, so two threads
that fault together report once; alternate signal stacks are at least
64 KiB and at least the system's SIGSTKSZ / _SC_SIGSTKSZ (128 KiB on
macOS; larger on CPUs with big vector state).

Tests: an older driver's refusal is reported with its own text, a new
driver without a closing line as ended from outside; the report followed
by runtime lines (forrtl, a libgfortran backtrace) is read as an abnormal
end; a library deck run leaves the host's signal dispositions and
unhandled-exception filter unchanged; each prepared thread has one
alternate stack of the system size; only app/cabledyn.f90 installs the
handlers; the real driver writes the exact start-up line.
@SMI-Lab-Inha

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5aebdc19eb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/cabledyn_fatal.c Resolved
…PACK test

A fault signal sent by a process is now recognised by its sender code
(SI_USER, SI_QUEUE, SI_TKILL, SI_TIMER, SI_ASYNCIO, SI_MESGQ where
defined, or any si_code <= 0), not by sign: on macOS SI_USER is positive,
so a SIGSEGV sent with kill() was taken for a hardware fault and the
handler returned instead of sending it again. Test (POSIX, forked
children): SIGSEGV and SIGFPE sent with kill() end the child by that
signal with one report.

On Windows a fault report now names the module and offset of the fault
and, for an access violation, the address read or written, so a fault
inside a library (such as the intermittent one seen in the modal tests
on some hosted runners) identifies itself.

New test lapack_guard_pages: DSYGV and DSBGVX through CableDyn's LAPACK
layer, with every argument array against a no-access page at its end and
at its start (n <= 96, half band widths 0..11, OPENBLAS_CORETYPE=Haswell),
with the argument sizes the modal code uses. It passes with OpenBLAS
0.3.33 and 0.3.34: the modal code's arguments and workspaces are within
bounds.
@SMI-Lab-Inha

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: 782e63e1c5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@SMI-Lab-Inha
SMI-Lab-Inha merged commit 95cc0af into dev Oct 9, 2026
15 checks passed
@SMI-Lab-Inha
SMI-Lab-Inha deleted the fix/silent-stall branch October 12, 2026 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant