Let a test fail when the program it runs crashes

The probe step reported nothing and took no time, which says the first probe
crashed and the step then died formatting its exit code. Two bugs, and the
formatting one is why the table nobody got would have named the statement:
in PowerShell 0xFFFFFFFF is Int32 -1, so the -band was a no-op, and [uint32]
threw on the very value the cast existed to display. `-f` with X8 renders a
negative Int32 as two's-complement hex by itself.

The first probe is CLS, and CLS is the one console statement whose exit code
no test checks. That is not a coincidence. compile_and_run_raw returns Ok
whatever the program did, so a crash arrives as truncated output -- and
test_cls_resets_the_column asserts that a newline does *not* appear after the
escape, which a program that died writing the escape satisfies perfectly. It
has been green on Windows throughout.

RunOutput::assert_ran_to_completion fixes the shape: it fails on any nonzero
exit and prints the code in hex, since Windows says what went wrong in it.
Every test that reads stdout for what a console statement wrote now calls it,
and a new test runs all sixteen console programs and asks only whether they
survived -- which a crash cannot fake. Verified by pointing one entry at
`PRINT 1/0`, which fails it.

The probes now split CLS by what its program links: alone it pulls in almost
nothing, while the same statement after a PRINT pulls in the whole file half
of the runtime. If one crashes and the other does not, the fault is in what
the linker kept rather than in the instructions, which is the question a
harness running the real Win64 output on Linux cannot answer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: jgarzik

.github/workflows/TestingCI.yml

@@ -80,11 +80,11 @@ jobs:
             # a different defect, and none of the three can be reproduced on
             # the Linux job.
             if ($LASTEXITCODE -ne 0) {
-              # Widen before masking: -band on two int32s stays int32, so the
-              # cast of a negative status threw and hid the number it was
-              # printed to reveal.
-              $code = [uint32]([int64]$LASTEXITCODE -band 0xFFFFFFFF)
-              throw ("run failed: {0} (exit {1}, 0x{2:X8})" -f $_.Name, $LASTEXITCODE, $code)
+              # `-f` with X8 renders a negative Int32 as its two's-complement
+              # hex on its own. Masking first needed no cast and got one: in
+              # PowerShell 0xFFFFFFFF is Int32 -1, so the -band was a no-op and
+              # [uint32] then threw on the very value it was there to show.
+              throw ("run failed: {0} (exit {1}, 0x{2:X8})" -f $_.Name, $LASTEXITCODE, $LASTEXITCODE)
             }
           }
 
@@ -102,6 +102,9 @@ jobs:
         run: |
           $probes = [ordered]@{
             'CLS'             = 'CLS'
+            'CLS after PRINT' = "PRINT `"x`"`nCLS"
+            'CLS twice'       = "CLS`nCLS"
+            'PRINT alone'     = 'PRINT "x"'
             'COLOR'           = 'COLOR 14, 1'
             'LOCATE'          = 'LOCATE 2, 5'
             'LOCATE row only' = 'LOCATE 3'
@@ -125,14 +128,13 @@ jobs:
             Set-Content -Path probe.bas -Value $p.Value
             & ./target/release/xbasic64.exe probe.bas -o probe.exe | Out-Null
             if ($LASTEXITCODE -ne 0) {
-              Write-Host ("{0,-16} DID NOT COMPILE" -f $p.Key)
+              Write-Host ("{0,-20} DID NOT COMPILE" -f $p.Key)
               $failed += $p.Key
               continue
             }
             & ./probe.exe | Out-Null
             $code = $LASTEXITCODE
-            $hex = [uint32]([int64]$code -band 0xFFFFFFFF)
-            Write-Host ("{0,-16} exit {1} (0x{2:X8})" -f $p.Key, $code, $hex)
+            Write-Host ("{0,-20} exit {1} (0x{2:X8})" -f $p.Key, $code, $code)
             if ($code -ne 0) { $failed += $p.Key }
           }
           if ($failed.Count -gt 0) { throw "crashed: $($failed -join ', ')" }

tests/common/mod.rs

@@ -32,6 +32,29 @@ impl RunOutput {
     pub fn lines(&self) -> Vec<&str> {
         self.stdout.trim().lines().collect()
     }
+
+    /// Panic unless the program ran to completion.
+    ///
+    /// [`compile_and_run_raw`] deliberately returns `Ok` whatever the program
+    /// did, which makes a crash look like truncated output -- and an assertion
+    /// phrased as "this must not appear" then passes *because* the program
+    /// died. The CLS test was written that way and stayed green on Windows
+    /// while the program it ran was aborting with an access violation.
+    ///
+    /// Any test that reads stdout for what a statement produced wants this
+    /// too. The exit code is printed in hex: Windows says what went wrong in
+    /// it, and 0xC0000005 is not a number the compiler ever chooses.
+    pub fn assert_ran_to_completion(&self, what: &str) {
+        assert_eq!(
+            self.exit_code,
+            Some(0),
+            "{what} did not run to completion: exit {:?} (0x{:08X}), stdout {:?}, stderr {:?}",
+            self.exit_code,
+            self.exit_code.unwrap_or(-1),
+            self.stdout,
+            self.stderr
+        );
+    }
 }
 
 /// A failure of the compiler itself (lexer, parser, sema, codegen, assembler, linker).

tests/control/mod.rs

@@ -1100,6 +1100,7 @@ fn test_system_ends_the_program() {
 #[test]
 fn test_beep_rings_the_bell() {
     let run = crate::common::compile_and_run_raw("BEEP\n", "").expect("should compile");
+    run.assert_ran_to_completion("BEEP");
     assert!(
         run.stdout.contains('\u{7}'),
         "BEEP writes BEL: {:?}",

tests/print/mod.rs

@@ -151,6 +151,7 @@ fn test_locate_and_color_emit_escapes() {
         "",
     )
     .expect("should compile");
+    out.assert_ran_to_completion("LOCATE then COLOR");
     assert!(
         out.stdout.contains("\u{1b}[5;10H"),
         "LOCATE 5,10 should home the cursor there: {:?}",
@@ -172,6 +173,7 @@ fn test_locate_and_color_emit_escapes() {
 #[test]
 fn test_locate_row_only() {
     let out = crate::common::compile_and_run_raw("LOCATE 7\n", "").expect("should compile");
+    out.assert_ran_to_completion("LOCATE with a row only");
     assert!(
         out.stdout.contains("\u{1b}[7;"),
         "row given, column preserved: {:?}",
@@ -209,6 +211,10 @@ fn test_cls_resets_the_column() {
         "",
     )
     .expect("should compile");
+    // Before this line the test could not fail on a crash: a program that died
+    // in CLS left nothing after the escape, which is exactly what the
+    // assertion below wants to see.
+    out.assert_ran_to_completion("PRINT then CLS then TAB");
     let after_cls = out.stdout.rsplit("\u{1b}[H").next().unwrap_or("");
     assert!(
         !after_cls.starts_with('\n'),
@@ -217,6 +223,42 @@ fn test_cls_resets_the_column() {
     );
 }
 
+/// Every console statement's program runs to completion.
+///
+/// Blunt on purpose. These helpers are written twice, and the Win64 half is
+/// the one no developer runs -- CI is the only place it executes at all. The
+/// tests above read what each statement *wrote*, which a crash can satisfy by
+/// writing nothing; this one only asks whether the program survived, which a
+/// crash cannot.
+#[test]
+fn test_console_statements_run_to_completion() {
+    for (what, source) in [
+        ("CLS", "CLS\n"),
+        ("CLS after PRINT", "PRINT \"x\"\nCLS\n"),
+        ("CLS twice", "CLS\nCLS\n"),
+        ("LOCATE", "LOCATE 2, 5\n"),
+        ("LOCATE row only", "LOCATE 3\n"),
+        ("LOCATE column only", "LOCATE , 8\n"),
+        ("COLOR", "COLOR 14, 1\n"),
+        ("POS", "PRINT POS(0)\n"),
+        ("BEEP", "BEEP\n"),
+        ("TIMER", "PRINT TIMER\n"),
+        ("RANDOMIZE", "RANDOMIZE 42\n"),
+        ("RANDOMIZE TIMER", "RANDOMIZE TIMER\n"),
+        ("RND", "PRINT RND\n"),
+        ("DATE$ and TIME$", "PRINT DATE$\nPRINT TIME$\n"),
+        ("FRE", "PRINT FRE(0)\n"),
+        (
+            "the lot together",
+            "CLS\nCOLOR 14, 1\nLOCATE 2, 5\nPRINT \"x\"; POS(0)\n",
+        ),
+    ] {
+        let run = crate::common::compile_and_run_raw(source, "")
+            .unwrap_or_else(|e| panic!("{what} should compile: {e}"));
+        run.assert_ran_to_completion(what);
+    }
+}
+
 /// `LOCATE` also sets the column the tracker believes, for the same reason.
 #[test]
 fn test_locate_sets_the_column() {