Make every problem standalone, and verify every documented example

Two gaps, both found by actually trying the things the docs claim.

Problems were not standalone. The demo ran anywhere, but the tests
depended on a root conftest.py, so a problem copied out of the
repository could not be tested -- which contradicts the README's claim
that each one is self-contained. Each problem now carries its own
conftest.py putting its src on sys.path, and the root one is gone. That
works for every invocation: from the repo root, from inside the problem,
by file, by node id, and from a copy with no repo files present at all.
A new CI job proves it by copying each problem to a temp directory and
running it there.

Documentation was unverified. scripts/check_examples.py now runs every
command in every bash fence and checks that every unlabelled block
following one really is what the command prints. Timestamps, epochs,
temp paths and generated ids are masked so genuinely variable output
still gets checked structurally rather than exempted wholesale.

It found four things on its first run:

- tic-tac-toe --human tracebacked with EOFError on piped stdin instead
  of ending the game. HumanPlayer now raises PlayerQuit on EOF or
  Ctrl-C and main exits cleanly; two tests cover it.
- payment-gateway's README dropped a transaction reference when pasting.
- text-editor's README had hand-realigned columns, so no line matched.
- logging-framework's README dropped a context field, abbreviated a
  path, and elided mid-line. I had originally exempted that problem from
  output checking because of its timestamps, which is exactly how the
  drift survived -- masking instead of exempting caught all three.

883 tests, 55 documented commands, 17 problems standalone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: abhaypaswan

.github/workflows/ci.yml

@@ -35,9 +35,10 @@ jobs:
       - name: Check the generated catalog, README table and patterns doc are current
         run: python scripts/build_catalog.py --check
 
-  demos:
-    # Every problem advertises a runnable demo. This makes sure they all still
-    # run, which the unit tests do not cover.
+  standalone:
+    # Each problem claims to be self-contained. This proves it: every problem
+    # is copied out of the repository, away from the root pyproject.toml, and
+    # its tests and demo are run with nothing else present.
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
@@ -46,22 +47,56 @@ jobs:
         with:
           python-version: "3.12"
 
-      - name: Run every problem's demo
+      - run: pip install pytest
+
+      - name: Run every problem in isolation
         run: |
-          set -e
+          set -u
           failed=0
+          workdir=$(mktemp -d)
+
           for problem in problems/*/; do
             slug=$(basename "$problem")
-            printf '%-22s' "$slug"
-            if (cd "$problem" && timeout 120 python src/main.py > /dev/null 2>&1); then
-              echo "ok"
+            cp -R "$problem" "$workdir/$slug"
+            find "$workdir/$slug" -name '__pycache__' -type d -exec rm -rf {} + 2>/dev/null || true
+            printf '%-24s' "$slug"
+
+            if (cd "$workdir/$slug" && timeout 300 python -m pytest -q > /dev/null 2>&1); then
+              tests=ok
+            else
+              tests=TESTS-FAILED
+              failed=1
+            fi
+
+            if (cd "$workdir/$slug" && timeout 300 python src/main.py > /dev/null 2>&1 < /dev/null); then
+              demo=ok
             else
-              echo "FAILED"
+              demo=DEMO-FAILED
               failed=1
             fi
+
+            echo "tests:$tests demo:$demo"
           done
+
           exit $failed
 
+  examples:
+    # Every command in the documentation is executed, and every output block
+    # the docs show is checked against what the command really prints.
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+
+      - run: pip install -r requirements-dev.txt
+
+      - name: Check every documented example
+        run: python scripts/check_examples.py
+
   lint:
     runs-on: ubuntu-latest
     steps:

CONTRIBUTING.md

@@ -39,10 +39,14 @@ problems/your-problem/
     └── test_your_problem.py
 ```
 
-**The package name must be the slug with underscores.** Every problem's `src/`
-goes on `sys.path` at once (see the root `conftest.py`), so a shared package
-name like `model` would mean the second problem to define it silently gets the
-first one's module.
+**The package name must be the slug with underscores.** Each problem's
+`conftest.py` puts its own `src/` on `sys.path`, and when the whole suite runs
+they are all there at once — so a shared package name like `model` would mean
+the second problem to define it silently gets the first one's module.
+
+That `conftest.py` is also what makes a problem **standalone**: copy the
+directory out of the repository and `python3 -m pytest` still works inside it.
+CI proves this for every problem on every push, so do not remove it.
 
 ### What each piece needs
 
@@ -91,9 +95,17 @@ python3 -m pytest              # all green
 python3 scripts/validate.py    # every problem complete and consistent
 python3 scripts/build_catalog.py   # refresh catalog.json, the README table, docs/patterns.md
 ruff check . && ruff format .
+python3 scripts/check_examples.py  # every documented command runs, every shown output is real
 ```
 
-CI runs all of these, plus every problem's demo, on Python 3.10 through 3.13.
+CI runs all of these on Python 3.10 through 3.13, plus every problem in
+isolation outside the repository.
+
+**If your README shows command output, paste what the command actually
+prints.** `check_examples.py` runs every ```bash block and checks every
+unlabelled block after one against the real output, so a hand-tidied paste
+fails the build. Timestamps, temporary paths and epoch values are masked, and
+a trailing `...` on a line means "the rest was trimmed".
 
 Never hand-edit `catalog.json`, the problem table in `README.md`, or
 `docs/patterns.md` — all three are generated from the `meta.json` files. The one

README.md

@@ -113,6 +113,13 @@ Or one problem's:
 python3 -m pytest problems/chess -v
 ```
 
+Each problem is genuinely self-contained — copy one out of the repository and
+its tests and demo still run, with nothing else present:
+
+```bash
+cd problems/chess && python3 -m pytest
+```
+
 ## How this repository is put together
 
 Each problem lives in `problems/<slug>/` and is self-contained — you can read
@@ -131,6 +138,7 @@ genuinely uses, and how long it should take. Those files validate against
 ```bash
 python3 scripts/validate.py        # check every problem is complete and consistent
 python3 scripts/build_catalog.py   # regenerate catalog.json, the table above, and docs/patterns.md
+python3 scripts/check_examples.py  # run every command in these docs and verify its output
 ```
 
 `catalog.json`, the problem table above, and `docs/patterns.md` are all
@@ -169,7 +177,10 @@ the design is meant to handle.
 4. Run `python3 scripts/build_catalog.py` to refresh the catalog
 5. Open a pull request
 
-CI runs the tests, the validator, and a check that the catalog is up to date.
+CI runs four jobs: the test suite on Python 3.10–3.13 plus the validator and
+the generated-file check; every problem in isolation, copied out of the
+repository; every command in the documentation, with its shown output verified;
+and lint.
 
 <p align="right">(<a href="#readme-top">back to top</a>)</p>
 

conftest.py

@@ -0,0 +1,22 @@
+"""Put this problem's `src` on sys.path, so its tests run from anywhere.
+
+pytest loads the nearest conftest.py before importing the test modules beneath
+it, which is what makes this work whether you run:
+
+    pytest                                  # from this directory
+    pytest problems/atm          # from the repo root
+    pytest tests/test_atm.py  # a single file
+
+It also means the directory is genuinely self-contained: copy it out of the
+repository and the tests still run, with no repo-level setup to remember.
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+SRC = Path(__file__).resolve().parent / "src"
+
+if str(SRC) not in sys.path:
+    sys.path.insert(0, str(SRC))

problems/atm/conftest.py

@@ -0,0 +1,22 @@
+"""Put this problem's `src` on sys.path, so its tests run from anywhere.
+
+pytest loads the nearest conftest.py before importing the test modules beneath
+it, which is what makes this work whether you run:
+
+    pytest                                  # from this directory
+    pytest problems/chat-room          # from the repo root
+    pytest tests/test_chat_room.py  # a single file
+
+It also means the directory is genuinely self-contained: copy it out of the
+repository and the tests still run, with no repo-level setup to remember.
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+SRC = Path(__file__).resolve().parent / "src"
+
+if str(SRC) not in sys.path:
+    sys.path.insert(0, str(SRC))

problems/chat-room/conftest.py

@@ -0,0 +1,22 @@
+"""Put this problem's `src` on sys.path, so its tests run from anywhere.
+
+pytest loads the nearest conftest.py before importing the test modules beneath
+it, which is what makes this work whether you run:
+
+    pytest                                  # from this directory
+    pytest problems/chess          # from the repo root
+    pytest tests/test_chess.py  # a single file
+
+It also means the directory is genuinely self-contained: copy it out of the
+repository and the tests still run, with no repo-level setup to remember.
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+SRC = Path(__file__).resolve().parent / "src"
+
+if str(SRC) not in sys.path:
+    sys.path.insert(0, str(SRC))

problems/chess/conftest.py

@@ -0,0 +1,22 @@
+"""Put this problem's `src` on sys.path, so its tests run from anywhere.
+
+pytest loads the nearest conftest.py before importing the test modules beneath
+it, which is what makes this work whether you run:
+
+    pytest                                  # from this directory
+    pytest problems/elevator-system          # from the repo root
+    pytest tests/test_elevator_system.py  # a single file
+
+It also means the directory is genuinely self-contained: copy it out of the
+repository and the tests still run, with no repo-level setup to remember.
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+SRC = Path(__file__).resolve().parent / "src"
+
+if str(SRC) not in sys.path:
+    sys.path.insert(0, str(SRC))

problems/elevator-system/conftest.py

@@ -0,0 +1,22 @@
+"""Put this problem's `src` on sys.path, so its tests run from anywhere.
+
+pytest loads the nearest conftest.py before importing the test modules beneath
+it, which is what makes this work whether you run:
+
+    pytest                                  # from this directory
+    pytest problems/file-system          # from the repo root
+    pytest tests/test_file_system.py  # a single file
+
+It also means the directory is genuinely self-contained: copy it out of the
+repository and the tests still run, with no repo-level setup to remember.
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+SRC = Path(__file__).resolve().parent / "src"
+
+if str(SRC) not in sys.path:
+    sys.path.insert(0, str(SRC))

problems/file-system/conftest.py

@@ -207,13 +207,14 @@ cd problems/logging-framework && python3 src/main.py
 ```
 
 ```
-2026-09-19 05:35:39 [INFO] app: GET /orders request_id=req-8814 user=alice
-2026-09-19 05:35:39 [ERROR] app: Upstream timeout request_id=req-8814 status=504 upstream=payments
+2026-09-19 06:27:43 [INFO] app: GET /orders request_id=req-8814 user=alice
+2026-09-19 06:27:43 [ERROR] app: Upstream timeout request_id=req-8814 status=504 upstream=payments user=alice
 
 7 records passed through the chain.
-2 of them reached errors.jsonl:
+2 of them reached /tmp/lld-logging-a1b2c3/errors.jsonl:
 
-{"level": "ERROR", "logger": "app", "message": "Upstream timeout", "status": 504, ...}
+{"level": "ERROR", "logger": "app", "message": "Upstream timeout", "request_id": "req-8814", "status": 504, "timestamp": 1789799263.605953, "upstream": "payments", "user": "alice"}
+{"level": "ERROR", "logger": "db", "message": "Connection pool exhausted", "size": 20, "timestamp": 1789799263.6060832, "waiting": 7}
 ```
 
 ## Tests