Skip to content

Commit c80d906

Browse files
authored
perf(ci): Cut Docker smoke-test time on large runners (#588)
## What Two changes to the Docker smoke-test job: - **Disk cleanup runs only when space is short.** Cleanup is skipped when both `/` and Docker's storage filesystem have at least 30 GiB free. Smaller runners still clean up as before. - **The generic PDF and `.gz` format checks use the one-page PDF** instead of the full paper. The `.gz` case compresses that PDF in pytest's temp directory, so the unused `layout-parser-paper.pdf.gz` fixture is removed. ``` Before: cleanup 183s (with 371 GiB free) · .pdf case 175s · .pdf.gz case 191s After: cleanup skipped · both cases process a one-page PDF ``` ## Why These checks only confirm that each file type is accepted and parsed. The full paper is still processed by the table-extraction, strategy, and parallel-mode smoke tests, so the generic format cases don't need it. All 37 cases and their explicit-MIME, inferred-MIME, and CSV assertions remain. ## Validation actionlint and Ruff pass, and all 37 smoke tests collect. Both changed cases were run against a controlled response, and the gzip decompresses byte-for-byte to the source PDF. Both disk branches (enough space and low space) were tested with mocked commands. Full timing comes from this PR's CI ([baseline](https://github.com/Unstructured-IO/unstructured-api/actions/runs/36955856924)).
1 parent cba716b commit c80d906

3 files changed

Lines changed: 24 additions & 21 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 15 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -102,26 +102,23 @@ jobs:
102102
cache-suffix: smoke
103103
- name: Set up Python ${{ env.PYTHON_VERSION }}
104104
run: uv python install ${{ env.PYTHON_VERSION }}
105-
- name: Free up disk space
105+
- name: Free up disk space when needed
106106
run: |
107-
# Clear some space (https://github.com/actions/runner-images/issues/2840)
108-
echo "Disk usage before cleanup:"
109-
df -h
110-
111-
# Remove unnecessary pre-installed software
112-
sudo rm -rf /usr/share/dotnet
113-
sudo rm -rf /opt/ghc
114-
sudo rm -rf /usr/local/share/boost
115-
sudo rm -rf /usr/local/lib/android
116-
sudo rm -rf /opt/hostedtoolcache/CodeQL
117-
sudo rm -rf /usr/local/.ghcup
118-
sudo rm -rf /usr/share/swift
119-
120-
# Clean up docker to ensure we start fresh
107+
# Large runners normally have hundreds of GiB free. Keep the fallback for
108+
# smaller runners, but avoid spending minutes deleting unused toolchains.
109+
# Docker images and the host toolchains/venv may live on different
110+
# filesystems, so both must have room before cleanup is skipped.
111+
docker_root=$(docker info --format '{{.DockerRootDir}}')
112+
available_kib=$(df -Pk / "$docker_root" | awk 'NR > 1 && (min == "" || $4 < min) {min = $4} END {print min}')
113+
if (( available_kib >= 30 * 1024 * 1024 )); then
114+
echo "Skipping cleanup: $((available_kib / 1024 / 1024)) GiB free on / and Docker storage"
115+
exit 0
116+
fi
117+
df -h / "$docker_root"
118+
sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/share/boost \
119+
/usr/local/lib/android /opt/hostedtoolcache/CodeQL /usr/local/.ghcup /usr/share/swift
121120
docker system prune -af --volumes
122-
123-
echo "Disk usage after cleanup:"
124-
df -h
121+
df -h / "$docker_root"
125122
- name: Install smoke test dependencies
126123
run: uv sync --only-group smoke --locked
127124
- name: Build Dockerfile
-4.17 MB
Binary file not shown.

‎scripts/smoketest.py‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ def send_document(
6767
(".md", "README.md", "text/markdown"),
6868
(".msg", "fake-email.msg", "application/x-ole-storage"),
6969
(".odt", "fake.odt", "application/vnd.oasis.opendocument.text"),
70-
(".pdf", "layout-parser-paper.pdf", "application/pdf"),
70+
(".pdf", "layout-parser-paper-fast.pdf", "application/pdf"),
7171
(".png", "english-and-korean.png", "image/png"),
7272
(".ppt", "fake-power-point.ppt", "application/vnd.ms-powerpoint"),
7373
(
@@ -89,12 +89,12 @@ def send_document(
8989
(".json", "spring-weather.html.json", "application/json"),
9090
(
9191
".gz",
92-
"layout-parser-paper.pdf.gz",
92+
"layout-parser-paper-fast.pdf",
9393
"application/gzip",
9494
),
9595
],
9696
)
97-
def test_happy_path_all_types(extension, example_filename: str, content_type: str):
97+
def test_happy_path_all_types(extension, example_filename: str, content_type: str, tmp_path: Path):
9898
"""
9999
For the files in sample-docs, verify that we get a 200
100100
and some structured response
@@ -113,6 +113,12 @@ def test_happy_path_all_types(extension, example_filename: str, content_type: st
113113
pytest.skip("emulated hardware")
114114

115115
test_file = str(Path("sample-docs") / example_filename)
116+
if extension == ".gz":
117+
# Gzip the one-page PDF at test time; full-document inference is covered
118+
# by the table and strategy tests.
119+
gzipped_file = tmp_path / f"{example_filename}.gz"
120+
gzip_file(test_file, str(gzipped_file))
121+
test_file = str(gzipped_file)
116122

117123
# Verify we can send with explicit content type
118124
response = send_document(filenames=[test_file], content_type=content_type)

0 commit comments

Comments
 (0)