operations.md

Operations, deployment, and debugging

This document is the practical companion to architecture.md. It covers how to run Symx locally, how the current deployment is wired, and how to inspect and debug production behavior. See domain-terminology.md for the Apple artifact and extraction vocabulary used by both documents.

1. Local setup

1.1 Base requirements

• Python 3.14+
• uv
• this repository checked out locally

For the newest Apple artifacts, keep both macOS and Xcode current. macOS 27 and Xcode 27 are recommended for extracting 27-era artifacts. Host requirements depend on the artifact; older artifacts may work with older toolchains. See extraction commands for the separate macOS and Xcode dependencies. Metadata, mirroring, and CLI/TUI administration can still run on Linux.

Install Python dependencies:

uv sync --dev

The project exposes a symx console script, so after uv sync the usual entrypoint is:

uv run symx --help

1.2 Tooling requirements by task

Admin / workflow inspection only

Needed for:

• the native app under apps/SymxAdmin
• uv run symx admin
• uv run symx admin sync
• direct GitHub Actions inspection with gh

Requirements:

• gh CLI installed
• gh auth login completed with access to this repository
• for the native app, macOS 27 and the Swift toolchain included with Xcode 27 are recommended; see apps/SymxAdmin/README.md

These commands do not need direct GCP credentials locally, because the actual metadata fetch happens inside a GitHub Actions workflow.

GCS-backed Symx runs

Needed for:

• uv run symx ipsw meta-sync ...
• uv run symx ipsw mirror ...
• uv run symx ipsw extract ...
• uv run symx ota mirror ...
• uv run symx ota extract ...
• uv run symx sim extract ...

Requirements:

• Google Cloud credentials available via ADC (gcloud auth application-default login) or GOOGLE_APPLICATION_CREDENTIALS
• a storage URI accepted by Symx, for example:
  • gs://my-bucket
  • gs://my-project@my-bucket

Caution

These commands read and write the configured bucket directly. Use a non-production bucket unless you intentionally want to operate on production data.

OTA metadata retrieval

Needed for:

• uv run symx ota mirror ...

Requirement:

• checksum-pinned ipsw 3.1.721 or newer installed and on PATH

The schema-1 OTA metadata envelope was introduced in ipsw 3.1.713. Symx standardizes metadata retrieval and extraction on the newer 3.1.721 production pin.

Extraction commands

Needed for:

• uv run symx ipsw extract-file ...
• uv run symx ota extract-file ...
• uv run symx ipsw extract ...
• uv run symx ota extract ...
• uv run symx sim extract ...

Requirements:

• a macOS host and selected full Xcode installation with the capabilities needed by the artifact
• checksum-pinned ipsw 3.1.721 or newer installed and on PATH
• executable ./symsorter at the repository root

For current artifacts, keep both macOS and Xcode current, not just ipsw:

• macOS supplies Apple Archive tooling and libraries needed to handle newer compression formats such as lzraven.
• Xcode supplies dsc_extractor.bundle, which ipsw dyld split uses for bulk DSC extraction. Xcode 27's extractor supports arm64e_x1 dyld shared caches; older extractors may reject these even when they can split an older artifact's caches. This applies to the cache variant, not just to macOS artifacts.

Installing an updated Xcode is not enough if another installation remains selected. Symx invokes ipsw dyld split without an explicit --xcode path, so ipsw uses the installation reported by xcode-select -p. Updating macOS or ipsw alone does not replace that selected Xcode extractor. DEVELOPER_DIR overrides the system-wide selection for an invocation and is honored by ipsw.

IPSW/OTA extraction commands log the actual host OS/kernel and architecture, DEVELOPER_DIR, macOS version/build, selected Xcode developer directory and version/build, and DSC extractor path/presence, alongside the existing ipsw and symsorter version logs. Host diagnostics are best-effort: failed probes warn but do not stop extraction, and no macOS or Xcode version gate is applied. These logs describe the environment used, not a compatibility verdict. Compare them with the artifact being extracted when investigating failures.

For IPSW extraction, Symx also ships a vendored AEA PEM DB snapshot at symx/ipsw/data/fcs-keys.json and passes it to ipsw via --pem-db before ipsw falls back to live Apple FCS-key lookup. Refresh that file from upstream ipsw/pkg/aea/data/fcs-keys.gz when newly mirrored IPSWs start failing with AEA/FCS-key 403s on GitHub macOS runners. Before the high-level ipsw mount sys / ipsw extract --dyld steps, Symx also does a small AEA preflight against the selected DMG member so failures can be classified as key-resolution problems earlier.

Production extraction runs on macOS because some inputs require Apple-provided macOS and Xcode binaries to handle encryption and compression algorithms that are unavailable in the Linux environment. Neither ipsw, DMG mounting, nor symsorter is inherently limited to macOS. Symx requires ipsw 3.1.721 or newer for per-family validation, structured error paths and hardened cache-family opening. The schema-1 ipsw download ota --json envelope first appeared in 3.1.713; the current pin also includes the OTA resolver, sequential macOS cryptex architecture search, and arm64e_x1 DSC/cryptex handling. ipsw owns any cryptex patching and temporary mount lifecycle.

Simulator extraction

Needed for:

• uv run symx sim extract ...

Additional requirement:

• a machine with Xcode simulator dyld caches at one of:
  • /Library/Developer/CoreSimulator/Caches/dyld
  • ~/Library/Developer/CoreSimulator/Caches/dyld

1.3 Quick sanity checks

Helpful local checks before debugging the actual workflows:

uv run symx --help
ipsw version
./symsorter --version
gh auth status

On extraction hosts, also check the OS and selected Xcode before investigating an artifact failure:

sw_vers
xcode-select -p
xcodebuild -version

2. Running Symx locally

2.1 Local, file-based reproduction

These commands are the fastest route when you already have the artifact locally and want to reproduce extraction behavior.

IPSW

uv run symx ipsw extract-file /path/to/file.ipsw -p iOS -o /tmp/symx-ipsw

What it does:

• logs host diagnostics and validates ipsw and ./symsorter,
• runs the IPSW extraction pipeline locally; macOS probes the shared arm64e, arm64e_x1, x86_64, and x86_64h DSC architecture set sequentially, skipping absent candidates without masking real extraction failures,
• uses the vendored IPSW PEM DB snapshot before any live Apple FCS-key lookup,
• plans every distinct SystemOS image from BuildManifest and carries a validated --device selector through both mount and DSC extraction when needed; shared Rosetta images are handled independently, not once per device,
• performs an AEA preflight for each encrypted SystemOS member before the higher-level ipsw mount/extract helpers,
• accumulates one source bundle using image-qualified splits and bounded restore/symsort batches,
• consumes/deletes the input IPSW after all image consumers finish: use a disposable copy/APFS clone, not your master file, a symlink, or a hard link,
• prints the output directory containing the symsorter result.

If mount cleanup cannot confirm detach, extraction stops and retains its owned workspace instead of recursively removing a potentially live mount. Inspect the logged workspace and hdiutil info; do not manually delete its mount or backing files until detach is confirmed. The worker still attempts the final failed-source metadata update but will not proceed to another source. Ordinary extraction failures retain the existing failed-source/continue policy.

OTA

uv run symx ota extract-file /path/to/file.zip -p ios -V 18.2 -b 22C152 -o /tmp/symx-ota

What it does:

• logs host diagnostics and validates ipsw and ./symsorter,
• for macOS, requests arm64e, arm64e_x1, x86_64, and x86_64h sequentially with exactly one --dyld-arch per operation; other platforms use the existing unfiltered operation,
• strictly parses each schema-1 report and validates every reported regular file beneath that attempt's temporary output root, including files belonging to excluded families,
• warns and continues past dsc-validation failures only when every error's structured primary path and source match a reported DriverKit or x86Support family; a supported System primary is still required,
• distinguishes a requested architecture that is absent from any other partial or real failure,
• treats remaining reports containing only dsc-validation failures as expected skips only when trusted metadata confirms a delta or recovery artifact; full and unknown artifacts are still failures,
• splits and compresses each successful macOS architecture before removing its materialization directory,
• restores all successful split archives and symsorts them in one invocation,
• preserves the caller-owned OTA file on both success and failure,
• prints the output directories containing the symsorter result.

Per-family validation and its optional error path are included in the pinned ipsw 3.1.721 release. Path-less errors cannot be excluded by diagnostic text.

There is no second materialization syntax or human-log-based fallback. A real failure from any architecture stops the operation, even if an earlier architecture succeeded. Structured errors and bounded stderr are retained for diagnostics; post-failure OTA probes do not materialize symbols. A materializer killed by a signal raises a distinct OtaDscProcessTerminatedError before JSON parsing, even if stdout contains a valid-looking report. Positive exit codes with missing/malformed JSON remain protocol errors.

2.2 GCS-backed runs from your machine

These are the same commands the production workflows ultimately run, just executed from your laptop or workstation.

IPSW

uv run symx ipsw meta-sync -s gs://my-project@my-bucket
uv run symx ipsw mirror -s gs://my-project@my-bucket -t 60
uv run symx ipsw extract -s gs://my-project@my-bucket -t 60
uv run symx ipsw migrate -s gs://my-project@my-bucket

OTA

uv run symx ota mirror -s gs://my-project@my-bucket
uv run symx ota extract -s gs://my-project@my-bucket -t 60
uv run symx ota migrate-storage -s gs://my-project@my-bucket

Simulator

uv run symx sim extract -s gs://my-project@my-bucket

Use these primarily for:

• development against a disposable bucket,
• one-off reproduction of a production workflow with explicit intent,
• validating changes before wiring them into GitHub Actions.

2.3 Admin workflow and clients

Sync the local snapshot cache

uv run symx admin sync

What happens:

1. Symx uses gh workflow run symx-admin-meta-sync.yml.
2. GitHub Actions fetches the remote IPSW and OTA metadata plus their GCS generations.
3. Symx downloads the workflow artifact locally.
4. Symx builds a SQLite snapshot under ~/.cache/symx/admin/snapshots/<snapshot_id>/snapshot.db.
5. manifest.json is updated to point to the active snapshot.

Start the native macOS app

macOS 27 and the Swift toolchain included with Xcode 27 are recommended for local administration:

cd apps/SymxAdmin
swift run

It reads the same local snapshot, can trigger a sync, provides complete searchable and sortable IPSW/OTA tables with a global modification-time preset (four weeks by default), shows relevant workflow runs, and applies curated rerun queues through GitHub Actions. See apps/SymxAdmin/README.md for its current feature list and model contract.

Start the TUI fallback

uv run symx admin

Current TUI capabilities:

• auto-sync on startup if no cached snapshot exists or the cached snapshot is older than 24h,
• separate IPSW and OTA failure tables,
• filter by processing state,
• show OTA GitHub run metadata when available,
• queue artifact downloads into ~/.cache/symx/admin/downloads/,
• build curated rerun batches for eligible rows and apply them through GitHub Actions.

Useful variants:

uv run symx admin --failure-state symbol_extraction_failed
uv run symx admin --failure-state mirroring_failed --failure-state mirror_corrupt
uv run symx admin --cache-dir /tmp/symx-admin

Query the SQLite snapshot directly

If you want a raw view instead of the TUI, inspect the local DB:

sqlite3 ~/.cache/symx/admin/snapshots/<snapshot_id>/snapshot.db '.tables'

Available tables:

• snapshot_info
• ipsw_artifacts
• ipsw_sources
• ota_artifacts

Generate the coverage HTML page

This report counts only rows in symbols_extracted, using the active local snapshot:

uv run symx stats coverage-html --output /tmp/symx-coverage.html

The current coverage page groups by platform and version for both IPSW and OTA.

It embeds the grouped rows as JSON and enhances the static tables with client-side filters plus a reset button for each table:

• platform
• major version
• minor version
• patch version

Selectors with no values for the current parent selection are hidden.

Table order is prepared in Python before HTML generation. Rows are sorted by platform, then by numeric version parts in descending order, with release-train tiebreakers within the same numeric base version such as release → RC → beta.

2.4 GitHub Actions inspection with gh

Use the GitHub CLI directly when you want to inspect workflow runs outside the admin TUI.

List known workflows:

gh workflow list

List recent runs for a workflow:

gh run list --workflow "Extract OTA symbols" --limit 20
gh run list --workflow "Mirror IPSW artifacts" --status failure --limit 20
gh run list --workflow symx-ota-extract.yml --limit 20

View or grep a specific run log:

gh run view 123456789 --log
gh run view 123456789 --log | grep -n "error" -C 2

3. Current deployment scenario

3.1 Source of truth

The current production system is rooted in three places:

• GitHub Actions for scheduling and execution,
• GCS for persisted state and artifacts,
• Sentry for observability.

There is no always-on service and no separate relational production database behind the workflows today.

3.2 Workflow entrypoints

Workflow file	Purpose	Command run inside workflow
.github/workflows/symx-ipsw-meta-sync.yml	refresh IPSW metadata	symx ipsw meta-sync -s $SYMX_STORE
.github/workflows/symx-ipsw-mirror.yml	mirror IPSWs	symx ipsw mirror -t 315 -s $SYMX_STORE
.github/workflows/symx-ipsw-extract.yml	extract IPSW symbols	symx ipsw extract -t 315 -s $SYMX_STORE
.github/workflows/symx-ota-mirror.yml	refresh OTA metadata and mirror OTAs	symx ota mirror -s $SYMX_STORE
.github/workflows/symx-ota-extract.yml	extract OTA symbols	symx ota extract -t 330 -s $SYMX_STORE
.github/workflows/symx-ota-migrate-storage.yml	reset failed OTA extractions	symx ota migrate-storage -s $SYMX_STORE
.github/workflows/symx-ipsw-migrate.yml	manual IPSW migration entrypoint	symx ipsw migrate -s $SYMX_STORE
.github/workflows/symx-simulator-extract.yml	upload simulator symbols	symx sim extract -s $SYMX_STORE
.github/workflows/symx-admin-meta-sync.yml	build admin snapshot inputs	shell script using gcloud storage
.github/workflows/symx-admin-apply.yml	apply curated admin rerun batches	symx admin apply-batch --storage "$SYMX_STORE" --request-json ... --result-path ...
.github/workflows/symx-coverage-pages.yml	publish coverage stats to GitHub Pages	symx admin sync --cache-dir ... then symx stats coverage-html --output site/index.html

For the workflows that use the reusable Symx runner wrappers, the command is not shell-interpolated directly. Those reusable workflows pass SYMX_RUN into scripts/run_symx_gha.py, which safely tokenizes it and executes python -m symx .... The admin meta-sync, admin apply, coverage-pages, and simulator workflows are custom wrappers around the same CLI.

The GitHub Pages deployment assumes the repository Pages source is set to GitHub Actions.

3.3 Reusable workflow layers

Ubuntu reusable workflow

File: .github/workflows/symx-runner-ubuntu.yml

It does the following:

• checks out the repository,
• authenticates to GCP via GitHub OIDC,
• installs the pinned Cloud SDK,
• installs pinned uv and Python dependencies,
• installs the pinned, checksum-verified ipsw CLI,
• installs the checksum-verified Apple root certificate,
• removes large preinstalled toolchains to free disk,
• runs scripts/run_symx_gha.py.

This is used for meta sync and mirroring stages.

macOS reusable workflow

File: .github/workflows/symx-runner-macos.yml

It does the following:

• checks out the repository,
• authenticates to GCP via GitHub OIDC,
• installs the pinned Cloud SDK,
• installs pinned uv and Python dependencies,
• installs the pinned, checksum-verified ipsw CLI,
• downloads the pinned, checksum-verified symsorter binary,
• logs macOS host, Xcode, AppleArchive (/usr/bin/aa), and ipsw versions for extraction diagnostics,
• runs scripts/run_symx_gha.py.

This is used for OTA and IPSW extraction stages. Both callers explicitly enable remove_simulators (default false for other callers). Before authentication, dependencies, or artifact downloads, the disposable CI runner uses .github/scripts/remove-ci-simulators.sh to shut down/delete simulator devices, remove generated simulator dyld caches, and delete runtime images together with their associated MobileAssets. CoreSimulator performs the unmount/removal lifecycle; the script does not recursively remove mounted paths, Xcode, SDKs, or OS components. Runtime inventories and workspace/temp-filesystem free space are logged before/after cleanup, including disk reporting on failure, to .github/logs/remove-simulators.log. A cleanup failure blocks Symx and is included in the normal failure notification. Actual reclaimed space depends on the runner image; no fixed saving is assumed.

Do not run this destructive preparation on a workstation or a persistent/shared runner. The script checks for macOS Actions and a workspace, but those environment checks are not proof that a host is disposable. It is not called by local extraction commands. The separate symx-simulator-extract.yml deliberately retains simulators and has neither this cleanup nor the disk watchdog.

Emergency extraction disk watchdog

The reusable macOS wrapper enables SYMX_CI_DISK_GUARD=1 for scripts/run_symx_gha.py. The guarded path is restricted to macOS Actions ota extract / ipsw extract jobs. It checks the workspace and temporary-directory filesystems before starting the worker, every one second, and at exit. Below 1 GiB free, it emits a DiskSpaceExhaustedError to Sentry and exits unsuccessfully. This is an emergency safety floor, not a prediction that an artifact will fit, and rapid writes can exhaust space between samples.

During a run the supervisor first freezes the worker's isolated process group and snapshots/freezes its owned descendants, including helpers in separate sessions, then kills them and reaps the worker. This deliberately does not let ordinary artifact-error handlers mark the item failed or continue to another item. There is no same-run retry or supervisor metadata mutation; the in-progress mirrored item remains eligible for a fresh worker without migration. Already completed storage writes are not rolled back.

The disk-specific event preserves the pre-intervention path/total/used/free samples, threshold, worker PID, GitHub run ID, and pipeline; recovery of disk space after process termination must not erase the diagnosis. Existing stdout/stderr still flows into symx.log. Unrelated external signals remain distinct and are not reclassified as disk exhaustion when free space is healthy.

Emergency termination does not guarantee detach of OS-managed mounts or cleanup of backing files: the disposable VM must be torn down rather than reused after an abort. No broad filesystem deletion or global process-name kill is performed. The guard is CI-only; local extract-file calls retain their existing lifecycle.

3.4 Variables, secrets, and credentials

From the workflow files, the important pieces are:

• repo variable SYMX_STORE – bucket URI used by Symx
• secret SENTRY_DSN – enables Sentry instrumentation
• implicit GITHUB_TOKEN – available inside GitHub Actions
• GCP auth via GitHub OIDC + Workload Identity, configured in the workflow files

For the authoritative details, read the workflow YAML files themselves.

4. Monitoring and debugging

4.1 GitHub Actions

GitHub Actions is the first place to answer:

• did the workflow trigger,
• which runner did it use,
• did it fail in bootstrap or in Symx itself,
• how long did it run,
• which run ID should I search for elsewhere.

Recommended commands

gh workflow list
gh run list --workflow "Sync IPSW meta-db" --limit 10
gh run list --workflow "Extract IPSW symbols" --status failure --limit 20
gh run view <run-id> --log | grep -n "Traceback" -C 3

What to look for

• bootstrap failures before Symx starts:
  • uv sync
  • ipsw install/download
  • symsorter setup
  • GCP auth / Cloud SDK setup
• resource issues:
  • disk pressure,
  • timeouts,
  • runner image problems
• Symx failures after the Running: ... python -m symx ... line from run_symx_gha.py

4.2 Admin clients and local snapshots

The native app and TUI are the easiest ways to inspect current rows without manually pulling JSON from GCS. They share the same cache and workflow-driven mutation path.

Typical workflow

1. open the native app from apps/SymxAdmin with swift run, or run uv run symx admin
2. let the client sync, or explicitly run uv run symx admin sync
3. inspect IPSW or OTA rows
4. build and apply a curated rerun queue when a retry is the right next step
5. use d in the TUI when an artifact must be downloaded for local extract-file reproduction

Default failure states shown by both clients

The default filter is:

• mirroring_failed
• mirror_corrupt
• symbol_extraction_failed
• indexed_invalid

The shared vocabulary still includes the manual/operator-only ignored state, but the current automation does not emit it and the admin defaults do not include it. Expected terminal skip states such as delta_ota, recovery_ota, and unsupported_ota_payload are also outside the default failure view.

Useful local paths

• snapshot manifest: ~/.cache/symx/admin/manifest.json
• active snapshots: ~/.cache/symx/admin/snapshots/
• downloaded artifacts: ~/.cache/symx/admin/downloads/
• GitHub run cache used by the TUI: ~/.cache/symx/admin/github_runs.json

The native app opens snapshot.db read-only. Both clients delegate sync and curated metadata mutations to the Python admin backend; shared metadata changes remain workflow-executed and generation-matched.

4.3 Sentry

Sentry is the live, per-run observability surface.

Transaction names / ops worth knowing

Top-level transactions include:

• ipsw.meta_sync
• ipsw.mirror
• ipsw.extract
• ipsw.extract_file
• ota.meta_sync
• ota.mirror
• ota.extract
• ota.extract_file
• sim.extract

Common child span ops include:

• http.download
• gcs.download
• gcs.upload
• gcs.upload_symbols
• subprocess.dyld_split
• subprocess.symsort
• subprocess.ipsw_extract
• ota.extract.materialize_dsc
• ota.extract.dsc_arch
• ota.extract.payload_probe

High-value tags

Global:

• github.run.id

IPSW-specific:

• ipsw.artifact.key
• ipsw.artifact.platform
• ipsw.artifact.version
• ipsw.artifact.build
• ipsw.artifact.source
• ipsw.version
• symsorter.version

OTA-specific:

• artifact.key
• artifact.platform
• artifact.version
• artifact.build
• artifact.url

Metrics worth watching

Representative counters/distributions/gauges emitted by the current code:

• ipsw.mirror.succeeded
• ipsw.mirror.failed
• ipsw.extract.succeeded
• ipsw.extract.failed
• ipsw.extract.mirror_corrupt
• ota.mirror.succeeded
• ota.mirror.failed
• ota.extract.succeeded
• ota.extract.failed
• ota.extract.process_terminated (platform and signal name; aborts without an artifact-state update)
• ota.extract.materialization.succeeded
• ota.extract.materialization.failed
• ota.extract.materialization.incomplete
• ota.extract.materialization.unavailable
• ota.extract.materialization.not_present
• ota.extract.materialized_dscs
• ota.extract.skipped_delta
• ota.extract.skipped_recovery
• ota.extract.skipped_unsupported_payload
• download.size_bytes
• download.duration_seconds
• gcs.upload.size_bytes
• gcs.download.size_bytes
• symbols.uploaded_new
• symbols.duplicates
• symbols.total_files
• disk.free_bytes
• disk.used_percent

Practical Sentry debugging loop

If you have a failed GitHub run:

1. get the run ID,
2. open the Sentry transaction tagged with github.run.id=<run-id>,
3. inspect the failing child span (http.download, gcs.download, subprocess.*, etc.),
4. use the artifact tags to identify the specific IPSW source or OTA artifact,
5. reproduce locally if needed.

4.4 Bucket-level escape hatches

When you cannot or do not want to authenticate directly to production GCS from your machine, there are helper workflows in .github/workflows/:

• bucket-ls.yml
• bucket-objects-list.yml
• bucket-query-meta.yml
• bucket-cp.yml
• bucket-rm.yml

These are useful for ad-hoc inspection of the bucket through GitHub Actions.

4.5 Common failure modes and first checks

mirroring_failed (IPSW and OTA)

Likely causes:

• Apple CDN download failure
• repeated transient download errors
• IPSW SHA-1/size verification mismatch

First checks:

• failing GitHub run log
• Sentry http.download span
• source URL and expected hash in admin snapshot

indexed_invalid (OTA)

Likely causes:

• OTA SHA-1 verification mismatch
• mirror/upload failure

First checks:

• failing GitHub run log
• Sentry OTA mirror transaction
• OTA URL/hash from the admin snapshot

mirror_corrupt (IPSW)

Likely causes:

• mirror object disappeared from GCS
• mirror object could be downloaded but failed verification

First checks:

• inspect mirror_path
• verify object existence via bucket workflow or GCS access
• check whether the source should be reset and mirrored again

OTA reset from mirrored back to indexed

Likely cause:

• load_ota() could not retrieve the mirrored zip from GCS during extraction

What Symx does:

• clears download_path
• resets state to indexed
• relies on a later mirror run to repopulate the mirror

OTA materializer signal termination (row remains mirrored)

OtaDscProcessTerminatedError means the direct materializer subprocess exited by a signal, not that its JSON schema or the OTA is invalid. The OTA worker reports the exception and aborts immediately, without same-run retry, symbol upload, or metadata update. last_run and last_modified remain unchanged: correlate this failure with Sentry and workflow logs rather than expecting a new persisted run attribution.

For SIGKILL, inspect runner memory/physical footprint, swap, and free disk before retrying; the signal alone is not proof of OOM. The CI watchdog reports its own low-disk interventions as DiskSpaceExhaustedError, rather than waiting for an unrelated-looking downstream failure. A fresh worker can select the still-mirrored row naturally, but a resource-bound artifact may fail repeatedly on the same runner size and block later work. Correct the resource constraint before repeated retries. This handling does not increase capacity or repair historical failed rows.

Symx unwinds its own temporary output/download scopes. A killed ipsw cannot be assumed to have cleaned its private mounts or files. On a local host, inspect owned mounts and detach them before removing backing files; do not delete unrelated temporary paths. Avoid continuing extraction on a potentially unhealthy runner.

symbol_extraction_failed

Likely causes:

• ipsw extraction failure, including an incomplete payload-extract report that may be transient or a cache-family dsc-validation failure on a full/unknown artifact
• dyld_split failure
• missing host Apple Archive compression support; check macOS as well as Xcode and ipsw versions
• symsorter failure
• unexpected artifact layout differences

Best next step:

• download the artifact locally via the admin TUI,
• run extract-file locally,
• compare local stdout/stderr and Sentry spans with the workflow run.

delta_ota / recovery_ota

These are usually expected skip states, not operator emergencies.

They mean the OTA does not contain a full DSC that Symx can process. If an extractor or classifier bug incorrectly created a delta_ota row, an operator can explicitly include that state in the admin filter and queue a curated extract retry, provided the mirrored artifact is still present. recovery_ota remains excluded from curated reruns.

unsupported_ota_payload

Existing rows are terminal skip states rather than operator emergencies. They record OTAs previously classified as referencing a DSC that the payloadv2 / Apple Archive tooling could not materialize or verify.

Classification uses trusted metadata rather than extraction-failure symptoms:

• recovery_ota comes from synced schema-1 provenance or release-type metadata. For historical rows and local files, Symx falls back to the canonical recovery fields in a ZIP or reconstructed AEA Info.plist.
• delta_ota comes from synced delivery=delta or prerequisite metadata. Historical rows and local files retain the typed plist and AEA-only PrereqBuild text fallbacks.
• A payload-extract failure and payload/BOM inventory are not enough to classify an OTA because transient failures can produce the same evidence. These failures remain symbol_extraction_failed and visible in the default view.
• If supported materialization remains unavailable, a report whose errors are exclusively dsc-validation can use the same trusted delta/recovery metadata; mixed validation and extraction errors are still failures, as are non-excluded validation failures on full or unknown artifacts. Errors positively attributed to excluded DriverKit or x86Support families do not prevent extracting usable System caches.
• CDN paths, image_patches/, and archive listings are never used to infer an artifact type.

Classification logs record which trusted source was used.

After a runner, macOS, ipsw, or AppleArchive tooling change, existing rows can be included in a curated admin extract rerun and reset to mirrored. They are outside the default failure view, so include unsupported_ota_payload in the admin state filter when reviewing them. The same explicit-filter requirement applies when retrying a misclassified delta_ota row.

5. Current recovery mechanisms

There is no free-form general-purpose state editor in production today.

The implemented recovery surface is intentionally narrower: curated admin rerun batches plus the workflow-driven migration tools below.

Admin curated reruns

Workflow: symx-admin-apply.yml

Current behavior:

• applies a curated batch reviewed from the local admin snapshot,
• validates the reviewed snapshot generation against the current remote generation,
• updates eligible rows back to indexed or mirrored depending on the selected action,
• dispatches the corresponding worker workflow when possible.

This is intentionally narrower than arbitrary state editing.

IPSW

Workflow: symx-ipsw-migrate.yml

Current behavior:

• runs symx ipsw migrate
• resets a hard-coded list of IPSW sources in symbol_extraction_failed back to mirrored

This is intentionally narrow and code-driven.

OTA

Workflow: symx-ota-migrate-storage.yml

Current behavior:

• runs symx ota migrate-storage
• resets all OTA artifacts in symbol_extraction_failed back to mirrored

This is broader than the IPSW path, but still workflow-driven rather than interactive.

6. Before shipping code changes

The repo convention is to use uv for everything and to run the full check suite:

uv run ruff check --fix
uv run ruff format
uv run ty check
uv run pytest