Skip to content

Surface AWF startup failures in firewall summaries - #8023

Open
lpcox with Copilot wants to merge 4 commits into
mainfrom
copilot/awf-codex-engine-fix
Open

Surface AWF startup failures in firewall summaries#8023
lpcox with Copilot wants to merge 4 commits into
mainfrom
copilot/awf-codex-engine-fix

Conversation

Copilot AI commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Codex workflows can fail before the agent starts, leaving no Squid access.log and making awf logs summary report only “no log sources found.” This adds a preserved startup diagnostic path for pre-egress failures.

  • Startup diagnostics

    • Write a redacted awf-startup-error.json into the proxy logs directory when AWF aborts during startup.
    • Capture timestamp, phase, and failure message.
  • Log discovery

    • Treat directories containing awf-startup-error.json as valid log sources, even without access.log.
    • Supports both AWF_LOGS_DIR and preserved /tmp/squid-logs-* discovery.
  • Summary output

    • Include startup diagnostics in JSON, markdown, and pretty summaries so failed startup runs are diagnosable from artifacts.

Example diagnostic surfaced by awf logs summary:

{
  "startupDiagnostics": [
    {
      "timestamp": "2026-09-02T12:45:57.000Z",
      "phase": "startup",
      "message": "Refusing to use symlink as bind mountpoint: /usr/local/bin/npm"
    }
  ]
}

Copilot AI changed the title [WIP] Fix codex engine exit issue in AWF sandbox Surface AWF startup failures in firewall summaries Sep 2, 2026
Copilot AI requested a review from lpcox September 2, 2026 17:23
@lpcox
lpcox marked this pull request as ready for review September 2, 2026 19:50
Copilot AI balanced review requested due to automatic review settings September 2, 2026 19:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Diagnostic persistence currently has symlink-safety, sensitive-data, lifecycle, coverage, and phase-classification defects.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds persisted, redacted startup-failure diagnostics to firewall summaries when Squid produces no access log.

Changes:

  • Records startup failures in awf-startup-error.json.
  • Discovers and aggregates diagnostic-only log sources.
  • Displays diagnostics in JSON, Markdown, and terminal summaries.
File summaries
File Description
src/types/logging.ts Defines startup diagnostic data.
src/types/index.ts Exports the diagnostic type.
src/logs/startup-diagnostics.ts Locates and reads diagnostic files.
src/logs/log-discovery.ts Discovers diagnostic-only sources.
src/logs/log-discovery.test.ts Tests diagnostic discovery.
src/logs/log-aggregator.ts Adds diagnostics to aggregated statistics.
src/logs/log-aggregator.test.ts Tests diagnostic aggregation.
src/logs/stats-formatter.ts Formats diagnostics in all outputs.
src/logs/stats-formatter.test.ts Tests diagnostic formatting.
src/commands/main-action.ts Writes diagnostics on failures.
src/commands/main-action.test.ts Tests writing and basic redaction.
Review details

Suppressed comments (4)

src/commands/main-action.ts:111

  • redactSecrets() does not redact secret-derived API target hosts from config.sensitiveAllowedDomains, so an exception mentioning a private base URL can leak it into this world-readable diagnostic artifact. Apply deriveSensitiveEndpointForms(config.sensitiveAllowedDomains) and redactSensitiveValues() here as config-writer.ts:413-426 does for other uploaded audit artifacts.
        message: redactSecrets(message),

src/commands/main-action.ts:107

  • The marker is only written on failure and is never cleared. If --proxy-logs-dir is reused, a later successful run retains the previous awf-startup-error.json; loadAndAggregate() then unconditionally reports that stale failure alongside the new run's access log. Remove the marker safely at the start of each run or associate diagnostics with a per-run directory/identifier.
    fs.writeFileSync(
      getStartupDiagnosticPath(proxyLogsDir),

src/commands/main-action.ts:438

  • This catch covers the entire workflow, including runAgentCommand() and cleanup after onContainersStarted has fired. Exceptions after the agent starts are therefore persisted with phase startup, and markdown incorrectly says they occurred before Squid emitted access.log. Track the current phase or only write this marker from the pre-agent startup paths.
    writeStartupFailureDiagnostic(config, error);

src/commands/main-action.ts:345

  • The diagnostic catch begins only after runDindBootstrap(config), which can reject during Docker staging or path validation. Those are startup aborts but still leave no marker, so the new summary path does not cover all promised pre-agent failures. Wrap the post-validation bootstrap sequence in the same diagnostic handling or move it inside this guarded startup flow.
    writeStartupFailureDiagnostic(config, error);
  • Files reviewed: 11/11 changed files
  • Comments generated: 1
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/commands/main-action.ts Outdated
Comment on lines +103 to +107
const proxyLogsDir = config.proxyLogsDir || path.join(config.workDir, 'squid-logs');
fs.mkdirSync(proxyLogsDir, { recursive: true, mode: 0o755 });
const message = error instanceof Error ? error.message : String(error);
fs.writeFileSync(
getStartupDiagnosticPath(proxyLogsDir),
@lpcox

lpcox commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

@copilot address review feedback

Copilot AI commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

@copilot address review feedback

Addressed the review feedback in 1b3a283: startup diagnostic writes now validate the log directory, refuse symlink/non-regular marker targets, redact sensitive endpoint forms, and only record failures before the agent command begins. Added focused regression coverage.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

✅ Copilot review passed with no inline comments.

@copilot Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

🔑 BYOK report filed by Smoke Copilot BYOK

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Copilot

@lpcox
lpcox deployed to aoai-model September 2, 2026 23:54 — with GitHub Actions Active
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Smoke Claude passed

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by Smoke Claude for #8023

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

Warning

Firewall blocked 7 domains

The following domains were blocked by the firewall during workflow execution:

  • accounts.google.com
  • android.clients.google.com
  • clients2.google.com
  • contentautofill.googleapis.com
  • msfeed25.pkgs.visualstudio.com
  • www.google.com
  • www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "msfeed25.pkgs.visualstudio.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Docker Sbx has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Docker Sbx

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

🚀 Security Guard has started processing this pull request

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Smoke Gemini reports failed. Facets need polishing...

💎 Faceted by Smoke Gemini

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Build Test Suite completed successfully!

Generated by Build Test Suite for #8023

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

📡 Smoke OTel Tracing reports failed. OTel tracing regression detected. ⚠️

OTEL Tracing Smoke Test — all scenarios passed:
✅ S1 Module Loading: otel.js loaded, exports startRequestSpan/setTokenAttributes/setBudgetAttributes/endSpan/endSpanError/shutdown/isEnabled + internals.
✅ S2 Test Suite: 3 suites (otel.test.js, otel-fanout.test.js, otel-workload-identity.test.js), 68/68 tests passed.
✅ S3 Env Var Forwarding: env-passthrough.ts forwards GITHUB_AW_OTEL_TRACE_ID/PARENT_SPAN_ID/COPILOT_OTEL_FILE_EXPORTER_PATH to agent; api-proxy-env-config.ts buildOtelEnv() forwards OTLP endpoint/headers/service name/trace context to api-proxy.
✅ S4 Token Tracker Integration: onUsage callback present in token-tracker-http.js (used as OTEL hook, sets budget attrs from normalized usage).
✅ S5 OTEL Diagnostics: no spans exported in this run (expected — no live LLM traffic proxied during smoke test); no errors, graceful no-op confirmed.
No unexpected failures found; not a PR-triggered event so no comment added.

📡 OTel tracing validated by Smoke OTel Tracing

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

Tested by Smoke Chroot

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Contribution Check failed. Please review the logs for details.

Generated by Contribution Check for #8023

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

🔌 Service connectivity validated by Smoke Services

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: Claude Engine Validation

Check Status
API ✅ PASS
gh CLI ✅ PASS
File ✅ PASS

Overall result: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by Smoke Claude for #8023 · claude · haiku45 · 58.3 AIC · ⊞ 4.5K ·
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot BYOK (Direct) Mode ✅

Test Result
GitHub MCP ✅ (PRs #8026, #8021 fetched)
GitHub.com HTTP ✅ (HTTP 200)
File Write/Read ✅ (/tmp/gh-aw/agent/smoke-test-copilot-byok.txt)
BYOK Inference ✅ (Direct mode via api-proxy → api.githubcopilot.com)

Status: PASS — Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY)

🔑 BYOK report filed by Smoke Copilot BYOK
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot Engine@lpcox

Overall: PASS

📰 BREAKING: Report filed by Smoke Copilot
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

EGRESS_RESULT allow=pass deny=pass

✅ Allowed domain (api.github.com) reachable — HTTP 200
✅ Non-allowed domain (example.com) blocked — CONNECT tunnel 403

Overall status: PASS

cc @lpcox

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • example.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation
Add label ready-for-aw to run again

@github-actions github-actions Bot added the smoke-copilot-network-isolation Copilot network-isolation egress smoke test label Sep 2, 2026
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: Docker Sbx@lpcox

Overall: PASS

📰 BREAKING: Report filed by Smoke Docker Sbx
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: GitHub Actions Services Connectivity

  • Redis PING: ❌ (DNS resolution failure for host.docker.internal)
  • PostgreSQL pg_isready: ❌ (no response)
  • PostgreSQL SELECT 1: ❌ (DNS resolution failure)

Overall: FAIL

REDIS=Could not connect to Redis at host.docker.internal:6379: Temporary failure in name resolution
PG_READY=host.docker.internal:5432 - no response
PG_QUERY=psql: error: could not translate host name "host.docker.internal" to address: Temporary failure in name resolution

🔌 Service connectivity validated by Smoke Services
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Chroot Version Comparison Results

Runtime Host Version Chroot Version Match?
Python Python 3.12.14 Python 3.12.14 ✅ YES
Node.js v24.19.0 v2.98.0 ❌ NO
Go go1.22.12 go1.22.12 ✅ YES

⚠️ Not all tests passed — Node.js version mismatch detected between host and chroot environments. smoke-chroot label not applied.

Tested by Smoke Chroot
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

@lpcox @Copilot

  • test: validate shared-gateway enclave smoke; Mount the configured container working directory when no other mount exposes it: ❌
  • GitHub.com connectivity: ✅
  • File write/read test: ✅
  • BYOK inference test: ✅
    Overall: FAIL
    Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw)

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Merged PRs: #8026 test: validate shared-gateway enclave smoke | #8021 Mount the configured container working directory when no other mount exposes it
Listed PRs: #8038 Fix Copilot proxy auth and GHEC model discovery | #8025 test: organize Cloud Hypervisor manager tests by feature
GitHub merged-PR review ✅
safeinputs-gh query ❌
Playwright title ✅
File write/read ✅
Discussion comment ❌
Build AWF ✅
Overall: FAIL

Warning

Firewall blocked 7 domains

The following domains were blocked by the firewall during workflow execution:

  • accounts.google.com
  • android.clients.google.com
  • clients2.google.com
  • contentautofill.googleapis.com
  • msfeed25.pkgs.visualstudio.com
  • www.google.com
  • www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "msfeed25.pkgs.visualstudio.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

test: validate shared-gateway enclave smoke: ✅
Mount the configured container working directory when no other mount exposes it: ✅
GitHub.com connectivity: ✅
File write/read: ✅
Direct BYOK inference: ✅
Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra
Overall: PASS
Author: @lpcox; Assignees: none

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia 1/1 passed ✅ PASS
Bun hono 1/1 passed ✅ PASS
C++ fmt N/A ✅ PASS
C++ json N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world N/A ✅ PASS
.NET json-parse N/A ✅ PASS
Go color ok ✅ PASS
Go env ok ✅ PASS
Go uuid ok ✅ PASS
Java gson 1/1 passed ✅ PASS
Java caffeine 1/1 passed ✅ PASS
Node.js clsx passed ✅ PASS
Node.js execa passed ✅ PASS
Node.js p-limit passed ✅ PASS
Rust fd 1/1 passed ✅ PASS
Rust zoxide 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — PASS

Notes:

  • Java: initial mvn run failed because the default ~/.m2/repository directory (owned by root) was not writable; worked around by pointing <localRepository> to a writable path (/tmp/m2repo). Proxy settings themselves were correct.
  • All other ecosystems built/tested successfully on the first attempt.

Generated by Build Test Suite for #8023 · copilot · auto · 46.2 AIC · ⊞ 12K ·
Add label ready-for-aw to run again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[awf] codex engine exits immediately with no output under AWF sandbox (transient failure)

3 participants