narwhal-aicode-risks

A field guide to security incidents caused by AI-generated code.

English | 中文 ·

This repository documents the risks of AI-generated code through layered evidence buckets:

• Technical Report — English PDF · 中文 PDF · arXiv
• cases/ — 143 verified real-world incidents with primary sources, evidence archives, and bilingual analysis.
• inferred/ — partial-evidence cases: event appears real, but key facts (vendor advisory / CVE / postmortem) not yet pinned down. (0 cases as of v1.0; submit one!)
• scenarios/ — illustrative scenarios for a real risk pattern, not tied to a confirmed event.
• Risk Taxonomy — 7 categories spanning supply chain, code-level vulnerabilities, cloud / IaC, agent risks, domain-specific risks, IP & compliance, and human factors. See docs/taxonomy.md.

Companion repository (defenses): narwhal-aicode-guardrails (coming soon) — defenses, evaluation benchmarks, and best practices for securing AI-generated code.

Verification policy. Every case has been independently fact-checked against primary sources. Each meta.yaml records severity_basis (cvss / quantifiable-impact / editorial) and verification_notes. See Verification Status below.

---

At a Glance

143 cases · 7 active categories · 2022 → 2026 · 345+ AI tools implicated · 90 cases anchored to public CVEs (CVSS 4 / 5 / 6 / 7 / 8 / 9 / 10)

---

Hall of Shame — Top 5 by Verified Real-World Impact

#1  Lovable AI Platform Data Exposure  (2026) 18,697 records leaked  ·  5,000+ vibe-coded apps unprotected Missing Supabase Row-Level Security + inverted-auth RPC on a Lovable-built EdTech app — then a platform-wide regression made public-project chats and source code accessible to any logged-in user for 77 days. RedAccess later found 380,000 public assets across the vibe-coding ecosystem (Lovable / Replit / Base44 / Netlify), ~5,000 with sensitive data. Read case →
#2  Moonwell cbETH Oracle Misconfig  (2026) $1,779,044.83  bad debt PR #578 (Claude-coauthored, Copilot-reviewed, 28 checks passed) shipped a cbETH oracle config that returned the cbETH/ETH ratio as a USD price — about $1.12 instead of ~$2,200. Liquidators drained 1,096.317 cbETH within minutes. Official Moonwell postmortem + BlockSec + Cointelegraph + rekt.news. Read case →	#3  n8n Path Traversal  (2025) Public PoC  ·  CWE-22 Introducing commit ff958e4 carries Co-Authored-By: Claude <noreply@anthropic.com> and ships os.path.join("workflows", filename) straight to a download endpoint — ..%5c walks out of the directory. NVD-confirmed; later picked up by Georgia Tech's Vibe Security Radar and The Register. Read case →
#4  EchoLeak — M365 Copilot  (2025) First zero-click AI-agent CVE A single crafted email — no click required — coerces M365 Copilot's RAG into emitting a markdown image URL that exfiltrates the most sensitive context it can find. Coined "LLM Scope Violation" by Aim Labs (now Cato Networks). Microsoft patched, MSRC scored 9.3. Read case →	#5  Moltbook RLS Data Exposure  (2026) 1.5M API tokens · 35K emails · 4,060 DMs Vibe-coded AI-agent social network whose founder publicly stated he doesn't write code. 88:1 agent-to-human ratio + Supabase RLS off + OpenClaw's optimistic-trust defaults = entire DB walkable from a public anon key. Karpathy's reaction shifted from "sci-fi takeoff" to "dumpster fire" within days. Read case →

---

Case Atlas

Hallucination & Supply Chain  (4) AI hallucinates non-existent dependencies, recommends squatted packages, or contaminates install chains. huggingface-cli · 2024 · react-codeshift agent self-spread · 2026 · InstallFix + Bing OpenClaw spoof · 2026 · LiteLLM PyPI supply chain poisoning · 2026 ·	Code-Level Vulnerabilities  (2) Snippet-level defects in AI-generated code: unsafe API use, missing validation, CVE pattern reintroduction. n8n CVE-2025-55526 · 2025 · Mass CVE reintroduction · 2026 ·	Agent Risks  (1) Prompt injection, tool-call hijacking, architectural biases that surface as security flaws. EchoLeak (M365 Copilot) · 2025 ·
Domain-Specific Risks  (2) Risks unique to specific domains: smart contracts, AI app platforms, no-code generators. Lovable platform data exposure · 2026 · Moonwell cbETH oracle ($1.78M) · 2026 ·	IP, License & Compliance  (2) Copyright lawsuits, license contamination, training-data IP disputes. Doe 1 v. GitHub — filing · 2022 · Doe 1 v. GitHub — 2026 update · 2026 ·	Human Factors  (2) Skill erosion, over-reliance on AI, security-culture decay in AI-assisted teams. Samsung ChatGPT data leak · 2023 · Moltbook RLS exposure · 2026 ·
Cloud & IaC Misconfiguration  (1) Cloud and infrastructure-as-code defaults or generated scripts that expose data or misconfigure access. AI-generated IaC public S3 bucket leak · 2025 · [Scenario] AI Terraform → S3 Public-Read ·	Submit a case  (+) Spotted an incident we should document? See CONTRIBUTING.md for the bilingual case template, the meta.yaml field reference, and verification expectations. We accept real incidents and labelled scenarios.	Auto-generated index The full table grouped by category is auto-rendered from meta.yaml files. → cases/README.md Run python3 scripts/render_index.py to regenerate after adding a case.

---

Verification Status

A 2026-05 verification pass against primary sources reshaped the library:

Verdict	Count	Notes
✅ Real, well-sourced	14	All cases/* after this round
📘 Illustrative scenario (downgraded from "case")	1	scenarios/2025-iac-s3-bucket-leak/ — phenomenon real, specific incident not corroborated; cited primary sources returned 404
🔁 Merged duplicates	-1	The earlier 2026-agent-architecture-bias-db covered the same Moltbook incident as 2026-security-culture-erosion; merged into the latter

Each case's meta.yaml carries a severity_basis field with one of three values:

• cvss — anchored to a public CVE / NVD / vendor advisory (3 cases: n8n, Lovable, EchoLeak)
• quantifiable-impact — anchored to a publicly disclosed loss / scope figure that we can cite (8 cases as of this index)
• editorial — judgment based on the nature of the risk where no CVSS scale applies (3 cases as of this index)

verification_notes in each meta.yaml records what was checked and what was changed during verification.

---

Reproduce

Cases marked reproducible: true in meta.yaml ship a code/ directory with the relevant PoC. As of now: 4 cases (n8n path traversal, hallucinated package, react-codeshift agent self-spread, Moltbook RLS exposure).

git clone https://github.com/Narwhal-Lab/narwhal-aicode-risks.git
cd narwhal-aicode-risks/cases/2025-n8n-path-traversal-cve/code
# follow the case README for run instructions

---

Contributing

We welcome new case submissions. There are two paths:

• Easy — Open the 📝 Submit a case Issue Form. No git/markdown needed; a maintainer will fact-check, convert to a draft PR, and credit you. SLA: 14 days.
• PR — Copy cases/_template/ (or inferred/_template/ / scenarios/_template/), fill meta.yaml and the bilingual README.md, and open a PR. See CONTRIBUTING.md for the field reference, verification policy, and PR checklist.

Local validation before opening a PR:

pip install pyyaml
python3 scripts/validate_cases.py             # schema check
python3 scripts/validate_cases.py --check-links   # also HEAD all reference URLs
python3 scripts/render_index.py               # regenerate index + SVGs

Every PR runs the same checks via GitHub Actions. See .github/workflows/validate.yml.

---

Key Findings (from the Technical Report)

• Phased Penetration — AI-generated code has moved through Explosive Exploration → Rational Regression → Stable Collaboration, settling on high-repeatability tasks (tests, docs, boilerplate).
• Language-Stack Asymmetry — High penetration in Python / JavaScript / TypeScript; cautious adoption in Rust / C++ and other systems languages.
• Dual Role in the Vulnerability Lifecycle — AI is observed as both a source of vulnerabilities (later reverted to human implementations during fixes) and an accelerator of remediation.
• Patterned Risk Profile — AI-introduced defects concentrate in input validation, unsafe API calls, and outdated cryptography; severity distribution mirrors human code, but network-exposed surfaces are over-represented.
• Three-in-One Mitigation Framework — Multi-dimensional evaluation benchmarks + intrinsic model security + human-machine collaborative governance, with developers retaining ultimate accountability.

The full technical report is at docs/report-en.pdf (English) / docs/report-cn.pdf (Chinese).

---

Citation

To cite this case library and technical report:

@techreport{narwhal2025_aicode_risks,
  title        = {AI-Generated Code in the Wild: Security Risk Study},
  author       = {Tencent Security Platform Department and Narwhal-Lab},
  year         = {2025},
  institution  = {Tencent Security Platform Department, Narwhal-Lab},
  type         = {Technical Report},
  url          = {https://github.com/Narwhal-Lab/narwhal-aicode-risks}
}

To cite the related arXiv paper:

@misc{wang2025aicodewildmeasuring,
  title={AI Code in the Wild: Measuring Security Risks and Ecosystem Shifts of AI-Generated Code in Modern Software},
  author={Bin Wang and Wenjie Yu and Yilu Zhong and Hao Yu and Keke Lian and Chaohua Lu and Hongfang Zheng and Dong Zhang and Hui Li},
  year={2025},
  eprint={2512.18567},
  archivePrefix={arXiv},
  primaryClass={cs.SE},
  url={https://arxiv.org/abs/2512.18567}
}

A CITATION.cff is also provided for GitHub's "Cite this repository" button.

---

Contributors

Thanks goes to these wonderful people. We use all-contributors — see CONTRIBUTING.md for how to get added.

Bin Wang 📆 🔬 📝 👀	Wenjie Yu 🔬 📝	Yilu Zhong 🔬 📝	Hao Yu 🔬 📝	Jiazheng Quan 📝 👀	Jianing Zhou 📝	Liangliang Qian 📝
Like Liu 📝	Zherong Zhang 📝

Contribution emoji key: 📝 Content · 🔬 Research · 👀 Review · 📆 Project management · 🛠 Infrastructure · 🌍 Translation · 🐛 Bug report

---

Team

Narwhal Lab

Bin Wang

Wenjie Yu

Yilu Zhong

Hao Yu

Jiazheng Quan

Like Liu

---

Acknowledgments and Feedback

Thanks to all contributors and reviewers. If you have suggestions, spot an error, or want to share field experience, open an Issue. For collaboration inquiries, email thebinking66@gmail.com.

License

© 2025 Narwhal Lab. Licensed under CC BY 4.0 — attribution to Narwhal Lab is required.