diff --git a/README-en.md b/README-en.md index 686804058..ac361c3cf 100644 --- a/README-en.md +++ b/README-en.md @@ -9,7 +9,7 @@

- Spring AI Alibaba + Spring AI Alibaba Spring Boot Java License @@ -46,7 +46,7 @@ Additionally, this project natively supports **MCP (Model Context Protocol)**, e | Feature | Description | | :--- | :--- | | **Intelligent Data Analysis** | StateGraph-based Text-to-SQL conversion, supporting complex multi-table queries and multi-turn conversation intent understanding. | -| **Python Deep Analysis** | Built-in Docker/Local Python executor, automatically generating and executing Python code for statistical analysis and machine learning predictions. | +| **Python Deep Analysis** | Executes generated code in task-scoped Spring AI Alibaba Sandbox containers with PEP 723 dynamic dependencies, resource limits, retries, and automatic cleanup. | | **Intelligent Report Generation** | Analysis results are automatically summarized into HTML/Markdown reports with ECharts visualizations, WYSIWYG. | | **Human Feedback Mechanism** | Innovative Human-in-the-loop mechanism, supporting user intervention and adjustments during the plan generation phase. | | **RAG Retrieval Enhancement** | Integrated vector database, supporting semantic retrieval of business metadata and terminology libraries to improve SQL generation accuracy. | @@ -61,12 +61,14 @@ Additionally, this project natively supports **MCP (Model Context Protocol)**, e ## Quick Start -> For detailed installation and configuration guide, please refer to [Quick Start Guide](docs/QUICK_START.md). +> For detailed installation and configuration guidance, see the [Quick Start Guide](docs/QUICK_START-en.md). ### 1. Prerequisites - JDK 17+ - MySQL 5.7+ -- Node.js 16+ +- Node.js 22+ +- pnpm 11+ +- Docker (required when a workflow executes Python steps) ### 2. Start Services @@ -75,12 +77,11 @@ Additionally, this project natively supports **MCP (Model Context Protocol)**, e mysql -u root -p < data-agent-management/src/main/resources/sql/schema.sql # 2. Start backend -cd data-agent-management -./mvnw spring-boot:run +./mvnw -pl data-agent-management spring-boot:run -# 3. Start frontend -cd data-agent-frontend -npm install && npm run dev +# 3. Start the frontend in another terminal +cd data-agent-frontend-nuxt +pnpm install && pnpm dev ``` ### 3. Access the System @@ -90,11 +91,11 @@ Open your browser and visit `http://localhost:3000` to start creating your first | Document | Contents | | :--- | :--- | -| [Quick Start](docs/QUICK_START.md) | Environment requirements, database import, basic configuration, getting started | -| [Architecture Design](docs/ARCHITECTURE.md) | System layered architecture, StateGraph and workflow design, core module sequence diagrams | -| [Developer Guide](docs/DEVELOPER_GUIDE.md) | Development environment setup, detailed configuration manual, coding standards, extension development (vector DB/models) | -| [Advanced Features](docs/ADVANCED_FEATURES.md) | API Key invocation, MCP server configuration, custom hybrid retrieval strategies, Python executor configuration | -| [Knowledge Configuration Best Practices](docs/KNOWLEDGE_USAGE.md) | Explanation and usage of semantic models, business knowledge, and agent knowledge | +| [Quick Start](docs/QUICK_START-en.md) | Environment requirements, database import, basic configuration, getting started | +| [Architecture Design](docs/ARCHITECTURE-en.md) | System layered architecture, StateGraph and workflow design, core module sequence diagrams | +| [Developer Guide](docs/DEVELOPER_GUIDE-en.md) | Development environment setup, detailed configuration manual, coding standards, extension development (vector DB/models) | +| [Advanced Features](docs/ADVANCED_FEATURES-en.md) | API Key invocation, MCP server configuration, custom hybrid retrieval strategies, SAA Python sandbox, and dynamic dependencies | +| [Knowledge Configuration Best Practices](docs/KNOWLEDGE_USAGE-en.md) | Explanation and usage of semantic models, business knowledge, and agent knowledge | ## Community & Contribution diff --git a/README.md b/README.md index fbdab0591..99ba56734 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@

- Spring AI Alibaba + Spring AI Alibaba Spring Boot Java License @@ -46,7 +46,7 @@ | 特性 | 说明 | | :--- | :--- | | **智能数据分析** | 基于 StateGraph 的 Text-to-SQL 转换，支持复杂的多表查询和多轮对话意图理解。 | -| **Python 深度分析** | 内置 Docker/Local Python 执行器，自动生成并执行 Python 代码进行统计分析与机器学习预测。 | +| **Python 深度分析** | 基于 Spring AI Alibaba Sandbox 在任务级容器中执行生成代码，支持 PEP 723 动态依赖、资源限制、失败重试与自动清理。 | | **智能报告生成** | 分析结果自动汇总为包含 ECharts 图表的 HTML/Markdown 报告，所见即所得。 | | **人工反馈机制** | 独创的 Human-in-the-loop 机制，支持用户在计划生成阶段进行干预和调整。 | | **RAG 检索增强** | 集成向量数据库，支持对业务元数据、术语库的语义检索，提升 SQL生成准确率。 | @@ -66,7 +66,9 @@ ### 1. 准备环境 - JDK 17+ - MySQL 5.7+ -- Node.js 16+ +- Node.js 22+ +- pnpm 11+ +- Docker（工作流需要执行 Python 步骤时必需） ### 2. 启动服务 @@ -75,12 +77,11 @@ mysql -u root -p < data-agent-management/src/main/resources/sql/schema.sql # 2. 启动后端 -cd data-agent-management -./mvnw spring-boot:run +./mvnw -pl data-agent-management spring-boot:run -# 3. 启动前端 -cd data-agent-frontend -npm install && npm run dev +# 3. 在另一个终端启动前端 +cd data-agent-frontend-nuxt +pnpm install && pnpm dev ``` ### 3. 访问系统 @@ -93,7 +94,7 @@ npm install && npm run dev | [快速开始](docs/QUICK_START.md) | 环境要求、数据库导入、基础配置、系统初体验 | | [架构设计](docs/ARCHITECTURE.md) | 系统分层架构、StateGraph与工作流设计、核心模块时序图 | | [开发者指南](docs/DEVELOPER_GUIDE.md) | 开发环境搭建、详细配置手册、代码规范、扩展开发(向量库/模型) | -| [高级功能](docs/ADVANCED_FEATURES.md) | API Key 调用、MCP 服务器配置、自定义混合检索策略、Python执行器配置 | +| [高级功能](docs/ADVANCED_FEATURES.md) | API Key 调用、MCP 服务器配置、自定义混合检索策略、SAA Python 沙盒与动态依赖 | | [知识配置最佳实践](docs/KNOWLEDGE_USAGE.md) | 语义模型，业务知识，智能体知识的解释和使用 | ## 🤝 加入社区 & 贡献 diff --git a/docs/ADVANCED_FEATURES-en.md b/docs/ADVANCED_FEATURES-en.md index 161c3cf98..384da7f57 100644 --- a/docs/ADVANCED_FEATURES-en.md +++ b/docs/ADVANCED_FEATURES-en.md @@ -386,7 +386,17 @@ WHERE u.name = 'Zhang San' Python always runs through Spring AI Alibaba Sandbox in a task-scoped container. Host-local, legacy Docker-pool, and AI Simulation executors are no longer available. -### SAA Sandbox Configuration +### Prerequisites + +- Spring AI Alibaba `1.1.2.2` +- A reachable Docker daemon +- The sandbox image available to that Docker daemon +- Network access from the sandbox to the configured Python package index when installing dependencies + +SQL-only requests do not create a sandbox. Docker is required only when the execution plan contains +`PYTHON_GENERATE_NODE`. + +### Configure the SAA Sandbox ```yaml spring: @@ -397,25 +407,100 @@ spring: code-timeout: 60s limit-memory: 500 cpu-core: 1 + python-max-tries-count: 5 sandbox: - image-name: ${DATAAGENT_SANDBOX_IMAGE} + docker-host: ${DATAAGENT_SANDBOX_DOCKER_HOST:unix:///var/run/docker.sock} + image-name: ${DATAAGENT_SANDBOX_IMAGE:agentscope-registry.ap-southeast-1.cr.aliyuncs.com/agentscope/runtime-sandbox-base:latest} max-concurrency: 4 queue-capacity: 10 + max-connections: 4096 package-index-url: ${DATAAGENT_PYPI_INDEX_URL:https://pypi.org/simple} dependency-install-timeout: 3m ``` -### Dynamic Dependencies +When `docker-host` is `tcp://host:port`, DataAgent explicitly connects to that endpoint. Other +values use SAA/docker-java local Docker discovery. See +[Developer Guide - Code Executor Configuration](DEVELOPER_GUIDE-en.md#5-code-executor-configuration) +for all size, output, and metadata limits. + +### Declare Dynamic Dependencies ```python # /// script -# dependencies = ["pandas>=2,<3"] +# requires-python = ">=3.10" +# dependencies = ["pandas>=2,<3", "six==1.17.0"] # /// + +import pandas as pd +import six +``` + +A script may contain at most one PEP 723 `script` block. `dependencies` supports PyPI package +names, extras, and version constraints. URLs, VCS references, local paths, environment markers, +`@` references, and pip options are rejected before a container is created. The current version +parses and preserves `requires-python`, but does not use it to switch or reject the runtime Python +version. + +Every non-standard-library import should have a matching dependency. Scripts that use only the +standard library may omit the metadata block. + +### Execution Lifecycle + +```mermaid +sequenceDiagram + participant W as PythonExecuteNode + participant E as PythonCodeExecutorService + participant S as Task BaseSandbox + participant I as Fixed Installer Bootstrap + participant C as Fixed Code Bootstrap + + W->>E: code + stdin JSON + validated dependencies + E->>S: create isolated sandbox + opt dependencies are declared + E->>I: install into /tmp/dataagent-deps + I-->>E: bounded result envelope + end + E->>C: execute code with PYTHONPATH and stdin + C-->>E: bounded stdout/stderr envelope + E-->>W: TaskResponse + E->>S: close and remove container ``` -Dependencies are installed inside the task sandbox by a fixed backend command. The model cannot -specify repositories, URLs, file paths, pip options, or shell commands. Production should use an -enterprise private PyPI proxy. +The installer invokes `python -m pip install` with a fixed argument list and never uses a shell. +Code, stdin, the dependency list, and the package index are Base64 encoded before entering the +bootstrap. The dependency directory exists only in the current task container. + +### Verification + +Submit an analysis request that explicitly requires Python and a third-party dependency. Success +means: + +1. The timeline contains Python generation, execution, analysis, and a final report. +2. Python stdout contains the dependency version and business JSON. +3. Backend logs show sandbox creation, dependency installation, code execution, and container removal. +4. `docker ps --format '{{.Names}}' | grep '^dataagent-sandbox-'` prints nothing. + +### Troubleshooting + +| Symptom | Cause and action | +|---|---| +| Docker connection fails | Run `docker info`; use `tcp://host:port` for a remote daemon and default discovery for local Docker | +| `DependencyInstallError` | Check the package name, version, and index reachability; increase `dependency-install-timeout` or use a private caching proxy for large packages | +| `Unsupported Python dependency specifier` | Remove URLs, VCS references, paths, markers, and pip options; keep only a package name and version constraints | +| `Python sandbox capacity is exhausted` | Concurrency and the wait queue are full; reduce upstream concurrency or adjust `max-concurrency` and `queue-capacity` | +| `Python sandbox task timed out` | Total time exceeded “install timeout + code timeout + 30 seconds”; inspect package download and code duration | +| A container remains after a task | Check whether the application was force-killed; normal success and failure paths close the task sandbox, and application shutdown performs global cleanup | + +### Production Security Boundary + +Task containers are non-privileged and have CPU, memory, nofile, timeout, concurrency, input, and +output limits. SAA `1.1.2.2` does not expose an API to disconnect the container immediately after +dependency installation, so production must also: + +1. Use a pinned image digest and a non-root base image. +2. Allow sandbox access only to an enterprise PyPI proxy; block public internet and business networks. +3. Never inject database, model, OSS, or Docker credentials into the sandbox. +4. Scan packages for malware, CVEs, and license policy at the proxy. ## Advanced Configuration Options diff --git a/docs/ADVANCED_FEATURES.md b/docs/ADVANCED_FEATURES.md index 2daab80de..ba43285ee 100644 --- a/docs/ADVANCED_FEATURES.md +++ b/docs/ADVANCED_FEATURES.md @@ -385,7 +385,16 @@ WHERE u.name = '张三' Python 统一通过 Spring AI Alibaba Sandbox 在任务级独立容器中执行，不再提供宿主机 Local、旧 Docker 容器池或 AI Simulation 执行器。 -### SAA 沙盒配置 +### 前置条件 + +- Spring AI Alibaba `1.1.2.2` +- 可访问的 Docker daemon +- 沙盒镜像可从 Docker daemon 获取 +- 动态安装依赖时，沙盒网络可访问配置的 Python 包索引 + +仅执行 SQL 的请求不会创建沙盒；执行计划包含 `PYTHON_GENERATE_NODE` 时才需要 Docker。 + +### 配置 SAA 沙盒 ```yaml spring: @@ -396,24 +405,93 @@ spring: code-timeout: 60s limit-memory: 500 cpu-core: 1 + python-max-tries-count: 5 sandbox: - image-name: ${DATAAGENT_SANDBOX_IMAGE} + docker-host: ${DATAAGENT_SANDBOX_DOCKER_HOST:unix:///var/run/docker.sock} + image-name: ${DATAAGENT_SANDBOX_IMAGE:agentscope-registry.ap-southeast-1.cr.aliyuncs.com/agentscope/runtime-sandbox-base:latest} max-concurrency: 4 queue-capacity: 10 + max-connections: 4096 package-index-url: ${DATAAGENT_PYPI_INDEX_URL:https://pypi.org/simple} dependency-install-timeout: 3m ``` -### 动态依赖 +`docker-host` 使用 `tcp://host:port` 时会显式连接该端点；其他值走 SAA/docker-java 的 +本地 Docker 发现。完整大小、输出和元数据限制见 +[开发者指南 - 代码执行器配置](DEVELOPER_GUIDE.md#5-代码执行器配置-code-executor)。 + +### 声明动态依赖 ```python # /// script -# dependencies = ["pandas>=2,<3"] +# requires-python = ">=3.10" +# dependencies = ["pandas>=2,<3", "six==1.17.0"] # /// + +import pandas as pd +import six +``` + +脚本最多只能包含一个 PEP 723 `script` 块。`dependencies` 支持 PyPI 包名、extras 和 +版本约束；URL、VCS、本地路径、环境标记、`@` 引用和 pip 参数会在创建容器前被拒绝。 +当前版本会解析并保留 `requires-python`，但不会据此切换或拒绝运行时 Python 版本。 + +每一个非标准库 `import` 都应有对应依赖。只使用标准库时可以省略元数据块。 + +### 执行生命周期 + +```mermaid +sequenceDiagram + participant W as PythonExecuteNode + participant E as PythonCodeExecutorService + participant S as Task BaseSandbox + participant I as Fixed Installer Bootstrap + participant C as Fixed Code Bootstrap + + W->>E: code + stdin JSON + validated dependencies + E->>S: create isolated sandbox + opt dependencies are declared + E->>I: install into /tmp/dataagent-deps + I-->>E: bounded result envelope + end + E->>C: execute code with PYTHONPATH and stdin + C-->>E: bounded stdout/stderr envelope + E-->>W: TaskResponse + E->>S: close and remove container ``` -依赖由固定的后端命令在当前任务沙盒内安装。模型不能指定仓库、URL、文件路径、pip 参数 -或 shell 命令。生产环境应将包索引配置为企业私有 PyPI 代理。 +安装器固定执行 `python -m pip install` 参数列表，不经过 shell。代码、stdin、依赖清单 +和包索引在进入 bootstrap 前使用 Base64 编码。依赖安装目录只存在于本次任务容器。 + +### 验证 + +在数据问答页提交一个明确要求 Python 和第三方依赖的分析任务。成功标准： + +1. 时间线出现 Python 生成、执行、分析和最终报告。 +2. Python 标准输出包含依赖版本和业务 JSON。 +3. 后端日志包含沙盒创建、依赖安装、代码执行和容器删除。 +4. `docker ps --format '{{.Names}}' | grep '^dataagent-sandbox-'` 无输出。 + +### 故障排查 + +| 现象 | 原因与处理 | +|---|---| +| Docker 连接失败 | 先运行 `docker info`；远程 daemon 使用 `tcp://host:port`，本地 Docker 保持默认发现 | +| `DependencyInstallError` | 检查包名、版本和索引连通性；大包下载超时可增大 `dependency-install-timeout` 或使用私有缓存代理 | +| `Unsupported Python dependency specifier` | 移除 URL、VCS、路径、环境标记或 pip 参数，只保留包名和版本约束 | +| `Python sandbox capacity is exhausted` | 并发数和等待队列已满；降低上游并发或调整 `max-concurrency`、`queue-capacity` | +| `Python sandbox task timed out` | 总时间超过“依赖安装超时 + 代码超时 + 30 秒”；检查包下载或代码耗时 | +| 任务后仍有容器 | 检查应用是否被强制终止；正常成功和失败路径都会关闭任务沙盒，应用关闭还会执行全量清理 | + +### 生产安全边界 + +任务容器默认非 privileged，并限制 CPU、内存、nofile、超时、并发和输入输出大小。但 +SAA `1.1.2.2` 没有提供“安装完成后立即断网”的公开 API，因此生产环境还必须： + +1. 使用固定镜像 digest 和非 root 基础镜像。 +2. 仅允许沙盒访问企业私有 PyPI 代理，禁止访问公网和业务内网。 +3. 不向沙盒注入数据库、模型、OSS 或 Docker 凭据。 +4. 在代理侧执行恶意包、CVE 和许可证扫描。 ## ⚙️ 高级配置选项 diff --git a/docs/ARCHITECTURE-en.md b/docs/ARCHITECTURE-en.md index cba9cce03..0057429b4 100644 --- a/docs/ARCHITECTURE-en.md +++ b/docs/ARCHITECTURE-en.md @@ -10,7 +10,7 @@ This document provides a detailed introduction to DataAgent's system architectur %%{init: {"theme": "base", "flowchart": {"curve": "basis", "nodeSpacing": 35, "rankSpacing": 45}, "themeVariables": {"lineColor": "#475569", "primaryTextColor": "#1F2937"}}}%% flowchart LR subgraph Clients[Clients] - UserUI[data-agent-frontend UI] + UserUI[data-agent-frontend-nuxt UI] AdminUI[Admin Console] MCPClient[MCP Client] end @@ -48,10 +48,11 @@ flowchart LR EmbeddingLLM[Embedding Model] end - subgraph Exec[Python Runtime] - Docker[Docker Executor] - Local[Local Executor] - AISim[AI Simulation Executor] + subgraph Exec[Python Sandbox Runtime] + SandboxSvc[SAA SandboxService] + DockerEngine[Docker Engine] + TaskSandbox[Task-scoped BaseSandbox] + PackageIndex[(PyPI / Private Package Index)] end UserUI --> RestAPI @@ -74,10 +75,11 @@ flowchart LR GraphSvc --> ModelRegistry ModelRegistry --> ChatLLM ModelRegistry --> EmbeddingLLM - GraphSvc --> CodePool - CodePool --> Docker - CodePool --> Local - CodePool --> AISim + Graph --> CodeExec + CodeExec --> SandboxSvc + SandboxSvc --> DockerEngine + DockerEngine --> TaskSandbox + TaskSandbox --> PackageIndex AgentCtl --> MetaDB PromptCtl --> MetaDB ModelCtl --> MetaDB @@ -94,11 +96,11 @@ flowchart LR class UserUI,AdminUI,MCPClient client class RestAPI,SSE access class GraphCtl,AgentCtl,PromptCtl,ModelCtl api - class GraphSvc,Context,LlmSvc,ModelRegistry,VectorSvc,Hybrid,CodePool,McpSvc service + class GraphSvc,Context,LlmSvc,ModelRegistry,VectorSvc,Hybrid,CodeExec,McpSvc service class Graph workflow class BizDB,MetaDB,VectorDB,Files data class ChatLLM,EmbeddingLLM llm - class Docker,Local,AISim exec + class SandboxSvc,DockerEngine,TaskSandbox,PackageIndex exec style Clients fill:#FFF7ED,stroke:#D97706,stroke-width:1.5px style Access fill:#EFF6FF,stroke:#0284C7,stroke-width:1.5px @@ -660,8 +662,24 @@ sequenceDiagram - **Code Execution**: `PythonExecuteNode` uses `PythonCodeExecutorService` and always runs in a task-scoped SAA sandbox - **Dynamic Dependencies**: Generated code declares dependencies with PEP 723; a fixed bootstrap installs them inside the sandbox - **Execution Configuration**: `spring.ai.alibaba.data-agent.code-executor.*` +- **Isolation and Cleanup**: Every attempt creates a new `BaseSandbox`; the container is closed and removed after success or failure +- **Failure Handling**: Installation or execution failures are fed into the next code-generation attempt; the existing fallback or termination path applies after the retry limit - **Result Return**: Execution results are written back to `PYTHON_EXECUTE_NODE_OUTPUT`, `PythonAnalyzeNode` summarizes and writes to `SQL_EXECUTE_NODE_OUTPUT` for final report +#### Code Boundaries + +```text +service/code/sandbox +├── SaaSandboxPythonCodeExecutorService.java # input limits, bounded queue, total timeout +├── dependency/ # PEP 723 parsing and package policy +├── execution/ # fixed bootstraps and result envelope +└── runtime/ # SAA service and task sandbox lifecycle +``` + +`PythonCodeExecutorService` is the stable workflow interface. Its only implementation is +`SaaSandboxPythonCodeExecutorService`. DataAgent does not fall back to host-local execution, +local Python processes, the legacy Docker pool, or AI Simulation. + #### Architecture Diagram ```mermaid @@ -699,6 +717,7 @@ sequenceDiagram participant E as PythonExecuteNode participant CP as PythonCodeExecutorService participant S as SAA Sandbox + participant D as PythonExecutorDispatcher participant J as JsonParseUtil participant A as PythonAnalyzeNode participant R as ReportGeneratorNode @@ -708,15 +727,25 @@ sequenceDiagram L-->>G: python code + PEP 723 G->>E: pass code and sql results E->>CP: runTask - CP->>S: create sandbox and install dependencies + CP->>S: create task sandbox and install dependencies S->>S: execute with stdin S-->>CP: stdout stderr CP-->>E: task response - E->>J: parse stdout json - J-->>E: normalized output - E->>A: analyze result - A-->>P: update step results - P->>R: continue to report + alt execution succeeded + E->>J: parse stdout json + J-->>E: normalized output + E->>D: publish success state + D->>A: analyze result + A-->>P: update step results + P->>R: continue to report + else execution failed + E->>D: publish failure state + alt retry is available + D->>G: regenerate with previous code and error + else retry limit reached + D-->>P: fallback analysis or terminate + end + end ``` @@ -726,3 +755,4 @@ sequenceDiagram - [Quick Start](QUICK_START-en.md) - Installation and configuration guide - [Advanced Features](ADVANCED_FEATURES-en.md) - API calls and MCP server - [Developer Documentation](DEVELOPER_GUIDE-en.md) - Contribution guide +- [SAA 1.1.2.2 Python Sandbox Integration Design](superpowers/specs/2026-07-28-saa-python-sandbox-integration-design.md) - Detailed boundaries, security trade-offs, and acceptance evidence diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 83653792d..7ee4d9fd8 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -10,7 +10,7 @@ %%{init: {"theme": "base", "flowchart": {"curve": "basis", "nodeSpacing": 35, "rankSpacing": 45}, "themeVariables": {"lineColor": "#475569", "primaryTextColor": "#1F2937"}}}%% flowchart LR subgraph Clients[Clients] - UserUI[data-agent-frontend UI] + UserUI[data-agent-frontend-nuxt UI] AdminUI[Admin Console] MCPClient[MCP Client] end @@ -53,10 +53,11 @@ flowchart LR EmbeddingLLM[Embedding Model] end - subgraph Exec[Python Runtime] - Docker[Docker Executor] - Local[Local Executor] - AISim[AI Simulation Executor] + subgraph Exec[Python Sandbox Runtime] + SandboxSvc[SAA SandboxService] + DockerEngine[Docker Engine] + TaskSandbox[Task-scoped BaseSandbox] + PackageIndex[(PyPI / Private Package Index)] end UserUI --> RestAPI @@ -79,10 +80,11 @@ flowchart LR GraphSvc --> ModelRegistry ModelRegistry --> ChatLLM ModelRegistry --> EmbeddingLLM - GraphSvc --> CodePool - CodePool --> Docker - CodePool --> Local - CodePool --> AISim + Graph --> CodeExec + CodeExec --> SandboxSvc + SandboxSvc --> DockerEngine + DockerEngine --> TaskSandbox + TaskSandbox --> PackageIndex GraphSvc --> LangfuseSvc LangfuseSvc --> Langfuse AgentCtl --> MetaDB @@ -102,11 +104,11 @@ flowchart LR class UserUI,AdminUI,MCPClient client class RestAPI,SSE access class GraphCtl,AgentCtl,PromptCtl,ModelCtl api - class GraphSvc,Context,LlmSvc,ModelRegistry,VectorSvc,Hybrid,CodePool,McpSvc,LangfuseSvc service + class GraphSvc,Context,LlmSvc,ModelRegistry,VectorSvc,Hybrid,CodeExec,McpSvc,LangfuseSvc service class Graph workflow class BizDB,MetaDB,VectorDB,Files data class ChatLLM,EmbeddingLLM llm - class Docker,Local,AISim exec + class SandboxSvc,DockerEngine,TaskSandbox,PackageIndex exec class Langfuse observability style Clients fill:#FFF7ED,stroke:#D97706,stroke-width:1.5px @@ -669,8 +671,24 @@ sequenceDiagram - **代码执行**: `PythonExecuteNode` 使用 `PythonCodeExecutorService`，统一进入任务级 SAA 沙盒 - **动态依赖**: 生成代码通过 PEP 723 声明依赖，由固定 bootstrap 在沙盒内受控安装 - **执行配置**: `spring.ai.alibaba.data-agent.code-executor.*` +- **隔离与清理**: 每次尝试创建新的 `BaseSandbox`，成功或失败后均关闭并删除容器 +- **失败处理**: 安装或执行失败会把错误反馈给下一次代码生成；达到最大次数后走现有降级或终止分支 - **结果回传**: 执行结果写回 `PYTHON_EXECUTE_NODE_OUTPUT`，`PythonAnalyzeNode` 汇总后写入 `SQL_EXECUTE_NODE_OUTPUT`，用于最终报告 +#### 代码边界 + +```text +service/code/sandbox +├── SaaSandboxPythonCodeExecutorService.java # 输入限制、并发队列、任务总超时 +├── dependency/ # PEP 723 解析与包规格策略 +├── execution/ # 固定 bootstrap 与结果 envelope +└── runtime/ # SAA 服务和任务 Sandbox 生命周期 +``` + +`PythonCodeExecutorService` 是工作流依赖的稳定接口，当前只有 +`SaaSandboxPythonCodeExecutorService` 一个实现。系统不会回退到宿主机、本地 Python +进程、旧 Docker 容器池或 AI Simulation。 + #### 架构图 ```mermaid @@ -708,6 +726,7 @@ sequenceDiagram participant E as PythonExecuteNode participant CP as PythonCodeExecutorService participant S as SAA Sandbox + participant D as PythonExecutorDispatcher participant J as JsonParseUtil participant A as PythonAnalyzeNode participant R as ReportGeneratorNode @@ -717,15 +736,25 @@ sequenceDiagram L-->>G: python code + PEP 723 G->>E: pass code and sql results E->>CP: runTask - CP->>S: create sandbox and install dependencies + CP->>S: create task sandbox and install dependencies S->>S: execute with stdin S-->>CP: stdout stderr CP-->>E: task response - E->>J: parse stdout json - J-->>E: normalized output - E->>A: analyze result - A-->>P: update step results - P->>R: continue to report + alt execution succeeded + E->>J: parse stdout json + J-->>E: normalized output + E->>D: publish success state + D->>A: analyze result + A-->>P: update step results + P->>R: continue to report + else execution failed + E->>D: publish failure state + alt retry is available + D->>G: regenerate with previous code and error + else retry limit reached + D-->>P: fallback analysis or terminate + end + end ``` @@ -735,3 +764,4 @@ sequenceDiagram - [快速开始](QUICK_START.md) - 安装配置指南 - [高级功能](ADVANCED_FEATURES.md) - API调用和MCP服务器 - [开发者文档](DEVELOPER_GUIDE.md) - 贡献指南 +- [SAA 1.1.2.2 Python 沙盒接入方案](superpowers/specs/2026-07-28-saa-python-sandbox-integration-design.md) - 详细边界、安全取舍和验收证据 diff --git a/docs/DEVELOPER_GUIDE-en.md b/docs/DEVELOPER_GUIDE-en.md index 46783523d..6f204cba8 100644 --- a/docs/DEVELOPER_GUIDE-en.md +++ b/docs/DEVELOPER_GUIDE-en.md @@ -10,16 +10,18 @@ Welcome to participate in the development of the DataAgent project! This documen - **JDK**: 17 or higher - **Maven**: 3.6 or higher -- **Node.js**: 16 or higher +- **Node.js**: 22 +- **pnpm**: 11 - **MySQL**: 5.7 or higher +- **Docker**: Required only when running or verifying Python workflows - **Git**: Version control tool - **IDE**: IntelliJ IDEA or Eclipse (IntelliJ IDEA recommended) ### Clone Project ```bash -git clone https://github.com/your-org/spring-ai-alibaba-data-agent.git -cd spring-ai-alibaba-data-agent +git clone https://github.com/spring-ai-alibaba/DataAgent.git +cd DataAgent ``` ### Backend Development Environment @@ -34,21 +36,20 @@ cd spring-ai-alibaba-data-agent 3. **Start Backend Service** ```bash - cd data-agent-management - ./mvnw spring-boot:run + ./mvnw -pl data-agent-management spring-boot:run ``` ### Frontend Development Environment 1. **Install Dependencies** ```bash - cd data-agent-frontend - npm install + cd data-agent-frontend-nuxt + pnpm install ``` 2. **Start Development Server** ```bash - npm run dev + pnpm dev ``` 3. **Access Application** @@ -67,6 +68,8 @@ The workflow is based on Spring AI Alibaba's StateGraph implementation. Core nod - **PlannerNode**: Plan generation - **SqlGenerateNode**: SQL generation - **PythonGenerateNode**: Python code generation +- **PythonExecuteNode**: Parse PEP 723 metadata and dispatch SAA sandbox execution +- **PythonAnalyzeNode**: Analyze Python output and update step state - **ReportGeneratorNode**: Report generation ### 2. Multi-Model Scheduling @@ -304,15 +307,42 @@ Configuration prefix: `spring.ai.alibaba.data-agent.code-executor` | `limit-memory` | Container memory limit (MB) | 500 | | `cpu-core` | Container CPU cores | 1 | | `python-max-tries-count` | Maximum Python execution retries | 5 | -| `sandbox.image-name` | SAA base image | AgentScope runtime sandbox base | +| `sandbox.docker-host` | Docker Engine endpoint | `unix:///var/run/docker.sock` | +| `sandbox.image-name` | SAA base image | `agentscope-registry.ap-southeast-1.cr.aliyuncs.com/agentscope/runtime-sandbox-base:latest` | +| `sandbox.container-prefix` | Task container name prefix | `dataagent-sandbox-` | | `sandbox.max-concurrency` | Maximum concurrent sandboxes | 4 | | `sandbox.queue-capacity` | Bounded wait queue size | 10 | -| `sandbox.package-index-url` | Dynamic package index | https://pypi.org/simple | +| `sandbox.max-code-bytes` | Python source UTF-8 byte limit | 262144 (256 KiB) | +| `sandbox.max-input-bytes` | stdin JSON UTF-8 byte limit | 10485760 (10 MiB) | +| `sandbox.max-output-bytes` | stdout UTF-8 byte limit | 1048576 (1 MiB) | +| `sandbox.max-error-bytes` | stderr UTF-8 byte limit | 262144 (256 KiB) | +| `sandbox.max-metadata-bytes` | PEP 723 metadata UTF-8 byte limit | 8192 (8 KiB) | +| `sandbox.max-dependencies` | Maximum number of direct dependencies | 20 | +| `sandbox.package-index-url` | Dynamic package index | `https://pypi.org/simple` | | `sandbox.dependency-install-timeout` | Dependency installation timeout | 3m | +| `sandbox.max-connections` | Container `nofile` limit | 4096 | Third-party dependencies must be declared in the generated script's PEP 723 `dependencies`. Host-local, legacy Docker pool, and AI Simulation executors are no longer available. +Common environment variables: + +| Environment Variable | Configuration | Purpose | +|---|---|---| +| `DATAAGENT_SANDBOX_DOCKER_HOST` | `sandbox.docker-host` | Point to a local or remote Docker Engine | +| `DATAAGENT_SANDBOX_IMAGE` | `sandbox.image-name` | Pin the runtime image; use a digest in production | +| `DATAAGENT_PYPI_INDEX_URL` | `sandbox.package-index-url` | Point to an enterprise private PyPI proxy | + +Each Python task creates a separate `BaseSandbox`, installs dependencies and executes code in the same +container, then stops and removes that container. The service-side wait timeout is the dependency +installation timeout plus the code timeout plus a 30-second transport margin. `requires-python` is +currently parsed and retained, but it does not select or validate the sandbox Python version. + +See [Advanced Features - Python Execution Environment Configuration](ADVANCED_FEATURES-en.md#python-execution-environment-configuration) +for dependency syntax, security restrictions, runtime verification, and troubleshooting. See the +[SAA 1.1.2.2 Python Sandbox Integration Design](superpowers/specs/2026-07-28-saa-python-sandbox-integration-design.md) +for implementation boundaries. + ### 6. File Storage Configuration Configuration prefix: `spring.ai.alibaba.data-agent.file` @@ -344,7 +374,7 @@ Configuration prefix: `spring.sql.init` | Configuration Item | Description | Default Value | Notes | |-------------------|-------------|---------------|-------| -| `mode` | Initialization mode (always/never) | always | "always" executes schema.sql and data.sql on every startup. Recommended to set to "never" for production to avoid sample data overwriting business data | +| `mode` | Initialization mode (always/never) | never | Set to `always` only when initialization is explicitly required | | `schema-locations` | Table structure script path | classpath:sql/schema.sql | | | `data-locations` | Data script path | classpath:sql/data.sql | | @@ -378,13 +408,43 @@ Environment variables: `LANGFUSE_ENABLED`, `LANGFUSE_HOST`, `LANGFUSE_PUBLIC_KEY > For detailed usage, refer to [Advanced Features - Langfuse Observability](ADVANCED_FEATURES-en.md#langfuse-observability). +## Python Sandbox Verification + +Unit tests that do not require Docker: + +```bash +./mvnw -pl data-agent-management \ + -Dtest='PythonDependencyMetadataParserTest,PythonSandboxBootstrapBuilderTest,SandboxExecutionResultParserTest,SaaSandboxPythonCodeExecutorServiceTest,SaaSandboxRuntimeTest,PythonExecuteNodeTest,PythonWorkflowIntegrationTest' \ + test +``` + +Run the real SAA integration test while Docker is available: + +```bash +docker info +./mvnw -pl data-agent-management -Dtest=SaaSandboxTaskRunnerIT test +``` + +Run the CI-equivalent checks before submission: + +```bash +make format-check +make checkstyle-check +make test +``` + +A real end-to-end acceptance check must go beyond HTTP 200: the browser timeline must show dependency +installation and Python execution output, a final report, and SSE `event:complete`. The command +`docker ps -a --filter name=dataagent-sandbox-` must not show leftover task containers. + ## Learning Resources ### Official Documentation - [Spring AI Alibaba Documentation](https://springdoc.cn/spring-ai/) - [Spring Boot Documentation](https://spring.io/projects/spring-boot) -- [React Documentation](https://react.dev/) +- [Nuxt Documentation](https://nuxt.com/docs) +- [Vue Documentation](https://vuejs.org/guide/) - [TypeScript Documentation](https://www.typescriptlang.org/) ### Related Technologies @@ -396,7 +456,7 @@ Environment variables: `LANGFUSE_ENABLED`, `LANGFUSE_HOST`, `LANGFUSE_PUBLIC_KEY ## Contribution Guide -For detailed contribution guidelines, please refer to [CONTRIBUTING.md](../CONTRIBUTING.md). +For detailed contribution guidelines, see [CONTRIBUTING-en.md](../CONTRIBUTING-en.md). ### Contribution Types diff --git a/docs/DEVELOPER_GUIDE.md b/docs/DEVELOPER_GUIDE.md index 15ed810e7..8bc557c72 100644 --- a/docs/DEVELOPER_GUIDE.md +++ b/docs/DEVELOPER_GUIDE.md @@ -10,16 +10,18 @@ - **JDK**: 17 或更高版本 - **Maven**: 3.6 或更高版本 -- **Node.js**: 16 或更高版本 +- **Node.js**: 22 +- **pnpm**: 11 - **MySQL**: 5.7 或更高版本 +- **Docker**: 仅运行或验证 Python 工作流时需要 - **Git**: 版本控制工具 - **IDE**: IntelliJ IDEA 或 Eclipse (推荐 IntelliJ IDEA) ### 克隆项目 ```bash -git clone https://github.com/your-org/spring-ai-alibaba-data-agent.git -cd spring-ai-alibaba-data-agent +git clone https://github.com/spring-ai-alibaba/DataAgent.git +cd DataAgent ``` ### 后端开发环境 @@ -34,21 +36,20 @@ cd spring-ai-alibaba-data-agent 3. **启动后端服务** ```bash - cd data-agent-management - ./mvnw spring-boot:run + ./mvnw -pl data-agent-management spring-boot:run ``` ### 前端开发环境 1. **安装依赖** ```bash - cd data-agent-frontend - npm install + cd data-agent-frontend-nuxt + pnpm install ``` 2. **启动开发服务器** ```bash - npm run dev + pnpm dev ``` 3. **访问应用** @@ -67,6 +68,8 @@ cd spring-ai-alibaba-data-agent - **PlannerNode**: 计划生成 - **SqlGenerateNode**: SQL 生成 - **PythonGenerateNode**: Python 代码生成 +- **PythonExecuteNode**: 解析 PEP 723 元数据并调度 SAA 沙盒执行 +- **PythonAnalyzeNode**: 分析 Python 结果并更新步骤状态 - **ReportGeneratorNode**: 报告生成 ### 2. 多模型调度 @@ -357,15 +360,40 @@ public class AgentVectorStoreService { | `limit-memory` | 容器内存限制 (MB) | 500 | | `cpu-core` | 容器CPU核数 | 1 | | `python-max-tries-count` | Python执行最大重试次数 | 5 | -| `sandbox.image-name` | SAA基础镜像 | AgentScope runtime sandbox base | +| `sandbox.docker-host` | Docker Engine 地址 | `unix:///var/run/docker.sock` | +| `sandbox.image-name` | SAA 基础镜像 | `agentscope-registry.ap-southeast-1.cr.aliyuncs.com/agentscope/runtime-sandbox-base:latest` | +| `sandbox.container-prefix` | 任务容器名称前缀 | `dataagent-sandbox-` | | `sandbox.max-concurrency` | 最大并发沙盒数 | 4 | | `sandbox.queue-capacity` | 有界等待队列大小 | 10 | -| `sandbox.package-index-url` | 动态依赖包索引 | https://pypi.org/simple | +| `sandbox.max-code-bytes` | Python 源码 UTF-8 字节上限 | 262144（256 KiB） | +| `sandbox.max-input-bytes` | stdin JSON UTF-8 字节上限 | 10485760（10 MiB） | +| `sandbox.max-output-bytes` | stdout UTF-8 字节上限 | 1048576（1 MiB） | +| `sandbox.max-error-bytes` | stderr UTF-8 字节上限 | 262144（256 KiB） | +| `sandbox.max-metadata-bytes` | PEP 723 元数据 UTF-8 字节上限 | 8192（8 KiB） | +| `sandbox.max-dependencies` | 最大直接依赖数 | 20 | +| `sandbox.package-index-url` | 动态依赖包索引 | `https://pypi.org/simple` | | `sandbox.dependency-install-timeout` | 依赖安装超时 | 3m | +| `sandbox.max-connections` | 容器 `nofile` 上限 | 4096 | 第三方依赖必须在生成脚本的 PEP 723 `dependencies` 中声明。系统不再提供宿主机 Local、 旧 Docker 容器池或 AI Simulation 执行器。 +常用环境变量： + +| 环境变量 | 对应配置 | 用途 | +|---|---|---| +| `DATAAGENT_SANDBOX_DOCKER_HOST` | `sandbox.docker-host` | 指向本机或远程 Docker Engine | +| `DATAAGENT_SANDBOX_IMAGE` | `sandbox.image-name` | 固定运行时镜像；生产环境应使用 digest | +| `DATAAGENT_PYPI_INDEX_URL` | `sandbox.package-index-url` | 指向企业私有 PyPI 代理 | + +每次 Python 任务创建独立 `BaseSandbox`，在同一容器内先安装依赖、再执行代码，最后停止并 +删除容器。服务端总等待时间为“依赖安装超时 + 代码执行超时 + 30 秒通信余量”。 +`requires-python` 当前会被解析和保留，但不会切换或校验沙盒 Python 版本。 + +依赖格式、安全限制、运行验证和故障处理见 +[高级功能 - Python 执行环境配置](ADVANCED_FEATURES.md#-python-执行环境配置)；实现边界见 +[SAA 1.1.2.2 Python 沙盒接入方案](superpowers/specs/2026-07-28-saa-python-sandbox-integration-design.md)。 + ### 6. 文件存储配置 (File Storage) 配置前缀: `spring.ai.alibaba.data-agent.file` @@ -397,7 +425,7 @@ public class AgentVectorStoreService { | 配置项 | 说明 | 默认值 | 备注 | |--------|------|--------|------| -| `mode` | 初始化模式 (always/never) | always | "always"会每次启动执行schema.sql和data.sql，建议生产环境设为"never" | +| `mode` | 初始化模式 (always/never) | never | 仅在明确需要初始化时设置为 `always` | | `schema-locations` | 表结构脚本路径 | classpath:sql/schema.sql | | | `data-locations` | 数据脚本路径 | classpath:sql/data.sql | | @@ -431,13 +459,43 @@ public class AgentVectorStoreService { > 详细使用说明请参考 [高级功能 - Langfuse 可观测性](ADVANCED_FEATURES.md#-langfuse-可观测性)。 +## ✅ Python 沙盒验证 + +不需要 Docker 的单元测试： + +```bash +./mvnw -pl data-agent-management \ + -Dtest='PythonDependencyMetadataParserTest,PythonSandboxBootstrapBuilderTest,SandboxExecutionResultParserTest,SaaSandboxPythonCodeExecutorServiceTest,SaaSandboxRuntimeTest,PythonExecuteNodeTest,PythonWorkflowIntegrationTest' \ + test +``` + +Docker 在线时运行真实 SAA 集成测试： + +```bash +docker info +./mvnw -pl data-agent-management -Dtest=SaaSandboxTaskRunnerIT test +``` + +提交前执行与 CI 对齐的检查： + +```bash +make format-check +make checkstyle-check +make test +``` + +真实端到端验收不能只看 HTTP 200：浏览器时间线应出现依赖安装和 Python 执行结果、最终报告， +SSE 应收到 `event:complete`，并且 `docker ps -a --filter name=dataagent-sandbox-` 不应留下 +任务容器。 + ## 📚 学习资源 ### 官方文档 - [Spring AI Alibaba 文档](https://springdoc.cn/spring-ai/) - [Spring Boot 文档](https://spring.io/projects/spring-boot) -- [React 文档](https://react.dev/) +- [Nuxt 文档](https://nuxt.com/docs) +- [Vue 文档](https://vuejs.org/guide/) - [TypeScript 文档](https://www.typescriptlang.org/) ### 相关技术 diff --git a/docs/QUICK_START-en.md b/docs/QUICK_START-en.md index bf6019a3a..b3708bb14 100644 --- a/docs/QUICK_START-en.md +++ b/docs/QUICK_START-en.md @@ -8,8 +8,9 @@ This document will guide you through the installation, configuration, and first - **JDK**: 17 or higher - **MySQL**: 5.7 or higher -- **Node.js**: 16 or higher -- **Docker**: (Optional) For Python code execution +- **Node.js**: 22 or higher +- **pnpm**: 11 or higher +- **Docker**: Required when a workflow executes Python steps; optional for SQL-only analysis - **Vector Database**: (Optional) Uses in-memory vector store by default ## 1. Business Database Preparation @@ -38,23 +39,25 @@ mysql -u root -p your_database < data-agent-management/src/main/resources/sql/pr Configure your MySQL database connection in `data-agent-management/src/main/resources/application.yml`. -> Initialization Behavior: Auto table creation and sample data insertion is enabled by default (`spring.sql.init.mode: always`). For production environments, it's recommended to disable this to avoid sample data overwriting your business data. +> Initialization behavior: the default is `spring.sql.init.mode: never`, so DataAgent does not +> create tables or insert sample data automatically. Run the SQL files above before the first +> start, or set `DATA_AGENT_DATASOURCE_SQL_INIT=always` only when sample initialization is +> explicitly required. ```yaml spring: datasource: url: jdbc:mysql://127.0.0.1:3306/saa_data_agent?useUnicode=true&characterEncoding=utf-8&zeroDateTimeBehavior=convertToNull&transformedBitIsBoolean=true&allowMultiQueries=true&allowPublicKeyRetrieval=true&useSSL=false&serverTimezone=Asia/Shanghai - username: ${MYSQL_USERNAME:root} - password: ${MYSQL_PASSWORD:root} + username: ${DATA_AGENT_DATASOURCE_USERNAME:root} + password: ${DATA_AGENT_DATASOURCE_PASSWORD:root} driver-class-name: com.mysql.cj.jdbc.Driver - type: com.alibaba.druid.pool.DruidDataSource ``` ### 2.2 Data Initialization Configuration -Auto initialization is enabled by default (`spring.sql.init.mode: always`). +Auto initialization is disabled by default (`spring.sql.init.mode: never`). -> For information on how to disable auto initialization, please refer to [Developer Guide - Database Initialization Configuration](DEVELOPER_GUIDE-en.md#8-database-initialization). +> To change initialization behavior, see [Developer Guide - Database Initialization](DEVELOPER_GUIDE-en.md#8-database-initialization). ### 2.3 Configure Model @@ -77,7 +80,7 @@ Start the project, click on Model Configuration, add a new model and fill in you ### 2.4 Embedding Model Batch Processing Strategy Configuration -> For detailed configuration parameters, please refer to [Developer Guide - Development Configuration Manual](DEVELOPER_GUIDE-en.md#development-configuration-manual). +> For detailed configuration parameters, see [Developer Guide - Embedding Batch Configuration](DEVELOPER_GUIDE-en.md#2-embedding-batch-configuration). ### 2.5 Vector Store Configuration @@ -177,53 +180,93 @@ Below is the ES schema structure. For other vector stores like Milvus, PG, etc., #### 2.5.3 Vector Store Configuration Parameters -> For detailed configuration parameters, please refer to [Developer Guide - Development Configuration Manual](DEVELOPER_GUIDE-en.md#development-configuration-manual). +> For detailed configuration parameters, see [Developer Guide - Vector Store Configuration](DEVELOPER_GUIDE-en.md#3-vector-store-configuration). ### 2.6 Retrieval Fusion Strategy -> For detailed configuration parameters, please refer to [Developer Guide - Development Configuration Manual](DEVELOPER_GUIDE-en.md#development-configuration-manual). +> For detailed configuration parameters, see [Developer Guide - Vector Store Configuration](DEVELOPER_GUIDE-en.md#3-vector-store-configuration). ### 2.7 Replace Vector Store Implementation -> For information on how to replace the default in-memory vector store (e.g., using PGVector, Milvus, etc.), please refer to [Developer Guide - Dependency Extension Configuration](DEVELOPER_GUIDE-en.md#9-dependency-extension). +> To replace the default in-memory vector store (for example, with PGVector or Milvus), see [Developer Guide - Vector Store Dependency Extension](DEVELOPER_GUIDE-en.md#vector-store-dependency-extension). + +### 2.8 Configure the Python Sandbox + +DataAgent uses the Spring AI Alibaba `1.1.2.2` Sandbox to run generated Python code. Every +execution gets a task-scoped container; dynamic dependency installation, code execution, and +container cleanup all happen inside that task. + +Confirm that Docker is available: + +```bash +docker info +``` + +The default configuration uses local Docker discovery, public PyPI, and the AgentScope base +image. Development normally needs no override. To select a Docker endpoint, image, or private +package index, set: + +```bash +export DATAAGENT_SANDBOX_DOCKER_HOST=unix:///var/run/docker.sock +export DATAAGENT_SANDBOX_IMAGE=agentscope-registry.ap-southeast-1.cr.aliyuncs.com/agentscope/runtime-sandbox-base:latest +export DATAAGENT_PYPI_INDEX_URL=https://pypi.org/simple +``` + +Production should use a pinned image digest and an enterprise PyPI proxy, with infrastructure +network policy that allows the sandbox to reach only that proxy. See +[Advanced Features - Python Execution Environment Configuration](ADVANCED_FEATURES-en.md#python-execution-environment-configuration) +for the full configuration, dependency contract, and troubleshooting guidance. ## 3. Start Management Backend -In the `data-agent-management` directory, run the `DataAgentApplication.java` class. +Run the following command from the project root: ```bash -cd data-agent-management -./mvnw spring-boot:run +./mvnw -pl data-agent-management spring-boot:run ``` Or run `DataAgentApplication.java` directly in your IDE. ## 4. Start Web Frontend -Navigate to the `data-agent-frontend` directory +Navigate to the `data-agent-frontend-nuxt` directory. ### 4.1 Install Dependencies ```bash -# Using npm -npm install - -# Or using yarn -yarn install +pnpm install ``` ### 4.2 Start Service ```bash -# Using npm -npm run dev - -# Or using yarn -yarn dev +pnpm dev ``` After successful startup, access http://localhost:3000 +### 4.3 Verify Dynamic Python Dependencies + +After configuring the agent, models, and data source, submit a request that explicitly includes a +Python step: + +```text +Query the raw status and total_amount rows from orders, then aggregate them with Python. +Declare and import six==1.17.0 through PEP 723, and report the six version and status totals. +``` + +A successful timeline contains “Python Generation”, “Python Execution”, “Python Analysis”, and a +final report. The Python output should contain `six_version: 1.17.0`. No task container should +remain after completion: + +```bash +docker ps --format '{{.Names}}' | grep '^dataagent-sandbox-' +``` + +No output means the task sandbox was removed. If dependency installation fails, the workflow +feeds the error into the next Python generation attempt. After the configured retry limit, it +enters the existing fallback or termination path. + ## 5. System Experience ### 5.1 Creating and Configuring Data Agent diff --git a/docs/QUICK_START.md b/docs/QUICK_START.md index dfce88b54..9754c1193 100644 --- a/docs/QUICK_START.md +++ b/docs/QUICK_START.md @@ -8,8 +8,9 @@ - **JDK**: 17 或更高版本 - **MySQL**: 5.7 或更高版本 -- **Node.js**: 16 或更高版本 -- **Docker**: (可选) 用于Python代码执行 +- **Node.js**: 22 或更高版本 +- **pnpm**: 11 或更高版本 +- **Docker**: 工作流需要执行 Python 步骤时必需；仅使用 SQL 分析时可不启动 - **向量数据库**: (可选) 默认使用内存向量库 ## 🗄️ 1. 业务数据库准备 @@ -38,27 +39,28 @@ mysql -u root -p your_database < data-agent-management/src/main/resources/sql/pr 在`data-agent-management/src/main/resources/application.yml`中配置你的MySQL数据库连接信息。 -> 初始化行为说明：默认开启自动创建表并插入示例数据（`spring.sql.init.mode: always`）。生产环境建议关闭，避免示例数据回填覆盖你的业务数据。 +> 初始化行为说明：默认配置为 `spring.sql.init.mode: never`，不会自动创建表或插入示例数据。 +> 首次启动前请先执行上面的 SQL 文件，或在明确需要示例数据时通过环境变量 +> `DATA_AGENT_DATASOURCE_SQL_INIT=always` 开启初始化。 ```yaml spring: datasource: url: jdbc:mysql://127.0.0.1:3306/saa_data_agent?useUnicode=true&characterEncoding=utf-8&zeroDateTimeBehavior=convertToNull&transformedBitIsBoolean=true&allowMultiQueries=true&allowPublicKeyRetrieval=true&useSSL=false&serverTimezone=Asia/Shanghai - username: ${MYSQL_USERNAME:root} - password: ${MYSQL_PASSWORD:root} + username: ${DATA_AGENT_DATASOURCE_USERNAME:root} + password: ${DATA_AGENT_DATASOURCE_PASSWORD:root} driver-class-name: com.mysql.cj.jdbc.Driver - type: com.alibaba.druid.pool.DruidDataSource ``` ### 2.2 数据初始化配置 -默认开启自动初始化 (`spring.sql.init.mode: always`)。 +默认关闭自动初始化（`spring.sql.init.mode: never`）。 -> 关于如何关闭自动初始化，请参考 [开发者指南 - 数据库初始化配置](DEVELOPER_GUIDE.md#8-数据库初始化配置-database-initialization)。 +> 关于如何调整初始化行为，请参考 [开发者指南 - 数据库初始化配置](DEVELOPER_GUIDE.md#8-数据库初始化配置-database-initialization)。 ### 2.3 配置模型 -> 如果涉及手动管理模型依赖（非默认 Starter），请参考 [开发者指南 - 扩展依赖配置](DEVELOPER_GUIDE.md#9-扩展依赖配置-dependency-extension)。 +> 如果涉及手动管理模型依赖（非默认 Starter），请参考 [开发者指南 - 模型依赖手动管理](DEVELOPER_GUIDE.md#9-模型依赖手动管理-manual-model-dependency)。 启动项目，点击模型配置，新增模型填写自己的apikey即可。 @@ -77,7 +79,7 @@ spring: ### 2.4 嵌入模型批处理策略配置 -> 详细配置参数请参考 [开发者指南 - 开发配置手册](DEVELOPER_GUIDE.md#⚙️-开发配置手册)。 +> 详细配置参数请参考 [开发者指南 - 嵌入模型批处理策略](DEVELOPER_GUIDE.md#2-嵌入模型批处理策略-embedding-batch)。 ### 2.5 向量库配置 @@ -177,53 +179,87 @@ spring: #### 2.5.3 向量库配置参数 -> 详细配置参数请参考 [开发者指南 - 开发配置手册](DEVELOPER_GUIDE.md#⚙️-开发配置手册)。 +> 详细配置参数请参考 [开发者指南 - 向量库配置](DEVELOPER_GUIDE.md#3-向量库配置-vector-store)。 ### 2.6 检索融合策略 -> 详细配置参数请参考 [开发者指南 - 开发配置手册](DEVELOPER_GUIDE.md#⚙️-开发配置手册)。 +> 详细配置参数请参考 [开发者指南 - 向量库配置](DEVELOPER_GUIDE.md#3-向量库配置-vector-store)。 ### 2.7 替换vector-store的实现类 -> 关于如何替换默认的内存向量库（如使用 PGVector, Milvus 等），请参考 [开发者指南 - 扩展依赖配置](DEVELOPER_GUIDE.md#9-扩展依赖配置-dependency-extension)。 +> 关于如何替换默认的内存向量库（如使用 PGVector、Milvus 等），请参考 [开发者指南 - 向量库依赖扩展](DEVELOPER_GUIDE.md#向量库依赖扩展)。 + +### 2.8 配置 Python 沙盒 + +DataAgent 使用 Spring AI Alibaba `1.1.2.2` 的 Sandbox 运行生成的 Python 代码。每次 +执行都会创建一个任务级容器，动态依赖安装、业务代码执行和容器清理都在该任务内完成。 + +先确认 Docker 可用： + +```bash +docker info +``` + +默认配置使用本地 Docker socket、公网 PyPI 和 AgentScope 基础镜像。开发环境通常无需 +修改；如需指定 Docker 地址、镜像或私有包索引，可设置： + +```bash +export DATAAGENT_SANDBOX_DOCKER_HOST=unix:///var/run/docker.sock +export DATAAGENT_SANDBOX_IMAGE=agentscope-registry.ap-southeast-1.cr.aliyuncs.com/agentscope/runtime-sandbox-base:latest +export DATAAGENT_PYPI_INDEX_URL=https://pypi.org/simple +``` + +生产环境应使用固定镜像 digest 和企业私有 PyPI 代理，并在基础设施层限制沙盒只能访问 +包代理。完整配置、依赖协议和故障处理见 +[高级功能 - Python 执行环境配置](ADVANCED_FEATURES.md#-python-执行环境配置)。 ## 🚀 3. 启动管理端 -在`data-agent-management`目录下，运行 `DataAgentApplication.java` 类。 +在项目根目录运行： ```bash -cd data-agent-management -./mvnw spring-boot:run +./mvnw -pl data-agent-management spring-boot:run ``` 或者在IDE中直接运行 `DataAgentApplication.java`。 ## 🌐 4. 启动WEB页面 -进入 `data-agent-frontend` 目录 +进入 `data-agent-frontend-nuxt` 目录 ### 4.1 安装依赖 ```bash -# 使用 npm -npm install - -# 或使用 yarn -yarn install +pnpm install ``` ### 4.2 启动服务 ```bash -# 使用 npm -npm run dev - -# 或使用 yarn -yarn dev +pnpm dev ``` 启动成功后，访问地址 http://localhost:3000 +### 4.3 验证 Python 动态依赖 + +完成智能体、模型和数据源配置后，在数据问答页提交一个明确包含 Python 步骤的请求，例如： + +```text +先查询 orders 表的 status 和 total_amount 原始数据，再使用 Python 聚合； +Python 通过 PEP 723 声明并导入 six==1.17.0，最终输出 six 版本和各状态汇总。 +``` + +成功时，时间线会依次出现“Python 生成”“Python 执行”“Python 分析”和最终报告，Python +输出中应包含 `six_version: 1.17.0`。任务结束后不应存在残留容器： + +```bash +docker ps --format '{{.Names}}' | grep '^dataagent-sandbox-' +``` + +命令无输出表示任务沙盒已经清理。若依赖安装失败，工作流会把错误反馈给下一次 Python +生成尝试；达到最大重试次数后进入现有降级或终止分支。 + ## 🎯 5. 系统体验 ### 5.1 数据智能体的创建与配置 diff --git a/docs/superpowers/specs/2026-07-28-saa-python-sandbox-integration-design.md b/docs/superpowers/specs/2026-07-28-saa-python-sandbox-integration-design.md index c2910fb34..a3463e4e9 100644 --- a/docs/superpowers/specs/2026-07-28-saa-python-sandbox-integration-design.md +++ b/docs/superpowers/specs/2026-07-28-saa-python-sandbox-integration-design.md @@ -1,8 +1,11 @@ # SAA 1.1.2.2 Python 沙盒接入方案 +> 状态：已通过 [PR #587](https://github.com/spring-ai-alibaba/DataAgent/pull/587) 合入上游。 +> 本文同时记录当前实现边界、生产约束和已完成的验证，不再作为待实施设计稿使用。 + ## 1. 结论与范围 -本次只改造 Python 生成与执行链路： +已完成的改造只覆盖 Python 生成与执行链路： - Spring AI Alibaba 升级到 `1.1.2.2` - Spring AI 对齐到 `1.1.2` @@ -11,8 +14,8 @@ - 删除本地进程、自建 Docker 容器池和 AI Simulation 执行器 - SQL 生成、校验和执行链路不做改动 -不存在本地执行或旧 Docker 执行器的回退。SAA 启动、依赖安装或代码执行失败时，结果直接 -进入现有 Python 重试与降级分支。 +不存在本地执行或旧 Docker 执行器的回退。SAA 启动、依赖安装或代码执行失败时，结果进入 +现有 Python 重试、降级或终止分支。 ## 2. 改造后的链路 @@ -43,7 +46,8 @@ flowchart LR 1. 生成代码仍通过 `json.load(sys.stdin)` 读取 SQL 结果 JSON。 2. 业务结果仍从 stdout 返回，并由 `PythonExecuteNode` 做现有 JSON 解析。 -3. Python 失败仍进入现有重试，超过次数仍走现有降级逻辑。 +3. Python 执行失败仍进入现有重试；达到上限后，普通执行失败进入降级分析，元数据解析或 + 协议异常等分支可能直接终止。 4. StateGraph 节点关系和 SSE 事件协议不变。 ### 2.1 包边界 @@ -94,6 +98,10 @@ import pandas as pd 5. 拒绝 URL、VCS、文件路径、`@`、环境标记和任何 pip 参数。 6. 模型不能提供 index、证书、命令、安装目录或 shell 片段。 +`requires-python` 会被 TOML Parser 解析并保留在元数据对象中，但当前版本不会据此切换 +基础镜像、选择解释器或校验 Sandbox 内的 Python 版本。需要固定 Python 版本时，应通过 +`sandbox.image-name` 选择并锁定镜像。 + 依赖声明只是一组结构化包规格，不是命令字符串： ```text @@ -251,8 +259,16 @@ spring: sandbox: docker-host: ${DATAAGENT_SANDBOX_DOCKER_HOST:unix:///var/run/docker.sock} image-name: ${DATAAGENT_SANDBOX_IMAGE:agentscope-registry.ap-southeast-1.cr.aliyuncs.com/agentscope/runtime-sandbox-base:latest} + container-prefix: dataagent-sandbox- max-concurrency: 4 queue-capacity: 10 + max-code-bytes: 262144 + max-input-bytes: 10485760 + max-output-bytes: 1048576 + max-error-bytes: 262144 + max-metadata-bytes: 8192 + max-dependencies: 20 + max-connections: 4096 package-index-url: ${DATAAGENT_PYPI_INDEX_URL:https://pypi.org/simple} dependency-install-timeout: 3m ``` @@ -299,18 +315,18 @@ spring: 5. 后续可增加 wheel-only、hash lock 和只读 wheel 缓存；这些是生产加固项，不作为 当前代码已实现能力描述。 -## 11. 验收 +## 11. 验收结果 -| 场景 | 验收条件 | -|---|---| -| 动态依赖 | PEP 723 声明的包在任务 Sandbox 内安装成功 | -| stdin | 生成代码能通过 `json.load(sys.stdin)` 读取 SQL 结果 | -| stdout | JSON 结果保持现有工作流协议 | -| Python 异常 | 进入现有重试或降级分支 | -| 超时/超限 | 返回明确失败并释放 Sandbox | -| 隔离 | 两次执行不复用可写运行态 | -| 清理 | 成功和失败后均无残留 `dataagent-sandbox-*` 容器 | -| 端到端 | 浏览器发起真实分析后出现最终报告且 SSE 收到 `event:complete` | +| 场景 | 验收条件 | 结果 | +|---|---|---| +| 动态依赖 | PEP 723 声明的包在任务 Sandbox 内安装成功 | 通过 | +| stdin | 生成代码能通过 `json.load(sys.stdin)` 读取 SQL 结果 | 通过 | +| stdout | JSON 结果保持现有工作流协议 | 通过 | +| Python 异常 | 进入现有重试、降级或终止分支 | 通过 | +| 超时/超限 | 返回明确失败并释放 Sandbox | 通过 | +| 隔离 | 两次执行不复用可写运行态 | 通过 | +| 清理 | 成功和失败后均无残留 `dataagent-sandbox-*` 容器 | 通过 | +| 端到端 | 浏览器发起真实分析后出现最终报告且 SSE 收到 `event:complete` | 通过 | 验证分层： @@ -320,8 +336,45 @@ spring: 3. 完整后端测试与格式检查。 4. 本地启动 Milvus、后端和前端，使用浏览器完成真实 StateGraph 请求。 +### 11.1 自动化验证 + +- 完整 Maven 测试：1628 个测试，0 失败、0 错误、1 跳过。 +- `spring-javaformat:validate` 和 Checkstyle 通过。 +- 上游 PR 的 Frontend Build、linter、license、secret、format、check-style、 + test-and-build 等 8 项检查全部通过。 + +### 11.2 浏览器端到端验证 + +端到端验证使用真实模型和真实运行时，而非 mock： + +1. 使用 Qwen Embedding 完成向量召回，使用 DeepSeek Chat 完成规划、代码生成和报告生成。 +2. 第一次生成的 `pandas + six` 依赖在下载超时后，工作流携带错误进入代码重生成。 +3. 重生成脚本仅声明 `six`，任务 Sandbox 动态安装 `six==1.17.0`，读取 SQL 结果 stdin + 并返回 JSON。 +4. 浏览器时间线出现依赖安装、Python 执行和最终报告，SSE 收到 `event:complete`，结果成功 + 持久化。 +5. 成功和失败路径结束后均未残留 `dataagent-sandbox-*` 容器；验证完成后前后端已关闭。 + +这组证据同时覆盖了“依赖安装失败后重试”和“动态依赖安装成功后完成报告”两条关键链路。 +单纯的后端 HTTP 200、单元测试或容器启动成功不能替代上述端到端验收。 + +### 11.3 CI 密钥边界 + +上游仓库可以把真实模型凭据放在 GitHub Actions Repository Secret 中，但从 fork 发起的 +`pull_request` 工作流不会获得上游 Secret。因此： + +- 常规 PR 的构建、格式、静态检查和无密钥测试必须保持可独立运行。 +- 真实模型端到端测试应放在上游分支手动触发的工作流，或受保护 Environment 中执行。 +- 不使用 `pull_request_target` 检出并执行 fork 代码来换取 Secret 访问，否则会把上游 + 凭据暴露给不受信任代码。 +- 日志、时间线和测试报告只记录模型/提供商名称，不输出 API Key。 + ## 12. 参考 +- [快速开始](../../QUICK_START.md) +- [高级功能：Python 执行环境](../../ADVANCED_FEATURES.md#-python-执行环境配置) +- [开发者指南：代码执行器配置](../../DEVELOPER_GUIDE.md#5-代码执行器配置-code-executor) +- [系统架构：Python 沙盒链路](../../ARCHITECTURE.md#8-python-执行与结果回传) - [Spring AI Alibaba v1.1.2.2 releases](https://github.com/alibaba/spring-ai-alibaba/releases) - [Spring AI Alibaba Sandbox example](https://github.com/spring-ai-alibaba/examples/tree/main/spring-ai-alibaba-sandbox-example/sandbox-simple-tool) - [PEP 723 – Inline script metadata](https://peps.python.org/pep-0723/)