Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 15 additions & 14 deletions README-en.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
</p>

<p>
<a href="https://github.com/alibaba/spring-ai-alibaba"><img src="https://img.shields.io/badge/Spring%20AI%20Alibaba-1.1.0.0-blue" alt="Spring AI Alibaba"></a>
<a href="https://github.com/alibaba/spring-ai-alibaba"><img src="https://img.shields.io/badge/Spring%20AI%20Alibaba-1.1.2.2-blue" alt="Spring AI Alibaba"></a>
<img src="https://img.shields.io/badge/Spring%20Boot-3.4.8+-green" alt="Spring Boot">
<img src="https://img.shields.io/badge/Java-17+-orange" alt="Java">
<img src="https://img.shields.io/badge/License-Apache%202.0-red" alt="License">
Expand Down Expand Up @@ -46,7 +46,7 @@ Additionally, this project natively supports **MCP (Model Context Protocol)**, e
| Feature | Description |
| :--- | :--- |
| **Intelligent Data Analysis** | StateGraph-based Text-to-SQL conversion, supporting complex multi-table queries and multi-turn conversation intent understanding. |
| **Python Deep Analysis** | Built-in Docker/Local Python executor, automatically generating and executing Python code for statistical analysis and machine learning predictions. |
| **Python Deep Analysis** | Executes generated code in task-scoped Spring AI Alibaba Sandbox containers with PEP 723 dynamic dependencies, resource limits, retries, and automatic cleanup. |
| **Intelligent Report Generation** | Analysis results are automatically summarized into HTML/Markdown reports with ECharts visualizations, WYSIWYG. |
| **Human Feedback Mechanism** | Innovative Human-in-the-loop mechanism, supporting user intervention and adjustments during the plan generation phase. |
| **RAG Retrieval Enhancement** | Integrated vector database, supporting semantic retrieval of business metadata and terminology libraries to improve SQL generation accuracy. |
Expand All @@ -61,12 +61,14 @@ Additionally, this project natively supports **MCP (Model Context Protocol)**, e

## Quick Start

> For detailed installation and configuration guide, please refer to [Quick Start Guide](docs/QUICK_START.md).
> For detailed installation and configuration guidance, see the [Quick Start Guide](docs/QUICK_START-en.md).

### 1. Prerequisites
- JDK 17+
- MySQL 5.7+
- Node.js 16+
- Node.js 22+
- pnpm 11+
- Docker (required when a workflow executes Python steps)

### 2. Start Services

Expand All @@ -75,12 +77,11 @@ Additionally, this project natively supports **MCP (Model Context Protocol)**, e
mysql -u root -p < data-agent-management/src/main/resources/sql/schema.sql

# 2. Start backend
cd data-agent-management
./mvnw spring-boot:run
./mvnw -pl data-agent-management spring-boot:run

# 3. Start frontend
cd data-agent-frontend
npm install && npm run dev
# 3. Start the frontend in another terminal
cd data-agent-frontend-nuxt
pnpm install && pnpm dev
```

### 3. Access the System
Expand All @@ -90,11 +91,11 @@ Open your browser and visit `http://localhost:3000` to start creating your first

| Document | Contents |
| :--- | :--- |
| [Quick Start](docs/QUICK_START.md) | Environment requirements, database import, basic configuration, getting started |
| [Architecture Design](docs/ARCHITECTURE.md) | System layered architecture, StateGraph and workflow design, core module sequence diagrams |
| [Developer Guide](docs/DEVELOPER_GUIDE.md) | Development environment setup, detailed configuration manual, coding standards, extension development (vector DB/models) |
| [Advanced Features](docs/ADVANCED_FEATURES.md) | API Key invocation, MCP server configuration, custom hybrid retrieval strategies, Python executor configuration |
| [Knowledge Configuration Best Practices](docs/KNOWLEDGE_USAGE.md) | Explanation and usage of semantic models, business knowledge, and agent knowledge |
| [Quick Start](docs/QUICK_START-en.md) | Environment requirements, database import, basic configuration, getting started |
| [Architecture Design](docs/ARCHITECTURE-en.md) | System layered architecture, StateGraph and workflow design, core module sequence diagrams |
| [Developer Guide](docs/DEVELOPER_GUIDE-en.md) | Development environment setup, detailed configuration manual, coding standards, extension development (vector DB/models) |
| [Advanced Features](docs/ADVANCED_FEATURES-en.md) | API Key invocation, MCP server configuration, custom hybrid retrieval strategies, SAA Python sandbox, and dynamic dependencies |
| [Knowledge Configuration Best Practices](docs/KNOWLEDGE_USAGE-en.md) | Explanation and usage of semantic models, business knowledge, and agent knowledge |

## Community & Contribution

Expand Down
19 changes: 10 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
</p>

<p>
<a href="https://github.com/alibaba/spring-ai-alibaba"><img src="https://img.shields.io/badge/Spring%20AI%20Alibaba-1.1.0.0-blue" alt="Spring AI Alibaba"></a>
<a href="https://github.com/alibaba/spring-ai-alibaba"><img src="https://img.shields.io/badge/Spring%20AI%20Alibaba-1.1.2.2-blue" alt="Spring AI Alibaba"></a>
<img src="https://img.shields.io/badge/Spring%20Boot-3.4.8+-green" alt="Spring Boot">
<img src="https://img.shields.io/badge/Java-17+-orange" alt="Java">
<img src="https://img.shields.io/badge/License-Apache%202.0-red" alt="License">
Expand Down Expand Up @@ -46,7 +46,7 @@
| 特性 | 说明 |
| :--- | :--- |
| **智能数据分析** | 基于 StateGraph 的 Text-to-SQL 转换,支持复杂的多表查询和多轮对话意图理解。 |
| **Python 深度分析** | 内置 Docker/Local Python 执行器,自动生成并执行 Python 代码进行统计分析与机器学习预测。 |
| **Python 深度分析** | 基于 Spring AI Alibaba Sandbox 在任务级容器中执行生成代码,支持 PEP 723 动态依赖、资源限制、失败重试与自动清理。 |
| **智能报告生成** | 分析结果自动汇总为包含 ECharts 图表的 HTML/Markdown 报告,所见即所得。 |
| **人工反馈机制** | 独创的 Human-in-the-loop 机制,支持用户在计划生成阶段进行干预和调整。 |
| **RAG 检索增强** | 集成向量数据库,支持对业务元数据、术语库的语义检索,提升 SQL生成准确率。 |
Expand All @@ -66,7 +66,9 @@
### 1. 准备环境
- JDK 17+
- MySQL 5.7+
- Node.js 16+
- Node.js 22+
- pnpm 11+
- Docker(工作流需要执行 Python 步骤时必需)

### 2. 启动服务

Expand All @@ -75,12 +77,11 @@
mysql -u root -p < data-agent-management/src/main/resources/sql/schema.sql

# 2. 启动后端
cd data-agent-management
./mvnw spring-boot:run
./mvnw -pl data-agent-management spring-boot:run

# 3. 启动前端
cd data-agent-frontend
npm install && npm run dev
# 3. 在另一个终端启动前端
cd data-agent-frontend-nuxt
pnpm install && pnpm dev
```

### 3. 访问系统
Expand All @@ -93,7 +94,7 @@ npm install && npm run dev
| [快速开始](docs/QUICK_START.md) | 环境要求、数据库导入、基础配置、系统初体验 |
| [架构设计](docs/ARCHITECTURE.md) | 系统分层架构、StateGraph与工作流设计、核心模块时序图 |
| [开发者指南](docs/DEVELOPER_GUIDE.md) | 开发环境搭建、详细配置手册、代码规范、扩展开发(向量库/模型) |
| [高级功能](docs/ADVANCED_FEATURES.md) | API Key 调用、MCP 服务器配置、自定义混合检索策略、Python执行器配置 |
| [高级功能](docs/ADVANCED_FEATURES.md) | API Key 调用、MCP 服务器配置、自定义混合检索策略、SAA Python 沙盒与动态依赖 |
| [知识配置最佳实践](docs/KNOWLEDGE_USAGE.md) | 语义模型,业务知识,智能体知识的解释和使用 |

## 🤝 加入社区 & 贡献
Expand Down
99 changes: 92 additions & 7 deletions docs/ADVANCED_FEATURES-en.md
Original file line number Diff line number Diff line change
Expand Up @@ -386,7 +386,17 @@ WHERE u.name = 'Zhang San'
Python always runs through Spring AI Alibaba Sandbox in a task-scoped container. Host-local,
legacy Docker-pool, and AI Simulation executors are no longer available.

### SAA Sandbox Configuration
### Prerequisites

- Spring AI Alibaba `1.1.2.2`
- A reachable Docker daemon
- The sandbox image available to that Docker daemon
- Network access from the sandbox to the configured Python package index when installing dependencies

SQL-only requests do not create a sandbox. Docker is required only when the execution plan contains
`PYTHON_GENERATE_NODE`.

### Configure the SAA Sandbox

```yaml
spring:
Expand All @@ -397,25 +407,100 @@ spring:
code-timeout: 60s
limit-memory: 500
cpu-core: 1
python-max-tries-count: 5
sandbox:
image-name: ${DATAAGENT_SANDBOX_IMAGE}
docker-host: ${DATAAGENT_SANDBOX_DOCKER_HOST:unix:///var/run/docker.sock}
image-name: ${DATAAGENT_SANDBOX_IMAGE:agentscope-registry.ap-southeast-1.cr.aliyuncs.com/agentscope/runtime-sandbox-base:latest}
max-concurrency: 4
queue-capacity: 10
max-connections: 4096
package-index-url: ${DATAAGENT_PYPI_INDEX_URL:https://pypi.org/simple}
dependency-install-timeout: 3m
```

### Dynamic Dependencies
When `docker-host` is `tcp://host:port`, DataAgent explicitly connects to that endpoint. Other
values use SAA/docker-java local Docker discovery. See
[Developer Guide - Code Executor Configuration](DEVELOPER_GUIDE-en.md#5-code-executor-configuration)
for all size, output, and metadata limits.

### Declare Dynamic Dependencies

```python
# /// script
# dependencies = ["pandas>=2,<3"]
# requires-python = ">=3.10"
# dependencies = ["pandas>=2,<3", "six==1.17.0"]
# ///

import pandas as pd
import six
```

A script may contain at most one PEP 723 `script` block. `dependencies` supports PyPI package
names, extras, and version constraints. URLs, VCS references, local paths, environment markers,
`@` references, and pip options are rejected before a container is created. The current version
parses and preserves `requires-python`, but does not use it to switch or reject the runtime Python
version.

Every non-standard-library import should have a matching dependency. Scripts that use only the
standard library may omit the metadata block.

### Execution Lifecycle

```mermaid
sequenceDiagram
participant W as PythonExecuteNode
participant E as PythonCodeExecutorService
participant S as Task BaseSandbox
participant I as Fixed Installer Bootstrap
participant C as Fixed Code Bootstrap

W->>E: code + stdin JSON + validated dependencies
E->>S: create isolated sandbox
opt dependencies are declared
E->>I: install into /tmp/dataagent-deps
I-->>E: bounded result envelope
end
E->>C: execute code with PYTHONPATH and stdin
C-->>E: bounded stdout/stderr envelope
E-->>W: TaskResponse
E->>S: close and remove container
```

Dependencies are installed inside the task sandbox by a fixed backend command. The model cannot
specify repositories, URLs, file paths, pip options, or shell commands. Production should use an
enterprise private PyPI proxy.
The installer invokes `python -m pip install` with a fixed argument list and never uses a shell.
Code, stdin, the dependency list, and the package index are Base64 encoded before entering the
bootstrap. The dependency directory exists only in the current task container.

### Verification

Submit an analysis request that explicitly requires Python and a third-party dependency. Success
means:

1. The timeline contains Python generation, execution, analysis, and a final report.
2. Python stdout contains the dependency version and business JSON.
3. Backend logs show sandbox creation, dependency installation, code execution, and container removal.
4. `docker ps --format '{{.Names}}' | grep '^dataagent-sandbox-'` prints nothing.

### Troubleshooting

| Symptom | Cause and action |
|---|---|
| Docker connection fails | Run `docker info`; use `tcp://host:port` for a remote daemon and default discovery for local Docker |
| `DependencyInstallError` | Check the package name, version, and index reachability; increase `dependency-install-timeout` or use a private caching proxy for large packages |
| `Unsupported Python dependency specifier` | Remove URLs, VCS references, paths, markers, and pip options; keep only a package name and version constraints |
| `Python sandbox capacity is exhausted` | Concurrency and the wait queue are full; reduce upstream concurrency or adjust `max-concurrency` and `queue-capacity` |
| `Python sandbox task timed out` | Total time exceeded “install timeout + code timeout + 30 seconds”; inspect package download and code duration |
| A container remains after a task | Check whether the application was force-killed; normal success and failure paths close the task sandbox, and application shutdown performs global cleanup |

### Production Security Boundary

Task containers are non-privileged and have CPU, memory, nofile, timeout, concurrency, input, and
output limits. SAA `1.1.2.2` does not expose an API to disconnect the container immediately after
dependency installation, so production must also:

1. Use a pinned image digest and a non-root base image.
2. Allow sandbox access only to an enterprise PyPI proxy; block public internet and business networks.
3. Never inject database, model, OSS, or Docker credentials into the sandbox.
4. Scan packages for malware, CVEs, and license policy at the proxy.

## Advanced Configuration Options

Expand Down
90 changes: 84 additions & 6 deletions docs/ADVANCED_FEATURES.md
Original file line number Diff line number Diff line change
Expand Up @@ -385,7 +385,16 @@ WHERE u.name = '张三'
Python 统一通过 Spring AI Alibaba Sandbox 在任务级独立容器中执行,不再提供宿主机
Local、旧 Docker 容器池或 AI Simulation 执行器。

### SAA 沙盒配置
### 前置条件

- Spring AI Alibaba `1.1.2.2`
- 可访问的 Docker daemon
- 沙盒镜像可从 Docker daemon 获取
- 动态安装依赖时,沙盒网络可访问配置的 Python 包索引

仅执行 SQL 的请求不会创建沙盒;执行计划包含 `PYTHON_GENERATE_NODE` 时才需要 Docker。

### 配置 SAA 沙盒

```yaml
spring:
Expand All @@ -396,24 +405,93 @@ spring:
code-timeout: 60s
limit-memory: 500
cpu-core: 1
python-max-tries-count: 5
sandbox:
image-name: ${DATAAGENT_SANDBOX_IMAGE}
docker-host: ${DATAAGENT_SANDBOX_DOCKER_HOST:unix:///var/run/docker.sock}
image-name: ${DATAAGENT_SANDBOX_IMAGE:agentscope-registry.ap-southeast-1.cr.aliyuncs.com/agentscope/runtime-sandbox-base:latest}
max-concurrency: 4
queue-capacity: 10
max-connections: 4096
package-index-url: ${DATAAGENT_PYPI_INDEX_URL:https://pypi.org/simple}
dependency-install-timeout: 3m
```

### 动态依赖
`docker-host` 使用 `tcp://host:port` 时会显式连接该端点;其他值走 SAA/docker-java 的
本地 Docker 发现。完整大小、输出和元数据限制见
[开发者指南 - 代码执行器配置](DEVELOPER_GUIDE.md#5-代码执行器配置-code-executor)。

### 声明动态依赖

```python
# /// script
# dependencies = ["pandas>=2,<3"]
# requires-python = ">=3.10"
# dependencies = ["pandas>=2,<3", "six==1.17.0"]
# ///

import pandas as pd
import six
```

脚本最多只能包含一个 PEP 723 `script` 块。`dependencies` 支持 PyPI 包名、extras 和
版本约束;URL、VCS、本地路径、环境标记、`@` 引用和 pip 参数会在创建容器前被拒绝。
当前版本会解析并保留 `requires-python`,但不会据此切换或拒绝运行时 Python 版本。

每一个非标准库 `import` 都应有对应依赖。只使用标准库时可以省略元数据块。

### 执行生命周期

```mermaid
sequenceDiagram
participant W as PythonExecuteNode
participant E as PythonCodeExecutorService
participant S as Task BaseSandbox
participant I as Fixed Installer Bootstrap
participant C as Fixed Code Bootstrap

W->>E: code + stdin JSON + validated dependencies
E->>S: create isolated sandbox
opt dependencies are declared
E->>I: install into /tmp/dataagent-deps
I-->>E: bounded result envelope
end
E->>C: execute code with PYTHONPATH and stdin
C-->>E: bounded stdout/stderr envelope
E-->>W: TaskResponse
E->>S: close and remove container
```

依赖由固定的后端命令在当前任务沙盒内安装。模型不能指定仓库、URL、文件路径、pip 参数
或 shell 命令。生产环境应将包索引配置为企业私有 PyPI 代理。
安装器固定执行 `python -m pip install` 参数列表,不经过 shell。代码、stdin、依赖清单
和包索引在进入 bootstrap 前使用 Base64 编码。依赖安装目录只存在于本次任务容器。

### 验证

在数据问答页提交一个明确要求 Python 和第三方依赖的分析任务。成功标准:

1. 时间线出现 Python 生成、执行、分析和最终报告。
2. Python 标准输出包含依赖版本和业务 JSON。
3. 后端日志包含沙盒创建、依赖安装、代码执行和容器删除。
4. `docker ps --format '{{.Names}}' | grep '^dataagent-sandbox-'` 无输出。

### 故障排查

| 现象 | 原因与处理 |
|---|---|
| Docker 连接失败 | 先运行 `docker info`;远程 daemon 使用 `tcp://host:port`,本地 Docker 保持默认发现 |
| `DependencyInstallError` | 检查包名、版本和索引连通性;大包下载超时可增大 `dependency-install-timeout` 或使用私有缓存代理 |
| `Unsupported Python dependency specifier` | 移除 URL、VCS、路径、环境标记或 pip 参数,只保留包名和版本约束 |
| `Python sandbox capacity is exhausted` | 并发数和等待队列已满;降低上游并发或调整 `max-concurrency`、`queue-capacity` |
| `Python sandbox task timed out` | 总时间超过“依赖安装超时 + 代码超时 + 30 秒”;检查包下载或代码耗时 |
| 任务后仍有容器 | 检查应用是否被强制终止;正常成功和失败路径都会关闭任务沙盒,应用关闭还会执行全量清理 |

### 生产安全边界

任务容器默认非 privileged,并限制 CPU、内存、nofile、超时、并发和输入输出大小。但
SAA `1.1.2.2` 没有提供“安装完成后立即断网”的公开 API,因此生产环境还必须:

1. 使用固定镜像 digest 和非 root 基础镜像。
2. 仅允许沙盒访问企业私有 PyPI 代理,禁止访问公网和业务内网。
3. 不向沙盒注入数据库、模型、OSS 或 Docker 凭据。
4. 在代理侧执行恶意包、CVE 和许可证扫描。

## ⚙️ 高级配置选项

Expand Down
Loading
Loading