refactor: enhance schema recall (#108)

* refactor: enhance schema recall
- adjust workflow before plannerNode
- call hybrid search in AgentVectorStoreService
- add aop for log
- clean unsed code

* clean unsed node

* feat: adapt flux output

* add semantic support

* fix: python bug

* fix unit test

* fix TableRelationDispatcher unit test

* fix method buildBusinessKnowledgePrompt

* delete unused code

* add metadata constant

* add semantic support

* update semantic crud cotroller

* fix MappersTest.java

* feat: add semantic frontend

* fix bugs

* feat: add search text-edit

* fix: bugs in schema.sql

* adjust semantic and business knowledge

* adjust by copilot feedback

* fix MapperTest type error.

* refactor: workflow

* chore update README.md

---------

Co-authored-by: VLSMB <2047857654@qq.com>
Co-authored-by: VLSMB <36068250+VLSMB@users.noreply.github.com>

Author: lzq986

README.md

@@ -117,7 +117,13 @@ spring-ai-alibaba-nl2sql/
 
 将表和数据导入到你的MySQL数据库中。
 
-### 2. 配置management数据库
+
+
+### 2. 配置
+
+#### 2.1 配置management数据库
+
+
 
 在`spring-ai-alibaba-nl2sql-management/src/main/resources/application.yml`中配置你的MySQL数据库连接信息。
 
@@ -133,7 +139,9 @@ spring:
     type: com.alibaba.druid.pool.DruidDataSource
 ```
 
-### 2.1 可选：启用/关闭自动初始化（schema.sql + data.sql）
+#### 2.2 可选：启用/关闭自动初始化（schema.sql + data.sql）
+
+
 
 - 默认配置：`application.yml` 中已设置为开启
 
@@ -162,7 +170,7 @@ spring:
 ![Snipaste_2025-09-18_14-35-29.jpg](img/Snipaste_2025-09-18_14-35-29.jpg)
 
 
-### 3. 配置 API Key
+#### 2.3 配置 API Key
 
 ```yaml
 spring:
@@ -178,15 +186,52 @@ spring:
 ```
 
 
-### 4. 启动管理端
+
+#### 2.4 嵌入模型批处理策略配置
+
+
+
+| 属性                                                        | 说明                                                         | 默认值      |
+| ----------------------------------------------------------- | ------------------------------------------------------------ | ----------- |
+| spring.ai.alibaba.nl2sql.embedding-batch.encoding-type      | 文本编码类型，可参考com.knuddels.jtokkit.api.EncodingType    | cl100k_base |
+| spring.ai.alibaba.nl2sql.embedding-batch.max-token-count    | 每批次最大令牌数 值越小，每批次文档越少，但更安全 值越大，处理效率越高，但可能超出API限制 建议值：2000-8000，根据实际API限制调整 | 2000        |
+| spring.ai.alibaba.nl2sql.embedding-batch.reserve-percentage | 预留百分比 用于预留缓冲空间，避免超出限制 建议值：0.1-0.2（10%-20%） | 0.2         |
+| spring.ai.alibaba.nl2sql.embedding-batch.max-text-count     | 每批次最大文本数量 适用于DashScope等有文本数量限制的API DashScope限制为10 | 10          |
+
+
+
+#### 2.5 向量库配置
+
+
+
+| 属性                                                         | 说明                                                         | 默认值    |
+| ------------------------------------------------------------ | ------------------------------------------------------------ | --------- |
+| spring.ai.alibaba.nl2sql.vector-store.similarity-threshold   | 相似度阈值配置，用于过滤相似度分数大于等于此阈值的文档       | 0.2       |
+| spring.ai.alibaba.nl2sql.vector-store.batch-del-topk-limit   | 一次删除操作中，最多删除的文档数量                           | 5000      |
+| spring.ai.alibaba.nl2sql.vector-store.topk-limit             | 查询返回最大文档数                                           | 30        |
+| **spring.ai.alibaba.nl2sql.vector-store.enable-hybrid-search** | 是否启用混合搜索。**注意**：**项目目前默认只提供ES的混合检索能力，<br />如需要扩展其他向量库可自行继承重写 com.alibaba.cloud.ai.service.hybrid.retrieval<br />.AbstractHybridRetrievalStrategy#retrieve 该方法**<br />**并且修改com.alibaba.cloud.ai.service.hybrid.<br />factory.HybridRetrievalStrategyFactory#getObject<br />注册相应的bean** | **false** |
+| spring.ai.alibaba.nl2sql.vector-store.elasticsearch-min-score | Elasticsearch最小分数阈值，用于es执行关键词搜索时过滤相关性较低的文档。<br />**开发时使用的es服务端版本 8.15.0** | 0.5       |
+
+
+
+#### 2.6 检索融合策略
+
+| 属性                                     | 说明                 | 默认值 |
+| ---------------------------------------- | -------------------- | ------ |
+| spring.ai.alibaba.nl2sql.fusion-strategy | 多路召回结果融合策略 | rrf    |
+|                                          |                      |        |
+
+
+
+### 3. 启动管理端
 
 在`spring-ai-alibaba-nl2sql-management`目录下，运行 `spring-ai-alibaba-nl2sql/spring-ai-alibaba-nl2sql-management/src/main/java/com/alibaba/cloud/ai/DataAgentApplication.java` 类。
 
-### 5. 启动WEB页面
+### 4. 启动WEB页面
 
 进入 `spring-ai-alibaba-nl2sql/spring-ai-alibaba-nl2sql-web-ui` 目录
 
-#### 安装依赖
+#### 4.1 安装依赖
 
 
 ```bash
@@ -197,7 +242,7 @@ npm install
 yarn install
 ```
 
-### 启动服务
+#### 4.2 启动服务
 
 ```bash
 # 使用 npm

spring-ai-alibaba-data-agent-chat/pom.xml

@@ -39,6 +39,12 @@
             <artifactId>spring-boot-starter-web</artifactId>
         </dependency>
 
+        <!-- Spring AOP依赖 -->
+        <dependency>
+            <groupId>org.springframework.boot</groupId>
+            <artifactId>spring-boot-starter-aop</artifactId>
+        </dependency>
+
         <dependency>
             <groupId>org.springframework.boot</groupId>
             <artifactId>spring-boot-starter-test</artifactId>
@@ -199,7 +205,10 @@
             <artifactId>elasticsearch-rest-client</artifactId>
             <version>${elasticsearch-client.version}</version>
         </dependency>
-
+        <dependency>
+            <groupId>jakarta.validation</groupId>
+            <artifactId>jakarta.validation-api</artifactId>
+        </dependency>
     </dependencies>
     <repositories>
         <repository>

spring-ai-alibaba-data-agent-chat/src/main/java/com/alibaba/cloud/ai/aop/NodeEntryLoggingAspect.java

@@ -0,0 +1,57 @@
+/*
+ * Copyright 2025 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.cloud.ai.aop;
+
+import com.alibaba.cloud.ai.graph.OverAllState;
+import lombok.extern.slf4j.Slf4j;
+import org.aspectj.lang.JoinPoint;
+import org.aspectj.lang.annotation.Aspect;
+import org.aspectj.lang.annotation.Before;
+import org.aspectj.lang.annotation.Pointcut;
+import org.springframework.stereotype.Component;
+
+/**
+ * AOP切面类，用于记录所有Node类的入口日志
+ *
+ *
+ */
+@Aspect
+@Component
+@Slf4j
+public class NodeEntryLoggingAspect {
+
+	@Pointcut("execution(* com.alibaba.cloud.ai.node..*.apply(com.alibaba.cloud.ai.graph.OverAllState))")
+	public void nodeEntry() {
+	}
+
+	/**
+	 * 在所有实现NodeAction接口的类的apply方法执行前记录日志
+	 * @param joinPoint 连接点
+	 */
+	@Before("nodeEntry()")
+	public void logNodeEntry(JoinPoint joinPoint) {
+		String className = joinPoint.getTarget().getClass().getSimpleName();
+		log.info("Entering {} node", className);
+
+		// 获取方法参数并打印状态信息
+		Object[] args = joinPoint.getArgs();
+		if (args != null && args.length > 0 && args[0] instanceof OverAllState state) {
+			log.debug("State: {}", state);
+		}
+	}
+
+}

spring-ai-alibaba-data-agent-chat/src/main/java/com/alibaba/cloud/ai/config/DataAgentConfiguration.java

@@ -108,22 +108,23 @@ public StateGraph nl2sqlGraph(NodeBeanUtil nodeBeanUtil) throws GraphStateExcept
 			keyStrategyHashMap.put(INPUT_KEY, new ReplaceStrategy());
 			// Agent ID
 			keyStrategyHashMap.put(AGENT_ID, new ReplaceStrategy());
-			// Business knowledge
-			keyStrategyHashMap.put(BUSINESS_KNOWLEDGE, new ReplaceStrategy());
+			// Intent recognition
+			keyStrategyHashMap.put(INTENT_RECOGNITION_NODE_OUTPUT, new ReplaceStrategy());
+			// QUERY_ENHANCE_NODE节点输出
+			keyStrategyHashMap.put(QUERY_ENHANCE_NODE_OUTPUT, new ReplaceStrategy());
 			// Semantic model
-			keyStrategyHashMap.put(SEMANTIC_MODEL, new ReplaceStrategy());
-			// queryWrite节点输出
-			keyStrategyHashMap.put(QUERY_REWRITE_NODE_OUTPUT, new ReplaceStrategy());
-			// keyword extract节点输出
-			keyStrategyHashMap.put(KEYWORD_EXTRACT_NODE_OUTPUT, new ReplaceStrategy());
-			keyStrategyHashMap.put(EVIDENCES, new ReplaceStrategy());
+			keyStrategyHashMap.put(GENEGRATED_SEMANTIC_MODEL_PROMPT, new ReplaceStrategy());
+			// EVIDENCE节点输出
+			keyStrategyHashMap.put(EVIDENCE, new ReplaceStrategy());
 			// schema recall节点输出
 			keyStrategyHashMap.put(TABLE_DOCUMENTS_FOR_SCHEMA_OUTPUT, new ReplaceStrategy());
-			keyStrategyHashMap.put(COLUMN_DOCUMENTS_BY_KEYWORDS_OUTPUT, new ReplaceStrategy());
+			keyStrategyHashMap.put(COLUMN_DOCUMENTS__FOR_SCHEMA_OUTPUT, new ReplaceStrategy());
 			// table relation节点输出
 			keyStrategyHashMap.put(TABLE_RELATION_OUTPUT, new ReplaceStrategy());
 			keyStrategyHashMap.put(TABLE_RELATION_EXCEPTION_OUTPUT, new ReplaceStrategy());
 			keyStrategyHashMap.put(TABLE_RELATION_RETRY_COUNT, new ReplaceStrategy());
+			// Feasibility Assessment 节点输出
+			keyStrategyHashMap.put(FEASIBILITY_ASSESSMENT_NODE_OUTPUT, new ReplaceStrategy());
 			// sql generate节点输出
 			keyStrategyHashMap.put(SQL_GENERATE_SCHEMA_MISSING_ADVICE, new ReplaceStrategy());
 			keyStrategyHashMap.put(SQL_GENERATE_OUTPUT, new ReplaceStrategy());
@@ -160,10 +161,12 @@ public StateGraph nl2sqlGraph(NodeBeanUtil nodeBeanUtil) throws GraphStateExcept
 		};
 
 		StateGraph stateGraph = new StateGraph(NL2SQL_GRAPH_NAME, keyStrategyFactory)
-			.addNode(QUERY_REWRITE_NODE, nodeBeanUtil.getNodeBeanAsync(QueryRewriteNode.class))
-			.addNode(KEYWORD_EXTRACT_NODE, nodeBeanUtil.getNodeBeanAsync(KeywordExtractNode.class))
+			.addNode(INTENT_RECOGNITION_NODE, nodeBeanUtil.getNodeBeanAsync(IntentRecognitionNode.class))
+			.addNode(EVIDENCE_RECALL_NODE, nodeBeanUtil.getNodeBeanAsync(EvidenceRecallNode.class))
+			.addNode(QUERY_ENHANCE_NODE, nodeBeanUtil.getNodeBeanAsync(QueryEnhanceNode.class))
 			.addNode(SCHEMA_RECALL_NODE, nodeBeanUtil.getNodeBeanAsync(SchemaRecallNode.class))
 			.addNode(TABLE_RELATION_NODE, nodeBeanUtil.getNodeBeanAsync(TableRelationNode.class))
+			.addNode(FEASIBILITY_ASSESSMENT_NODE, nodeBeanUtil.getNodeBeanAsync(FeasibilityAssessmentNode.class))
 			.addNode(SQL_GENERATE_NODE, nodeBeanUtil.getNodeBeanAsync(SqlGenerateNode.class))
 			.addNode(PLANNER_NODE, nodeBeanUtil.getNodeBeanAsync(PlannerNode.class))
 			.addNode(PLAN_EXECUTOR_NODE, nodeBeanUtil.getNodeBeanAsync(PlanExecutorNode.class))
@@ -175,13 +178,19 @@ public StateGraph nl2sqlGraph(NodeBeanUtil nodeBeanUtil) throws GraphStateExcept
 			.addNode(SEMANTIC_CONSISTENCY_NODE, nodeBeanUtil.getNodeBeanAsync(SemanticConsistencyNode.class))
 			.addNode(HUMAN_FEEDBACK_NODE, nodeBeanUtil.getNodeBeanAsync(HumanFeedbackNode.class));
 
-		stateGraph.addEdge(START, QUERY_REWRITE_NODE)
-			.addConditionalEdges(QUERY_REWRITE_NODE, edge_async(new QueryRewriteDispatcher()),
-					Map.of(KEYWORD_EXTRACT_NODE, KEYWORD_EXTRACT_NODE, END, END))
-			.addEdge(KEYWORD_EXTRACT_NODE, SCHEMA_RECALL_NODE)
+		stateGraph.addEdge(START, INTENT_RECOGNITION_NODE)
+			.addConditionalEdges(INTENT_RECOGNITION_NODE, edge_async(new IntentRecognitionDispatcher()),
+					Map.of(EVIDENCE_RECALL_NODE, EVIDENCE_RECALL_NODE, END, END))
+			.addEdge(EVIDENCE_RECALL_NODE, QUERY_ENHANCE_NODE)
+			.addConditionalEdges(QUERY_ENHANCE_NODE, edge_async(new QueryEnhanceDispatcher()),
+					Map.of(SCHEMA_RECALL_NODE, SCHEMA_RECALL_NODE, END, END))
 			.addEdge(SCHEMA_RECALL_NODE, TABLE_RELATION_NODE)
 			.addConditionalEdges(TABLE_RELATION_NODE, edge_async(new TableRelationDispatcher()),
-					Map.of(PLANNER_NODE, PLANNER_NODE, END, END, TABLE_RELATION_NODE, TABLE_RELATION_NODE)) // retry
+					Map.of(FEASIBILITY_ASSESSMENT_NODE, FEASIBILITY_ASSESSMENT_NODE, END, END, TABLE_RELATION_NODE,
+							TABLE_RELATION_NODE)) // retry
+			.addConditionalEdges(FEASIBILITY_ASSESSMENT_NODE, edge_async(new FeasibilityAssessmentDispatcher()),
+					Map.of(PLANNER_NODE, PLANNER_NODE, END, END))
+
 			// The edge from PlannerNode now goes to PlanExecutorNode for validation and
 			// execution
 			.addEdge(PLANNER_NODE, PLAN_EXECUTOR_NODE)
@@ -214,16 +223,15 @@ public StateGraph nl2sqlGraph(NodeBeanUtil nodeBeanUtil) throws GraphStateExcept
 			.addConditionalEdges(SQL_EXECUTE_NODE, edge_async(new SQLExecutorDispatcher()),
 					Map.of(SQL_GENERATE_NODE, SQL_GENERATE_NODE, SEMANTIC_CONSISTENCY_NODE, SEMANTIC_CONSISTENCY_NODE))
 			.addConditionalEdges(SQL_GENERATE_NODE, edge_async(new SqlGenerateDispatcher()),
-					Map.of(KEYWORD_EXTRACT_NODE, KEYWORD_EXTRACT_NODE, END, END, SQL_EXECUTE_NODE, SQL_EXECUTE_NODE))
+					Map.of(FEASIBILITY_ASSESSMENT_NODE, FEASIBILITY_ASSESSMENT_NODE, END, END, SQL_EXECUTE_NODE,
+							SQL_EXECUTE_NODE))
 			.addConditionalEdges(SEMANTIC_CONSISTENCY_NODE, edge_async(new SemanticConsistenceDispatcher()),
 					Map.of(SQL_GENERATE_NODE, SQL_GENERATE_NODE, PLAN_EXECUTOR_NODE, PLAN_EXECUTOR_NODE));
 
 		GraphRepresentation graphRepresentation = stateGraph.getGraph(GraphRepresentation.Type.PLANTUML,
 				"workflow graph");
 
-		log.info("\n\n");
-		log.info(graphRepresentation.content());
-		log.info("\n\n");
+		log.info("workflow in PlantUML format as follows \n\n" + graphRepresentation.content() + "\n\n");
 
 		return stateGraph;
 	}

spring-ai-alibaba-data-agent-chat/src/main/java/com/alibaba/cloud/ai/config/DataAgentProperties.java

@@ -37,6 +37,8 @@ public class DataAgentProperties {
 	 */
 	private EmbeddingBatch embeddingBatch = new EmbeddingBatch();
 
+	private VectorStoreProperties vectorStore = new VectorStoreProperties();
+
 	@Getter
 	@Setter
 	public static class EmbeddingBatch {
@@ -63,4 +65,35 @@ public static class EmbeddingBatch {
 
 	}
 
+	@Getter
+	@Setter
+	public static class VectorStoreProperties {
+
+		/**
+		 * 相似度阈值配置，用于过滤相似度分数大于等于此阈值的文档
+		 */
+		private double similarityThreshold = 0.2;
+
+		/**
+		 * 一次删除操作中，最多删除的文档数量
+		 */
+		private int batchDelTopkLimit = 5000;
+
+		/**
+		 * 查询时返回的最大文档数量
+		 */
+		private int topkLimit = 30;
+
+		/**
+		 * 是否启用混合搜索
+		 */
+		private boolean enableHybridSearch = false;
+
+		/**
+		 * Elasticsearch最小分数阈值，用于es执行关键词搜索时过滤相关性较低的文档
+		 */
+		private double elasticsearchMinScore = 0.5;
+
+	}
+
 }

spring-ai-alibaba-data-agent-chat/src/main/java/com/alibaba/cloud/ai/constant/Constant.java

@@ -35,27 +35,27 @@ private Constant() {
 
 	public static final String NL2SQL_GRAPH_NAME = "nl2sqlGraph";
 
-	public static final String QUERY_REWRITE_NODE_OUTPUT = "QUERY_REWRITE_NODE_OUTPUT";
+	public static final String INTENT_RECOGNITION_NODE_OUTPUT = "INTENT_RECOGNITION_NODE_OUTPUT";
 
-	public static final String KEYWORD_EXTRACT_NODE_OUTPUT = "KEYWORD_EXTRACT_NODE_OUTPUT";
+	public static final String QUERY_ENHANCE_NODE_OUTPUT = "QUERY_ENHANCE_NODE_OUTPUT";
 
-	public static final String EVIDENCES = "EVIDENCES";
+	public static final String FEASIBILITY_ASSESSMENT_NODE_OUTPUT = "FEASIBILITY_ASSESSMENT_NODE_OUTPUT";
+
+	public static final String EVIDENCE = "EVIDENCE";
 
 	public static final String TABLE_DOCUMENTS_FOR_SCHEMA_OUTPUT = "TABLE_DOCUMENTS_FOR_SCHEMA";
 
 	public static final String SCHEMA_RECALL_NODE_OUTPUT = "SCHEMA_RECALL_NODE_OUTPUT";
 
-	public static final String COLUMN_DOCUMENTS_BY_KEYWORDS_OUTPUT = "COLUMN_DOCUMENTS_BY_KEYWORDS_OUTPUT";
+	public static final String COLUMN_DOCUMENTS__FOR_SCHEMA_OUTPUT = "COLUMN_DOCUMENTS__FOR_SCHEMA_OUTPUT";
 
 	public static final String TABLE_RELATION_OUTPUT = "TABLE_RELATION_OUTPUT";
 
 	public static final String TABLE_RELATION_EXCEPTION_OUTPUT = "TABLE_RELATION_EXCEPTION_OUTPUT";
 
 	public static final String TABLE_RELATION_RETRY_COUNT = "TABLE_RELATION_RETRY_COUNT";
 
-	public static final String BUSINESS_KNOWLEDGE = "BUSINESS_KNOWLEDGE";
-
-	public static final String SEMANTIC_MODEL = "SEMANTIC_MODEL";
+	public static final String GENEGRATED_SEMANTIC_MODEL_PROMPT = "GENEGRATED_SEMANTIC_MODEL_PROMPT";
 
 	public static final String SQL_GENERATE_OUTPUT = "SQL_GENERATE_OUTPUT";
 
@@ -93,6 +93,14 @@ private Constant() {
 
 	public static final String PLAN_EXECUTOR_NODE = "PLAN_EXECUTOR_NODE";
 
+	public static final String INTENT_RECOGNITION_NODE = "INTENT_RECOGNITION_NODE";
+
+	public static final String EVIDENCE_RECALL_NODE = "EVIDENCE_RECALL_NODE";
+
+	public static final String QUERY_ENHANCE_NODE = "QUERY_ENHANCE_NODE";
+
+	public static final String FEASIBILITY_ASSESSMENT_NODE = "FEASIBILITY_ASSESSMENT_NODE";
+
 	public static final String QUERY_REWRITE_NODE = "QUERY_REWRITE_NODE";
 
 	public static final String REPORT_GENERATOR_NODE = "REPORT_GENERATOR_NODE";
@@ -144,16 +152,4 @@ private Constant() {
 	// 人类复核相关
 	public static final String HUMAN_REVIEW_ENABLED = "HUMAN_REVIEW_ENABLED";
 
-	// column
-	public static final String COLUMN = "column";
-
-	// table
-	public static final String TABLE = "table";
-
-	// vectorType
-	public static final String VECTOR_TYPE = "vectorType";
-
-	// knowledgeId
-	public static final String KNOWLEDGE_ID = "knowledgeId";
-
 }