Skip to content

feat: KnowledgeFS and New Knowledge - #44165

Draft
hyoban wants to merge 1194 commits into
mainfrom
deploy/konwledge
Draft

hyoban wants to merge 1194 commits into
mainfrom
deploy/konwledge

Conversation

@hyoban

@hyoban hyoban commented Oct 15, 2026

Copy link
Copy Markdown
Member

Summary

Integration branch for KnowledgeFS and the New Knowledge experience (backend proxy/services plus the web/features/new-rag frontend).

Main dropped its outdated copy of this feature in #44163. This branch carries the current version, and its latest main merge records that commit without applying its deletions.

Screenshots

Before After
... ...

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've verified the change and added or updated tests where meaningful regression risk justifies coverage.
  • I've updated the documentation accordingly.
  • I ran make lint && make type-check (backend) and vp staged (frontend) to appease the lint gods

From Claude Code

🤖 Generated with Claude Code

hyoban and others added 30 commits September 8, 2026 12:47
Safely normalize semantic chunk boundaries with bounded retries and accurate error classification. Synchronize failed processing tasks and overview attention states.

Allow safe retrieval model changes before publication, create fresh compilation attempts when profiles change, and preserve explicit concurrency and revision-conflict safeguards.
…nwledge

Preserve import recovery and model-settings fixes alongside upstream knowledge naming changes. Regenerate the KnowledgeFS contract lock and verify the merged API contracts.
# Conflicts:
#	.agents/skills/how-to-write-component/references/data.md
#	web/app/components/workflow-app/hooks/__tests__/use-available-nodes-meta-data.spec.ts
#	web/app/components/workflow-app/hooks/use-available-nodes-meta-data.ts
#	web/features/agent-v2/agent-detail/configure/components/__tests__/agent-prompt-editor.spec.tsx
#	web/features/agent-v2/agent-detail/configure/components/orchestrate/prompt-editor/index.tsx
#	web/features/new-rag/__tests__/add-source-page.spec.tsx
#	web/features/new-rag/__tests__/crawl-selection-form.spec.tsx
#	web/features/new-rag/__tests__/create-knowledge-page.spec.tsx
#	web/features/new-rag/__tests__/document-detail-page.spec.tsx
#	web/features/new-rag/__tests__/knowledge-space-shell.spec.tsx
#	web/features/new-rag/__tests__/new-knowledge-list.spec.tsx
#	web/features/new-rag/__tests__/processing-task-events.spec.ts
#	web/features/new-rag/__tests__/sources-page.spec.tsx
#	web/features/new-rag/add-source-page.tsx
#	web/features/new-rag/crawl-selection-form.tsx
#	web/features/new-rag/create-knowledge-page.tsx
#	web/features/new-rag/create-knowledge-workflow.ts
#	web/features/new-rag/document-detail-page.tsx
#	web/features/new-rag/document-detail-queries.ts
#	web/features/new-rag/documents-page.tsx
#	web/features/new-rag/knowledge-space-shell.tsx
#	web/features/new-rag/new-knowledge-list.tsx
#	web/features/new-rag/processing-tasks-drawer.tsx
#	web/features/new-rag/services/processing-task-events.ts
#	web/features/new-rag/sources-page.tsx
#	web/features/new-rag/sources/create/website-crawl-preview.tsx
#	web/features/new-rag/use-document-reindex.ts
#	web/features/new-rag/use-document-task-status.ts
#	web/service/console-openapi-url.ts
#	web/service/console-router-loader.ts
#	web/service/console/client.spec.ts
#	web/service/console/query-policies.ts
Accept bounded optional file metadata and cache publication-verified download receipts for conservative unchanged-file checks. Preserve the existing download and hash comparison path for legacy plugins.

This is platform-side groundwork; metadata-aware SDK, daemon and provider support is still required for the standard plugin chain. Update the KnowledgeFS contract lock and cover metadata parsing and verification.
hyoban and others added 3 commits October 16, 2026 11:42
The interactive mock boundary forbids mocking Dify UI primitives; assert the
Escape-close and draft reset behavior through the real Popover instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@lyzno1 lyzno1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for the remaining frontend state-loss, access-state, async-session, cache invalidation, pagination, and accessibility defects detailed inline. The comments are anchored to b0d81094d5d7ac2b1a37f46d5a4cabc1226317b2; they do not repeat the original frontend issues already addressed in the previous review. Minor violations of explicit Dify UI/Web component contracts are marked P3 separately.

Validation covered 1,265 passing existing focused tests and 19 targeted defect probes across the reviewed frontend areas. Relevant keyboard/focus interactions were also checked in Chromium. Probes use real production owners with mocked external responses, with generated Query keys tested against a real QueryClient where applicable; this was not a complete live-provider/backend E2E run. The simulated-DOM numeric-validation candidate and the custom-interval focus candidate were excluded after browser verification.

Comment thread web/features/new-rag/settings/draft.ts Outdated
Comment thread web/features/new-rag/list/knowledge-space-card-tags.tsx Outdated
Comment thread web/features/new-rag/quality/golden-question-import-dialog.tsx Outdated
Comment thread web/features/new-rag/quality/golden-question-dialog.tsx
Comment thread web/features/new-rag/sources/create/connected-source-workflow.tsx
Comment thread web/features/new-rag/quality/golden-questions-panel.tsx Outdated
Comment thread web/features/new-rag/sources/create/connected-source-workflow.tsx Outdated
Comment thread web/features/new-rag/quality/golden-question-dialog.tsx Outdated
Comment thread web/features/new-rag/quality/quality-row-menu-trigger.tsx Outdated
Comment thread web/app/components/datasets/creator-filter.tsx Outdated
hjlarry and others added 23 commits October 16, 2026 12:14
Adopt main's generated oRPC console calls (#44169) in branch code: the deleted
use-dataset/use-datasource/useMembers hooks become consoleQuery queries, the
KnowledgeFS dataset list, scope dialog and card header keep their branch
behavior, and branch specs mock the new operations instead of the removed
modules. Contracts, OpenAPI markdown and oxlint suppressions are regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er settings mutations

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Freeze the default newest-record selection once the composer is edited so
  history that resolves later cannot replace an unsent draft; explicit record
  selection still switches the composer.
- Treat restored query images as retest readiness so image-only traces retest.
- Gate bad-case creation on the trace id instead of query text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stop the import continuation once the creation session is disposed, defer
cleanup until a pending submission settles and delete against the last
confirmed source revision. Give creation and the custom sync interval editor
real form submission boundaries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed status read kept the job handle but a retry overwrote it, leaving
the first crawl running. Release the held job before starting a new one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep the latest submitted tag selection in the card until queued replacements
and the cache sync finish, so the picker never reopens from stale tags and
replacements cannot complete out of order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Bind CSV reads to the latest selection, withdraw stale rows while reading,
  and lock file replacement during import.
- Freeze the golden-question draft while a save is pending and show pending
  through Button loading.
- Reuse the evaluation idempotency key for unknown-outcome retries.
- Invalidate golden-question pages and the active count with list mutations.
- Render golden-question, bad-case and evaluation lists as native tables.
- Leave trigger focus styling to Dify UI primitives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e and surface its failures

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Represent a running crawl through Button loading with a stable labelled name,
and drop duplicate focus ring classes from the source actions trigger and the
source sort button so the primitives own their focus indicator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd dataset list

- Track variables referenced by Knowledge Retrieval v2 metadata-filter values in
  getNodeUsedVars, rename/removal, server pass keys, and single-run inputs.
- Keep the knowledge-space removal action in the tab order and reveal it on focus.
- Paginate the API key scope picker sources and show retryable per-source errors
  instead of an empty result.
- Use the shared SearchInput for the dataset creator filter.
- Prove the generated dataset list key is what creation/settings flows invalidate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e list spec

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hyoban and others added 2 commits October 17, 2026 09:32
Keep the branch's dataset card types and agent knowledge dialog against main's
alias removal (#44225), moving remaining branch uses of the removed aliases to
the types main adopted. Keep the external API endpoint validation in main's
dialog anatomy. Branch dialogs that predate main's Dialog/AlertDialog
restyling contract are added to the oxlint suppression baseline, as main did
for its own legacy dialogs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adopt main's real-behavior test setup (#44243) in the conflicted specs: the
rag-pipeline node list and agent prompt editor use the real nodes and main's
render helper, with the KnowledgeFS flags seeded explicitly; the citation popup
uses main's console responses plus the branch's logical-document mocks. The
datasets list spec keeps the branch's rewritten version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review: top ghfind author score; see https://ghfind.com size:XXL This PR changes 1000+ lines, ignoring generated files. web This relates to changes on the web.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants