Skip to content

fix(api): write built-in document metadata in a single format - #44218

Open
cty-ut wants to merge 1 commit into
langgenius:mainfrom
cty-ut:fix/built-in-metadata-format
Open

cty-ut wants to merge 1 commit into
langgenius:mainfrom
cty-ut:fix/built-in-metadata-format

Conversation

@cty-ut

@cty-ut cty-ut commented Oct 16, 2026 •

Copy link
Copy Markdown

Important

  1. Make sure you have read our contribution guidelines
  2. Ensure there is an associated issue
  3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

Fixes #44217

Documents created after built-in metadata was enabled stored upload_date / last_update_date as "%Y-%m-%d %H:%M:%S" strings and source as the raw data_source_type. Enabling built-in fields and batch metadata edits store epoch seconds and the MetadataDataSource label instead. Time filters cast the value to FLOAT, so on PostgreSQL one mismatched document made the whole retrieval query fail. source filters silently skipped those documents.

Changes:

  • Add build_built_in_field_values() in models/dataset.py, next to the Document model that owns doc_metadata. All three writers now call it:
    • SQLAlchemyMetadataRepository._built_in
    • DocumentService.build_document
    • PipelineGenerator._build_document
  • Convert time fields with the existing libs.datetime_utils.to_utc_timestamp. Behaviour is unchanged under the app (ext_timezone forces TZ=UTC), but the helper no longer depends on the process timezone, so it gives the same result in tests and scripts.
  • Why models/ and not services/: pipeline_generator lives in core, and adding a new core -> services entry to the .importlinter baseline would go against its "keep every exception exact" rule.

Notes:

  • Existing rows are not migrated. Disabling and re-enabling built-in fields for a knowledge base rewrites them in the new format.
  • Time values are now stored as int instead of float. Filters cast to FLOAT, so filtering behaviour does not change.

Testing

  • New tests/unit_tests/services/knowledge/test_metadata_built_in_fields.py:
    • epoch conversion does not depend on the process TZ
    • source mapping for all six data source types
    • build_document output matches the shared helper
    • no built-in fields when the feature is disabled
  • test_build_document_sets_metadata_for_builtin_fields now asserts the exact values instead of only checking that doc_metadata is truthy.
  • ruff format / ruff check on the changed files, lint-imports (68 kept, 0 broken), api-contract-lint, and pyrefly on the changed files all pass.
  • Unit tests under service_api, console datasets, services/knowledge, pipeline, repositories/knowledge and models: 2318 passed.

Screenshots

N/A (backend only)

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've verified the change and added or updated tests where meaningful regression risk justifies coverage.
  • I've updated the documentation accordingly.
  • I ran make lint && make type-check (backend) and vp staged (frontend) to appease the lint gods

🤖 Generated with Claude Code

From Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Oct 16, 2026
@ghfind-review ghfind-review Bot added the review: medium ghfind author score; see https://ghfind.com label Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review: medium ghfind author score; see https://ghfind.com size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Built-in metadata written at document creation uses a different format, breaking time filters on upload_date / last_update_date

1 participant