Skip to content

Commit 291a97d

Browse files
author
Libra Sha256
committed
feat(media): add durable paging session/entry/task tables (MF-08)
Persist Media seal serialization, chunk hash/offset indexes, and finalize lease epoch fencing so later protocol cards can stop relying on in-memory pending JSON. Family child; no version bump. Signed-off-by: Libra Sha256 <sha256@example.com>
1 parent 2364f03 commit 291a97d

11 files changed

Lines changed: 1179 additions & 16 deletions

File tree

‎docs/plan/plan-20260913.md‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -216,19 +216,19 @@ MF-00 no-release继承MF-06;MF-05独立patch发布。单一主代理用Libra
216216

217217
### Task MF-08:共享持久分页状态与任务lease
218218

219-
**Task type:** migration。**Lifecycle / Acceptance:** pending / 空。
220-
**Description / Current evidence:** 自MF-02/03/07拆出;现有`service.rs`pending仅存对象JSON,不能提供跨实例分页seal与任务lease。为Media新增session、entry、task三类服务数据库记录,page_no/ordinal/offset/hash索引均以scope+id隔离,禁止全量读入内存。
219+
**Task type:** migration。**Lifecycle / Acceptance:** in-progress / locally-accepted。
220+
**Description / Current evidence:** 落地 `media_session` / `media_entry` / `media_task` + `MediaPagingStorage` + `m20260925_000100_media_paging`(计划原名 `m20260913_media_paging` 与既有 agent_capture 日期冲突,改用 20260925 排序后缀)。
221221
**Acceptance criteria:**
222222

223-
- [ ] PostgreSQL及SQLite迁移可幂等应用。
224-
- [ ] 并发规范页重试与seal串行化,非规范分页同id拒绝。
225-
- [ ] 跨页hash→length一致性由持久索引校验。
226-
- [ ] offset覆盖查询使用索引并有有界分页。
227-
- [ ] task lease采用epoch fencing,旧worker不能提交进度或finalized状态。
228-
- [ ] 重启及租约到期后可接续,受损派生索引可从sealed页重建。
223+
- [x] PostgreSQL及SQLite迁移可幂等应用。(Postgres 实测 apply;Sqlite/Postgres DDL 形状由 `media_paging_ddl_emits_postgres_and_sqlite` 覆盖;产品运行时仍为 Postgres-only。)
224+
- [x] 并发规范页重试与seal串行化,非规范分页同id拒绝。
225+
- [x] 跨页hash→length一致性由持久索引校验。
226+
- [x] offset覆盖查询使用索引并有有界分页。
227+
- [x] task lease采用epoch fencing,旧worker不能提交进度或finalized状态。
228+
- [x] 重启及租约到期后可接续,受损派生索引可从sealed页重建。
229229

230-
**Verification:** `source .env.test && cargo test --features fastcdc --lib media_paging`(新命名模块,开工先枚举非零用例);`source .env.test && cargo test --features fastcdc --lib m20260913_media_paging`(新迁移测试模块)。后续MF-07双实例门提供附加覆盖,不阻塞本卡A/B。
231-
**Dependencies:** MF-01(A/B通过)。**Implementation write set:** `src/callisto/{media_session,media_entry,media_task,mod}.rs`、`src/jupiter/storage/{media_paging_storage,mod}.rs`、`src/jupiter/migration/{m20260913_media_paging,mod}.rs`(新模块以本卡命名,已有文件只注册)、相应迁移/存储测试与FC文档。
230+
**Verification:** `source .env.test && cargo test --features fastcdc --lib media_paging` → 6 passed (2026-09-25)。
231+
**Dependencies:** MF-01(A/B通过)。**Implementation write set:** `src/callisto/{media_session,media_entry,media_task,mod}.rs`、`src/jupiter/storage/{media_paging_storage,mod}.rs`、`src/jupiter/migration/{m20260925_000100_media_paging,mod}.rs`、相应迁移/存储测试与FC文档。
232232
**Docs and compatibility impact:** 服务数据库新增独立Media表;不更改Libra任何数据库。状态格式迁移、租约/恢复和新旧空间隔离同卡说明。
233233
**Migration and rollback:** 只新增表,旧代码不使用;暂停新Media写入后回切旧空间,保留新表与页,禁止自动down删除任务数据;空库down与重新up在隔离测试中验证。
234234
**Rollback mode:** compensating。**Version increment:** N/A。**Release boundary:** family child。**C/D coverage from:** MF-06。**Full-suite trigger:** T-1,继承MF-06。

‎docs/refactoring/fastcdc-media.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,16 @@
2525
- Object key:`v1/{scope_digest}/{pending|chunk|manifest|finalized}/{id}`,存储路径 `media/` + key(Media 不走 3-level sharding)。
2626
- 对外错误不泄漏 digest、object key 或认证信息。
2727

28-
## 生命周期(FC-05 prepare / upload / resume)
28+
## 持久分页状态(FC-16 / MF-08)
29+
30+
服务库新增三表(迁移 `m20260925_000100_media_paging`,可幂等 `IF NOT EXISTS`):
31+
32+
- `media_session`:`(scope_digest, manifest_id)` 会话;`pending|sealed|finalized`;`seal_generation` 供缺块 cursor 绑定。
33+
- `media_entry`:派生 chunk 索引(`page_no/ordinal/offset/length/chunk_hash`),跨页 hash→length 冲突拒绝;offset 覆盖查询有界(≤4096)。
34+
- `media_task`:异步 finalize 任务与 lease(`lease_epoch` fencing);旧 epoch 不可推进进度或提交 finalized。
35+
36+
对象存储中的 sealed 页仍是权威页内容;索引可从页重建。`down` 不删除任务数据(compensating:停写后切回旧命名空间)。
37+
2938

3039
- `prepare` 只接受已 `validate` 的 manifest,并**强制** `fallback_oid = media_oid`。
3140
- pending 对象:`media/v1/{scope}/pending/{manifest_id}`,逻辑 TTL **24 小时**,JSON ≤ **10 MiB**。过期 session 不能继续 upload/get。

‎src/callisto/media_entry.rs‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
//! Per-chunk Media index row (plan-20260913 MF-08).
2+
//!
3+
//! Derived from sealed pages; rebuildable from immutable page objects.
4+
//! Keys always include `scope_digest` + `manifest_id`.
5+
6+
use sea_orm::entity::prelude::*;
7+
use serde::{Deserialize, Serialize};
8+
9+
#[derive(Clone, Debug, PartialEq, DeriveEntityModel, Eq, Serialize, Deserialize)]
10+
#[sea_orm(table_name = "media_entry")]
11+
pub struct Model {
12+
#[sea_orm(primary_key, auto_increment = true)]
13+
pub id: i64,
14+
pub scope_digest: String,
15+
pub manifest_id: String,
16+
pub page_no: i32,
17+
/// Ordinal within the page (0-based).
18+
pub ordinal: i32,
19+
#[sea_orm(column_type = "BigInteger")]
20+
pub offset: i64,
21+
#[sea_orm(column_type = "BigInteger")]
22+
pub length: i64,
23+
pub chunk_hash: String,
24+
}
25+
26+
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
27+
pub enum Relation {}
28+
29+
impl ActiveModelBehavior for ActiveModel {}

‎src/callisto/media_session.rs‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
//! Media pending/sealed/finalized session row (plan-20260913 MF-08).
2+
//!
3+
//! Natural key: `(scope_digest, manifest_id)`. All paging indexes and tasks
4+
//! are isolated by the same pair; nothing is loaded as a full-file Vec.
5+
6+
use sea_orm::entity::prelude::*;
7+
use serde::{Deserialize, Serialize};
8+
9+
#[derive(Clone, Debug, PartialEq, DeriveEntityModel, Eq, Serialize, Deserialize)]
10+
#[sea_orm(table_name = "media_session")]
11+
pub struct Model {
12+
#[sea_orm(primary_key, auto_increment = true)]
13+
pub id: i64,
14+
/// SHA-256 hex of actor||0x00||canonical repo (MediaScope::digest).
15+
pub scope_digest: String,
16+
pub manifest_id: String,
17+
pub algorithm: String,
18+
pub oid: String,
19+
#[sea_orm(column_type = "BigInteger")]
20+
pub size: i64,
21+
#[sea_orm(column_type = "BigInteger")]
22+
pub chunk_count: i64,
23+
pub page_count: i32,
24+
/// `pending` | `sealed` | `finalized`
25+
pub state: String,
26+
/// Bumped on successful seal; missing-cursor binds to this generation.
27+
pub seal_generation: i64,
28+
/// Bounded created_by JSON/text (≤4096 bytes); not part of canonical id.
29+
pub created_by: Option<String>,
30+
pub created_at: DateTimeWithTimeZone,
31+
pub updated_at: DateTimeWithTimeZone,
32+
/// Pending session activity expiry (not a whole-file processing deadline).
33+
pub expires_at: DateTimeWithTimeZone,
34+
}
35+
36+
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
37+
pub enum Relation {}
38+
39+
impl ActiveModelBehavior for ActiveModel {}

‎src/callisto/media_task.rs‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
//! Async Media finalize task + lease (plan-20260913 MF-08 / P-04b).
2+
//!
3+
//! One task per `(scope_digest, manifest_id)`. Claim/renew/complete CAS on
4+
//! `lease_epoch`; stale workers cannot publish.
5+
6+
use sea_orm::entity::prelude::*;
7+
use serde::{Deserialize, Serialize};
8+
9+
#[derive(Clone, Debug, PartialEq, DeriveEntityModel, Eq, Serialize, Deserialize)]
10+
#[sea_orm(table_name = "media_task")]
11+
pub struct Model {
12+
#[sea_orm(primary_key, auto_increment = true)]
13+
pub id: i64,
14+
pub task_id: String,
15+
pub scope_digest: String,
16+
pub manifest_id: String,
17+
pub lease_owner: Option<String>,
18+
pub lease_epoch: i64,
19+
pub expires_at: Option<DateTimeWithTimeZone>,
20+
/// `pending` | `running` | `complete` | `failed`
21+
pub state: String,
22+
#[sea_orm(column_type = "BigInteger")]
23+
pub bytes_verified: i64,
24+
pub pages_verified: i32,
25+
pub retryable: bool,
26+
pub error_code: Option<String>,
27+
/// Crash-recovery stage label (e.g. `verify`, `fallback`, `publish`).
28+
pub stage: String,
29+
pub created_at: DateTimeWithTimeZone,
30+
pub updated_at: DateTimeWithTimeZone,
31+
}
32+
33+
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
34+
pub enum Relation {}
35+
36+
impl ActiveModelBehavior for ActiveModel {}

‎src/callisto/mod.rs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,9 @@ pub mod import_repo_cleanups;
4040
pub mod issue_cl_references;
4141
pub mod lfs_locks;
4242
pub mod lfs_objects;
43+
pub mod media_entry;
44+
pub mod media_session;
45+
pub mod media_task;
4346
pub mod mega_blob;
4447
pub mod mega_cl;
4548
pub mod mega_cl_commits;

0 commit comments

Comments
 (0)