Skip to content

Commit 95a61fd

Browse files
author
Libra Sha256
committed
feat(media): MF-07 multi-layout atomic publication
Publish immutable finalized-by-manifest_id via put_metadata_atomic, keep by-media replaceable, and accept concurrent same-oid layouts without 409. Signed-off-by: Libra Sha256 <sha256@example.com>
1 parent 55c3bb0 commit 95a61fd

14 files changed

Lines changed: 1696 additions & 159 deletions

File tree

‎docs/plan/plan-20260913.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -252,15 +252,15 @@ MF-00 no-release继承MF-06;MF-05独立patch发布。单一主代理用Libra
252252

253253
### Task MF-07:多布局原子发布
254254

255-
**Task type:** implementation。**Lifecycle / Acceptance:** pending / 空。
256-
**Description / Current evidence:** 自MF-03拆出;修复`finalize.rs:205-215`不同id同oid409与非原子检查后写入。
255+
**Task type:** implementation。**Lifecycle / Acceptance:** locally-accepted / locally-accepted。
256+
**Description / Current evidence:** 自MF-03拆出;`publication.rs` + `put_metadata_atomic`;immutable key=`(scope,manifest_id)`;by-media 可替换;同oid多布局并发与故障窗口已测。
257257
**Acceptance criteria:**
258-
- [ ] 同oid不同布局并发finalize均成功。
259-
- [ ] 同id重试返回等价canonical身份记录。
260-
- [ ] by-media只暴露完整已finalized布局。
261-
- [ ] C-04每个发布故障窗口可重试收敛。
262-
- [ ] 本地与S3模式均不暴露半写记录。
263-
**Verification:** `source .env.test && cargo test --features fastcdc --lib ceres::lfs::media`;`source .env.test && cargo test --features fastcdc --test integration_fastcdc_libra`(本仓新增发布故障/并发用例不设ignored)。
258+
- [x] 同oid不同布局并发finalize均成功。
259+
- [x] 同id重试返回等价canonical身份记录。
260+
- [x] by-media只暴露完整已finalized布局。
261+
- [x] C-04每个发布故障窗口可重试收敛。
262+
- [x] 本地与S3模式均不暴露半写记录。
263+
**Verification:** `source .env.test && cargo test --features fastcdc --lib ceres::lfs::media` → 34 passed;`media_publication_ -- --list` → `media_publication_local`/`media_publication_s3` 各一次且非ignored;两用例 `--exact` 各 `1 passed, 0 failed, 0 ignored`(2026-09-25)。
264264
**Dependencies:** MF-03(A/B通过)。**Implementation write set:** `src/ceres/lfs/media/{finalize,service,scope,publication,mod}.rs`(publication新);`src/jupiter/storage/lfs_db_storage.rs`(fallback同oid幂等);`src/orbit_api/object_storage.rs`、`src/orbit/adapter/{object,mod}.rs`(有界原子metadata接口与适配);强制同步集。
265265
**Docs and compatibility impact:** FC提交点、多布局、幂等及旧空间恢复。**Rollback mode:** compensating。**Version increment:** N/A。**Release boundary:** family child。**C/D coverage from:** MF-06。**Full-suite trigger:** 继承MF-06。
266266
**Granularity:** `type=implementation; axis=multi-layout publication; recovery=compensating; complete=yes; self-contained=yes; AC=5/8; VER=2/8; landing=4; prod-files=9; scope=M; deps=MF-03; writeset=serial; release=family child; split-from=MF-03; exception=N/A`

‎docs/refactoring/fastcdc-media.md‎

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,9 @@
4444
- Scope = 服务端 actor + canonical 绝对仓库路径;digest = SHA-256(`actor || 0x00 || repo`)。
4545
Actor 取 `AccessTokenUser` 的 `website_user_id`(非空)否则 `username`。请求体 actor/repository/`created_by` 不得覆盖。
4646
- Object key:`fastcdc-v2020-32k/{scope_digest}/{pending|chunk|manifest|finalized|page}/{id}`,存储路径 `media/` + key(Media 不走 3-level sharding)。旧 `v1/` 对象隔离保留。
47+
- `finalized/{manifest_id}`:不可变身份(完整 `ManifestResponse`)
48+
- `manifest/{media_oid}`:可替换 by-media 发现
49+
- `page/{manifest_id}-p{n}`:不可变页 blob
4750
- 对外错误不泄漏 digest、object key 或认证信息。
4851

4952
## 持久分页状态(FC-16 / MF-08)
@@ -64,21 +67,28 @@
6467
- chunk 上传必须命中 pending 声明的 hash/length,并校验实际 SHA-256(≤ 256 KiB)。同 scope 同 hash 的正确对象可幂等复用;已存错误内容返回 Conflict。
6568
- 领域错误:`Invalid` / `NotFound` / `Conflict` / `Storage` / `Io` / `Json`。存储错误对外固定为 `media object store error`。
6669

67-
## Finalize / fallback(FC-06 / MF-03)
70+
## Finalize / fallback / multi-layout publication(FC-06 / MF-03 / MF-07)
6871

6972
- 同时最多 **2** 个 finalize(semaphore);活跃排队上限 **128**(超限 429 + `Retry-After`)。
7073
- `POST …/manifests/{id}/finalize` **仅接受 sealed 会话**,返回 **202** `{task_id,manifest_id,state,status_url}`;`status_url` 为同前缀 `libra/media/v1/tasks/{task_id}`(客户端不得跨 origin 跟随或转发 token)。
7174
- `GET …/tasks/{task_id}` 每请求重新做 scope 认证,返回 `state/bytes_verified/pages_verified/retryable/error_code`;`complete` 另含 `oid/size`。
7275
- 验证:按 sealed 页序遍历 `media_entry`,逐块 scoped 读并校验 length/hash;连续覆盖 `media_size`;整对象 SHA-256 须等于 `media_oid`。**不**要求新鲜 FastCDC 冷切边界相等。
7376
- 磁盘写与哈希在 `spawn_blocking`;单次 I/O ≤120s、无进展 ≤10min、lease 60s(约每 20s 续租);持续进展不受整文件墙钟限制。
7477
- 缺失或损坏 chunk / gap / overlap / overflow:**不**写 LFS namespace、**不**写 `lfs_objects`、**不**发布 finalized;临时文件与 lease 在成功/失败/取消路径均释放。
75-
- 通过后:发布到 `lfs/{oid}`,`lfs_objects` 幂等插入,再写 `media/fastcdc-v2020-32k/{scope}/finalized/{media_oid}`(多布局原子发布见 MF-07)。
76-
- 已存在的 finalized 若 `manifest_id`/`media_oid` 不一致则 Conflict;重复 finalize 在内容一致时成功。
78+
- 通过后的发布顺序(C-04 / ADR-MF-05):
79+
1. 完整 LFS fallback `lfs/{oid}`(`put_stream_bounded`)+ `lfs_objects` 同 oid 幂等(大小一致才接受);
80+
2. 不可变身份记录 `media/…/finalized/{manifest_id}`(`put_metadata_atomic`,≤1 MiB 完整对象 PUT);
81+
3. 持久 `media_session.state=finalized`;
82+
4. 可替换发现 `media/…/manifest/{media_oid}`(by-media;任一完整已发布布局可成为当前值,无需胜者 CAS);
83+
5. 仅在本调用**新写入**不可变记录且已可读后发 `lfs.media.finalized`。
84+
- 同 oid、不同合法布局(不同 `manifest_id`)可并发成功;同 id 重试返回等价 canonical 身份(`created_by` 不参与比较)。不得暴露半 JSON / 错 id。
85+
- `GET …/manifests/by-media/{oid}` 只返回会话已 `finalized` 的完整 `ManifestResponse`。固定 id HTTP 路由见 MF-04;此前可通过存储直读 `finalized/{manifest_id}`。
86+
- 不支持原子完整对象 PUT 的后端必须对 `put_metadata_atomic` 返回明确错误,禁止退化为边写边可见。
7787

7888
## 出站事件(plan-20260912 / WH-06,已交付)
7989

80-
- 既有 finalize 路径在**本次实际写入 finalized manifest 成功**后发一次 `lfs.media.finalized`(契约见 [`storage-events.md`](storage-events.md)):scope 只填服务端 `MediaScope` 的 canonical `repo_path`,data 为 `oid,size,manifest_id,transfer="fastcdc"`。
81-
- prepare / chunk / fallback 中间步骤不发;已存在 finalized 的 no-op 不发。
90+
- 既有 finalize 路径在**本次实际新写入 immutable finalized 记录成功且可读后**发一次 `lfs.media.finalized`(契约见 [`storage-events.md`](storage-events.md)):scope 只填服务端 `MediaScope` 的 canonical `repo_path`,data 为 `oid,size,manifest_id,transfer="fastcdc"`。
91+
- prepare / chunk / fallback 中间步骤不发;同 id 重试(immutable 已存在)不发。
8292
- 认证不改:Media 路由仍全部要求 `AccessTokenUser`。
8393

8494
## HTTP / auth / OpenAPI(FC-07)

‎src/api/router/lfs_media.rs‎

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -511,12 +511,7 @@ async fn load_published(
511511
scope: &MediaScope,
512512
oid: &str,
513513
) -> Result<crate::ceres::lfs::media::protocol::ManifestResponse, MediaError> {
514-
let key = match scope.object_key(MediaObjectKind::Finalized, oid) {
515-
Ok(key) => key,
516-
Err(_) => return Err(MediaError::NotFound),
517-
};
518-
let bytes = media.read_bytes(&key, MAX_ENVELOPE_SIZE).await?;
519-
serde_json::from_slice(&bytes).map_err(|e| MediaError::Json(e.to_string()))
514+
crate::ceres::lfs::media::publication::load_by_media(media, scope, oid).await
520515
}
521516

522517
#[cfg(test)]

‎src/ceres/lfs/media/finalize.rs‎

Lines changed: 29 additions & 133 deletions
Original file line numberDiff line numberDiff line change
@@ -15,27 +15,21 @@ use std::{
1515
use bytes::Bytes;
1616
use sha2::{Digest, Sha256};
1717
use tokio::sync::Semaphore;
18-
use tokio_util::io::ReaderStream;
1918
use uuid::Uuid;
2019

2120
use crate::{
22-
callisto::lfs_objects,
23-
ceres::lfs::{
24-
digest::LfsDigest,
25-
media::{
26-
protocol::{
27-
FinalizeAcceptedResponse, FinalizeTaskResponse, MAX_ENVELOPE_SIZE, ManifestError,
28-
ManifestResponse, MediaManifest,
29-
},
30-
scope::{MediaObjectKind, MediaScope, redact_storage_error},
31-
service::{MediaError, MediaService, map_store, scope_key, unix_now},
21+
ceres::lfs::media::{
22+
protocol::{
23+
FinalizeAcceptedResponse, FinalizeTaskResponse, ManifestError, ManifestResponse,
24+
MediaManifest,
3225
},
26+
scope::MediaScope,
27+
service::{MediaError, MediaService, unix_now},
3328
},
3429
jupiter::storage::{
3530
lfs_db_storage::LfsDbStorage,
3631
media_paging_storage::{MediaPagingStorage, TASK_COMPLETE, TASK_FAILED, TASK_PENDING},
3732
},
38-
orbit_api::object_storage::{ObjectByteStream, ObjectKey, ObjectMeta, ObjectNamespace},
3933
};
4034

4135
/// Persistent finalize concurrency (P-04b / ADR-MF-03).
@@ -396,13 +390,22 @@ async fn run_finalize_body(
396390
p.update(manifest.media_size, sealed.page_count, "fallback")
397391
.await?;
398392
}
399-
put_fallback_from_path(media, lfs_db, &manifest, &tmp_path).await?;
400393
check_cancel(&mut cancel)?;
401394
if let Some(p) = progress {
402395
p.update(manifest.media_size, sealed.page_count, "publish")
403396
.await?;
404397
}
405-
publish_finalized(media, scope, manifest_id, &manifest, now, emitter).await
398+
crate::ceres::lfs::media::publication::publish_verified_layout(
399+
media,
400+
lfs_db,
401+
scope,
402+
manifest_id,
403+
&manifest,
404+
&tmp_path,
405+
now,
406+
emitter,
407+
)
408+
.await
406409
}
407410
.await;
408411

@@ -551,115 +554,6 @@ async fn rebuild_and_verify(
551554
Ok(())
552555
}
553556

554-
async fn put_fallback_from_path(
555-
media: &MediaService,
556-
lfs_db: &LfsDbStorage,
557-
manifest: &MediaManifest,
558-
path: &Path,
559-
) -> Result<(), MediaError> {
560-
let key = ObjectKey {
561-
namespace: ObjectNamespace::Lfs,
562-
key: manifest.media_oid.clone(),
563-
};
564-
let file = tokio::fs::File::open(path).await?;
565-
let stream: ObjectByteStream = Box::pin(ReaderStream::new(file));
566-
let meta = ObjectMeta {
567-
size: manifest.media_size as i64,
568-
..ObjectMeta::default()
569-
};
570-
media
571-
.object_store()
572-
.inner
573-
.put_stream_bounded(&key, stream, meta)
574-
.await
575-
.map_err(map_store)?;
576-
577-
lfs_db
578-
.new_lfs_object(lfs_objects::Model {
579-
oid: manifest.media_oid.clone(),
580-
size: manifest.media_size as i64,
581-
exist: true,
582-
})
583-
.await
584-
.map_err(map_db)?;
585-
586-
let stored = lfs_db
587-
.get_lfs_object(&manifest.media_oid)
588-
.await
589-
.map_err(map_db)?
590-
.ok_or(MediaError::NotFound)?;
591-
if stored.oid != manifest.media_oid
592-
|| stored.size != manifest.media_size as i64
593-
|| !stored.exist
594-
{
595-
return Err(MediaError::Conflict(
596-
"lfs_objects metadata does not match the published fallback".to_string(),
597-
));
598-
}
599-
if !media.exists(&key).await? {
600-
return Err(MediaError::NotFound);
601-
}
602-
Ok(())
603-
}
604-
605-
async fn publish_finalized(
606-
media: &MediaService,
607-
scope: &MediaScope,
608-
manifest_id: &str,
609-
manifest: &MediaManifest,
610-
now: u64,
611-
emitter: &crate::jupiter::service::storage_event_emitter::StorageEventEmitter,
612-
) -> Result<ManifestResponse, MediaError> {
613-
let response = ManifestResponse {
614-
manifest_id: manifest_id.to_string(),
615-
manifest: manifest.clone(),
616-
};
617-
let key = scope_key(scope, MediaObjectKind::Finalized, &manifest.media_oid)?;
618-
if media.exists(&key).await? {
619-
let existing = media.read_bytes(&key, MAX_ENVELOPE_SIZE).await?;
620-
let parsed: ManifestResponse =
621-
serde_json::from_slice(&existing).map_err(|e| MediaError::Json(e.to_string()))?;
622-
if parsed.manifest_id != manifest_id || parsed.manifest.media_oid != manifest.media_oid {
623-
return Err(MediaError::Conflict(
624-
"finalized manifest does not match media oid or scope session".to_string(),
625-
));
626-
}
627-
return Ok(parsed);
628-
}
629-
let payload =
630-
Bytes::from(serde_json::to_vec(&response).map_err(|e| MediaError::Json(e.to_string()))?);
631-
if payload.len() > MAX_ENVELOPE_SIZE {
632-
return Err(MediaError::Invalid(
633-
"finalized manifest exceeds size limit".to_string(),
634-
));
635-
}
636-
media.put_bytes(&key, payload).await?;
637-
638-
// WH-06 (plan-20260912 / ADR-WH-04/05): exactly one `lfs.media.finalized`
639-
// when THIS finalize actually put the finalized manifest (the exists
640-
// no-op above returns before this point; intermediate chunk/fallback
641-
// steps never emit). Scope metadata comes from the server-side
642-
// MediaScope's canonical repository, never from the request body; the
643-
// actor never leaves the process. A repeat cross-process finalize may
644-
// still rewrite the fallback and re-notify (no uniqueness lock added).
645-
let event = crate::jupiter::service::storage_event::CommittedEvent {
646-
event_id: uuid::Uuid::new_v4(),
647-
event_type: crate::jupiter::service::storage_event::EventType::LfsMediaFinalized,
648-
occurred_at: now,
649-
source: crate::jupiter::service::storage_event::EventSource::Lfs,
650-
scope: crate::jupiter::service::storage_event::EventScope::Media {
651-
repo_path: scope.repository().to_owned(),
652-
},
653-
data: crate::jupiter::service::storage_event::EventData::LfsMediaFinalized {
654-
oid: manifest.media_oid.clone(),
655-
size: manifest.media_size,
656-
manifest_id: manifest_id.to_owned(),
657-
},
658-
};
659-
let _ = emitter.try_emit(event);
660-
Ok(response)
661-
}
662-
663557
fn map_manifest(err: ManifestError) -> MediaError {
664558
match err {
665559
ManifestError::Invalid(msg) => MediaError::Invalid(msg),
@@ -684,11 +578,6 @@ fn map_paging(err: crate::jupiter::storage::media_paging_storage::MediaPagingErr
684578
}
685579
}
686580

687-
fn map_db(err: crate::common::errors::MegaError) -> MediaError {
688-
let _ = redact_storage_error(&err);
689-
MediaError::Storage
690-
}
691-
692581
/// Test helper: put P-01a pages for a prepared manifest and seal.
693582
pub async fn put_pages_and_seal(
694583
media: &MediaService,
@@ -722,10 +611,14 @@ mod tests {
722611

723612
use super::*;
724613
use crate::{
725-
ceres::lfs::media::{
726-
chunker,
727-
protocol::{ChunkEntry, CreatedBy},
728-
service::MediaService,
614+
ceres::lfs::{
615+
digest::LfsDigest,
616+
media::{
617+
chunker,
618+
protocol::{ChunkEntry, CreatedBy},
619+
scope::MediaObjectKind,
620+
service::{MediaService, scope_key},
621+
},
729622
},
730623
jupiter::{
731624
service::storage_event_emitter::StorageEventEmitter,
@@ -736,7 +629,10 @@ mod tests {
736629
},
737630
tests::test_db_connection,
738631
},
739-
orbit_api::factory::{LocalConfig, ObjectStorageBackend, ObjectStorageConfig},
632+
orbit_api::{
633+
factory::{LocalConfig, ObjectStorageBackend, ObjectStorageConfig},
634+
object_storage::{ObjectKey, ObjectNamespace},
635+
},
740636
};
741637

742638
/// Recording fake transport (WH-09 seam): exact bodies + call counter.

‎src/ceres/lfs/media/mod.rs‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
pub mod chunker;
22
pub mod finalize;
33
pub mod protocol;
4+
pub mod publication;
45
pub mod scope;
56
pub mod service;
67

@@ -440,9 +441,14 @@ mod tests {
440441
assert_eq!(again.manifest_id, published.manifest_id);
441442
let finalized = fx
442443
.scope
443-
.object_key(MediaObjectKind::Finalized, &manifest.media_oid)
444+
.object_key(MediaObjectKind::Finalized, &prepared.manifest_id)
444445
.unwrap();
445446
assert!(fx.service.exists(&finalized).await.unwrap());
447+
let by_media = fx
448+
.scope
449+
.object_key(MediaObjectKind::Manifest, &manifest.media_oid)
450+
.unwrap();
451+
assert!(fx.service.exists(&by_media).await.unwrap());
446452
}
447453

448454
#[tokio::test]
@@ -500,7 +506,7 @@ mod tests {
500506
assert!(!fx.service.exists(&lfs_key).await.unwrap());
501507
let finalized = fx
502508
.scope
503-
.object_key(MediaObjectKind::Finalized, &manifest.media_oid)
509+
.object_key(MediaObjectKind::Finalized, &prepared.manifest_id)
504510
.unwrap();
505511
assert!(!fx.service.exists(&finalized).await.unwrap());
506512
assert!(

0 commit comments

Comments
 (0)