Skip to content

feat(s3-blob): implement BlobStore put/get/exists with content-addres… - #2

Open
Ashwin-3cS wants to merge 2 commits into
abhinavg6:mainfrom
Ashwin-3cS:main
Open

feat(s3-blob): implement BlobStore put/get/exists with content-addres…#2
Ashwin-3cS wants to merge 2 commits into
abhinavg6:mainfrom
Ashwin-3cS:main

Conversation

@Ashwin-3cS

Copy link
Copy Markdown

What this does

Fills in the three todo!() method bodies in backends/s3-blob that were listed as outstanding in crates/STATUS.md:

Method Implementation
put PutObject keyed by hex(sha256_digest); force_path_style enabled automatically when AWS_ENDPOINT_URL is set (MinIO, Cloudflare R2, Localstack)
get GetObject + sha2::Sha256 integrity check on the fetched bytes before returning; 404 → Ok(None), digest mismatch → Err(BlobError::AddressMismatch)
exists HeadObject; 404 → Ok(false)

Verified locally against MinIO — object keys are hex-encoded SHA-256 digests as expected.

Screenshot From 2026-05-30 15-31-12

To run against MinIO locally:

AWS_ACCESS_KEY_ID=minioadmin \
AWS_SECRET_ACCESS_KEY=minioadmin \
AWS_REGION=us-east-1 \
AWS_ENDPOINT_URL=http://localhost:9000 \
YUTHA_S3_TEST_BUCKET=yutha-test \
cargo test -p yutha-backend-s3-blob --test integration

Open design question — Postgres + S3 wiring

backends/s3-blob/README.md describes the intended architecture as:

Evidence above a configurable size threshold (default 256 KiB) is stored in S3 keyed by content-address. The Postgres row stores the evidence's content-address; the actual bytes live in S3.

The current backends/postgres-receipt schema stores evidence bytes inline in receipt_evidence.value BYTEA. The S3 offload wiring (threshold check → route large evidence to S3, write only the hash pointer into Postgres) is not yet implemented.

Question: Is the Postgres-side wiring (nullable value, separate blob_ref column, or a decorator layer) in scope for Phase 1, or is BlobStore intended to land first as a standalone component and the integration follow in a later PR? Happy to take that on if it is in scope — wanted to confirm the intended schema change before touching the Postgres backend.

@Ashwin-3cS
Ashwin-3cS requested a review from abhinavg6 as a code owner May 30, 2026 10:12
@Ashwin-3cS
Ashwin-3cS marked this pull request as draft May 30, 2026 10:36
@Ashwin-3cS
Ashwin-3cS marked this pull request as ready for review May 30, 2026 11:32
@abhinavg6

Copy link
Copy Markdown
Owner

Thanks for the PR, @Ashwin-3cS. I'm doing a bit of refactoring over the next couple of days while adding a couple of enterprise readiness capabilities. I'll take a look at this then, and would be good to see if there's any change needed here after the refactor. Let me get back to you.

@Ashwin-3cS

Copy link
Copy Markdown
Author

Thanks for the PR, @Ashwin-3cS. I'm doing a bit of refactoring over the next couple of days while adding a couple of enterprise readiness capabilities. I'll take a look at this then, and would be good to see if there's any change needed here after the refactor. Let me get back to you.

Hey @abhinavg6, thanks for the heads-up! Take your time with the refactoring - happy to revisit this.
In the meantime I've been looking into the Walrus backend and sketching out an approach (HTTP calls to the local daemon + Postgres side-index for query support). it'd be good to see how the enterprise readiness changes shape the receipt store interface before going too deep there.

…s verification

Implements the three todo!() bodies in backends/s3-blob: PutObject keyed
by hex(sha256_digest) with path-style forced for non-AWS endpoints (MinIO,
R2, Localstack); GetObject with sha2 integrity check on read; HeadObject
for existence checks. Renames _client/_bucket fields now that they are used.

Adds 7 integration tests in tests/integration.rs gated on
YUTHA_S3_TEST_BUCKET, covering round-trip, idempotency, missing-key
handling, large payloads (300 KiB), and error display. Follows the same
env-var-or-skip pattern as the Postgres conformance suite.
@Ashwin-3cS

Copy link
Copy Markdown
Author

Hey @abhinavg6 , just pulled your latest and rebased the branch; checked everything locally and the S3 blob store change is still clean.

@abhinavg6

Copy link
Copy Markdown
Owner

Thanks @Ashwin-3cS - My sincere apologies, I was preparing for another release today with a bunch of simulation related capabilities. I'll merge your changes with the next release for next week, where I intend to add a newer capability for observability - it would also align well since both would be infra level changes.

@Ashwin-3cS

Copy link
Copy Markdown
Author

Thanks @Ashwin-3cS - My sincere apologies, I was preparing for another release today with a bunch of simulation related capabilities. I'll merge your changes with the next release for next week, where I intend to add a newer capability for observability - it would also align well since both would be infra level changes.

No problem, @abhinavg6 .
Thanks for the update.
Happy to check back next week and revisit anything that needs to be adjusted after the observability changes land.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants