feat(s3-blob): implement BlobStore put/get/exists with content-addres… - #2
feat(s3-blob): implement BlobStore put/get/exists with content-addres…#2Ashwin-3cS wants to merge 2 commits into
Conversation
|
Thanks for the PR, @Ashwin-3cS. I'm doing a bit of refactoring over the next couple of days while adding a couple of enterprise readiness capabilities. I'll take a look at this then, and would be good to see if there's any change needed here after the refactor. Let me get back to you. |
Hey @abhinavg6, thanks for the heads-up! Take your time with the refactoring - happy to revisit this. |
…s verification Implements the three todo!() bodies in backends/s3-blob: PutObject keyed by hex(sha256_digest) with path-style forced for non-AWS endpoints (MinIO, R2, Localstack); GetObject with sha2 integrity check on read; HeadObject for existence checks. Renames _client/_bucket fields now that they are used. Adds 7 integration tests in tests/integration.rs gated on YUTHA_S3_TEST_BUCKET, covering round-trip, idempotency, missing-key handling, large payloads (300 KiB), and error display. Follows the same env-var-or-skip pattern as the Postgres conformance suite.
|
Hey @abhinavg6 , just pulled your latest and rebased the branch; checked everything locally and the S3 blob store change is still clean. |
|
Thanks @Ashwin-3cS - My sincere apologies, I was preparing for another release today with a bunch of simulation related capabilities. I'll merge your changes with the next release for next week, where I intend to add a newer capability for observability - it would also align well since both would be infra level changes. |
No problem, @abhinavg6 . |
What this does
Fills in the three
todo!()method bodies inbackends/s3-blobthat were listed as outstanding incrates/STATUS.md:putPutObjectkeyed byhex(sha256_digest);force_path_styleenabled automatically whenAWS_ENDPOINT_URLis set (MinIO, Cloudflare R2, Localstack)getGetObject+sha2::Sha256integrity check on the fetched bytes before returning;404 → Ok(None), digest mismatch →Err(BlobError::AddressMismatch)existsHeadObject;404 → Ok(false)Verified locally against MinIO — object keys are hex-encoded SHA-256 digests as expected.
To run against MinIO locally:
AWS_ACCESS_KEY_ID=minioadmin \ AWS_SECRET_ACCESS_KEY=minioadmin \ AWS_REGION=us-east-1 \ AWS_ENDPOINT_URL=http://localhost:9000 \ YUTHA_S3_TEST_BUCKET=yutha-test \ cargo test -p yutha-backend-s3-blob --test integrationOpen design question — Postgres + S3 wiring
backends/s3-blob/README.mddescribes the intended architecture as:The current
backends/postgres-receiptschema stores evidence bytes inline inreceipt_evidence.value BYTEA. The S3 offload wiring (threshold check → route large evidence to S3, write only the hash pointer into Postgres) is not yet implemented.Question: Is the Postgres-side wiring (nullable
value, separateblob_refcolumn, or a decorator layer) in scope for Phase 1, or isBlobStoreintended to land first as a standalone component and the integration follow in a later PR? Happy to take that on if it is in scope — wanted to confirm the intended schema change before touching the Postgres backend.