Skip to content

Commit 1bfc45b

Browse files
milanmajchrakclaude
andcommitted
Port the CLARIN S3 stack to AWS SDK v2 (decision D1)
7.6.7 rewrote S3BitStoreService from com.amazonaws (v1) to software.amazon.awssdk (v2 CRT client). The original plan rejected that and kept v1; the user reversed it: "if 7.6.7 is on AWS2, go to AWS2". v1 is past AWS end-of-support anyway. The fork owns three S3 classes vanilla does not know, so this was not a file takeover: - S3BitStoreService = vanilla 7.6.7 + exactly three CLARIN items (47 added lines): getFile(Bitstream), `protected` client so the subclasses can reuse it, and a configurable pathStyleAccessEnabled (vanilla hardcodes forcePathStyle(true) on an endpoint override; keeping the property preserves the fork's v1 behaviour and keeps bitstore.xml valid). getFile is byte-identical to the fork's own v9 implementation (5e930d1) so the branches do not diverge. - SyncS3BitStoreService - hand-ported, including the explicit multipart path. - S3DirectDownloadServiceImpl - presigning moved to S3Presigner, built from the same credentials/region/endpoint as the bitstore client. - S3BitStoreServiceIT - taken from 7.6.7 wholesale; the fork's copy was byte-identical to 7.6.5, so no CLARIN test was lost. LocalStack replaces s3mock, which only ever spoke v1. - ClarinS3BitStoreServiceIT - new. The CLARIN-only paths had no test at all. - bitstore.xml / assetstore.cfg - the merge had dropped vanilla's four new tuning keys (targetThroughputGbps, minPartSizeBytes, maxConcurrency, s3ChecksumAlgorithm). Two defects the new tests caught, neither visible by reading the code: 1. v1 returned part ETags unquoted, v2 returns them quoted. The multipart checksum comparison would have failed on every part, and since that path logs and swallows, completeMultipartUpload would never run - a silent upload failure. 2. SyncS3BitStoreService.put() never closed the FileOutputStream on the local assetstore copy. Pre-existing, not a port regression, but it leaks a handle per stored bitstream and made remove() silently fail to delete the local file. Evidence (local, this head): - mvn test-compile -DskipTests: 13/13 modules SUCCESS - S3BitStoreServiceIT Tests run: 15, Failures: 0, Errors: 0 - ClarinS3BitStoreServiceIT Tests run: 6, Failures: 0, Errors: 0 - S3DirectDownloadServiceTest Tests run: 8, Failures: 0, Errors: 0 - checkstyle 0 violations; `git grep com.amazonaws` finds nothing s3Store is lazy-init and nothing else instantiates it, so springWiringBindsStoreProperties loads the bean from bitstore.xml on purpose - it is the only check that the blank maxConcurrency binds to a null Integer and that s3ChecksumAlgorithm binds to the enum. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent b284571 commit 1bfc45b

9 files changed

Lines changed: 829 additions & 517 deletions

File tree

dspace-api/pom.xml

Lines changed: 24 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -742,9 +742,25 @@
742742

743743
<!-- S3 -->
744744
<dependency>
745-
<groupId>com.amazonaws</groupId>
746-
<artifactId>aws-java-sdk-s3</artifactId>
747-
<version>1.12.792</version>
745+
<groupId>software.amazon.awssdk</groupId>
746+
<artifactId>s3</artifactId>
747+
<version>2.42.40</version>
748+
<exclusions>
749+
<exclusion>
750+
<groupId>software.amazon.awssdk</groupId>
751+
<artifactId>netty-nio-client</artifactId>
752+
</exclusion>
753+
<exclusion>
754+
<groupId>software.amazon.awssdk</groupId>
755+
<artifactId>apache-client</artifactId>
756+
</exclusion>
757+
</exclusions>
758+
</dependency>
759+
760+
<dependency>
761+
<groupId>software.amazon.awssdk.crt</groupId>
762+
<artifactId>aws-crt</artifactId>
763+
<version>0.45.1</version>
748764
</dependency>
749765

750766
<dependency>
@@ -860,21 +876,12 @@
860876
</exclusions>
861877
</dependency>
862878

879+
<!-- Replaces io.findify:s3mock, which only ever spoke AWS SDK v1. -->
863880
<dependency>
864-
<groupId>io.findify</groupId>
865-
<artifactId>s3mock_2.13</artifactId>
866-
<version>0.2.6</version>
867-
<scope>test</scope>
868-
<exclusions>
869-
<exclusion>
870-
<groupId>com.amazonawsl</groupId>
871-
<artifactId>aws-java-sdk-s3</artifactId>
872-
</exclusion>
873-
<exclusion>
874-
<groupId>com.amazonaws</groupId>
875-
<artifactId>aws-java-sdk-s3</artifactId>
876-
</exclusion>
877-
</exclusions>
881+
<groupId>org.testcontainers</groupId>
882+
<artifactId>testcontainers-localstack</artifactId>
883+
<version>2.0.5</version>
884+
<scope>test</scope>
878885
</dependency>
879886

880887
<dependency>

0 commit comments

Comments
 (0)