How to use AWS S3 as a remote cache?

[jparismorgan]

Hi there, I'm looking to migrate a project from using GCP as a remote cache to using AWS S3. GCP was very easy to set up using --google_credentials.

But I'm now looking to switch over to AWS S3. I have this in my .bazelrc:

build --remote_cache=https://s3.us-west-2.amazonaws.com/bazel-remote-cache --remote_upload_local_results=true

And I have a new S3 bucket with Block all public access = On and this IAM policy, which lets me read and write:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ReadOnlyAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<my_id>:user/<my_name>"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::bazel-remote-cache/*"
        },
        {
            "Sid": "WriteAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<my_id>:user/<my_name>"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::bazel-remote-cache/*"
        },
        {
            "Sid": "ListBucketAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<my_id>:user/<my_name>"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::bazel-remote-cache"
        }
    ]
}

And finally I configured aws CLI locally, and I can see it working. But even with this, I get:

bazel clean && bazel build //examples:hello
INFO: Invocation ID: 82218df6-fd0b-49c9-8051-26a82d7c1c00
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
INFO: Invocation ID: 515b323c-19fa-4ba9-a63d-a008dcda4661
INFO: Analyzed target //examples:hello (194 packages loaded, 2733 targets configured).
INFO: Found 1 target...
WARNING: Remote Cache: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>foo1</RequestId><HostId>t...=</HostId></Error>
WARNING: Remote Cache: 4 errors during bulk transfer:
com.google.devtools.build.lib.remote.http.HttpException: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>foo2</RequestId><HostId>K...etc=</HostId></Error>
com.google.devtools.build.lib.remote.http.HttpException: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>foo3</RequestId><HostId>j...etc=</HostId></Error>
com.google.devtools.build.lib.remote.http.HttpException: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>foo4</RequestId><HostId>w...etc=</HostId></Error>
com.google.devtools.build.lib.remote.http.HttpException: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>foo5</RequestId><HostId>S...etc=</HostId></Error>

If I change from Block all public access = On to Block all public access = Off then I can see the bucket being written to and read from. But I can't do that, I need to limit to certain people only.

So, my questions:

1. Is directly using AWS S3 as a Bazel cache supported?
2. Am I missing some CLI arguments or other Bazel config?
3. Or is this a misconfiguration of my S3 bucket?
4. Are there any suggested Bazel flags to debug this?

Thank you!

PS. I also tried adding --remote_default_exec_properties=aws_profile=default based on https://stackoverflow.com/questions/79009812/bazel-remote-caching-to-s3-returning-remote-cache-403-forbidden, but that didn't work.

[JaredNeil]

I don't think you can use S3 directly. The most lightweight solution I know of is to run https://github.com/buchgr/bazel-remote locally and configure it to use S3 as a backing store.

[cnunciato]

Yep -- @JaredNeil's suggestion may work (haven't tested it myself, but sounds reasonable). Alternatively, you could also add a CloudFront endpoint, which does support HTTP Basic auth, by way of AWS Lambda.

I set this up this recently myself and have a public repo you can use if you like. It handles all of the infra provisioning with Pulumi (S3 bucket, CloudFront CDN, and edge Lambda for the auth support), and it actually works quite well; we use it internally at Buildkite for testing. You'll find it here -- hope it helps!

https://github.com/cnunciato/bazel-remote-cache-pulumi-aws