Ambry

Ambry is a distributed object store that supports storage of trillions of small immutable objects (50K -100K) as well as billions of large objects. It was specifically designed to store and serve media objects in web companies. However, it can be used as a general purpose storage system to store DB backups, search indexes or business reports. The system has the following characterisitics:

1. Highly available and horizontally scalable
2. Low latency and high throughput
3. Optimized for both small and large objects
4. Cost effective
5. Easy to use

Requires JDK version 1.11 - 1.14.

Documentation

Detailed documentation is available at https://github.com/linkedin/ambry/wiki

Research

Paper introducing Ambry at SIGMOD 2016 -> http://dprg.cs.uiuc.edu/data/files/2016/ambry.pdf

Reach out to us at ambrydev@googlegroups.com if you would like us to list a paper that is based off of research on Ambry.

Getting Started

Step 1: Download the code, build it and prepare for deployment.

To get the latest code and build it, do

$ git clone https://github.com/linkedin/ambry.git 
$ cd ambry
$ ./gradlew allJar
$ cd target
$ mkdir logs

Ambry uses files that provide information about the cluster to route requests from the frontend to servers and for replication between servers. We will use a simple clustermap that contains a single server with one partition. The partition will use /tmp as the mount point.

Step 2: Deploy a server.

$ nohup java -Dlog4j2.configurationFile=file:../config/log4j2.xml -jar ambry.jar --serverPropsFilePath ../config/server.properties --hardwareLayoutFilePath ../config/HardwareLayout.json --partitionLayoutFilePath ../config/PartitionLayout.json > logs/server.log &

Through this command, we configure the log4j properties, provide the server with configuration options and cluster definitions and redirect output to a log. Note down the process ID returned (serverProcessID) because it will be needed for shutdown.
The log will be available at logs/server.log. Alternately, you can change the log4j properties to write the log messages to a file instead of standard output.

Step 3: Deploy a frontend.

$ nohup java -Dlog4j2.configurationFile=file:../config/log4j2.xml -cp "*" com.github.ambry.frontend.AmbryFrontendMain --serverPropsFilePath ../config/frontend.properties --hardwareLayoutFilePath ../config/HardwareLayout.json --partitionLayoutFilePath ../config/PartitionLayout.json > logs/frontend.log &

Note down the process ID returned (frontendProcessID) because it will be needed for shutdown. Make sure that the frontend is ready to receive requests.

$ curl http://localhost:1174/healthCheck
GOOD

The log will be available at logs/frontend.log. Alternately, you can change the log4j properties to write the log messages to a file instead of standard output.

Step 4: Interact with Ambry !

We are now ready to store and retrieve data from Ambry. Let us start by storing a simple image. For demonstration purposes, we will use an image demo.gif that has been copied into the target folder.

POST

$ curl -i -H "x-ambry-service-id:CUrlUpload"  -H "x-ambry-owner-id:`whoami`" -H "x-ambry-content-type:image/gif" -H "x-ambry-um-description:Demonstration Image" http://localhost:1174/ --data-binary @demo.gif
HTTP/1.1 201 Created
Location: AmbryID
Content-Length: 0

The CUrl command creates a POST request that contains the binary data in demo.gif. Along with the file data, we provide headers that act as blob properties. These include the size of the blob, the service ID, the owner ID and the content type.
In addition to these properties, Ambry also has a provision for arbitrary user defined metadata. We provide x-ambry-um-description as user metadata. Ambry does not interpret this data and it is purely for user annotation. The Location header in the response is the blob ID of the blob we just uploaded.

GET - Blob Info

Now that we stored a blob, let us verify some properties of the blob we uploaded.

$ curl -i http://localhost:1174/AmbryID/BlobInfo
HTTP/1.1 200 OK
x-ambry-blob-size: {Blob size}
x-ambry-service-id: CUrlUpload
x-ambry-creation-time: {Creation time}
x-ambry-private: false
x-ambry-content-type: image/gif
x-ambry-owner-id: {username}
x-ambry-um-desc: Demonstration Image
Content-Length: 0

GET - Blob

Now that we have verified that Ambry returns properties correctly, let us obtain the actual blob.

$ curl http://localhost:1174/AmbryID > demo-downloaded.gif
$ diff demo.gif demo-downloaded.gif 
$

This confirms that the data that was sent in the POST request matches what we received in the GET. If you would like to see the image, simply point your browser to http://localhost:1174/AmbryID and you should see the image that was uploaded !

DELETE

Ambry is an immutable store and blobs cannot be updated but they can be deleted in order to make them irretrievable. Let us go ahead and delete the blob we just created.

$ curl -i -X DELETE http://localhost:1174/AmbryID
HTTP/1.1 202 Accepted
Content-Length: 0

You will no longer be able to retrieve the blob properties or data.

$ curl -i http://localhost:1174/AmbryID/BlobInfo
HTTP/1.1 410 Gone
Content-Type: text/plain; charset=UTF-8
Content-Length: 17
Connection: close

Failure: 410 Gone

Step 5: Stop the frontend and server.

$ kill -15 frontendProcessID
$ kill -15 serverProcessID

You can confirm that the services have been shut down by looking at the logs.

Additional information:

In addition to the simple APIs demonstrated above, Ambry provides support for GET of only user metadata and HEAD. In addition to the POST of binary data that was demonstrated, Ambry also supports POST of multipart/form-data via CUrl or web forms. Other features of interest include:

• Time To Live (TTL): During POST, a TTL in seconds can be provided through the addition of a header named x-ambry-ttl. This means that Ambry will stop serving the blob after the TTL has expired. On GET, expired blobs behave the same way as deleted blobs.
• Private: During POST, providing a header named x-ambry-private with the value true will mark the blob as private. API behavior can be configured based on whether a blob is public or private.

Testing

WARNING: Tests currently can take upwards of 40 minutes to run

Ambry requires azurite and MySQL for testing. To install on MacOS:

azurite:

$ npm install -g azurite
$ azurite

mysql:

$ brew install mysql
$ brew services start mysql
$ mysql -uroot
mysql> CREATE USER 'travis'@'localhost';
mysql> GRANT ALL PRIVILEGES ON *.* to 'travis'@'localhost';
mysql> FLUSH PRIVILEGES;
mysql> CREATE DATABASE AmbryRepairRequests;
mysql> USE AmbryRepairRequests; SOURCE ./ambry-mysql/src/main/resources/AmbryRepairRequests.ddl;

Then run ./gradlew build to build and run all unit tests.

Named-blob cleaner purge (review proposal)

Normal named-blob cleanup retains the latest version. The optional NamedBlobsCleanupRunner constructor accepting a Predicate<Container> purgeEligibility and boolean purgeDryRun can instead delete every unexpired, non-deleted mapping in selected containers, including latest READY versions and recent IN_PROGRESS versions. Existing constructors never enable purge. Container/account exclusions and disabled named-blob mode still take precedence; container lifecycle status is not changed.

The caller owns the eligibility policy. It must require explicit per-container opt-in, verify that the source data is no longer needed, and prevent source reads and writes before returning true. For a migration, write routing alone is not proof of read cutover or destination completeness. Keep this policy live so removing approval acts as a kill switch; it is checked before every page and every blob deletion. A check cannot cancel an already-issued deletion, so it does not replace fencing source traffic. Start with purgeDryRun=true: this reports only the first bounded page per eligible container, not a full-container count, and performs no blob or metadata deletion for those containers. Other containers continue normal cleanup.

MySqlNamedBlobDb scans purge candidates in ascending primary-key order rather than normal cleanup's descending version order. The last blob name remains an inclusive cursor so a name with more versions than a page is revisited. Purge waits for metadata soft-deletion before fetching another page and stops on deletion or metadata failure, or if the next page repeats versions that should have been removed; the next scheduled run restarts its purge sweep. Unlike normal cleanup, it never skips a name after repeated query kills. Primary-key ordering avoids the mixed-order filesort, but filtering already-deleted rows can still require scanning many index entries: this is not a fixed query-latency guarantee or a fix for normal-cleanup scans.

Database implementations without purge support fail explicitly through the new default NamedBlobDb.pullStaleBlobs overload. Blob deletion continues to use the supplied router's existing offline deleteBlob(blobId, serviceId) path. This OSS change does not guarantee primary-only routing in a composite router; that must be verified by its integration. Migration-state/completeness checks, deployment configuration, composite database delegation, and any secondary metadata cleanup are deliberately outside this OSS proposal. Expired/already-deleted mappings remain under their existing cleanup mechanisms; metadata soft-deletion does not itself immediately reclaim storage bytes.

PurgePreviewBlobCount counts dry-run candidates and PurgedBlobCount counts versions processed after metadata cleanup succeeds. Failures use the existing container failure counter and logs. These are cumulative counters, not distinct-blob inventories.

Focused validation (the integration tests require the local MySQL setup above and the AmbryNamedBlobs schema from ambry-named-mysql/src/main/resources/NamedBlobsSchema.ddl):

./gradlew -PdisableShipkit :ambry-frontend:test --tests com.github.ambry.frontend.NamedBlobsCleanupRunnerTest \
  :ambry-named-mysql:test --tests com.github.ambry.named.MySqlNamedBlobDbTest
./gradlew -PdisableShipkit :ambry-named-mysql:intTest \
  --tests 'com.github.ambry.named.MySqlNamedBlobDbIntegrationTest.testPurge*'