How Shelby stores data

You don't need any of this to use Shelby: the S3 API handles it for you. It's here for when you want to know what your data goes through.

Where your data lives

Shelby runs storage nodes in our own datacenters, connected by a private network. By default, objects go to the default location, which spreads them across all of our datacenters. Every request, from anywhere, goes to the same endpoint and sees the same namespace.

Writing an object

When you upload an object:

1. The S3 endpoint receives it and checks it against Content-MD5, if you sent one.
2. The object is split into fixed-size pieces, and each piece is erasure-coded (below).
3. The coded chunks are written to a group of storage nodes.
4. Once every node has confirmed its chunks, the write is committed and your PUT returns.

Because the PUT waits for that commit, writes take longer than on Amazon S3. Once it returns, reads see the new object straight away, though it can take a few seconds to appear in listings. Known compatibility issues explains how to handle that in code.

Every object is encrypted at rest with keys Shelby manages. There's nothing to configure.

Erasure coding

Rather than keeping several full copies of your data, Shelby uses erasure coding, the same family of techniques large storage systems use to stay durable without the cost of full replication.

Each piece of an object becomes 16 chunks: 10 that hold the data and 6 parity chunks computed from it. Any 10 of the 16 are enough to rebuild the piece, so it survives the loss of up to 6 storage nodes at once. That costs 1.6 times the original size, where three full copies would cost 3 times.

Shelby uses Clay codes, which repair a lost chunk by reading a small part of many chunks rather than 10 whole ones. That cuts the network traffic needed for recovery by up to 4 times compared with standard Reed-Solomon codes.

Reading an object

When you read an object, the S3 endpoint finds the storage nodes that hold its chunks, fetches enough of them over the private network to rebuild the bytes you asked for, checks them, and returns them. Range reads fetch only the pieces that cover the range.

Coordination

Shelby keeps track of which objects exist and where their chunks are on the Aptos blockchain, which acts as its coordination layer. You never interact with it directly: an access key and the S3 API are all you need.

Going deeper

The white paper describes the original protocol design in full.