Overview

Shelby is a data platform for distributed compute. It gives you one consistent place to store and read data from wherever your compute runs, using familiar S3-compatible tools.

Shelby is built for workloads where:

• Compute moves between locations, providers or environments.
• Data is collected, processed or used for training in different places.
• New compute creates more staging, copying or pipeline work.
• The same dataset needs to serve multiple workloads.

These challenges are particularly relevant to AI training and inference, large-scale data analytics and other data-intensive applications.

Compute is dynamic, and the best location for a workload can change as capacity, performance and cost fluctuate. Shelby reduces the operational complexity and cost of moving between environments by reducing the need to repeatedly stage, copy and synchronize datasets. Teams can choose the compute that best suits each workload without rebuilding their data infrastructure each time.

What's under the hood

• One address, global access. A single global namespace lets workloads read the same data through the same S3-compatible interface. By default, data is stored in the default location, which spreads it across all of our datacenters, so you don't manage separate regional copies.
• Efficient erasure coding. Data is encoded across many storage nodes to support durability and recovery without the overhead of storing complete replicas.
• Integrity checks on every write. Uploads are checked against Content-MD5 when you send it, and ETags are MD5s, so you can confirm the bytes that arrived are the bytes you sent.
• High-performance data access. Shelby is designed for data-intensive workloads across distributed compute environments.
• Built by experienced teams. Shelby is built by teams from Jump Trading Group and Aptos Labs, with years of experience in high-performance storage and globally distributed systems.

How Shelby stores data goes into more detail.

Ways to use Shelby

Ready to try it? Start with the Quickstart.