S3 API
Use the S3 tools you already have with Shelby. Change the endpoint and credentials, and keep your code.
Shelby has an S3-compatible endpoint. If your code or tools talk to Amazon S3 today, they can talk to Shelby with a different endpoint and access key. There's nothing else to set up.
Connect
| Setting | Value |
|---|---|
| Endpoint | https://s3.beta.shelby.xyz |
| Access key ID | Your access key ID (starts with SHLB) |
| Secret access key | Your access key's secret |
| Bucket | The bucket's name, for example shelby-managed-account-1 |
| Region | default |
| Addressing | Path-style |
Find your bucket name, and an access key ID and secret for that bucket, at console.shelby.xyz. Each access key reaches only its own bucket. Your default project already has a bucket, so there's nothing to create.
Pick your client
AWS CLI
aws s3 cp, ls, sync and rm.
Python (boto3)
boto3 and anything built on botocore.
JavaScript (AWS SDK v3)
@aws-sdk/client-s3 in Node.js.
rclone
Copy and sync directories.
DuckDB
Write Parquet and CSV, and query them in place.
Each quickstart takes about five minutes. Moving existing code or pipelines over instead? See Migrating from S3.
What works well
Shelby is designed for data-intensive workloads where datasets need to stay accessible as compute moves between locations:
- AI training datasets, training shards and model checkpoints
- Large datasets used across distributed compute environments
- Parquet and CSV files you query in place with DuckDB
- Media archives, logs and backups
Uploads, downloads, range reads, multipart uploads up to 3,000 parts, listing and deletes all use the standard S3 calls. To share an object without giving out your credentials, sign a presigned URL that expires.
How Shelby differs from S3
Most differences come from how Shelby stores data:
- You create buckets in the console. A project can have many buckets, and each access key reaches only its own bucket. You can't create, delete or rename buckets over the S3 API, and you use path-style addressing.
- Every write is a durable commit. A PUT returns once your object is erasure-coded across many storage nodes. Writes take longer than S3, an overwrite is a full new write, and a new object can take a few seconds to show up in listings.
- Encryption is always on. Every object is encrypted at rest. There is nothing to configure.
- There is no server-side copy. Copying or renaming an object means downloading it and uploading it again.
Two kinds of workload need extra care, or a different design:
- Browser uploads. Shelby has no presigned uploads, and your secret must never reach a browser. Upload from a server. Downloads work: a presigned URL lets a browser open or download one object without credentials.
- Optimistic concurrency. Writes that rely on
If-Matchare not protected yet on the hosted endpoint. See Known compatibility issues.
The compatibility matrix lists every S3 feature and how Shelby handles it.
Running your own gateway
The hosted endpoint above is the recommended way to use Shelby's S3 API. If you need the gateway inside your own network, talk to us.