Migrating from S3

What to change when you move code or pipelines from Amazon S3, Cloudflare R2 or MinIO to Shelby.

Most code that works with S3 works with Shelby after a configuration change. This page lists that change, then the S3 features worth checking your code for.

1. Change the client configuration

Setting	Value
Endpoint	`https://s3.beta.shelby.xyz`
Access key ID	Your access key ID (starts with `SHLB`)
Secret access key	Your access key's secret
Bucket	The bucket's name, for example `shelby-managed-account-1`
Region	`default`
Addressing	Path-style

Find your bucket name, and an access key ID and secret for that bucket, at console.shelby.xyz. Each access key reaches only its own bucket. Your default project already has a bucket, so there's nothing to create.

For a boto3 client, the change looks like this:

s3 = boto3.client(
    "s3",
    endpoint_url="https://s3.beta.shelby.xyz",  
    aws_access_key_id=SHELBY_ACCESS_KEY_ID,  
    aws_secret_access_key=SHELBY_SECRET_ACCESS_KEY,  
    region_name="default",  
    config=Config(
        s3={"addressing_style": "path"},  
        request_checksum_calculation="when_required",  
    ),
)

Each quickstart shows the same change for its client.

2. Check for features Shelby handles differently

Search your code for these calls and settings. Each one needs a change, or a decision.

If your code uses	On Shelby	What to do
`CopyObject`, `aws s3 mv`, `s3.copy`	Not supported	Download and upload again
`x-amz-meta-*` user metadata over 2 KB	Rejected with `MetadataTooLarge`, as on S3	Keep larger metadata in your application or database
Presigned download URLs (`generate_presigned_url`, `getSignedUrl`, `aws s3 presign`)	Supported for GET and HEAD, for up to 7 days	Sign with region `default`. In JavaScript, also set `responseChecksumValidation: "WHEN_REQUIRED"` on the client. `response-cache-control` and `response-expires` are refused. See Presigned URLs.
Presigned uploads, or uploads from a browser	Not supported (no presigned PUT, and you can't configure CORS)	Upload from your server
`If-Match` for optimistic concurrency	Not enforced yet	Coordinate writers in your application. Create-only writes with `If-None-Match: *` do work.
Conditional GET for caching (`If-None-Match`, `If-Modified-Since`)	Ignored	Each read downloads the full object
Lifecycle rules, object expiry	Not available	Delete objects yourself when you no longer need them
Versioning	Not available	Write each version to its own key, for example `model/v12/weights.bin`
Tags, storage classes	Ignored	Encode anything you need in the key
`x-amz-checksum-*`	Accepted, not verified	Use `Content-MD5`, which Shelby verifies, or compare SHA-256 after download
A paginator with `PageSize` over 1,000	Rejected	Use 1,000 or less
Several buckets, or `CreateBucket` in code	Buckets are created in the console, and each access key reaches only its own bucket	Create the buckets in the console, and make a client for each bucket with that bucket's access key
Listing right after writing	A new object can take a few seconds to appear	Retry the listing, or track the keys you wrote
`DeleteObjects` from the AWS SDK for JavaScript	Not recognised yet; nothing is deleted	Delete keys one at a time with `DeleteObjectCommand`

3. Copy your existing data

rclone can copy straight from S3, R2 or MinIO into Shelby. Configure a remote for the source as usual, then:

rclone copy source:my-bucket "shelby:$SHELBY_BUCKET" --checksum --progress

rclone streams each object through your machine, because Shelby can't copy server-side. Run it close to the source (for example, in the same cloud region) to keep transfer costs down.

4. Verify

After copying, compare the two sides:

rclone check source:my-bucket "shelby:$SHELBY_BUCKET" --one-way

rclone check compares sizes, and MD5 hashes wherever both sides have them. Large files that rclone uploads in parts (over 200 MiB by default) have no whole-file MD5, so for those it compares size only. For a byte-for-byte check, add --download.

The migrate-from-s3 recipe runs steps 3 and 4 end to end on a public dataset. It copies with rclone, checks with rclone check, then runs the same DuckDB query against both sides and compares the results.

Migrating from S3

1. Change the client configuration

2. Check for features Shelby handles differently

3. Copy your existing data

4. Verify

On this page