# Migrating from S3 (/tools/s3-gateway/migrating)

Most code that works with S3 works with Shelby after a configuration change. This page lists
that change, then the S3 features worth checking your code for.

## 1. Change the client configuration

| Setting           | Value                                                     |
| ----------------- | --------------------------------------------------------- |
| Endpoint          | `https://s3.beta.shelby.xyz`                              |
| Access key ID     | Your access key ID (starts with `SHLB`)                   |
| Secret access key | Your access key's secret                                  |
| Bucket            | The bucket's name, for example `shelby-managed-account-1` |
| Region            | `default`                                                 |
| Addressing        | Path-style                                                |

Find your bucket name, and an access key ID and secret for that bucket, at
[console.shelby.xyz](https://console.shelby.xyz). Each access key reaches only its own
bucket. Your default project already has a bucket, so there's nothing to create.

For a boto3 client, the change looks like this:

```python
s3 = boto3.client(
    "s3",
    endpoint_url="https://s3.beta.shelby.xyz",  # [!code ++]
    aws_access_key_id=SHELBY_ACCESS_KEY_ID,  # [!code ++]
    aws_secret_access_key=SHELBY_SECRET_ACCESS_KEY,  # [!code ++]
    region_name="default",  # [!code ++]
    config=Config(
        s3={"addressing_style": "path"},  # [!code ++]
        request_checksum_calculation="when_required",  # [!code ++]
    ),
)
```

Each [quickstart](/tools/s3-gateway/quickstarts/aws-cli) shows the same change for its
client.

## 2. Check for features Shelby handles differently

Search your code for these calls and settings. Each one needs a change, or a decision.

| If your code uses                                                                    | On Shelby                                                                           | What to do                                                                                                                                                                                                                            |
| ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `CopyObject`, `aws s3 mv`, `s3.copy`                                                 | Not supported                                                                       | Download and upload again                                                                                                                                                                                                             |
| `x-amz-meta-*` user metadata over 2 KB                                               | Rejected with `MetadataTooLarge`, as on S3                                          | Keep larger metadata in your application or database                                                                                                                                                                                  |
| Presigned download URLs (`generate_presigned_url`, `getSignedUrl`, `aws s3 presign`) | Supported for GET and HEAD, for up to 7 days                                        | Sign with region `default`. In JavaScript, also set `responseChecksumValidation: "WHEN_REQUIRED"` on the client. `response-cache-control` and `response-expires` are refused. See [Presigned URLs](/tools/s3-gateway/presigned-urls). |
| Presigned uploads, or uploads from a browser                                         | Not supported (no presigned PUT, and you can't configure CORS)                      | Upload from your server                                                                                                                                                                                                               |
| `If-Match` for optimistic concurrency                                                | Not enforced yet                                                                    | Coordinate writers in your application. Create-only writes with `If-None-Match: *` do work.                                                                                                                                           |
| Conditional GET for caching (`If-None-Match`, `If-Modified-Since`)                   | Ignored                                                                             | Each read downloads the full object                                                                                                                                                                                                   |
| Lifecycle rules, object expiry                                                       | Not available                                                                       | Delete objects yourself when you no longer need them                                                                                                                                                                                  |
| Versioning                                                                           | Not available                                                                       | Write each version to its own key, for example `model/v12/weights.bin`                                                                                                                                                                |
| Tags, storage classes                                                                | Ignored                                                                             | Encode anything you need in the key                                                                                                                                                                                                   |
| `x-amz-checksum-*`                                                                   | Accepted, not verified                                                              | Use `Content-MD5`, which Shelby verifies, or compare SHA-256 after download                                                                                                                                                           |
| A paginator with `PageSize` over 1,000                                               | Rejected                                                                            | Use 1,000 or less                                                                                                                                                                                                                     |
| Several buckets, or `CreateBucket` in code                                           | Buckets are created in the console, and each access key reaches only its own bucket | Create the buckets in the console, and make a client for each bucket with that bucket's access key                                                                                                                                    |
| Listing right after writing                                                          | A new object can take a few seconds to appear                                       | Retry the listing, or track the keys you wrote                                                                                                                                                                                        |
| `DeleteObjects` from the AWS SDK for JavaScript                                      | Not recognised yet; nothing is deleted                                              | Delete keys one at a time with `DeleteObjectCommand`                                                                                                                                                                                  |

## 3. Copy your existing data

[rclone](/tools/s3-gateway/quickstarts/rclone) can copy straight from S3, R2 or MinIO into
Shelby. Configure a remote for the source as usual, then:

```bash
rclone copy source:my-bucket "shelby:$SHELBY_BUCKET" --checksum --progress
```

rclone streams each object through your machine, because Shelby can't copy server-side.
Run it close to the source (for example, in the same cloud region) to keep transfer costs
down.

## 4. Verify

After copying, compare the two sides:

```bash
rclone check source:my-bucket "shelby:$SHELBY_BUCKET" --one-way
```

`rclone check` compares sizes, and MD5 hashes wherever both sides have them. Large files
that rclone uploads in parts (over 200 MiB by default) have no whole-file MD5, so for those
it compares size only. For a byte-for-byte check, add `--download`.

The [migrate-from-s3 recipe](https://github.com/shelby/shelby-quickstart/tree/main/recipes/migrate-from-s3)
runs steps 3 and 4 end to end on a public dataset. It copies with rclone, checks with
`rclone check`, then runs the same DuckDB query against both sides and compares the
results.