rclone

Prefer to run it? shelby-quickstart has this page as a script that checks every step and cleans up after itself.

Before you start

rclone 1.60 or later

Copy your bucket name, and an access key ID and secret for that bucket, from the console. Export them so the commands below can use them:

export SHELBY_ACCESS_KEY_ID="<your access key ID, SHLB…>"
export SHELBY_SECRET_ACCESS_KEY="<your secret>"
export SHELBY_BUCKET="<your bucket name>"

Create a `shelby` remote

rclone config create shelby s3 \
  provider=Other \
  endpoint=https://s3.beta.shelby.xyz \
  access_key_id="$SHELBY_ACCESS_KEY_ID" \
  secret_access_key="$SHELBY_SECRET_ACCESS_KEY" \
  region=default \
  force_path_style=true \
  chunk_size=10Mi \
  max_upload_parts=3000 >/dev/null

chunk_size=10Mi uploads large files in parts of at least 10 MiB, up from rclone's default of 5 MiB, because smaller parts each count as 10 MiB of storage. max_upload_parts=3000 matches Shelby's part limit. rclone raises the part size for large files to fit.

rclone config create prints the new remote's settings, secret included. Redirecting the output to /dev/null keeps your secret out of your terminal's scrollback, and out of an AI agent's transcript.

Copy a directory

mkdir -p data && echo "Hello, Shelby" > data/hello.txt
rclone copy ./data "shelby:$SHELBY_BUCKET/data" --checksum --progress

Always pass --checksum (or --size-only) when copying to an existing destination. See Why --checksum below.

List and verify

rclone ls "shelby:$SHELBY_BUCKET/data"
rclone check ./data "shelby:$SHELBY_BUCKET/data" --one-way

rclone check compares sizes and MD5 hashes, and should report 0 differences found. For a byte-for-byte check of large files, add --download.

Download and clean up

rclone copy "shelby:$SHELBY_BUCKET/data" ./data-copy --checksum
rclone delete "shelby:$SHELBY_BUCKET/data"

Keep a directory in sync

rclone sync ./data "shelby:$SHELBY_BUCKET/data" --checksum

On the second run, unchanged files are skipped and nothing is uploaded.

Why `--checksum`

By default rclone decides whether a file changed by comparing size and modification time, which it stores with each object it uploads. When the sizes match but the times don't, for example after you re-extract an archive, or for objects another tool uploaded, rclone updates the stored time with a server-side copy. Shelby doesn't support server-side copy, so that fails with Failed to set modification time ... 501 NotImplemented.

--checksum compares size and MD5 instead, so unchanged files are skipped. Files rclone uploads in parts (over 200 MiB by default) have no whole-file MD5, so for those it compares size only. --size-only also works and is faster, but it can't detect an edit that keeps the same size.

What's different with rclone

Every upload is a full write. Re-uploading an unchanged file repeats the whole upload, which is another reason to always use --checksum.
Server-side rclone copy or move between two Shelby paths fails. Download and upload again.
-vv doesn't show HTTP traffic. To debug a request, use --dump headers.

Compatibility matrix · Troubleshooting

rclone

On this page