AWS CLI

Prefer to run it? shelby-quickstart has this page as a script that checks every step and cleans up after itself.

Before you start

AWS CLI v2

Copy your bucket name, and an access key ID and secret for that bucket, from the console. Export them so the commands below can use them:

export SHELBY_ACCESS_KEY_ID="<your access key ID, SHLB…>"
export SHELBY_SECRET_ACCESS_KEY="<your secret>"
export SHELBY_BUCKET="<your bucket name>"

Create a `shelby` profile

aws configure set aws_access_key_id "$SHELBY_ACCESS_KEY_ID" --profile shelby
aws configure set aws_secret_access_key "$SHELBY_SECRET_ACCESS_KEY" --profile shelby
aws configure set region default --profile shelby
aws configure set endpoint_url https://s3.beta.shelby.xyz --profile shelby
aws configure set s3.addressing_style path --profile shelby
aws configure set s3.multipart_chunksize 10MB --profile shelby
aws configure set request_checksum_calculation when_required --profile shelby

s3.multipart_chunksize uploads large files in 10 MiB parts, up from the CLI's default of 8 MiB, because smaller parts each count as 10 MiB of storage.

The last setting stops the CLI from sending uploads in aws-chunked framing, which it does by default (CLI 2.23 and later) and Shelby doesn't accept yet. Without it, uploads fail with 501 NotImplemented.

Upload a file

export AWS_PROFILE=shelby
echo "Hello, Shelby" > hello.txt
aws s3 cp hello.txt "s3://$SHELBY_BUCKET/examples/hello.txt"

List and download it

aws s3 ls "s3://$SHELBY_BUCKET/examples/"
aws s3 cp "s3://$SHELBY_BUCKET/examples/hello.txt" hello-copy.txt

A new object can take a few seconds to appear in listings. If the listing is empty, run it again.

Check the round trip

shasum -a 256 hello.txt hello-copy.txt

Both lines should show the same hash.

url=$(aws s3 presign "s3://$SHELBY_BUCKET/examples/hello.txt" --expires-in 300)
curl -sS "$url"

aws s3 presign signs a link that anyone can use to download the object for the next 5 minutes, without credentials. curl sends none, and prints Hello, Shelby. Until it expires, treat the link like a password. See Presigned URLs.

Clean up

aws s3 rm "s3://$SHELBY_BUCKET/examples/hello.txt"

Sync a directory

aws s3 sync ./data "s3://$SHELBY_BUCKET/data/"

Large files are uploaded in parts automatically.

What's different with the AWS CLI

aws s3 mv, and aws s3 cp from one Shelby key to another, fail. Both need server-side copy. Download the object and upload it again instead.
aws s3 ls, sync and rm --recursive can miss objects written in the last few seconds. They work from listings, and a new object takes a few seconds to appear in one. A sync straight after an upload can upload the same file again.
Files over about 29 GiB need a larger part size. Shelby allows 3,000 parts per upload, and the profile above uses 10 MiB parts. Raise it with aws configure set s3.multipart_chunksize 64MB --profile shelby.

Compatibility matrix · Troubleshooting

AWS CLI

On this page