Python (boto3)

Prefer to run it? shelby-quickstart has this page as a script that checks every step and cleans up after itself.

Before you start

Python 3.10 or later, with pip install boto3

Copy your bucket name, and an access key ID and secret for that bucket, from the console. Export them so the commands below can use them:

export SHELBY_ACCESS_KEY_ID="<your access key ID, SHLB…>"
export SHELBY_SECRET_ACCESS_KEY="<your secret>"
export SHELBY_BUCKET="<your bucket name>"

Create a client

shelby_s3.py

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url="https://s3.beta.shelby.xyz",
    aws_access_key_id=os.environ["SHELBY_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["SHELBY_SECRET_ACCESS_KEY"],
    region_name="default",
    config=Config(
        s3={"addressing_style": "path"},
        request_checksum_calculation="when_required",
    ),
)
bucket = os.environ["SHELBY_BUCKET"]

request_checksum_calculation stops boto3 from sending uploads in aws-chunked framing, which it does by default (botocore 1.36 and later) and Shelby doesn't accept yet. Without it, uploads fail with 501 NotImplemented.

Upload and read back

import hashlib

body = b"Hello, Shelby\n"
s3.put_object(Bucket=bucket, Key="examples/hello.txt", Body=body)

data = s3.get_object(Bucket=bucket, Key="examples/hello.txt")["Body"].read()
assert hashlib.sha256(data).digest() == hashlib.sha256(body).digest()

# A new object can take a few seconds to appear in listings.
for obj in s3.list_objects_v2(Bucket=bucket, Prefix="examples/").get("Contents", []):
    print(obj["Key"], obj["Size"])

import urllib.request

url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": bucket, "Key": "examples/hello.txt"},
    ExpiresIn=300,
)
with urllib.request.urlopen(url) as response:  # sends no credentials
    print(response.read().decode())

Anyone with url can download the object for the next 5 minutes, without credentials. The link works because the client signs with region_name="default". Until it expires, treat the link like a password. See Presigned URLs.

Clean up

s3.delete_object(Bucket=bucket, Key="examples/hello.txt")

Upload large files

upload_file switches to multipart automatically, in 8 MiB parts by default. Set a part size of at least 10 MiB, because smaller parts each count as 10 MiB of storage, and large enough for your largest file to fit in Shelby's 3,000 parts:

from boto3.s3.transfer import TransferConfig

s3.upload_file(
    "dataset.parquet",
    bucket,
    "data/dataset.parquet",
    Config=TransferConfig(multipart_chunksize=64 * 1024 * 1024),  # 64 MiB parts
)

64 MiB parts fit files up to about 187 GiB.

What's different with boto3

copy_object and s3.copy fail with 501. Shelby has no server-side copy. Download and upload again.
Paginators need PageSize of 1,000 or less. Larger values are rejected.

Compatibility matrix · Troubleshooting