Downloads

The S3 Gateway supports downloading files from Shelby storage using standard S3 read operations. This guide covers single file downloads, listing objects, and partial downloads with range requests.

Prerequisites

Before using the examples on this page, you need a running S3 Gateway with a valid configuration. See Configuration to set up your shelby.config.yaml and Integrations to configure your S3 client (i.e., rclone, boto3, AWS CLI).

Supported Read Operations

Operation	Description
`GetObject`	Download file contents
`HeadObject`	Get object metadata without downloading
`ListObjectsV1/V2`	List objects in a bucket with pagination
`ListBuckets`	List all configured buckets
`HeadBucket`	Check if a bucket exists

GetObject

Download a file from Shelby storage.

# Download a single file
rclone copy shelby:<YOUR_ACCOUNT_ADDRESS>/file.txt ./

# Download to a specific local path
rclone copy shelby:<YOUR_ACCOUNT_ADDRESS>/data/report.pdf ./downloads/

# Download with progress
rclone copy shelby:<YOUR_ACCOUNT_ADDRESS>/large-file.zip ./ -P

# Download a single file
aws --profile shelby --endpoint-url http://localhost:9000 \
  s3 cp s3://<YOUR_ACCOUNT_ADDRESS>/file.txt ./

# Download to a specific path
aws --profile shelby --endpoint-url http://localhost:9000 \
  s3 cp s3://<YOUR_ACCOUNT_ADDRESS>/data/report.pdf ./downloads/

import boto3

s3 = boto3.client(
    's3',
    endpoint_url='http://localhost:9000',
    aws_access_key_id='AKIAIOSFODNN7EXAMPLE',
    aws_secret_access_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',
    region_name='shelbyland',
)

# Download to file
s3.download_file(
    '<YOUR_ACCOUNT_ADDRESS>',
    'path/to/file.txt',
    'local-file.txt'
)

# Get object content directly
response = s3.get_object(
    Bucket='<YOUR_ACCOUNT_ADDRESS>',
    Key='path/to/file.txt'
)
content = response['Body'].read()

import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";

const client = new S3Client({
  endpoint: "http://localhost:9000",
  region: "shelbyland",
  credentials: {
    accessKeyId: "AKIAIOSFODNN7EXAMPLE",
    secretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  },
  forcePathStyle: true,
});

const response = await client.send(new GetObjectCommand({
  Bucket: "<YOUR_ACCOUNT_ADDRESS>",
  Key: "path/to/file.txt",
}));

const content = await response.Body?.transformToString();

Range Requests

Download partial content using HTTP Range headers. Useful for:

Resuming interrupted downloads
Streaming video/audio
Reading specific parts of large files

# Download first 1KB
rclone cat shelby:<YOUR_ACCOUNT_ADDRESS>/large-file.bin --count 1000 > partial.bin

# Download from byte 1000 to end
rclone cat shelby:<YOUR_ACCOUNT_ADDRESS>/large-file.bin --offset 1000 > rest.bin

# Download 500 bytes starting at byte 1000
rclone cat shelby:<YOUR_ACCOUNT_ADDRESS>/large-file.bin --offset 1000 --count 500 > chunk.bin

# Download specific byte range
response = s3.get_object(
    Bucket='<YOUR_ACCOUNT_ADDRESS>',
    Key='large-file.bin',
    Range='bytes=0-999'  # First 1KB
)
partial_content = response['Body'].read()
print(f"Got {len(partial_content)} bytes")

const response = await client.send(new GetObjectCommand({
  Bucket: "<YOUR_ACCOUNT_ADDRESS>",
  Key: "large-file.bin",
  Range: "bytes=0-999", // First 1KB
}));

const partialContent = await response.Body?.transformToByteArray();
console.log(`Got ${partialContent?.length} bytes`);

ListObjects

List objects in a bucket with optional prefix filtering and pagination.

# List all objects
rclone ls shelby:<YOUR_ACCOUNT_ADDRESS>

# List with sizes and dates
rclone lsl shelby:<YOUR_ACCOUNT_ADDRESS>

# List only directories
rclone lsd shelby:<YOUR_ACCOUNT_ADDRESS>

# List objects with a prefix
rclone ls shelby:<YOUR_ACCOUNT_ADDRESS>/data/

# List all objects
aws --profile shelby --endpoint-url http://localhost:9000 \
  s3 ls s3://<YOUR_ACCOUNT_ADDRESS>/

# List with prefix
aws --profile shelby --endpoint-url http://localhost:9000 \
  s3 ls s3://<YOUR_ACCOUNT_ADDRESS>/data/

# Recursive listing
aws --profile shelby --endpoint-url http://localhost:9000 \
  s3 ls s3://<YOUR_ACCOUNT_ADDRESS>/ --recursive

# List objects (v2)
response = s3.list_objects_v2(
    Bucket='<YOUR_ACCOUNT_ADDRESS>',
    Prefix='data/',  # Optional: filter by prefix
    MaxKeys=100      # Optional: limit results
)

for obj in response.get('Contents', []):
    print(f"{obj['Key']} - {obj['Size']} bytes")

# Handle pagination
paginator = s3.get_paginator('list_objects_v2')
for page in paginator.paginate(Bucket='<YOUR_ACCOUNT_ADDRESS>'):
    for obj in page.get('Contents', []):
        print(obj['Key'])

import { ListObjectsV2Command } from "@aws-sdk/client-s3";

const response = await client.send(new ListObjectsV2Command({
  Bucket: "<YOUR_ACCOUNT_ADDRESS>",
  Prefix: "data/",  // Optional
  MaxKeys: 100,     // Optional
}));

for (const obj of response.Contents ?? []) {
  console.log(`${obj.Key} - ${obj.Size} bytes`);
}

HeadObject

Get object metadata without downloading the content. Returns size, content type, ETag, and expiration.

response = s3.head_object(
    Bucket='<YOUR_ACCOUNT_ADDRESS>',
    Key='path/to/file.txt'
)

print(f"Size: {response['ContentLength']} bytes")
print(f"ETag: {response['ETag']}")
print(f"Content-Type: {response['ContentType']}")
print(f"Expiration: {response.get('Expiration')}")

import { HeadObjectCommand } from "@aws-sdk/client-s3";

const response = await client.send(new HeadObjectCommand({
  Bucket: "<YOUR_ACCOUNT_ADDRESS>",
  Key: "path/to/file.txt",
}));

console.log(`Size: ${response.ContentLength} bytes`);
console.log(`ETag: ${response.ETag}`);
console.log(`Content-Type: ${response.ContentType}`);

ListBuckets

List all configured buckets (Shelby account addresses).

rclone lsd shelby:

aws --profile shelby --endpoint-url http://localhost:9000 s3 ls

response = s3.list_buckets()
for bucket in response['Buckets']:
    print(bucket['Name'])

Sync Directories

Synchronize a remote directory to local storage.

# Sync remote to local
rclone sync shelby:<YOUR_ACCOUNT_ADDRESS>/data/ ./local-backup/

# Sync with progress
rclone sync shelby:<YOUR_ACCOUNT_ADDRESS>/data/ ./local-backup/ -P

# Dry run (see what would be copied)
rclone sync shelby:<YOUR_ACCOUNT_ADDRESS>/data/ ./local-backup/ --dry-run

Query with DuckDB

DuckDB can query files directly from Shelby storage without downloading them first.

-- Configure S3 settings
INSTALL httpfs;
LOAD httpfs;

SET s3_endpoint = 'localhost:9000';
SET s3_access_key_id = 'AKIAIOSFODNN7EXAMPLE';
SET s3_secret_access_key = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY';
SET s3_use_ssl = false;
SET s3_url_style = 'path';
SET s3_region = 'shelbyland';

-- Query a Parquet file
SELECT * FROM read_parquet(
  's3://<YOUR_ACCOUNT_ADDRESS>/data/dataset.parquet'
);

-- Query CSV
SELECT * FROM read_csv('s3://<YOUR_ACCOUNT_ADDRESS>/reports/sales.csv', header = true);

-- Query multiple files with glob
SELECT * FROM read_parquet('s3://<YOUR_ACCOUNT_ADDRESS>/data/*.parquet');

Error Handling

Common Errors

Error Code	Description	Solution
`NoSuchKey`	Object does not exist	Verify the key path is correct
`NoSuchBucket`	Bucket not configured	Add the account address to your config
`AccessDenied`	Authentication failed	Check credentials and signature
`InvalidRange`	Invalid range request	Verify range is within file size

Troubleshooting

"NoSuchKey" but object exists:

Check for typos in the key path
Keys are case-sensitive
Ensure you're using the correct bucket (account address)

Slow downloads:

For large files, consider using range requests to download in parallel
Check network connectivity to the Shelby RPC endpoint

"SignatureDoesNotMatch":

Ensure your region setting matches the gateway config
Check that your system clock is accurate (within 15 minutes)

Downloads

On this page