JavaScript (AWS SDK v3)

Upload, list, download and delete Shelby objects with @aws-sdk/client-s3 in Node.js.

Prefer to run it? shelby-quickstart has this page as a script that checks every step and cleans up after itself.

Before you start

Node.js 22.18 or later, which runs TypeScript files directly

A project set up for ES modules, with the AWS SDK installed:

npm init -y
npm pkg set type=module
npm install @aws-sdk/client-s3 @aws-sdk/lib-storage

Copy your bucket name, and an access key ID and secret for that bucket, from the console. Export them so the commands below can use them:

export SHELBY_ACCESS_KEY_ID="<your access key ID, SHLB…>"
export SHELBY_SECRET_ACCESS_KEY="<your secret>"
export SHELBY_BUCKET="<your bucket name>"

Create a client

shelby-s3.ts

import { S3Client } from "@aws-sdk/client-s3";

export const s3 = new S3Client({
  endpoint: "https://s3.beta.shelby.xyz",
  region: "default",
  credentials: {
    accessKeyId: process.env.SHELBY_ACCESS_KEY_ID!,
    secretAccessKey: process.env.SHELBY_SECRET_ACCESS_KEY!,
  },
  forcePathStyle: true,
  requestChecksumCalculation: "WHEN_REQUIRED",
  responseChecksumValidation: "WHEN_REQUIRED",
});

export const Bucket = process.env.SHELBY_BUCKET!;

requestChecksumCalculation is optional. It stops the SDK from computing CRC checksums that Shelby accepts but doesn't verify.

responseChecksumValidation is needed for presigned download links. Without it, the SDK adds x-amz-checksum-mode to them, which Shelby refuses with 501 NotImplemented. Shelby returns no response checksums, so no check is lost.

Upload and read back

quickstart.ts

import {
  GetObjectCommand,
  ListObjectsV2Command,
  PutObjectCommand,
} from "@aws-sdk/client-s3";
import { Bucket, s3 } from "./shelby-s3.ts";

const Key = "examples/hello.txt";

await s3.send(new PutObjectCommand({ Bucket, Key, Body: "Hello, Shelby\n" }));

const object = await s3.send(new GetObjectCommand({ Bucket, Key }));
console.log(await object.Body?.transformToString());

// A new object can take a few seconds to appear in listings.
const listing = await s3.send(
  new ListObjectsV2Command({ Bucket, Prefix: "examples/" }),
);
for (const item of listing.Contents ?? []) console.log(item.Key, item.Size);

Run it:

node quickstart.ts

npm install @aws-sdk/s3-request-presigner

share.ts

import { GetObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { Bucket, s3 } from "./shelby-s3.ts";

const url = await getSignedUrl(
  s3,
  new GetObjectCommand({ Bucket, Key: "examples/hello.txt" }),
  { expiresIn: 300 },
);

const response = await fetch(url); // sends no credentials
console.log(response.status, await response.text());

node share.ts

Anyone with url can download the object for the next 5 minutes, without credentials. Until it expires, treat the link like a password. See Presigned URLs.

Clean up

clean-up.ts

import { DeleteObjectCommand } from "@aws-sdk/client-s3";
import { Bucket, s3 } from "./shelby-s3.ts";

await s3.send(new DeleteObjectCommand({ Bucket, Key: "examples/hello.txt" }));

node clean-up.ts

Upload large files

Upload from @aws-sdk/lib-storage splits large bodies into parts, 5 MiB each by default. Set partSize to at least 10 MiB, because smaller parts each count as 10 MiB of storage, and large enough for your largest file to fit in Shelby's 3,000 parts:

import { createReadStream } from "node:fs";
import { Upload } from "@aws-sdk/lib-storage";
import { Bucket, s3 } from "./shelby-s3.ts";

await new Upload({
  client: s3,
  params: {
    Bucket,
    Key: "data/dataset.parquet",
    Body: createReadStream("dataset.parquet"),
  },
  partSize: 64 * 1024 * 1024, // 64 MiB parts
}).done();

What's different with the AWS SDK for JavaScript

Run it on a server. Your secret must never reach a browser. To let a browser download an object, give it a presigned URL instead. Browsers can't upload, because Shelby has no presigned uploads.
CopyObjectCommand fails with 501. Download and upload again.
DeleteObjectsCommand doesn't work yet. The SDK sends batch deletes in a form Shelby doesn't recognise, so nothing is deleted. Delete keys one at a time with DeleteObjectCommand. See Known compatibility issues.
Authentication failures surface as XML parse errors. Shelby returns a plain-text 401, which the SDK can't parse. See Troubleshooting.

Compatibility matrix · Troubleshooting