Cavalier Setup

What storage operators need to do to stand up `cavalier` from the provided tarball

These instructions:

are valid only for Linux x64
assume familiarity with udev, bash, systemd
assume you have sufficient capabilities, or the ability to escalate to root
assume you already have obtained shelby-sp-operator-source-bundle-${CAV_COMMIT}.zip

Unpack the Bundle

set -euxo pipefail

export CAV_COMMIT="LATEST_VERSION_SHA"

unzip shelby-sp-operator-source-bundle-${CAV_COMMIT}.zip
tar -xzf shelby-sp-operator-source-bundle-${CAV_COMMIT}.tar.gz

Install Dependencies to Build from Source

Refer to deps.sh for the up-to-date list of things to install.

Ubuntu/Debian

apt install unzip git build-essential pkgconf autoconf automake autopoint bison flex gettext libtool cmake python3 python3-pip python3-venv

RHEL

dnf install unzip git make pkgconf gcc gcc-c++ perl autoconf automake bison flex gettext-devel libtool cmake python3

Build the Binaries

Build the cavalier binary and other necessary executables.

set -euxo pipefail

./shelby/deps.sh fetch check install |& tee cav_deps.log
./shelby/firedancer/deps.sh fetch install |& tee fd_deps.log
make -C shelby -j bin |& tee cav_build.log

export PATH="$PATH:$(readlink -f shelby/build/native/gcc/bin)"

The binary is built at shelby/build/native/gcc/bin/cavalier. If you move the cavalier binary, you MUST ensure that hugetlbfs.py and mountfs.py (in the same directory) are alongside the cavalier binary. It's RECOMMENDED that the cavalier binary is built on the machine it runs on, to avoid issues with CPU instruction discrepancies.

One Time Setup

These instructions should only be used when setting up a new node. Some steps are destructive to stored data. They should not be run once you have joined a Shelby network and are storing data.

Direct Disk Access

Decide if you wish to give cavalier direct access to disks. This is optional, but may improve performance.

Ensure:

the user that will run cavalier has access to the block device
the block device you want to use is in /dev/disk/by-id (or some other stable identifier)

This option WILL cause data on chosen disks to be WIPED!

In the config file (described later on), you will set storage_config.storages to point to the block devices accordingly. e.g.,

storage_snippet.toml

[storage_config]
storages = [
    "/dev/disk/by-id/nvme-nvme.01de-7373746f72616765312d6469736b2d32--00000001"
]

Via `udev`

One way of achieving this is by modifying your udev rules.

Populate the {{variables}} in the following udev snippet with appropriate values from your environment. You can use udevadm info to retrieve the wwid/ID_WWN of your disk (udevadm info -a -n /dev/nvme3n1 | grep wwid).

/usr/lib/udev/rules.d/70-cavalier-disk-ownership-{{wwid}}.rules

SUBSYSTEM=="block", ATTRS{wwid}=="{{wwid}}", OWNER="{{cav_user}}", TAG+="uaccess"

Bounce udevadm to apply changes.

udevadm control --reload-rules
udevadm trigger

If you run cavalier as root you can skip the udev updates.

Make an Aptos Account

Download + install the Aptos CLI: https://aptos.dev/build/cli/install-cli/install-cli-linux

Run

$ aptos init --profile cavtest
Configuring for profile cavtest
Choose network from [devnet, testnet, mainnet, local, custom | defaults to devnet]
testnet
Enter your private key as a hex literal (0x...) [Current: None | No input: Generate new key (or keep one if present)]

No key given, generating key...

---
Aptos CLI is now set up for account 0x00000000000 as profile cavtest!
---

The account has not been funded on chain yet. To fund the account and get APT on testnet you must visit https://aptos.dev/network/faucet?address=0x00000000000

This creates a profile for your cavalier instance.

Generate/Retrieve Necessary Keys

ED25519

Cavalier needs your Aptos private key as a standalone file.

grep private_key .aptos/config.yaml | cut -f3 -d- > cav_privkey

realpath cav_privkey will be the value of signature_config.aptos_priv_key_path in the config file.

Retrieve the account address as well.

echo "0x$(grep account .aptos/config.yaml | tr -d ' ' | cut -f2 -d:)"

You will need the account address for client_config.storage_provider in the config file.

BLS12-381

You will also need to generate a BLS12-381 keypair with the following command. The pubkey will be compressed and in G2.

python3 -m venv .venv
source ./.venv/bin/activate
pip install py-ecc
python3 shelby/move/scripts/generate_bls_keypairs.py generate --count 1 --filename cav_bls

realpath cav_bls_privkey will be the value of signature_config.priv_key_path in the config file.

Geomi API Key

Follow most of the instructions from https://aptos.dev/build/guides/build-e2e-dapp#setup-api-key to get a Geomi API key.

You'll need to make a couple changes:

Set the network to testnet
Disable Client usage (may be disabled by default)

Inform us what organization/email you used during this sign up.

Config

Write the config file. You WILL need to modify some of these values to match your environment. You can grep PROVIDE_THIS for values that need updating. Refer to previous sections of this document for values that were generated/computed.

testnet.toml

name = "Cavalier, a Shelby Storage Engine by Jump Crypto"

# The user that will be used to run Cavalier. The hugetlbfs must be writeable by
# this user.
username = "YOU_PROVIDE_THIS"

tiles = ["SP_SERVER", "SP_ENGINE", "SP_CLIENT", "SP_SIGN", "SP_REBUILD"]

# All memory that will be used in Cavalier is pre-allocated in huge pages which
# are 2 MiB. This is done to prevent TLB misses which can have a high
# performance cost.
#
# A typical layout of the mounts looks like so:
#
#  /mnt/.cavalier               [Mount parent directory specified below]
#    +-- .huge                  [Files created in this mount use 2 MiB pages]
#        +-- scratch1.wksp
#        +-- scratch2.wksp
[hugetlbfs]

# The absolute path to a directory in the file system.  Cavalier
# will mount the hugetlbfs file system for gigantic pages at a
# subdirectory named .gigantic under this path, or if the entire
# path already exists, will use it as-is.  Cavalier will also
# mount the hugetlbfs file system for huge pages at a subdirectory
# named .huge under this path, or if the entire path already exists,
# will use it as-is.  If the mount already exists it should be
# writable by the Cavalier user.
mount_path = "/mnt/.cavalier"

# SP server config
[server_config]

# cavalier listens on this port. this value must match how you register with
# the smart contract.
port = 39431

max_conns = 40
max_requests_in_flight = 20

# Example of how to restrict the server to listening on a single interface
# Note: Only accepts IPv4 at the moment
# interface              = "64.130.58.54"

# metadata checks
enable_md_checks = 1

# This only makes sense to enable if the above is enabled.
enable_chunk_proofs = 1

# SP client config
[client_config]
rest_endpoint_url = "WE_PROVIDE_THIS"
grpc_endpoint_host = "WE_PROVIDE_THIS"
grpc_endpoint_port = "WE PROVIDE THIS" # an integer, so remove the quotes.
grpc_use_tls = "WE PROVIDE THIS" # an integer, so remove the quotes.
grpc_starting_version = "WE PROVIDE THIS" # an integer, so remove the quotes.

# the aptos account address (NOT pubkey and NOT privkey) from the
# `Generate/Retrieve Necessary Keys` section above
storage_provider = "YOU_PROVIDE_THIS"

# get an API key from geomi
api_key = "YOU_PROVIDE_THIS"

# SP engine config
[engine_config]

# Path to the file or block device (but NOT a directory) to store engine state.
# We recommend a disk or file on a device/filesystem that can:
#  * maintain high IOPS,
#  * is separated from your OS or cavalier's storage, and
#  * has journaling capabilities
engine_state = "YOU_PROVIDE_THIS"
max_read_per_storage = 2
max_write_per_storage = 2

# SP storage config
[storage_config]

# *Total* user data storage capacity, in MiB, for this *entire SP*.
storage_capacity_MiB = "YOU_PROVIDE_THIS" # an integer, so remove the quotes.

# List of storages to be used by this SP. User data will be written here.
#
# Each storage can be either a block device or a regular file, but NOT a
# directory.
#
# If a storage is a block device, it will be *wiped*, re-partitioned and
# formatted with cavalier internal content at time of initialization.
# Only raw block device paths (i.e. /dev/sda) should be provided in this
# config, not partition paths (e.g. /dev/sda1) or other variants.
#
# If a storage is a regular file, it will be *wiped*, re-sized
# against configured capacity and formatted with cavalier internal
# content at time of initialization.
#
# If no entity exists at provided path, cavalier will attempt to create a
# regular file at that location and follow regular file behavior.
storages = ["YOU_PROVIDE_THIS"]

# SP signature config
[signature_config]

# Full path to file containing BLS private key (hex string with 0x prefix) for
# signing acknowledgments. Refer to Generate/Retrieve Necessary Keys section
# above.
priv_key_path = "YOU_PROVIDE_THIS"

# Full path to file containing Aptos private key (hex string with 0x prefix) for
# signing transactions. Refer to Generate/Retrieve Necessary Keys section above.
aptos_priv_key_path = "YOU_PROVIDE_THIS"

Initialize Cavalier

Run the init subcommand. This does the following:

wipes + initializes disks, if configured to do so
prepares workspaces

sudo shelby/build/native/gcc/bin/cavalier --config testnet.toml init

Register the SP

Before registration, ensure your account is funded with APT for gas fees. You can get testnet APT from the faucet.

Using the Registration Script (Recommended)

The sp_register.py script in the sp_operator/ directory automates registration in two phases:

Phase 1 — Initialize your storage provider on-chain and set up payment channels. Run this first.
Phase 2 — Activate your placement group slot(s). Run this only after the Shelby team has assigned your SP to a placement group and asked you to complete registration.

# Create a config file
cat > my_sp_config.json << 'EOF'
{
    "sp_index": 0,
    "sp_ed25519_key": "ed25519-priv-0xYOUR_PRIVATE_KEY",
    "sp_bls_public_key": "YOUR_BLS_PUBLIC_KEY_HEX",
    "sp_ip_address": "YOUR_PUBLIC_IP",
    "sp_port": 39431,
    "stake": 10000
}
EOF

# Phase 1: initialize SP and payment channels
python3 sp_operator/sp_register.py --network testnet --config-file my_sp_config.json --phase 1

After phase 1 completes, tell the Shelby team that you are ready to have your SP registered on-chain and provide your SP address (your Aptos account address). Once they have assigned your SP to a placement group, they will ask you to run phase 2:

# Phase 2: activate placement group slot(s)
python3 sp_operator/sp_register.py --network testnet --config-file my_sp_config.json --phase 2

Phase 2 will find and activate your slot(s) automatically. To activate a specific slot, you can pass --pg-address and --slot-index (or add pg_address and slot_index to your config file).

The sp_index is just a local identifier for organizing your CLI profiles — you can use any number.

Manual Registration

Alternatively, you can run the registration commands manually:

export micropayments_deployer="WE_PROVIDE_THIS"
export shelby_contract_deployer="WE_PROVIDE_THIS"

# the following will be how other cavaliers talk to you. if you're doublezero'd:
export my_ipv4_addr="$(doublezero status --json | jq -r '.[0].response.doublezero_ip')"
# otherwise, you may need to talk over the public internet
# export my_ipv4_addr="$(curl ifconfig.me)"

# initialize payment channels
aptos move run --profile cavtest --function-id "${micropayments_deployer}::micropayments::initialize_payment_channels" --assume-yes

# initialize storage provider
aptos move run --profile cavtest --function-id "${shelby_contract_deployer}::storage_provider::initialize_storage_provider" --args hex:$(cat cav_bls_pubkey) string:${my_ipv4_addr} u64:39431 --max-gas 100000 --assume-yes

Run

Start cavalier. You must be funded.

shelby/build/native/gcc/bin/cavalier --config testnet.toml run

systemd

An example service unit for systemd that starts cavalier and keeps it up:

~/.config/systemd/user/cavalier.service

[Unit]
Description=Cavalier Bootstrap Client
After=network.target

[Service]
Type=exec
ExecStart=shelby/build/native/gcc/bin/cavalier --config testnet.toml run
TimeoutStopSec=30
Restart=on-failure

[Install]
WantedBy=default.target

Cavalier Setup

On this page