Streaming a 50 GB MariaDB dump into R2 with a few hundred megabytes of RAM

The first version of backupxo did what every backup script does: dump to a file, compress the file, upload the file, delete the file. It worked until a customer pointed it at a 48 GB database on a server with 60 GB of free disk. The dump filled the disk, the compressor failed, and the cleanup step never ran.

The fix was to stop touching the disk at all.

The pipeline

Three processes, two pipes, one upload:

mariadb-dump --single-transaction ... | zstd -T0 -3 | (multipart upload to S3)

mariadb-dump writes SQL to stdout. zstd reads it and writes compressed bytes to its stdout. backupxo reads that stream in chunks and hands each chunk to an S3 multipart upload as one part.

The memory footprint is bounded by two numbers you can set: the part size (default 64 MB) and the number of parts allowed in flight (default 4). That is 256 MB for the upload buffer, plus whatever zstd needs, which at level 3 is small. A 50 GB database and a 500 MB database use the same amount of memory; the big one just takes longer.

Verifying what landed

While the chunks pass through, backupxo keeps a running SHA-256 and a byte count. When the upload completes it writes both into the object metadata and into its own catalog. A restore reads the object back through the same hashing step and refuses to continue if the digest differs. This caught a corrupt object exactly once, on a MinIO instance with a failing disk, which is one more time than we expected.

Two things that went wrong

The dump can fail after the upload started. mariadb-dump streams tables one by one. If it loses the connection halfway, it exits non-zero, but by then several gigabytes are already in the bucket. The multipart upload has to be aborted, not completed, or you end up with a truncated file that looks like a backup. backupxo watches the exit codes of both the dump and the compressor and aborts the upload if either is non-zero. The incomplete parts are deleted so you are not billed for them.

Credentials on the command line are visible to every process on the box. ps aux would show the database password. backupxo now writes a temporary --defaults-extra-file with mode 600, points the dump at it, and removes it afterwards. The same file carries --no-tablespaces for MySQL 8, where the default dump wants the PROCESS privilege that most backup users do not have.

What it looks like from the outside

The job page shows bytes read, bytes uploaded and the current table, updating every second. A Cancel button sends SIGTERM to the dump, which makes everything downstream exit, which makes the upload abort. The whole thing is a few hundred lines of Python around subprocess and boto3; the hard part was deciding what to do when one of the three pieces dies.