A backup that is 97% complete is 0% useful
What backupxo does when the dump dies mid-stream, and why partial objects are the dangerous case.
Short post about a specific failure, because it is the one that makes streaming backups scary and the reason we spent a week on a few lines of error handling.
The failure
backupxo streams a dump through zstd into a multipart upload, with no file in between. That means the bucket starts receiving data within seconds of the job starting. If the dump process dies at 97%, because the connection dropped or a table was locked for too long, the bucket already holds most of a backup.
Most of a backup looks like a backup. It has a name, a size that seems about right, a timestamp. The failure mode is not “the backup is missing”, which you would notice. It is “the backup is there and restores a database with the last three tables empty”, which you notice much later.
What happens now
Three processes run: mariadb-dump, zstd, and the uploader. The uploader watches both exit codes. If the dump exits non-zero, or zstd does, the uploader aborts the multipart upload. An aborted multipart upload is not an object; the parts are deleted and nothing with that key exists in the bucket. The job is marked failed with the stderr of whichever process died, and a Telegram message goes out.
The dump is also run with --single-transaction so that a consistent snapshot is taken at the start and a long-running job does not see later writes. If a table engine does not support that, the job page shows a warning before the first run rather than quietly producing an inconsistent dump.
The trailing check
Even a clean exit is not enough on its own. backupxo compares the number of bytes that went into the uploader with the size reported by the completed multipart upload, and compares the SHA-256 it computed while streaming with a hash of the object read back. A mismatch marks the job as failed and deletes the object. In practice this has fired once, on a MinIO instance with a failing drive, and we would rather have the check and never need it.
Retention does not delete the last good one
Retention by count keeps the N most recent successful backups. Failed ones do not count, and the pruner never deletes a successful backup if doing so would leave zero. A job that fails for a month straight leaves the last good backup where it was, and the Telegram messages make sure the month does not pass unnoticed.