Backups and restore

A backup is two things taken together: one archive of the files that make an install (/etc/logline and the engine's data directory) and a ClickHouse backup of the events table. logline install sets up a nightly timer for it; logline restore puts one back and checks that it worked.

sudo logline backup                    # take one now
sudo logline backup list               # what there is, newest first
sudo logline restore --dry-run logline-20261106T023512Z   # verify everything, change nothing
sudo logline restore logline-20261106T023512Z             # put it back

Both commands need root: the archive holds secrets (see below).

What is in a backup#

Names are logline- and the UTC time, so they sort by age. The archive is a gzip tar whose last member, MANIFEST.json, lists every file with its size, mode, owner and SHA-256, and every file of the ClickHouse part with its SHA-256. Each new archive is read back and verified before the backup counts as done.

Not in a backup: Vector's disk buffers and file positions (rebuilt), the download cache, ClickHouse's own logs and the binaries (re-run logline install to get those).

The ClickHouse backups disk is declared in the generated config.xml, and it is the only destination ClickHouse accepts for BACKUP and RESTORE. It lives outside the root-only backups directory because ClickHouse, which runs as its own user, writes it.

Secrets: who can read a backup#

A backup contains every secret of the install in plain text: secrets.json (HEC tokens and the ClickHouse password; plain text until the master key lands, #8), engine.env and install.json (the ClickHouse password), the CA's private key and the server key, and the password hashes and session tokens of every account. That is why:

• the archive is root-owned, mode 0600, in a root-only 0700 directory; logline backup repairs the directory's mode every run, and --to DIR creates its directory 0700 and warns if an existing one is open to others;
• logline backup and logline restore refuse to run without root, and restore refuses an archive that is not root-owned or is writable by group or others;
• nothing a backup or restore prints or logs carries a secret: paths, names, sizes, counts and errors only.

Reading the trees, root opens each file without following a symlink and without blocking on a FIFO, and records the owner and mode of the file it actually read; root's audit lines go through the same no-follow opens. An unprivileged account that owns part of a tree cannot make a backup read, or a restore write, a file elsewhere.

Treat a copy of an archive the way you would treat /etc/logline/tls/ca.key. The ClickHouse part holds your events but no platform secrets.

The nightly backup#

logline install writes and enables two units:

• logline-backup.timer: every night at 02:30 local time plus up to 30 random minutes; a night the host was off is caught up at the next boot.
• logline-backup.service: one logline backup --scheduled --keep N, as root (it reads root-only files), at low CPU and I/O priority, with the same sandbox as the other units and write access to /var/lib/logline only.

systemctl list-timers logline-backup.timer     # the next run
journalctl -u logline-backup.service           # what the last runs said
sudo logline install --backup-keep 14          # change retention (re-run)

Retention: after a successful backup the oldest archives beyond N (default 7; the install keeps the last value of --backup-keep) are removed with their ClickHouse parts. Only root-owned archives count. A ClickHouse part with no archive in the directory being pruned is left alone (it may belong to a --to backup); a failed backup removes its own part. A manual logline backup applies the same retention to its directory; --keep 0 keeps everything. Every backup is a full one: no archive depends on another, so removing any of them never breaks the rest.

Freshness: doctor and the Health page#

Every backup, scheduled or not, writes /var/lib/logline/backup-status.json (root:logline 0640; no secrets): when, whether it worked, the name, the size, whether the events table was in it, and the error if not. One check, shared by logline doctor and the Health page, reads it:

So a restore is checked by doctor every time it runs, not assumed. The Health page runs unprivileged and cannot read the archives; it shows the first two.

Restore#

sudo logline restore --dry-run <name>      # every check below, then stop
sudo logline restore <name>                # asks once; --yes does not
sudo logline restore --no-clickhouse <name>  # files only; the events table stays
sudo logline restore /mnt/usb/logline-20261106T023512Z.tar.gz   # an archive elsewhere

What it does, in order:

1. Checks the archive file: a regular file (a symlink is not followed), owned by root, not writable by group or others.
2. Verifies everything, writing nothing: reads the whole archive and rejects absolute paths, .., names outside etc/ and engine/, symlinks, hard links, devices, duplicates, any entry not in the manifest or missing from it, and any checksum, mode or owner that differs; then checksums every file of the ClickHouse part against the manifest. A damaged backup stops here with the file that is wrong.
3. Asks (unless --yes).
4. Pre-restore copy: the current files as pre-restore-<time>.tar.gz in the backups directory (the last three are kept), so a restore can be undone with another restore.
5. Stages the files into /etc/logline.restore and /var/lib/logline/engine.restore, checking every hash again as it writes, and applies modes and owners from the manifest (only the platform's own accounts; anything else becomes root; no setuid).
6. Restores the events table next to the live one (RESTORE TABLE logline.events AS logline.events_restore) while the live one keeps serving. A failure here changes nothing.
7. Stops the engine and Vector.
8. Swaps the staged directories in and removes the old ones, keeping the live audit log: it is a record, so it is not rolled back. A failure here puts the old directories back, drops the staged table and starts the engine and Vector again.
9. Swaps the tables with EXCHANGE TABLES and keeps the replaced one as logline.events_pre_restore (replacing an older one), so the events ingested since the backup are not lost yet. From the file swap on, the restore runs to the end even on Ctrl-C.
10. Restarts ClickHouse (its users.xml may have changed), the helper, Vector and the engine, and runs doctor.

Searches after a restore see the backup's events. Those ingested since are in logline.events_pre_restore until you drop it, which doctor reminds you of; it takes as much disk as it did as the live table:

sudo logline restore --drop-replaced     # once you are happy with the restore

On a new host#

Install first (same version), copy both parts over, then restore:

sudo logline install --yes
sudo install -o root -g root -m 0600 logline-20261106T023512Z.tar.gz /var/lib/logline/backups/
sudo cp -a clickhouse-backups/logline-20261106T023512Z /var/lib/logline/clickhouse-backups/
sudo chown -R logline-clickhouse: /var/lib/logline/clickhouse-backups/logline-20261106T023512Z
sudo logline restore logline-20261106T023512Z

The restored /etc/logline brings the old host's CA, certificate and ClickHouse password with it. If the host name changed, re-run logline install --name <new name> afterwards to reissue the certificate.

Audit#

Both actions are written to the audit log and so to index=_audit:

index=_audit event=backup.* | table _time, event, actor, via, ok, target, error

Not yet#

• Copies off the host: S3 targets, and --to for the ClickHouse part (it always stays on the backups disk; copy the directory yourself).
• Incremental ClickHouse backups (base_backup); every backup is full, which costs disk but keeps every archive independent.
• Encrypted archives; they wait for the master key (#8).
• A pre-upgrade backup taken by logline upgrade.
• A backups panel on the Health page with the list and a restore button; restore stays a root command until the helper can do it safely.
• An alert on a failed backup (the built-in alerts in DESIGN.md section 7).