System Backup to Compressed TAR Archive

Here is where you can post tips and tricks to share with other users of MX. Do not ask for help in this Forum.
Message
Author
User avatar
figueroa
Posts: 1097
Joined: Thu Dec 20, 2018 11:20 pm

System Backup to Compressed TAR Archive

#1 Post by figueroa »

Another user suggested that I put this in Tips and Tricks. So, here goes, only slightly edited.

Original post was responding to the subject title question "which hidden home folders exactly shouldn't be restored in a running system?" in the forums at viewtopic.php?f=104&t=61872

Everything in home is important. If you want to restore a user's home and have it be exactly like it was when the backup was made, backup and restore everything.

However, you can save quite a bit of space by NOT backing up the following:
home/*/.cache/*
home/*/.dbus/session-bus/*
home/*/.thumbnails/*

As written above, this is modeled as a tar exclude file and applies to EVERY users' home directory. If you are only dealing with one user, replace the first asterisk (*) with the actual username. Excluding this in your /home backups backs up the directories, .cache, .dbus/session-bus, and .thumbnails, but does not backup the contents.

If you are backing up an MX user with the intent on restoring to MX, especially with the intent of running the same desktop (i.e. XFCE), backp and restore everything in order to preserve the user's and the system's user configuration choices.

The directories .mozilla/firefox and .thunderbird in a user's home directory are special cases when restoring the user's files to a different Linux OS, because a different OS may be running different and incompatible versions of Firefox and Thunderbird. Furthermore, these hidden directories tend to be large, full of caches and copies of emails. I can only say your mileage may vary.

If your interest is really only a user's created and saved files, only backup the user's main directories where those files are saved: Documents, Downloads, Music, Photos, and so on. Note that some user's personal content will be huge.

System Backup to Compressed TAR Archive:

My problem with backup programs like Timeshift, is their lack of granularity and lack of compression. Programs that offer both granularity and compression are difficult to learn and use well, so most users might be better off in the long run to backup and restore from the commandline. Once scripted, the backup can be run as easily as pulling a trigger or automatically via the user's or root's crontab.

My MX-Linux weekly OS-only backup I call stage4, which backups up everything except for users' personal files. (Personal Files are backed up separately with a different, but similar script.) Everything, except for excluded content, is backed up into a single 2.1 GB (currently) compressed file, using the zstd compression program in less than 10 minutes. Here is the script and the exclude file.

Code: Select all

$ cat /home/USERNAME/bin/stage4.scr
#!/bin/sh
#stage4.scr
# Also note: https://wiki.archlinux.org/index.php/rsync#Full_system_backup
# If the directory being written to is not already mounted, uncomment and edit the line below to match your backup destination. As shown below, the backup will be written to /mnt/backup0 which is already mounted. (Uses /mnt/storage1 on DA Business Office PC) 
#mount /mnt/backup0
cd /
date > /mnt/storage1/stage4/stage4date.txt
tar cpf /mnt/storage1/stage4/stage4.tar.zst -I "zstd -9" --acls --xattrs --numeric-owner --no-wildcards-match-slash -X /home/USERNAME/bin/exclude.stage4 /
date >> /mnt/storage1/stage4/stage4date.txt
cd
#umount /mnt/backup0
NOTES: The script runs from my personal bin directory. If you use this as a model, replace USERNAME and the path to the script and the exclude file to wherever you operate from. Since what is being backed up is the system root (/) directory, the exclude file is vitally importance. Adjust to suit your tastes. Here is the exclude file:

Code: Select all

$ cat /home/USERNAME/bin/exclude.stage4
lost+found
dev/*
home/*/*
home/lost+found
media/*/*
mnt/*/*
proc/*
root/.cache/*
root/.dbus/session-bus/*
run/*
sys/*
tmp/*
var/cache/apt/archives/*.deb
var/tmp/*
NOTES: In this use case, /home is on a separate directory, so don't backup the lost+found directories. Doing so will waste space and confound matters when used to restore. The notation home/*/* causes all /home directories to be included in the backup but NONE of their contents. The notations mnt/*/* and media/*/* backups up any created mountpoints in /mnt and /media if they exist, but nothing from any devices actually mounted at those mountpoints. For example, if a user creates a mountpoint directory like /mnt/storage1, the directory /mnt/storage1 will be backed up, but if any drive or partition is mounted on /mnt/storage1 the content of that drive or partition will not be included in the backups.

I run this script and other versions of this script tailored to other distributions, automatically via root's crontab, while we sleep, on a weekly and monthly rotating basis, storing the archive on a secondary hard drive. Finally, users' personal data are backed up separately EVERY night, using a similar script customized for my group of office users.

The backup can be restored in whole, or in part, selectively with very fine granularity. In a perfect world, you never need backups. The world is not perfect, so backups are vital.

The stage4.tar.zst can be restored to a different partition, tweak the /etc/fstab to match the identity of the new partition, run update-grub in a way to match your situation and the new install will boot and run, with all of the users able to log in, but with NO customization or personal files. If you also restore user's personal files, running from the new partition instead of the former partition will be completely transparent to the users running on what is essentially a clone of the backed up source.

The stage4 archive is very close to what one gets from MX-Snapshot when not including personal files, but instead of an iso output, you get a tar archive compressed with zstd. To use zstd, you'd have to install it, "apt install zstd" or modify script and use gzip or other compression method. zstd is both faster and gives better compression. Level 9 zstd compression is a good balance between speed and compression.

A model restore script that restores everything to what is assumed to be a freshly formatted partition, /dev/sda6 mounted at /mnt/sda6:

Code: Select all

#!/bin/sh
#restorestage4.scr
cd /mnt/sda6
tar xpf /mnt/backup0/stage4.tar.zst -I "zstd" --acls --xattrs-include='*.*' --numeric-owner
cd
Enough! If interested, please ask questions. I would also be pleased for suggestions and corrections.

At the small school where this is run, our full backup archives (about 80GB) are also encrypted and put on off-line portable media and removed from the building. Some of our most critical user files are also sent to the cloud every night. Examples: payroll files, QuickBooks backup (from QuickBooks that runs in a Windows virtual machine), and such. In the event of a disaster, we could be back up and running within less than a day of acquiring replacement location and hardware.
Andy Figueroa
Using Unix from 1984; GNU/Linux from 1993

User avatar
Electroguard
Posts: 54
Joined: Mon Oct 07, 2019 9:07 pm

Re: System Backup to Compressed TAR Archive

#2 Post by Electroguard »

Thanks for that info, was very interesting - apparently a method for creating compressed system backups with individual files/folders restore capabilities.
Seemed to need much personalised manual tailoring, resulting in non-bootable system backups, with ability for full or partial (granular) restores.
It's strange that guru's need to do such things for themselves because there is nothing suitable already available.

If forced down a dark rabbit hole, I'd prefer to choose the direction... MX Snapshot already creates a compressed bootable system image with many preference options, and there's no technical reason that it couldn't in time have a gui Restore from ISO capability, and perhaps eventually even be able to selectively restore files and folders. But that's just a dream for tomorrow.

More immediately for today, perhaps minstall can offer a restore solution.
It is launched using minstal-launcher, but I didn't see the -launcher option listed in the help... suggesting it may be a later undocumented mod.
The help does mention gui options, but I didn't understand how to launch minstall in gui mode.
The help mentions several experimental options, but also several references to antix, which suggests minstall may not have been updated for a while.
But it does mention a config file and unattended option, including reformatting the original destination partition, so it is (or was) very comprehensive.
Therefore it may not actually be a million miles away from being able to do an install from a specified iso with the original configuration options.
Is there any other up-to-date information available for the desktop Installers "minstall" ?

Post Reply

Return to “Tips & Tricks by users”