Setting up for Successful Recovery...

[CharlesV]

As a mod here, (and a tech for 35+ years). I have seen many people struggling with recovery when things go wrong, and many new Linux users just don’t yet have the understanding of how to prepare. But there are a couple steps that are easy enough to do and can save you MAJOR frustration and speed up your recovery.

When I rebuilt my last laptop, I decided I was going to try a ‘little experiment’ to see how quickly I could recover it if things went wrong. When I travel, often I find I have left my trusty LiveUSB at home, so I decided to build my new laptop with a ‘recovery partition’. I created a very small partition and installed MX-21 in it, leaving it all generic. ( Yes, I like MX-21 and still run it :-) )

Planning is all good, but unless you test your backups etc.. your still not ‘safe’. I tested this partition and all good.

On my laptop, I have TimeShift set to run every morning at 9am, cause I am *not* going to be doing anything much before 9am on my machines! (On my big machines, TimeShift runs at 4pm, saving ‘that day’.)

So.. this am.. at 9:12, I checked to verify I had a timeshift of 9am, which I did, and so I updated my machine – leaning into the firmware updates and checking out the ‘new updates’. Upon restarting, my laptop gave me two lines of text after the grub menu, and then announced it was “out of memory” … uh huh.

Cool! A chance to test my new recovery system! I rebooted into the recovery partition, started TimeShift, canceled out of the setup, and the last 10 timeshift backups were listed. I chose this am’s 9:00 backup, and hit restore.

Five minutes later I was back at my regular MX Linux desktop and cruising away as normal. ( And ALL this on less than a full cup of coffee! )

Now, your results may vary, but I can tell you this was the best restore I have ever had from a 100% OS down! Planning it out was simple, implementing it was a little more work, testing it was easy… and recovery this morning proved how good it can be!

I always urge people to use TimeShift and have a good, known backup as well. And, if you really want to kick recovery into high gear… save yourself with a simple, quick ‘recovery partition’.

[davidy]

Wow. Pretty cool. I'm going to stick with my clonezilla usb method for now. Using timeshift and a separate partition is genius. MX should incorporate this into the installer and then I'd run timeshift for sure. OTH, clonezilla will do a bare disc restore which is why I like it. If your partitions get borked you will still need a usb drive and access to the files. I've run clonezilla so many times not just to backup but also to restore I can almost do it with my eyes closed. I restored my pc probably 3 times alone in just this past week to correct my errors because of issues with opensnitch and kernel upgrades etc.

[dreamer]

A recovery partition sounds good. Just wondering how your TimeShift snapshots could be listed in a different OS. Also I would worry about snapshots installing on the wrong partition since "sda" seems locked to the running OS.

I think apt should have "rollback", but it would be a mess because of dependency chains and old packages being removed from repos. Anyway, that wouldn't save a machine that isn't booting. grub-btrfs is pretty cool for that, but then one has to use btrfs...

Interesting stuff... Then there is the whole trend with immutable distros and "image" updates. I think both ChromeOS and Android use A/B root partitions that swap after an update. If the updated root partition can connect to update servers then it becomes the "new" root partition and the other root partition is ready for updates. These partitions can resize on the fly. I think some overlay FS is involved for monthly updates. Never had a bad update on Android so they are doing something right.

And every motherboard should incorporate dual BIOS for safe updates.

[MadMax]

That's a pretty neat setup. Best-case you can accomplish the same with a LiveUSB key, but you are right, you sometimes don't have one at hand or it got overwritten with a different distro/you use it for something else at the moment etc.

Also depending on the UEFI setup it might be easier to boot into a system on the local drive instead of a USB key.

In theory you could set up a super-minimalistic recovery system with only Gparted, Timeshift etc. installed. Would probably take less than 5 GB of space.

[CharlesV]

@MadMax Interesting.. I am going to have to try that!

@dreamer I have only once had to 'look' for my TimeShift location.. and it was just as easy as Settings - Location and pointing to where they were. (of course... I dont run encrypted, so that would be another small issue -) )

[davidy]

What about just having your main OS disc with 2 equal partitions for root (and one boot/efi part) a perfect clone of each other. Then just "sync" (daily or whatnot) to the one your not booted to. If anything goes wrong just reboot into the (other)clone and keep working. When something goes wrong, reboot. The actual boot/efi part can essentially remain static, and backed up to an actual secondary drive. Not sure of the exact commands or software to be used to sync your running system but it sounds doable. Clonezilla could set it up as long as you can get the grub menu working with both OS's. It would negate the need to ever use a usb drive unless grub itself fails. Maybe I'm missing something but I think it might work unless there are some hidden files in session that are impossible to copy or that might prevent any subsequent successful boots if missing between partitions.

I like CharlesV's method but if I have to reboot to get it to work why not just (re)boot to a working system in the 1st place? You would of course have to have separation between clones as there would only be one 'backup' so that the error/s are not duped.
You guys are much smarter than I. I'm just brainstoriming possibilities for a single user with limited resources. My current method requires me manually booting to backup, and then if something goes awry, I have to boot twice over with a space in between for the restore.

[MadMax]

You would of course have to have separation between clones as there would only be one 'backup' so that the error/s are not duped.

I think that's the caveat here. Keeping this second installation up to date requires work and might be prone to other errors. Also it takes up a lot more space naturally. I think a dedicated recovery system is the more manageable approach.

[davidy]

Requires work? Timeshift requires no work and will copy most everything, I think. I just barely ran it but deleted it because it takes up tons of space. 2 jobs and I was over 50gigs already whereas a snapshot is smaller and a clonezilla backup even smaller still. I only need one good backup and that as recent as can be. Having 5 or more backups when all a user usually wants is to go back before the problem existed is a waste of space. The problem I see with timeshift is it doesn't simply duplicate folders which is what would be required. A simple rsync type affair of some sort on a schedule, like timeshift (freefilesync as root might work). My OS disc is wasted space just so my backups are small. Less than 7% of 1TB is used. As for errors maybe. You could easily leave it unmounted until you sync and then unmount again. For me the hardest part would be getting grub to see both clones. Wasted space is really the part of a large disc that never gets used. Any actual backups never amount to anywhere near the actual disc size being used. My clonezilla backup is 10GB atm, much smaller than a snap or a timeshift by far.

The real trick is in making sure the most recent backup even exists. I deleted both my backups on my laptop by accident after creating one and had to use one a month old off my external drive, or I would have had to install clean just to fix a mistake I made. I rather like the idea of rebooting and movin on. Most people I think who require recovery don't even know what went wrong much less how to fix it and just want to go back when the issue didn't exist.
Really I think the bottleneck is in that for most users they only think of one user, one os, one disc, etc., when only recently have pc's been shipping standard with 2 hd's. The 2nd being for backup and data. IMO it should be mandatory that computers ship with at least 2 hd's. Phones have 2 parts by default and yet if you ask the average person they would not know that at all. Cloning the OS to the 2nd part of the 1st disc would really be the dogs you know what. Next time I install clean I'm going to try it out. If phones can do it why not?

[DukeComposed]

Having 5 or more backups when all a user usually wants is to go back before the problem existed is a waste of space.

Timeshift backups are a waste of space, but keeping an identical copy of literally everything on a second partition isn't?

[MadMax]

Timeshift shouldn't waste that much space anyway. You either have btrfs and proper snapshots with deduplication or timeshift with rsync that utilizes hardlinks to deduplicate and keep space requirements to a minimum.

As you described you'd have to set up the sync job to keep the second OS up-to-date. That's enough room for error for me to prefer a static recovery partition, that never changes and always will boot.