MX Linux Forum

Having made multiple changes to my disk configurations, I needed to rewrite all my LuckyBackup tasks. Basically, my external backup disk /sdc has 3 partitions - General, Pictures and WinFiles (ntfs). I backup all the folders in my /home partition to one or other of these.

So I set up 'Copy the source directory' for /home/phil/Pictures to /mnt/sdc/ and ran the backup. Checked the backup drive and all seemed OK - I could view all the files.

Then it struck me that the process had completed rather quickly, so I looked at the log which said that only 9 files had been copied of about 9000 files I could see there.

You're probably ahead of me, but I'm a 'one step at a time' operator, so I deleted from the source one file that I didn't need. Sure enough, it was deleted from the backup.

Memo to self (which may help others): If you symlink your data to /home, don't use /home/phil/Pictures as the source, use /mnt/sda7. Lucky Backup will then copy the files, and not just the symlinks!

Phil

Yeah, I have screwed up as well when symlinks were involved in backup scenarios. Managed one time to clear my entire Desktop that way, where I kept a lot of temporary but important stuff. Luckily, I also use Amazon S3 for a daily backup, so could recover.

I use LuckyBackup to backup /home/don to /archives/Backup and also to /media/disk. /archives/Backup/don goes well. /media/disk/don has about 330 errors. /archives/Backup/don is owned by don:users and /media/disk/don is owned by don:root. I don't know if that is the problem or not. I tried changing the owner but it was'nt allowed.

symlinks are a great asset and can solve a lot of problems, but it's often better to find out why something isn't working and fix it, then use symlinks to solve the problem because the unseen consequences, like this, can bite you when you least expect it.

I have been using Wuala for online backup. Automatically syncs folders you choose. As far as security:

All files are directly encrypted on your desktop. Your password never leaves your computer. Not even we as the provider can access your files or your password.

Also available for Android and iPhone, so I have access to my desktop files from my phone via wifi or cell.

http://www.wuala.com/

richb wrote:I have been using Wuala for online backup. Automatically syncs folders you choose. As far as security:

All files are directly encrypted on your desktop. Your password never leaves your computer. Not even we as the provider can access your files or your password.

I also use Wuala, but the free storage is only enough for key files. The other 95GB has to be local backup.

Phil

GoManutd wrote:symlinks are a great asset and can solve a lot of problems, but it's often better to find out why something isn't working and fix it, then use symlinks to solve the problem because the unseen consequences, like this, can bite you when you least expect it.

I don't think the lesson from this cautionary tale is to avoid using symlinks. The lesson here is to make sure you know what you're backing up (and what you aren't). Obviously, backing up a symlink won't back up the data it points to.

I'm a huge fan of symlinks, but I don't use them "to fix things that aren't working." In fact, I doubt you could use symlinks to fix anything that isn't working. Their main purpose is to avoid having to maintain multiple copies of the same data when different applications need to utilize the same files, such as library files. Symlinks can also free up space in /home. For example, instead of keeping 15GB virtual machine files in my /home partition, I keep those files in a separate partition and create symbolic links in /home instead.

joany wrote:I don't think the lesson from this cautionary tale is to avoid using symlinks. The lesson here is to make sure you know what you're backing up (and what you aren't). Obviously, backing up a symlink won't back up the data it points to.

Definitely, joany. The problem was not with my use of symlinks, but with the use of my ageing brain!

Phil

not saying don't use them, the lesson about symlinks is to understand that there may be unforeseen issues. too often people use them because it provides a quick fix for certain types of problems, e.g. software that was packaged for a different platform, placing data in a different location, etc.

i used symlinks in prior versions of myth because it was far easier that trying to get myth configured to put data where i wanted it (this issue is not an issue in current versions).

symlinks are cool, useful, but the average linux user isn't likely to under that there are potential issues to using them.

I use Lucky Backup for some things. For example, when I upgraded my wife's laptop from Mepis 8.5 to Mepis 11, I used it to make copies of her home directory, then restored what she needed after performing a clean install (folders and docs, hidden .mozilla folder so that she'd have all of her firefox related stuff).

But, I also make backups of her entire home partition from time to time.

The most reliable way I've found to do that is using ddrescue after booting into a Mepis Live CD, since it performs a sector by sector copy of either individual partitions or entire disks, and since it's a sector by sector copy (ignoring the file system entirely), you get an exact replica of the original drive or partition.

For example, this command would copy /dev/sda1, and write it to an image file named 20120724_sda1.img, located on /mnt/sdb1 (a mounted partition on an external drive). Note that the third parameter (log file named /mnt/sdb1/20120724.log in the below example) is optional:

su
ddrescue /dev/sda1 /mnt/sdb1/20120724_sda1.img /mnt/sdb1/20120724.log

That way, I have an *exact* sector by sector copy of the partition (/dev/sda1 in that example) that I can restore to another drive, simply by reversing the process like this:

su
ddrescue /mnt/sdb1/20120724_sda1.img /dev/sda1

When you make an image file of a partition (20120724_sda1.img in that example, since I usually use the backup date as a file name), you can also mount it as a loop device (so that you can read and write to it, just like you can do with a physical drive). That lets you easily copy and paste needed folders and files from it if you don't need to restore an entire partition.

You can do the same thing using dd (only the syntax is different). Since ddrescue is already installed in Mepis, I prefer it since I can boot into a Live CD and run it, and it handles errors better than dd if you have a failing drive you're trying to backup (since it will make multiple passes if needed due to errors by default, only copying what it didn't copy OK with previous passes,which is the purpose of the optional log file (third parameter in the above example).

I also use clonezilla live to do the same type of thing (including making full disk image backups, not just partition backups), and it has some default compression options built in to save disk space. But, like dd, it won't handle hard errors like ddrescue will, and if you want to do things like mount partitions as a loop device, making a full sector by sector copy without using any compression is a better bet. Basically, there are pros and cons to any backup solution (including disk space requirements on the backup device).

Gaer Boy wrote:

richb wrote:I have been using Wuala for online backup. Automatically syncs folders you choose. As far as security:

All files are directly encrypted on your desktop. Your password never leaves your computer. Not even we as the provider can access your files or your password.

I also use Wuala, but the free storage is only enough for key files. The other 95GB has to be local backup.

Phil

LOL: I only backup my key files, my entire documents directory, and they only consume 1.8 GB. I guess it depends on what you consider key files.

Very rarely use symlinks, although they are a neat tool, I have not seen a great need to use them.

richb wrote:Very rarely use symlinks, although they are a neat tool, I have not seen a great need to use them.

I make extensive use of them. I have a partition labeled "BigStuff" where I keep

• .mozilla
• .thunderbird
• .VirtualBox
• Pictures
• Music
• Documents
• Flash
• Java

and a handful of other folders/files. I can boot into any of five Linux installations and see exactly the same data. Also, if I update Java or Flash, I only have to do it once.

DBeckett wrote:

richb wrote:Very rarely use symlinks, although they are a neat tool, I have not seen a great need to use them.

I make extensive use of them. I have a partition labeled "BigStuff" where I keep

• .mozilla
• .thunderbird
• .VirtualBox
• Pictures
• Music
• Documents
• Flash
• Java

and a handful of other folders/files. I can boot into any of five Linux installations and see exactly the same data. Also, if I update Java or Flash, I only have to do it once.

That's a really neat idea, as I tend to replicate a lot of stuff between different distros installed on the same system. Do you ever run into conflicts between distros due to different versions of Firefox being installed; or because you're trying to share the same configuration files between different types of distros (those based on Debian versus Ubuntu versus Fedora versus OpenSUSE, etc.)?

DBeckett wrote:Also, if I update Java or Flash, I only have to do it once.

Thanks for that input. That's exactly why symlinks are so useful.

If you poke around your / (root) partition, you'll find there are symlinks everywhere. They probably number in the hundreds, and they were created at the time your system was installed or when software packages were installed. So everybody who uses Linux also "uses" symlinks, whether they know it or not.

JimC wrote:That's a really neat idea, as I tend to replicate a lot of stuff between different distros installed on the same system. Do you ever run into conflicts between distros due to different versions of Firefox being installed?

I neglected to mention that I use the Firefox tarball method, and it too is on the BigStuff partition. Therefore, if it gets updated once, it's updated for all. The same with Thunderbird, although I don't update that.

I'm careful with some apps, VirtualBox for instance, to make sure I have the same version on each. But they all share the same .virtualbox. That would be 4GB on each distro if they were independent, a waste of 16GB. My HDD is 160GB, so 16GB is a significant chunk.

I did get jammed up once when I decided to try one of the many new versions of Thunderbird to see what all the fuss was about. It worked okay, but then I discovered that .thunderbird had been trashed for all the other distros still using 2.0.0.24. Thank goodness for backups.

I use a script to do my backups. I have to remember that cp /home/dale/Documents will not work because that's a link. It has to be cp /mnt/sda3/Documents.

I like CrashPlan. It backs up locally and on the cloud (all encrypted). I'm paying $5/mo for unlimited backup. I store all of my important files on their servers and back up everything to an external drive too. I also back up my wife's computer to my external drive. It all happens automatically in the background so I don't have to think about it. I can also access my online files from any web browser if needed. Nice to know if something catastrophic happens such as my house burns down.

Wuala looks good too though.

Years ago I had backup software fail me, so now I just copy my data to a second hard drive. I don't worry about configuration files. Just email (which is now on the web) and a bookmark file is all I need beyond the normal /home files. kiss

fbt wrote:Years ago I had backup software fail me, so now I just copy my data to a second hard drive. kiss

Exactly.

I guess it depends on how you use your system. I still am quite happy not using symlinks I do not create. That is just the way it is for me.
But I appreciate the functionality it provides for you who use them.

I prefer the Clonezilla or PartedMagic gtkrsync solution. It's GUI and rsync will not copy empty space. With compression the resultant archive is very small. You can also have the program make 4 GB files for burning to DVD.
A portable external HDD is also a really good investment.

Yes, an external drive is definitely a must, I think. Now they are so cheap there's no excuse. My latest desktop has an E-SATA connection so transfers are really fast. As for manual copies, I get what you are saying. What I found was, however, that I'd forget to do it as often as I should have. Having it automatic means I don't have to worry about it. The most I loose is maybe a few hours. I run a decent sized company from my computer, so if something happens to my data it costs us real $$.

I'll agree that a business should have an automatic backup system. It should probably try to be as good as a mirrored array.

I installed a second HDD for backups (an external drive would work, but transferring data to the external would be too slow with my machine).

My VirtualBox folders and my / (root) and /home are on three separate partitions so I can back them up separately using Acronis True Image. (There's no point in backing up everything if I only make a change to one of these.)

All other data, such as Thunderbird email folders, photographs, music, and important documents are stored in sub-directories in a fourth partition. I use Lucky Backup for these (clone mode).

If the main HDD crashes, everything can be retrieved from the backup HDD. If the backup HDD crashes, the original data are still on the main HDD. No worries, unless some catastrophic event takes out both HDDs.

If I had multiple computers or a laptop, I might try other backup methods. But for me this works best.

joany wrote:No worries, unless some catastrophic event takes out both HDDs.

FWIW, I did have that happen -- a blown PSU took out the mobo and both hard drives. That's what forced my migration from M6.5 to M8.0.

I now back up to an internal drive, back that up to an external drive, and back that up to a NAS box. If my house burns down I'm hosed, but otherwise I think I'm covered.

if you really have sensitive stuff that you absolutely cannot afford to lose, ever, uncle mark's approach, along with some form of offsite backup is really what you would need.

keep in mind that if you use DVDs or CDs as backup media that these don't last as long as everyone originally had predicted. the better quality can last years, lesser quality can begin failing within 18 months.

one crucial component to backup schemes is periodically, and religiously testing restoration procedures to ensure the procedures are correct AND the backups are valid. cannot tell you the number of sys admins that i've talked to who did daily backups only to find that something went wrong with the media, the backup process silently failed, etc.

uncle mark wrote: FWIW, I did have that happen -- a blown PSU took out the mobo and both hard drives. That's what forced my migration from M6.5 to M8.0.

I now back up to an internal drive, back that up to an external drive, and back that up to a NAS box. If my house burns down I'm hosed, but otherwise I think I'm covered.

Holy cow!! What a bummer. I might have to reconsider getting that external HDD after all.

My routine is almost identical to uncle mark's except I occasionally make DVD copies too.

I'm with DBeckett on this. I use top of the line media and only store optical disks in jewel cases. I have backups of data on DVD that are over ten years old. I expect them to outlive me.

you'll certainly get a lot more life out of the top quality discs, but they don't last forever. archiving companies, like iron mountain, have been rethinking the use of optical discs for longterm storage because they do fail.

joany wrote: Holy cow!! What a bummer. I might have to reconsider getting that external HDD after all.

My external is a USB-to-SATA dock I got from NewEgg for like $20, with a 200G drive stuck in it that I salvaged from a fried machine.

I back up about 100G with Lucky Backup. The first run took forever of course, but now they take 3-10 minutes depending on whether or not I've added ISOs since the last.

The back up I do to the NAS box takes forever. I gave up trying to get Lucky to work across the LAN, so now just do it manually on a weekly basis on a smaller subset of my data (I omit all the ISOs I've collected -- most could be recreated from download or disk if needed).

.
I made two copies of my one .virtualbox VDI a couple of years ago, one on external HDD and another on DVD. I've never bothered to back it up since then. It's replaceable.

I keep other large things and the few ISOs I have on a special partition for such things, but don't back them up either. I either don't care about them that much or they're replaceable, not worth tying up a bunch of disk space.

I am however religious about making fairly frequent images.

GoManutd wrote:you'll certainly get a lot more life out of the top quality discs, but they don't last forever. archiving companies, like iron mountain, have been rethinking the use of optical discs for longterm storage because they do fail.

Archiving companies are in the business of making money. So if you think that they'd not exaggerate any faults of any other solution, and/or enhance any benefits of their own solution, I've got a bridge I want to sell you.

I might worry beyond 20 years for top-quality media stored correctly. If I needed to archive family photographs I'd expect more than 100 years for gold archive disks.

The people (customers) you are talking about expect/want as unlimited a time as possible.

The only truly safe method (for home users) at present is to make copies at intervals. Keep multiple copies. Have at least one copy off-site. For the home a good optical burner is tech that's in reach.

There are better technologies for archiving than DVD or BlueRay-type burners.

My brother (who runs Windows and MacOS) has had 4 HDDs quit on his Windows machine in the last 8 or 9 years. He's not real good at backing up. His solution was to use a mirrored array. He hasn't lost a bit of data in those years.

Okay, I'm convinced -- I've got two external drives, at least one of which is reliable (the other had a failure and has some bad blocks most likely related to the computer it was mounted in falling out of a van onto concrete, but with the bad blocks marked/locked out, it should be okay); neither one is large by modern standards, but both are big enough to do multiple "live" mirrors of my entire MEPIS install (root and complete /home partition), never mind compressed backups -- though neither comes close to being able to back up my NTFS volumes (one has almost 200 GB of photos on it, for instance). Next time I have a few dollars to spare, I'll get a new terabyte drive (around $75 shipped and sales taxed) and drop into the case with the damaged drive and be able to back up my terabyte NTFS drive (probably two rotating backups, though data on that drive doesn't change rapidly any more except for e-mail, bookmarks, etc. that are still there from before MEPIS).

For insurance, I'd like one of the MEPIS backups to be literally live, bootable and fully mirrored for both root and /home -- I'll start another thread in another location on how to do that (I can think of a couple potential ways). A USB external drive is pretty slow, but it'd be better than a thumb drive, because persistence is the normal setup for a hard drive -- and lots better than a live DVD, especially since my machine doesn't have a DVD drive.

Thanks for writing. I haven't yet adopted the practice of using symlinks, so your story won't apply to me. But am surprised to hear luckybackup already comes with MEPIS. I understand it is a user interface for rsync. I have never used it, but will give it a whirl when it comes time to backing up files. First, I will look for a tutorial on the web.