[Bug Report]Backup/Restore Web Interface

[RandomUsername]

On the backup management page, if you tick the "Encrypt Data" check box and save the settings, when the page is reloaded the box is no longer ticked. Therefore, if you make a change to the job and save it the encryption is turned off unless you remember to tick the box again. As the data encryption options are hidden until you choose the show them, this is very easy to miss and is a potential security hole.

[EDIT]Whilst I'm reporting this I may as well mention that there seems to be a limit of 24 characters for the encryption key. Any more and the web interface can't seem to load the job data.

[RandomUsername]

New bug: the username for my ftp server is my email address but the web interface doesn't allow "." or "@" in the username. I have to manually edit the jobdata file.

[pa]

Hi,

Thanks for highlighting these issues to us, then at least we will have the chance to do something about it.

/PA

[RandomUsername]

No problem Pa (how am I supposed to write your name? Is it Pa like a name or PA like initials?). Thought about using Launchpad or something like that to track bugs?

[pa]

Hi, regarding my name it is initials so it is "PA".

We are debating whether to expose our bugtracker or not, but at the moment we have decided to have it internal only.

/PA

[RandomUsername]

PA, sorry to sound like I'm ranting here but the entire backup and restore setup seems to be totally screwed. Every time I isolate and fix one bug I find another. The result is that I just can't get a stable and reliable backup.

I've got examples in logfiles where the uploading of a volume has failed but the duplicity backend then indicates the volume has been processed successfully - wtf?

Several times a big backup (I'm talking a few GBs) has ran all the way through and then bombed out uploading the signatures file right at the end (I suspect due to the size of it but I don't know why yet). I run the same backup again and all it does is upload the signatures file so it seems it's recovered quite gracefully but then none of the files show up as available to restore in the web interface.

If I remove all the local signature files (which I've done to test a couple of bug fixes) the option "Click to retrieve backup information from backup target" shows in the web interface but this option hasn't worked for me once.

I've already made several fixes to both the backup.pl and the ftpbackend.py scripts - e.g. I discovered a problem where if duplicity failed to upload a file but failed, when it retried ncftp was just hanging waiting for confirmation to overwrite the existing file. I fixed this by adding -f to all the "put" commands in ftpbackend.py.

I realise some of these bugs lie in duplicity itself rather than the scripts you've written for the Bubba but having the backup feature built in was one of the things that attracted me to it. You need to get this fixed asap - get a more up to date version of duplicity in there for one thing. The newest versions don't even use ncftp but ncftpput/ncftpget which is far more sensible as the former is meant for interactive use and the latter for scripting.

90% of the time I've been interacting with by Bubba | Two has been spent trying to get the damn backups to work. I haven't touched on any of the more interesting features of it yet because I want to be sure I've got a good set of offsite backups as a priority.

Frankly, this whole situation is very frustrating.

Darren.

[pa]

Hi Darren,

I have to give you a lot of credit, not many users actually even bothers use a backup, and very few would even try to find out what is going wrong if it did not work. So we definitely appreciate both your work, and very much that you do tell us in a sensible way so that we can add it to our bug-tracker.

The example you state with the logfiles saying that it failed and duplicity then reporting success is in my opinion one of the more troublesome, since that totally lies within duplicity. That you can not retrieve backup information is also worrying, but that is at least our own code... Not that is is any better, but a bit easier for us to track down.

Upgrading duplicity is pretty high on our list of things todo, we are aware that quite a few bugs have been fixed there, especially when it comes to backups using ftp.

/PA

[RandomUsername]

Backups are very important to me, I've seen too many people lose family photos and such like because they couldn't even be bothered to burn a spare CD or whatever.

Again, sorry for the venting but I'm a persistent bastard and I'll get this working if it's the last thing I do. If I make any serious progress I will let you have any work I've done if it helps.

Don't get me wrong, I love the whole concept of the Bubba and only want to make it better

[asparak]

Keep going. I'm following your work as well.

I stopped using Duplicity as my backup engine, because it was just too slow and full of bugs and instead switched to an rsync based backup strategy for now. Nowhere near ideal, but at least I can incrementally back-up some 450GB of data fairly quickly.

I too have seen way too many people lose data and be willing to pay anything for me to try to recover their files.

If you have bug fixes you want somebody else to sanity check, let me know. Quite happy to work with you on things.

Even considering buying a second Bubba so I can test in a proper sandbox. This little tool has such possibilities, very willing to help develop it.

[pa]

Hi,

A question here, we are considering adding a "sync" option, where the selected files would be rsynced on a remote target. In this case, the history of the files would be lost, but if we decide to go that way, would that be enough of backup for you?

Just curious on how you guys use backup-systems.

/PA

[RandomUsername]

Hi PA.

I think some kind of versioning system should be in place otherwise you might not notice a corrupted file until it's already been synced.

[asparak]

Doing incrementals via rsync is fairly easy to implement, I do it via crontab and maintain 1 full plus 2 incrementals at any one time.

http://www.mikerubel.org/computers/rsync_snapshots/ is a fairly comprehensive example of how to do this, or http://www.samba.org/rsync/examples.html.

[RandomUsername]

The problem with rsync is that it's rarely found on remote servers (unless you're able to install it yourself). There are any number of companies supplying ftp based storage, very few supplying rsync.

[asparak]

Yep,

I agree. Rsync isn't a solution for everyone or everything. I'm lucky, I control the remote server as well as this end, so can control both ends of the process.

Don't want to get into a heated discussion about backing up over FTP or SFTP but I agree, we do need a working solution.

Having duplicity fixed would be good, or replace it with one of the other apps out there that do the same thing, so that we cater for those less technically able to hack Linux from a command line.

[jmv]

Hi all,

Just few words on my usage of backups with bubba.

with my 2 previous bubbasV), I used rdiff (http://rdiff-backup.nongnu.org/) to backup from one machine to another one (cross-backups)

rdiff-backup backs up one directory to another, possibly over a network. The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago. The idea is to combine the best features of a mirror and an incremental backup. rdiff-backup also preserves subdirectories, hard links, dev files, permissions, uid/gid ownership, modification times, extended attributes, acls, and resource forks. Also, rdiff-backup can operate in a bandwidth efficient manner over a pipe, like rsync. Thus you can use rdiff-backup and ssh to securely back a hard drive up to a remote location, and only the differences will be transmitted. Finally, rdiff-backup is easy to use and settings have sensical defaults.

with BubbaV2, i tried duplicity. Unfortunately, i got some errors,too (fo example, in changing the the name of the machine ... the backup failed) AND I am not satisfied by the .gz files instead of clear and browsable miror directories.

my 2 cents

jmv