Performance of different backup options

mountaindude · Post by **mountaindude** » 25 Aug 2007, 12:07

I am looking at syncing directories of files on my PC with my Bubba.
The idea is to keep an identical copy of the PC's directory structure on the Bubba.
I want the Bubba to initiate the sync, and have been looking at rsync and Samba for solving this.

With Samba, I would use a Samba client (smbclient) on the Bubba, mounting the Windows share into /mnt/...

With rsync, I would need some rsync client (server?) running on the PC as well, I guess.

My question: Which of the above options are best from a performance point of view? The directory structure on the PC that I want to sync to the Bubba is a few hundred directories with a few thousand files.
Anyone has experience of this kind of setup?

Thanks,
MD

Mange · Post by **Mange** » 25 Aug 2007, 16:14

Hey dude.

You absolutely want to go with the rsync approach. Rsync uses a very clever incremental backup algorithm, which means that, if you change something on your PC, your Bubba only has to download the part of the file which has changed. Samba on the otherhand, would (I guess) see all files as atoms, with the only option to overwrite.

I use this approach myself, and the speedup gained from this incremental backup is amazing. I have also written a small bash script that e-mails me the rsync output after each backup job completes, very nice!

You will have to install a Rsync server on each PC you want to backup, and then have the Bubba logon to these periodically and pull files. One of my machines run Windows XP, and it's not very difficult to setup Rsyncd to run as a service.

I personally use a pre-compiled Rsync server for Windows, provided by the good folks of Sourceforge project BackupPC. You can fetch it here cygwin-rsyncd-2.6.8_0.zip. It has several patches to improve speed when using Cygwin (basically a framework that allows you to compile Linux apps in Windows environments).

The included Readme-file is pretty straightforward. But the general idea is:
1) Unzip contents of cygwin-rsyncd.zip
2) Open rsyncd.conf (this is the main config file). Define "network shares" as you wish (they are called Modules in rsync). By looking at the examples, you should figure out the syntax.
3) Make sure the config directive "secrets file" points to an existing file, preferably in your rsyncd-dir. This file contains all users and passwords who are allowed to connect to each Module (share). Use slashes in paths, so lets say you have C:\program\rsyncd\secrets.txt, the path would be c:/program/rsyncd/secrets.txt
5) It might be a good idea to only allow connections from your bubba, so enter its ip in the hosts allow directive.
6) Open your secrets file, create if it doesn't exist. Add users, one per line with format username:password. Like "bubba:mypassword".
7) To start and make the rsyncd server autostart with windows, you have to edit the service.bat file. It basically registers the service with windows, but you have to make sure all paths correspond to your installation. Please note the forward slashes in pathnames!

This should be it, now you can connect with Rsync from your bubba server and let it do the initial backup. Once that is finished, you can simply let it run again and it will automatically try to make everything up-to-date.

You can verify that you server works, and list all files within a share (module) by this command:

Code: Select all

rsync --list-only username@X.X.X.X::MYMODULE

Where MYMODULE is the modulename, X.X.X.X is server ip (your windows box) and username is, well, username!

To perform the actual backup, simply drop the --list-only parameter and add -av (that means -a for archive, -v for verbose).

Other options I find useful:

Code: Select all

--delete
If you delete a file on your PC, and your bubba has backup'd it before, you may want the file to remain on the bubba. However, using this option, rsync will remove all deleted from its backup archive. Essentially, you get an exact mirror of your PC.

--stats
Shows some statistics at the end of each job. I use this and pipe the output to an email. It shows stuff like files transfers, amount transfers, and speed-up thanks to the smart algorithm.

--exclude-from=filename
Excludes certain files, matching patterns found in the given filename. This can be good if you have a share of a complete drive, and want to backup EVERYTHING except, lets say C:\Temp .. then you can add temp to the exclude file, and it will skip it. The patterns are standard regex-ish.

Hope this helps![/list]

mountaindude · Post by **mountaindude** » 29 Aug 2007, 02:22

Hey Mange, awesome post - many thanks. Worked great except that non-English characters in file names aren't transferred correctly, but I guess that's part of Bubbas localization. Maybe it's possible to install enhanced character set support..

Still have some work to do finetuning security and access levels of the various files, but once that is done it's a neat solution.

The bash script you made for emailing the results, is that something you could share?

Edit:
Found another topic giving a basic example of how to do this http://excito.org/forum/viewtopic.php?t=415

Tada,
/MD

Mange · Post by **Mange** » 29 Aug 2007, 07:18

Yo

Glad you liked it. I had totally missed the howto by Excito on this, but I guess this post can remain since it covers a different non-ssh & windows approach. I should mention that, using this approach your backup transfers are not encrypted, but you do however gain a lot of performance from this.

Just for the reference, here is my bash script, it includes a log (from rsyncd) of which files that were transferred, and also a summary at the end. Note that this may sometimes cause your bubba to send very large emails (like several megs of text as non-attachments) and thus you may need to make sure your e-mail client can handle it

I personally use Opera for e-mail reading, and it sometimes has problems handling emails like this (freezes for a few mins upon arrival). Its usually not a big deal, because it only happens when something really big has happened in my dir structure, triggering a major backup.

Code: Select all

#!/bin/sh

##### RSYNC SETTINGS #####
MODULE="docs"
RSYNC_PASSWORD="1backup1"
RSYNC_USER="backup"
RSYNC_ADDR="192.168.0.90"

##### BACKUP SETTINGS ##### 
EMAIL="my@email.com"              # Address to receive backup-reports
DATAROOT="/mnt/hd"                     # NO TRAILING SLASH! This is where we store all backupjobs
LOGROOT="/mnt/hd/backup/logs"          # NO TRAILING SLASH! Dir for logfiles
EXCLUDEFILE="/mnt/hd/backup/wonk.exl"  # List of files to NOT backup. Point to empty file to exclude none
JOBNAME="wonk"                         # Jobname, just some unique name (typically name of machine being backed up)

#### DO NOT EDIT ANYTHING BELOW THIS LINE #### 

export RSYNC_PASSWORD 

if [ "$1" = "list" ]; then
  rsync --exclude-from=$EXCLUDEFILE --list-only $RSYNC_USER@$RSYNC_ADDR::$MODULE
elif [ -n "$1" ]; then
  CURRTIME=`date +%Y%m%d_%H%M`
  N_CURRDATE=`date +%Y%-m%-d`
  N_CURRTIME=`date +%H:%m`  
  rsync --log-format="%b of %l: %f" --exclude-from=$EXCLUDEFILE --delete --stats -av $RSYNC_USER@$RSYNC_ADDR::$MODULE $DATAROOT/$1/ > $LOGROOT/$JOBNAME-$CURRTIME.log 2>&1 
  cat $LOGROOT/$JOBNAME-$CURRTIME.log | mail -s "$JOBNAME ($1): ($N_CURRDATE $N_CURRTIME)" $EMAIL
else
  echo "usage: backup.sh <backupname|list>"
fi

And its usage is typically like this:

Code: Select all

backup.sh list

Lists all files to be backed up, with exclude file in effect. I sometimes use this to "debug" my exclude patterns.

Code: Select all

backup.sh somedir

Causes the script to make a full backup and place it into /mnt/hd/somedir (you can change the root dir by DATAROOT variable). The reason to why this is configurable is that I do several backups, for instance I do both backup.sh daily (once per day) and backup.sh weekly (once per week), both using the same script.

mountaindude · Post by **mountaindude** » 29 Aug 2007, 07:59

Good point about the size of the email. If the idea is to send an SMS to your phone with a backup status report you may want to make it shorter...

Mange · Post by **Mange** » 29 Aug 2007, 13:46

Hehe yeah

In 99% of the cases, the stats summary generated by --stats is more than enough to get on email... and that would probably fit into a SMS or a few concatted ones.

mountaindude · Post by **mountaindude** » 17 Sep 2007, 18:00

Got troubles getting this to work with multiple source directories/modules..

Setup is a WinXp machine with rsync server installed.
4-5 different directories/ rsyncd modules defined, that are to be backed up to the bubba.
Guess you could do a loop in a backup script on the bubba, calling rsync individually for those modules, but I recall rsync should be able to handle multiple sources... Anyone done this?

Thx

John W · Post by **John W** » 04 Nov 2007, 08:10

Thanks alot Mange.

But i wonder what "%b" represents in "rsync --log-format="%b of %l:"

Thanks.

John W · Post by **John W** » 22 Nov 2007, 16:10

I thought i share my batch script!

This is what it looks like:

Code: Select all

#!/bin/sh
RSYNC_PASSWORD="YOUR_PWD"
MESSAGE="/tmp/rsync_log.txt"
if [ -n "$1" ]; then
/etc/init.d/led_blink > /dev/null 2>&1
echo "Your backup of $1 started:" > $MESSAGE
DATE=`date +"%Y/%m/%d %H:%M"`
echo $DATE >> $MESSAGE
echo "With the following log information:" >> $MESSAGE
export RSYNC_PASSWORD
rsync -av --delete --stats YOUR_USR@YOUR_ADDR::$1 /var/rsync/backup/$1/ >> $MESSAGE
echo "And ended at:" >> $MESSAGE
DATE=`date +"%Y/%m/%d %H:%M"`
echo $DATE >> $MESSAGE
LOGDATE=`date +%Y%m%d_%H%M`
cp /tmp/rsync_log.txt /var/rsync/log/$1/$LOGDATE.log
cat /tmp/rsync_log.txt | smbclient -M JOHN -U RSYNC
/etc/init.d/led_off > /dev/null 2>&1
else
echo "usage: backup.sh <backupname>"
fi

To backup "document" i run "backup.sh dokument".
The logfile ends up in "/var/rsync/log/dokument/" with the filename (in this case) "20071121_2204.log".

As you could see i added:

Code: Select all

cat /tmp/rsync_log.txt | smbclient -M JOHN -U RSYNC

What it will do is to send the content of the logfile to my WS as a popup-message.

You will have to install the smbclient-package to get it to work.

My cron-tab:

Code: Select all

0 21 * * * /var/rsync/cron/backup.sh dump
0 22 * * * /var/rsync/cron/backup.sh dokument

// John W

ian · Post by **ian** » 23 Nov 2007, 09:29

I'm actually using a suite of REXX programs (written by me), to back up either an entire disk, or selected folders, to my bubba. This approach works pretty well, and doesn't require anything to be installed on the bubba, which is an advantage if you're a Unix-noob like me! It can be quite slow, though I suspect a lot of it has to do with our old 11MBps WiFi network...

If anyone already has IBM ooREXX installed (which is now free and available from the sourceforge website), then feel free to try my programs, available from http://www.ian-chapple.com. They have only been tried from a Windows PC, so I'm not sure if they work for any other OS.

Cheers, Ian.

dsp76 · Post by **dsp76** » 02 Apr 2008, 08:12

mountaindude wrote:... Worked great except that non-English characters in file names aren't transferred correctly, but I guess that's part of Bubbas localization. Maybe it's possible to install enhanced character set support..
...
/MD

Hi,
I can agree, its a pretty forward setup working fine. However I also experience problems with non-english characters. Is this a failure of cygwin or the bubba debian sarge release? Like other codepage problems?

... here is an error message caused by that...

Code: Select all

file has vanished: "ygdrive/c/WINDOWS/system32/D:/$privat/SOMEPATH/E-Mail-Empf+Ã±nger.MAPIMail" (in dDrive)

Excito team - please comment, would an Etch upgrade be a possible solution?