Backup using CrashPlan (installation guide/howto)

Got problems with your B2 or B3? Share and get helped!
Post Reply
rxmd
Posts: 22
Joined: 14 Jun 2011, 10:32

Backup using CrashPlan (installation guide/howto)

Post by rxmd » 18 Mar 2012, 03:25

Here's a little installation guide for the CrashPlan backup software. I use this to have my B3 make automated backups.

CrashPlan (http://www.crashplan.com) is a service for backing up data to the cloud. It runs continuously, monitoring files for changes and making continuous incremental backups that are encrypted and stored on CrashPlan's servers. CrashPlan uses its own client software written in Java. The nifty thing about the client software is that you can run it in headless mode on a headless server (such as a B3), and run the client portion on another PC with an encrypted connection to the headless server via a SSH tunnel.

So here's how to get the Java client running on the B3 for automatic backups. This guide uses information from some other sources, including here and here.

0. Prerequisites: Installing the Java runtime environment

CrashPlan is a Java application, so we need to install the Java runtime environment (JRE). On our B3 with its ARM processor architecture, we cannot use the original Sun JRE, so we will use OpenJDK instead. Since our B3 is a headless server, it is sufficient to install the headless version of the OpenJDK JRE.

Code: Select all

$ sudo apt-get install openjdk-6-jre-headless
1. Downloading and installing CrashPlan

Download the Linux installation version for CrashPlan. It is sufficient to do this once, CrashPlan will autoupdate itself. You can find out what the latest version is by looking at the CrashPlan download site, http://www.crashplan.com/consumer/downl ... l?os=Linux.

The choice you have to make at this point is whether you want to install and run CrashPlan as root or under some other user account. If you run CrashPlan under some user account other than root, it will obviously be able to backup only those files that are readable by this user; so if you have multiple users on your B3 and want all their data backed up separately, you'll have to create an extra "backup" user and think up a permissions scheme that will allow the backup user to see the data of everybody else, while still hiding their data from each other. Running CrashPlan as root is somewhat more convenient in that root can read everything by default, but it also means that you trust a third party binary blob piece of software to do basically anything it wants on your server. In my case, I only have data stored under one user account on the B3 anyway, so I simply install CrashPlan under that user account.

Code: Select all

$ cd ~/downloads
$ wget http://download.crashplan.com/installs/linux/install/CrashPlan/CrashPlan_3.2_Linux.tgz
$ tar xfzv CrashPlan_3.2_Linux.tgz
$ cd CrashPlan-install
$ ./install.sh
CrashPlan will ask you where to install the software. I chose to install it to /opt/crashplan. You may have to create this directory beforehand, using something like this:

Code: Select all

$ sudo mkdir /opt/crashplan
[sudo] password for your-user-name:
$ sudo chown your-user-name.users /opt/crashplan
The installer will probably also tell you that your inotify limit (the number of files that your Linux kernel can simultaneously watch for changes) is set rather low. CrashPlan needs to be able to do continuous monitoring of file changes in order to do incremental backups, and since full backup sets can easily run to hundreds of thousands of files, the default value of 8192 is indeed often insufficient. We will change this later.

After installing, CrashPlan will tell you that it has already been launched, but in reality it hasn't.

2. Patch the CrashPlan files

At this point CrashPlan will not work yet. The problem is that CrashPlan still assumes that it is running on an Intel-compatible machine. In particular, it makes use of the Java-to-Unix library (libjtux) via an interface known as Java Native Access (JNA). For this, CrashPlan includes a precompiled Intel version of libjtux, which of course is useless on the ARM architecture of our B3.

Install the Debian version of JNA, and tell CrashPlan where to find it. In order to do this, you'll have to make some changes to the CrashPlan launcher script:

Code: Select all

$ sudo apt-get install libjna-java
$ cd /opt/crashplan/bin
$ nano CrashPlanEngine
Find the line starting with FULL_CP, and edit it to include the Debian JNA library we just installed:

Code: Select all

FULL_CP="/usr/share/java/jna.jar:$TARGETDIR/lib/com.backup42.desktop.jar:$TARGETDIR/lang"
Now the system will still try to use the Intel-compiled version of libjtux that cae with CrashPlan. To install an ARM-compiled version of libjtux, first make a backup of the old Intel version:

Code: Select all

$ cd /opt/crashplan
$ cp libjtux.so libjtux.so.bak
Either download the binary libjtux.so for ARM that is attached to this post for convenience, or compile it yourself.

2A. Alternative 1: Use an ARM-precompiled version of libjtux.so

If you downloaded the attached version, just copy it over:

Code: Select all

$ cd wherever-you-downloaded-the-libjtux.so.gz-file-on-your-B3
$ gunzip libjtux.so.gz
$ cp libjtux.so /opt/crashplan
2B. Alternative 2: Compiling libjtux.so yourself

If you want to compile it yourself, you'll need the libjtux sources from http://basepath.com/aup/jtux/jtux.tar.gz. Also you'll need a patch for libjtux that contains some bugfixes as well as a makefile (the original jtux sources are missing a makefule). You can find the patch attached to this post, or you can download it from the discussion thread at https://crashplan.zendesk.com/entries/3 ... sheevaplug (you may need to be logged in to the CrashPlan site to see that link).

Code: Select all

$ cd wherever-you-downloaded-the-jtux.PS3-YDL6.1.patch.txt.gz-file-on-your-B3
$ gunzip jtux.PS3-YDL6.1.patch.txt.gz
$ sudo apt-get install openjdk-6-jdk
$ sudo apt-get install patch
$ sudo apt-get install make
$ sudo apt-get install gcc
$ cd ~/downloads
$ wget http://basepath.com/aup/jtux/jtux.tar.gz
Download the jtux patch using the browser, and put it into the same downloads directory.

Code: Select all

$ tar xfzv jtux.tar.gz
$ cd jtux
$ patch < ~/downloads/jtux.PS3-YDL6.1.patch.txt (or wherever you put it)
patching file jtux_file.c
patching file jtux_network.c
patching file jtux_posixipc.c
patching file jtux_process.c
patching file jtux_sysvipc.c
patching file jtux_util.c
patching file Makefile
The Makefile is still written for the Sun Java JDK, so that we need to edit it to make it point to the OpenJDK for ARM that we just installed on our B3.

Code: Select all

$ nano Makefile
Comment out the first line (with JAVA_INCLUDE) and replace it with

Code: Select all

JAVA_INCLUDE=/usr/lib/jvm/java-6-openjdk/include
Compile libjtux:

Code: Select all

$ make
The compiler will throw some warnings that we will ignore.
Copy the generated libjtux binary into the Crashplan directory:

Code: Select all

$ cp libjtux.so /opt/crashplan
3. Other server-side configuration

3.1 Filename encodings

My file names are in UTF-8, and I have some files with non-ASCII characters in their name. So in order for them to be stored correctly, I have to make sure that everything, the Java VM included, is running in UTF-8 encoding:

Code: Select all

$ sudo nano /etc/profile
Make sure to add the following lines near the end:

Code: Select all

LANG=en_US.utf8
LC_ALL=en_US.utf8
export LANG LC_ALL
3.2 inotify Open files monitor limit

CrashPlan uses the inotify kernel module to keep track of changed files. By default, inotify is configured to monitor 8192 files. Depending on the files you want to back up, this may not be enough (my backup set currently contains about 100.000 files). So it's a good idea to increase the inotify count.

Code: Select all

$ cat /proc/sys/fs/inotify/max_user_watches
8192
$ sudo echo 1048576 > /proc/sys/fs/inotify/max_user_watches
$ sudo nano /etc/sysctl.conf
Add a line towards the end:

Code: Select all

# Set up inotify kernel module to keep watch on up to 1M files
fs.inotify.max_user_watches=1048576
4. Accessing CrashPlan from your client computer

Download the CrashPlan desktop for your client computer from http://www.crashplan.com/consumer/download.html and install it.

Start up CrashPlan on the B3:

Code: Select all

$ /opt/crashplan/bin/CrashPlanEngine start
In order to connect the CrashPlan Desktop from your client computer to the headless engine running on your B3, you'll have to set up an SSH tunnel with port forwarding on your client computer. The detailed instructions how to do this vary, depending on the operating system on your client computer. There is a relatively detailed set of guidelines how to do this on the CrashPlan site: http://support.crashplan.com/doku.php/h ... ess_client

I use a Windows client and have set up the Putty SSH client to open an SSH tunnel from port 4200 on the local machine to port 4243 on the B3.

Follow the instructions on the CrashPlan site, connect the CrashPlan Desktop on your client computer to the CrashPlan engine on the B3 and set up your user account and files to back up.

5. Cleanup (optional)

If you followed alternative 2B above and compiled libjtux.so yourself, you may be left with all sorts of development-related software. If you normally don't do any software development yourself, you can remove them:

Code: Select all

$ sudo apt-get remove make
$ sudo apt-get remove patch
$ sudo apt-get remove gcc
$ sudo apt-get remove openjdk-6-jdk
$ sudo apt-get autoremove (WARNING this might be dangerous, check displayed package list before proceeding!)
6. Automatically starting up CrashPlan with the B3

In order to start up CrashPlan automatically at system startup with the B3 so that it will back up all the time, you have to add it to the system startup scripts. Fortunately, CrashPlan already includes a Debian startup script that we can copy and modify for our purposes. What we have to do is to modify the script so that instead of starting up the CrashPlan engine as root, it will start up the engine under whatever username we installed it:

Code: Select all

$ cd /opt/crashplan/bin
$ cp crashplan crashplan.user
$ nano crashplan.user
In order to do this, we have, to take all the instances where it says $SCRIPTNAME start, $SCRIPTNAME stop etc. (five of them) and modify them so that they say

Code: Select all

su your-user-name -c "$SCRIPTNAME start" (stop, restart etc., with replacing your-user-name with your username)
This will start the CrashPlan engine under your chosen username.

Now link this file into /etc/init.d and enable automatic startup:

Code: Select all

$ sudo ln -s /opt/crashplan/bin/crashplan.user /etc/init.d/crashplan
$ sudo update-rc.d crashplan defaults
At the next system startup, you can fire up htop or whatever process monitor you use to see that CrashPlan is indeed hard at work.
Attachments
jtux.PS3-YDL6.1.patch.txt.gz
Patch for libjtux for compiling on ARM, also contains some bugfixes
(1.45 KiB) Downloaded 950 times
libjtux.so.gz
JTUX library libjtux.so, compiled for ARM
(34.54 KiB) Downloaded 473 times

RandomUsername
Posts: 904
Joined: 09 Oct 2009, 18:49

Re: Backup using CrashPlan (installation guide/howto)

Post by RandomUsername » 18 Mar 2012, 05:47

Kudos. I looked at Crashplan myself a while ago but had heard bad things about the performance of java on the Arm platform. How are you finding it? I went with a storage provider that has SSH access in the end.

rxmd
Posts: 22
Joined: 14 Jun 2011, 10:32

Re: Backup using CrashPlan (installation guide/howto)

Post by rxmd » 21 Mar 2012, 08:24

RandomUsername wrote:Kudos. I looked at Crashplan myself a while ago but had heard bad things about the performance of java on the Arm platform. How are you finding it? I went with a storage provider that has SSH access in the end.
Well it isn't exactly fast, but I find it's workable. CPU load is certainly high when the system is backing up (around 80-90% currently when backing up, it goes up to 100 at times), but the CrashPlan process itself is behaving nicely, prioritizes itself rather low and the system remains responsive. I've been running this for over a month now and I never had problems; that's with CrashPlan, a Funambol instance and the Transmission bittorrent client running in parallel.

I guess it could be a little faster still if the B3 made use of the ARM CPU's built-in hardware encryption accelerator, since most of CrashPlan's CPU load seems to be due to on-the-fly encryption of data as it gets sent out.

JazzyB
Posts: 3
Joined: 18 Nov 2011, 07:36

Re: Backup using CrashPlan (installation guide/howto)

Post by JazzyB » 14 Apr 2012, 19:06

Thanks rxmd for sharing this info.

Now have crashplan running and backing up to the cloud.
Only twelve more days to go before its done :D

Bravo!

--
JazzyB

karl-petter
Posts: 29
Joined: 05 Jan 2011, 18:35
Location: Åre

Re: Backup using CrashPlan (installation guide/howto)

Post by karl-petter » 13 Jul 2012, 19:00

Hi,

thanks for a very good guide/howto, detailed and easy to follow. But I couldnt get it to work:(

It seems CrashPlan starts without problem on my B3, cannot see anything odd in the logs. But when I try to setup the tunnel and CrashPlan Desktop tries to connect I get:

channel 3: open failed: connect failed: Connection refused

And if I do

Code: Select all

netstat -an | grep LISTEN
there is nothing listening to 4243...

Anyone have had similar problems and solved it? Otherwise I guess I need to start digging...

karl-petter
Posts: 29
Joined: 05 Jan 2011, 18:35
Location: Åre

Re: Backup using CrashPlan (installation guide/howto)

Post by karl-petter » 14 Jul 2012, 12:59

Noticed two things;

1) it takes really LONG time for Crashplan to start. Read somewhere 4 minutes but mine takes about 9! So one thing I did wrong yesterday was not to wait long enough.

2) now when I did I saw that libmd5 crashes. Apparently this was added in 3.2 of CrashPlan and I found this page to get that compiled and libffi which is also now needed; http://www.droboports.com/app-repositor ... plan-3-2-1

It seems now to run without problems but I still cannot connect. Can this be an issue -> "critical error converting IP address into int"? I can telnet to port 4243 when I do it to localhost and I see through netstat it listens to this port but when I try to connect to LAN IP using that port it does not work, nor from a remote machine on that LAN network. Need to investigate further.

karl-petter
Posts: 29
Joined: 05 Jan 2011, 18:35
Location: Åre

Re: Backup using CrashPlan (installation guide/howto)

Post by karl-petter » 15 Jul 2012, 07:01

Finally got it to work!!

Somehow I had setup the wrong SSH tunnel. The correct is ssh -N -L 4200:localhost:4243 username@192.168.1.1. Important is that it is localhost since Crashplan only binds to localhost for the GUI, and then the second IP is to the machine to setup the tunnel to with your username on that machine.

Squeezeboxserver took so much CPU that Crashplan didnt had a chance to reply in time so had to shut it down. Hopefully it wont interfere with my future backups.

Hope this might come handy for someone. Thanks again for a nice tutorial!

JazzyB
Posts: 3
Joined: 18 Nov 2011, 07:36

Re: Backup using CrashPlan (installation guide/howto)

Post by JazzyB » 05 May 2013, 06:52

Brilliant - I've found a link which explains how to access a headless set up WITHOUT an ssh tunnel.

But I cannot tell you what it is because the forum software says I am not allowed :(
Like this:
Your post looks too spamy for a new user, please remove off-site URLs.
Found it on the liquidstate net blog site.
Google search: headless crashplan

Am trying this out now and all appears to be working OK. :)

--
JazzyB

Binkem
Posts: 388
Joined: 10 Jul 2008, 02:26

Post by Binkem » 08 May 2013, 16:06

Otherwise p.m. me and I will put it on the site.

Sorry about this spam control, but things were getting out of hand a while ago.

jesperw
Posts: 21
Joined: 23 Jul 2007, 07:08

Re: Backup using CrashPlan (installation guide/howto)

Post by jesperw » 12 May 2013, 12:55

Thanks for a great how to!

I am considering to backup to the cloud as well and considering crashplan. One question though: the backup will continue to run on Bubba even if the client on my PC is disconnected?
It is nice to be important, but more important to be nice.

JazzyB
Posts: 3
Joined: 18 Nov 2011, 07:36

Re: Backup using CrashPlan (installation guide/howto)

Post by JazzyB » 24 May 2013, 11:09

To confirm: Yes it continues to run without the PC client connected
Hurrah! :D

Post Reply