Backup using CrashPlan (installation guide/howto)
Posted: 18 Mar 2012, 03:25
Here's a little installation guide for the CrashPlan backup software. I use this to have my B3 make automated backups.
CrashPlan (http://www.crashplan.com) is a service for backing up data to the cloud. It runs continuously, monitoring files for changes and making continuous incremental backups that are encrypted and stored on CrashPlan's servers. CrashPlan uses its own client software written in Java. The nifty thing about the client software is that you can run it in headless mode on a headless server (such as a B3), and run the client portion on another PC with an encrypted connection to the headless server via a SSH tunnel.
So here's how to get the Java client running on the B3 for automatic backups. This guide uses information from some other sources, including here and here.
0. Prerequisites: Installing the Java runtime environment
CrashPlan is a Java application, so we need to install the Java runtime environment (JRE). On our B3 with its ARM processor architecture, we cannot use the original Sun JRE, so we will use OpenJDK instead. Since our B3 is a headless server, it is sufficient to install the headless version of the OpenJDK JRE.
1. Downloading and installing CrashPlan
Download the Linux installation version for CrashPlan. It is sufficient to do this once, CrashPlan will autoupdate itself. You can find out what the latest version is by looking at the CrashPlan download site, http://www.crashplan.com/consumer/downl ... l?os=Linux.
The choice you have to make at this point is whether you want to install and run CrashPlan as root or under some other user account. If you run CrashPlan under some user account other than root, it will obviously be able to backup only those files that are readable by this user; so if you have multiple users on your B3 and want all their data backed up separately, you'll have to create an extra "backup" user and think up a permissions scheme that will allow the backup user to see the data of everybody else, while still hiding their data from each other. Running CrashPlan as root is somewhat more convenient in that root can read everything by default, but it also means that you trust a third party binary blob piece of software to do basically anything it wants on your server. In my case, I only have data stored under one user account on the B3 anyway, so I simply install CrashPlan under that user account.
CrashPlan will ask you where to install the software. I chose to install it to /opt/crashplan. You may have to create this directory beforehand, using something like this:
The installer will probably also tell you that your inotify limit (the number of files that your Linux kernel can simultaneously watch for changes) is set rather low. CrashPlan needs to be able to do continuous monitoring of file changes in order to do incremental backups, and since full backup sets can easily run to hundreds of thousands of files, the default value of 8192 is indeed often insufficient. We will change this later.
After installing, CrashPlan will tell you that it has already been launched, but in reality it hasn't.
2. Patch the CrashPlan files
At this point CrashPlan will not work yet. The problem is that CrashPlan still assumes that it is running on an Intel-compatible machine. In particular, it makes use of the Java-to-Unix library (libjtux) via an interface known as Java Native Access (JNA). For this, CrashPlan includes a precompiled Intel version of libjtux, which of course is useless on the ARM architecture of our B3.
Install the Debian version of JNA, and tell CrashPlan where to find it. In order to do this, you'll have to make some changes to the CrashPlan launcher script:
Find the line starting with FULL_CP, and edit it to include the Debian JNA library we just installed:
Now the system will still try to use the Intel-compiled version of libjtux that cae with CrashPlan. To install an ARM-compiled version of libjtux, first make a backup of the old Intel version:
Either download the binary libjtux.so for ARM that is attached to this post for convenience, or compile it yourself.
2A. Alternative 1: Use an ARM-precompiled version of libjtux.so
If you downloaded the attached version, just copy it over:
2B. Alternative 2: Compiling libjtux.so yourself
If you want to compile it yourself, you'll need the libjtux sources from http://basepath.com/aup/jtux/jtux.tar.gz. Also you'll need a patch for libjtux that contains some bugfixes as well as a makefile (the original jtux sources are missing a makefule). You can find the patch attached to this post, or you can download it from the discussion thread at https://crashplan.zendesk.com/entries/3 ... sheevaplug (you may need to be logged in to the CrashPlan site to see that link).
Download the jtux patch using the browser, and put it into the same downloads directory.
The Makefile is still written for the Sun Java JDK, so that we need to edit it to make it point to the OpenJDK for ARM that we just installed on our B3.
Comment out the first line (with JAVA_INCLUDE) and replace it with
Compile libjtux:
The compiler will throw some warnings that we will ignore.
Copy the generated libjtux binary into the Crashplan directory:
3. Other server-side configuration
3.1 Filename encodings
My file names are in UTF-8, and I have some files with non-ASCII characters in their name. So in order for them to be stored correctly, I have to make sure that everything, the Java VM included, is running in UTF-8 encoding:
Make sure to add the following lines near the end:
3.2 inotify Open files monitor limit
CrashPlan uses the inotify kernel module to keep track of changed files. By default, inotify is configured to monitor 8192 files. Depending on the files you want to back up, this may not be enough (my backup set currently contains about 100.000 files). So it's a good idea to increase the inotify count.
Add a line towards the end:
4. Accessing CrashPlan from your client computer
Download the CrashPlan desktop for your client computer from http://www.crashplan.com/consumer/download.html and install it.
Start up CrashPlan on the B3:
In order to connect the CrashPlan Desktop from your client computer to the headless engine running on your B3, you'll have to set up an SSH tunnel with port forwarding on your client computer. The detailed instructions how to do this vary, depending on the operating system on your client computer. There is a relatively detailed set of guidelines how to do this on the CrashPlan site: http://support.crashplan.com/doku.php/h ... ess_client
I use a Windows client and have set up the Putty SSH client to open an SSH tunnel from port 4200 on the local machine to port 4243 on the B3.
Follow the instructions on the CrashPlan site, connect the CrashPlan Desktop on your client computer to the CrashPlan engine on the B3 and set up your user account and files to back up.
5. Cleanup (optional)
If you followed alternative 2B above and compiled libjtux.so yourself, you may be left with all sorts of development-related software. If you normally don't do any software development yourself, you can remove them:
6. Automatically starting up CrashPlan with the B3
In order to start up CrashPlan automatically at system startup with the B3 so that it will back up all the time, you have to add it to the system startup scripts. Fortunately, CrashPlan already includes a Debian startup script that we can copy and modify for our purposes. What we have to do is to modify the script so that instead of starting up the CrashPlan engine as root, it will start up the engine under whatever username we installed it:
In order to do this, we have, to take all the instances where it says $SCRIPTNAME start, $SCRIPTNAME stop etc. (five of them) and modify them so that they say
This will start the CrashPlan engine under your chosen username.
Now link this file into /etc/init.d and enable automatic startup:
At the next system startup, you can fire up htop or whatever process monitor you use to see that CrashPlan is indeed hard at work.
CrashPlan (http://www.crashplan.com) is a service for backing up data to the cloud. It runs continuously, monitoring files for changes and making continuous incremental backups that are encrypted and stored on CrashPlan's servers. CrashPlan uses its own client software written in Java. The nifty thing about the client software is that you can run it in headless mode on a headless server (such as a B3), and run the client portion on another PC with an encrypted connection to the headless server via a SSH tunnel.
So here's how to get the Java client running on the B3 for automatic backups. This guide uses information from some other sources, including here and here.
0. Prerequisites: Installing the Java runtime environment
CrashPlan is a Java application, so we need to install the Java runtime environment (JRE). On our B3 with its ARM processor architecture, we cannot use the original Sun JRE, so we will use OpenJDK instead. Since our B3 is a headless server, it is sufficient to install the headless version of the OpenJDK JRE.
Code: Select all
$ sudo apt-get install openjdk-6-jre-headless
Download the Linux installation version for CrashPlan. It is sufficient to do this once, CrashPlan will autoupdate itself. You can find out what the latest version is by looking at the CrashPlan download site, http://www.crashplan.com/consumer/downl ... l?os=Linux.
The choice you have to make at this point is whether you want to install and run CrashPlan as root or under some other user account. If you run CrashPlan under some user account other than root, it will obviously be able to backup only those files that are readable by this user; so if you have multiple users on your B3 and want all their data backed up separately, you'll have to create an extra "backup" user and think up a permissions scheme that will allow the backup user to see the data of everybody else, while still hiding their data from each other. Running CrashPlan as root is somewhat more convenient in that root can read everything by default, but it also means that you trust a third party binary blob piece of software to do basically anything it wants on your server. In my case, I only have data stored under one user account on the B3 anyway, so I simply install CrashPlan under that user account.
Code: Select all
$ cd ~/downloads
$ wget http://download.crashplan.com/installs/linux/install/CrashPlan/CrashPlan_3.2_Linux.tgz
$ tar xfzv CrashPlan_3.2_Linux.tgz
$ cd CrashPlan-install
$ ./install.sh
Code: Select all
$ sudo mkdir /opt/crashplan
[sudo] password for your-user-name:
$ sudo chown your-user-name.users /opt/crashplan
After installing, CrashPlan will tell you that it has already been launched, but in reality it hasn't.
2. Patch the CrashPlan files
At this point CrashPlan will not work yet. The problem is that CrashPlan still assumes that it is running on an Intel-compatible machine. In particular, it makes use of the Java-to-Unix library (libjtux) via an interface known as Java Native Access (JNA). For this, CrashPlan includes a precompiled Intel version of libjtux, which of course is useless on the ARM architecture of our B3.
Install the Debian version of JNA, and tell CrashPlan where to find it. In order to do this, you'll have to make some changes to the CrashPlan launcher script:
Code: Select all
$ sudo apt-get install libjna-java
$ cd /opt/crashplan/bin
$ nano CrashPlanEngine
Code: Select all
FULL_CP="/usr/share/java/jna.jar:$TARGETDIR/lib/com.backup42.desktop.jar:$TARGETDIR/lang"
Code: Select all
$ cd /opt/crashplan
$ cp libjtux.so libjtux.so.bak
2A. Alternative 1: Use an ARM-precompiled version of libjtux.so
If you downloaded the attached version, just copy it over:
Code: Select all
$ cd wherever-you-downloaded-the-libjtux.so.gz-file-on-your-B3
$ gunzip libjtux.so.gz
$ cp libjtux.so /opt/crashplan
If you want to compile it yourself, you'll need the libjtux sources from http://basepath.com/aup/jtux/jtux.tar.gz. Also you'll need a patch for libjtux that contains some bugfixes as well as a makefile (the original jtux sources are missing a makefule). You can find the patch attached to this post, or you can download it from the discussion thread at https://crashplan.zendesk.com/entries/3 ... sheevaplug (you may need to be logged in to the CrashPlan site to see that link).
Code: Select all
$ cd wherever-you-downloaded-the-jtux.PS3-YDL6.1.patch.txt.gz-file-on-your-B3
$ gunzip jtux.PS3-YDL6.1.patch.txt.gz
$ sudo apt-get install openjdk-6-jdk
$ sudo apt-get install patch
$ sudo apt-get install make
$ sudo apt-get install gcc
$ cd ~/downloads
$ wget http://basepath.com/aup/jtux/jtux.tar.gz
Code: Select all
$ tar xfzv jtux.tar.gz
$ cd jtux
$ patch < ~/downloads/jtux.PS3-YDL6.1.patch.txt (or wherever you put it)
patching file jtux_file.c
patching file jtux_network.c
patching file jtux_posixipc.c
patching file jtux_process.c
patching file jtux_sysvipc.c
patching file jtux_util.c
patching file Makefile
Code: Select all
$ nano Makefile
Code: Select all
JAVA_INCLUDE=/usr/lib/jvm/java-6-openjdk/include
Code: Select all
$ make
Copy the generated libjtux binary into the Crashplan directory:
Code: Select all
$ cp libjtux.so /opt/crashplan
3.1 Filename encodings
My file names are in UTF-8, and I have some files with non-ASCII characters in their name. So in order for them to be stored correctly, I have to make sure that everything, the Java VM included, is running in UTF-8 encoding:
Code: Select all
$ sudo nano /etc/profile
Code: Select all
LANG=en_US.utf8
LC_ALL=en_US.utf8
export LANG LC_ALL
CrashPlan uses the inotify kernel module to keep track of changed files. By default, inotify is configured to monitor 8192 files. Depending on the files you want to back up, this may not be enough (my backup set currently contains about 100.000 files). So it's a good idea to increase the inotify count.
Code: Select all
$ cat /proc/sys/fs/inotify/max_user_watches
8192
$ sudo echo 1048576 > /proc/sys/fs/inotify/max_user_watches
$ sudo nano /etc/sysctl.conf
Code: Select all
# Set up inotify kernel module to keep watch on up to 1M files
fs.inotify.max_user_watches=1048576
Download the CrashPlan desktop for your client computer from http://www.crashplan.com/consumer/download.html and install it.
Start up CrashPlan on the B3:
Code: Select all
$ /opt/crashplan/bin/CrashPlanEngine start
I use a Windows client and have set up the Putty SSH client to open an SSH tunnel from port 4200 on the local machine to port 4243 on the B3.
Follow the instructions on the CrashPlan site, connect the CrashPlan Desktop on your client computer to the CrashPlan engine on the B3 and set up your user account and files to back up.
5. Cleanup (optional)
If you followed alternative 2B above and compiled libjtux.so yourself, you may be left with all sorts of development-related software. If you normally don't do any software development yourself, you can remove them:
Code: Select all
$ sudo apt-get remove make
$ sudo apt-get remove patch
$ sudo apt-get remove gcc
$ sudo apt-get remove openjdk-6-jdk
$ sudo apt-get autoremove (WARNING this might be dangerous, check displayed package list before proceeding!)
In order to start up CrashPlan automatically at system startup with the B3 so that it will back up all the time, you have to add it to the system startup scripts. Fortunately, CrashPlan already includes a Debian startup script that we can copy and modify for our purposes. What we have to do is to modify the script so that instead of starting up the CrashPlan engine as root, it will start up the engine under whatever username we installed it:
Code: Select all
$ cd /opt/crashplan/bin
$ cp crashplan crashplan.user
$ nano crashplan.user
Code: Select all
su your-user-name -c "$SCRIPTNAME start" (stop, restart etc., with replacing your-user-name with your username)
Now link this file into /etc/init.d and enable automatic startup:
Code: Select all
$ sudo ln -s /opt/crashplan/bin/crashplan.user /etc/init.d/crashplan
$ sudo update-rc.d crashplan defaults