Ever since Dropbox, I’ve been searching for a self-hosted, secure (and now Condi-free) way of keeping my machines synchronised and backed up. There are lots. I tried many, wrote a couple myself, but none were exactly what I wanted.

My problem was thinking Windows, looking for a single program. Once I started thinking Unix, looking for modular components, the answers were obvious.

Storage

First we need a remote master storage to sync against, somewhere to backup our files. And we want that exposed as a local filesystem. I use the most obvious answer, sshfs:

sudo apt-get install sshfs
mkdir -p /home/graham/.backup/crypt  # Why 'crypt'? Read on.

sshfs server.example.com:backup /home/graham/.backup/crypt

You can use any storage that can appear as a filesytem, such as FTP (via curlftpfs), NTFS, and many others.

Encryption

There’s two kinds of data: public data, and encrypted data. We want the second kind. Just layer encfs:

sudo apt-get install encfs
mkdir -p /home/graham/.backup/clear

encfs /home/graham/.backup/crypt/ /home/graham/.backup/clear/

The first time you run encfs, answer ‘p’ for paranoia mode.

At this point anything you copy into .backup/clear will be encrypted and copied to your remote server. Not bad for two lines!

Sync

If you’re offline, .backup/crypt and .backup/clear will be empty. We want a local copy of the files. If you’re just doing backup (i.e. all changes come from a single local machine), use rsync. I want true two-way sync (desktop<->server and laptop<->server). The tool of choice for that is Unison:

sudo apt-get install unison
mkdir /home/graham/backup   # Local copy

unison -auto /home/graham/backup/ /home/graham/.backup/clear/

And that’s it! Three lines, an encrypted, remote, synchronising file system. Thanks Unix!

Bash glue

You need to re-mount the sshfs and encfs filesystems whenever you reboot your machine, and you probably don’t want to type that unison command line all that often. Let’s make a little bash script. I called mine sy, in my ~/bin/ directory.

#!/bin/bash

sync_mount () {
    if [[ $(mount | grep .backup/crypt) ]];then
        echo ".backup/crypt is mounted"
    else
        sshfs server.example.com:backup /home/graham/.backup/crypt
    fi
    if [[ $(mount | grep .backup/clear) ]];then
        echo ".backup/clear is mounted"
    else
        encfs /home/graham/.backup/crypt/ /home/graham/.backup/clear/
    fi
}
sync_umount () {
    fusermount -u /home/graham/.backup/clear
    fusermount -u /home/graham/.backup/crypt
}
sync_install () {
    sudo apt-get install sshfs encfs unison
    mkdir -p /home/graham/.backup/crypt
    mkdir -p /home/graham/.backup/clear
    mkdir /home/graham/backup
}
sync_full () {
    sync_mount
    unison -auto /home/graham/backup/ /home/graham/.backup/clear/
}

case "$1" in
umount)
    sync_umount ;;
install)
    sync_install ;;
*)
    sync_full ;;
esac

Further options

I’ve been using this exact script and setup for a month now, and I’m very happy with it. It feels so much simpler than everything else I tried.

If you have files in your backup folder that change rarely, Unison has a powerful -ignore syntax, for example -ignore 'Path Documents'. The fewer files Unison has to check, the faster it runs. Also -batch will stop Unison asking you for input.

A next stage could be to trigger sy automatically whenever you change something. That would use something based on inotify, maybe lsyncd or incrontab. I thought I would want this when I started, but so far it hasn’t bothered me. I just type sy once in a while. Let me know if you go that route.