Linux backup and restore with Duplicity

Duplicity backs directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. Because duplicity uses GnuPG to encrypt and/or sign these archives, they will be safe from spying and/or modification by the server.

Installation

Duplicity can be installed on Debian/Ubuntu as simply as:

$ sudo apt-get install duplicity ncftp python-boto python-paramiko

Note

To avoid the Import of duplicity.backends.giobackend Failed: No module named gio warning, you need to install the python-gobject-2 package:

$ apt-get install python-gobject-2

But it is not in the CentOS/Red Hat default repositories, so you need to install it from another source. This script will help in this installation process:

#!/bin/bash

wget http://dl.fedoraproject.org/pub/epel/6/x86_64/duplicity-0.6.18-1.el6.x86_64.rpm
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/ncftp-3.2.4-1.el6.x86_64.rpm
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/librsync-0.9.7-15.el6.x86_64.rpm
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/python-GnuPGInterface-0.3.2-6.el6.noarch.rpm
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/python-boto-2.5.2-1.el6.noarch.rpm

rpm -Uvh duplicity-0.6.18-1.el6.x86_64.rpm ncftp-3.2.4-1.el6.x86_64.rpm librsync-0.9.7-15.el6.x86_64.rpm python-GnuPGInterface-0.3.2-6.el6.noarch.rpm python-boto-2.5.2-1.el6.noarch.rpm

Command line basics

To make a backup, you can run:

$ duplicity --no-encryption /home/openkm file:///path/to/backup

To verify it:

$ duplicity --no-encryption verify file:///path/to/backup /home/openkm

To list backed-up files:

$ duplicity --no-encryption list-current-files file:///path/to/backup

To restore a single file:

$ duplicity --no-encryption --file-to-restore tomcat-9.0.76/openkm.properties file:///path/to/backup RestoredBackup

Note

The path to the file that is to be restored is relative to the directory on which the backup set is based. So, in the command above, tomcat-9.0.76/openkm.properties plus the directory on which we based our backup (/home/openkm) equals /home/openkm/tomcat-9.0.76/openkm.properties

Preliminaries

These scripts assume the following default values:

• The value of $TOMCAT_HOME is “/home/openkm/tomcat-9.0.76”
• The OpenKM database is named “okmdb”.
• The OpenKM database password is “*secret*”.
• The location of the OpenKM start & stop service script is “/etc/init.d/tomcat”

Sample script

Caution

As a good practice, the backup should be done by the root user.

Note

The script below performs a backup to USB, and the application database is MySQL.

The USB disk mount point can be defined in /etc/fstab as:

/dev/sdb1    /mnt/backup    ext4    defaults    0    0

These are the global script sections:

• Global variable configuration under the ## BEGIN CONFIG ## section.
• Checking for the root user under the # Check root section.
• Deleting older database backups under the # Delete older local database backup section.
• Mount disk (optional section) under the # Mount disk section.
• Stop application (optional section) under the # Stop tomcat section.
• Clean logs (optional section) under the # Clean logs section.
• Backup the database to the file system under the # Backup database section.
• Backup repository and database dump to the backup destination under the # Backup and purge old backups section.
• Start application (optional section) under the # Start tomcat section.
• Show statistics (optional section) under the # Status section.
• Unmount disk (optional section) under the # Umount disk section.

Explanation of configuration parameters:

Parameter	Description
HOST	The server host name.
DATABASE_PASS	The database password. The database user used to perform the backup is always the root user.
OPENKM_DB	The application database name. It is usually named “okmdb”.
OPENKM_HOME	The OpenKM home folder.
TOMCAT_HOME	The Tomcat home folder. Usually inside the OPENKM_HOME folder.
DATABASE_EXP	The database dump folder. Usually on the same server.
BACKUP_DIR	Backup destination.

Create a backup script /root/backup.sh

$ sudo su
$ vim /root/backup.sh

#!/bin/bash
#
## BEGIN CONFIG ##
HOST=$(uname -n)
DATABASE_PASS="*secret*"
OPENKM_DB="okmdb"
OPENKM_HOME="/home/openkm"
TOMCAT_HOME="$OPENKM_HOME/tomcat-9.0.76"
DATABASE_EXP="$OPENKM_HOME/db"
BACKUP_DIR="file:///mnt/backup"
## END CONFIG ##

# Check root user
if [ $(id -u) != 0 ]; then echo "You should run this script as root"; exit; fi

# Delete older local database backup
echo -e "### BEGIN: $(date +"%x %X") ###\n"
rm -rf $DATABASE_EXP
mkdir -p $DATABASE_EXP

# Mount disk
if mount | grep "$BACKUP_DIR type" > /dev/null; then
  echo "$BACKUP_DIR already mounted";
else
  mount "$BACKUP_DIR";
  if mount | grep "$BACKUP_DIR type" > /dev/null; then
    echo "$BACKUP_DIR mounted";
  else
    echo "$BACKUP_DIR error mounting";
    exit -1;
  fi
fi

# Stop Tomcat
/etc/init.d/tomcat stop

# Clean logs
#echo "Clean Tomcat temporal files."
#rm -rf $TOMCAT_HOME/logs/*
#rm -rf $TOMCAT_HOME/temp/*
#rm -rf $TOMCAT_HOME/work/Catalina/localhost

# Backup database
if [ -n "$DATABASE_PASS" ]; then
  echo "* Backuping MySQL data from $OPENKM_DB..."
  mysqldump -h localhost -u root -p$DATABASE_PASS $OPENKM_DB > $DATABASE_EXP/mysql_$OPENKM_DB.sql
  echo "-------------------------------------";
fi

# Backup and purge old backups
duplicity remove-older-than 1M --force $BACKUP_DIR/$HOST

if [ $(date +%u) -eq 7 ]; then
  echo "*** Full Backup ***"
  duplicity full --no-encryption $OPENKM_HOME $BACKUP_DIR/$HOST
  RETVAL=$?
else
  echo "*** Incremental Backup ***"
  duplicity --no-encryption $OPENKM_HOME $BACKUP_DIR/$HOST
  RETVAL=$?
fi

[ $RETVAL -eq 0 ] && echo "*** SUCCESS ***"
[ $RETVAL -ne 0 ] && echo "*** FAILURE ***"

# Start Tomcat
/etc/init.d/tomcat start
echo -e "\n### END: $(date +"%x %X") ###"

# Status
echo "=================================";
duplicity collection-status $BACKUP_DIR/$HOST
echo "*********************************";
df -h | grep "$BACKUP_DIR"
echo "=================================";

# Umount disk
sync
umount "$BACKUP_DIR"

Increase the backup period

Modify the # Backup and purge old backups section, increasing the parameter value:

Note

The “1M” value indicates that backups older than one month will be removed. Changing to “3M” or “1Y” indicates backups older than 3 months or 1 year, etc.

--remove-older-than 3M

Perform the remote backup

Modify the ## Config section.

BACKUP_DIR="ftp://user@ftp.domain.es/backup"
export FTP_PASSWORD="WhateverPasswordYouSetUp"

Modify the # Backup and purge old backups section.

# Backup and purge old backups
duplicity remove-older-than 1M --force $BACKUP_DIR/$HOST

if [ $(date +%u) -eq 7 ]; then
  echo "*** Full Backup ***"
  duplicity full --no-encryption $OPENKM_HOME $BACKUP_DIR/$HOST
  RETVAL=$?
else
  echo "*** Incremental Backup ***"
  duplicity --no-encryption $OPENKM_HOME $BACKUP_DIR/$HOST
  RETVAL=$?
fi

[ $RETVAL -eq 0 ] && echo "*** SUCCESS ***"
[ $RETVAL -ne 0 ] && echo "*** FAILURE ***"

Modify the # Status section.

duplicity collection-status $BACKUP_DIR/$HOST
unset FTP_PASSWORD

Perform the remote backup to SMB or CIFS

Modify the # Mount disk section.

# Mount disk
if mount | grep "$BACKUP_DIR type" > /dev/null; then
  echo "$BACKUP_DIR already mounted";
else
  echo "Mounting $BACKUP_DIR ...";
  mount -t cifs //REMOTE_SERVER/REMOTE_DIR $BACKUP_DIR -o username=REMOTE_USER,password=REMOTE_PASSWORD,iocharset=utf8,file_mode=0777,dir_mode=0777
  if mount | grep "$BACKUP_DIR type" > /dev/null; then
    echo "$BACKUP_DIR mounted";
  else
    echo "Error mounting $BACKUP_DIR.";
    exit -1;
  fi
fi

Do the PostgreSQL backup

Modify the # Backup database section

# Backup de PostgreSQL
echo "* Backuping PostgreSQL data from $OPENKM_DB..."
su postgres -c "pg_dump $OPENKM_DB" > $DATABASE_EXP/pg_$OPENKM_DB.sql

# Databases optimizations
su postgres -c "vacuumdb -a -z" > /dev/null
su postgres -c "reindexdb -a -q" 2> /dev/null

Do the remote backup to Amazon s3

Modify the ## Config section section.

BACKUP_DIR="s3+http://somebucket/somedirectory"
export AWS_ACCESS_KEY_ID=""
export AWS_SECRET_ACCESS_KEY=""
export PASSPHRASE=""

Modify the # Backup and purge old backups section.

# Backup and purge old backups
if [ $(date +%u) -eq 7 ]; then
  echo "*** Full Backup ***"
  duplicity full --no-encryption --s3-use-new-style $OPENKM_HOME $BACKUP_DIR/$HOST
  RETVAL=$?
else
  echo "*** Incremental Backup ***"
  duplicity --no-encryption --s3-use-new-style $OPENKM_HOME $BACKUP_DIR/$HOST
  RETVAL=$?
fi

[ $RETVAL -eq 0 ] && echo "*** SUCCESS ***"

Modify the # Status section.

# Status
duplicity collection-status --s3-use-new-style $BACKUP_DIR/$HOST
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
unset PASSPHRASE

Configure crontab

To install the cron job, run:

$ sudo mkdir /root/logs
$ sudo crontab -e

Add these lines according to your personal configuration:

MAILTO=nomail@openkm.com
@daily /root/backup.sh | tee /root/logs/backup.$(date +\%Y.\%m.\%d_\%H.\%M.\%S).log

Note

More information at Crontab quick reference

If you want to be notified by mail, you should install the postfix service on your server.

Troubleshooting deleting old backups

You have configured Duplicity to make a full backup every 7 days and to delete backups older than 14 days to save space. What happens if you see a message like this?

Tue May  1 00:00:00 2012
Wed May  2 00:00:00 2012
Which can't be deleted because newer sets depend on them

Duplicity uses rsync, which records incremental changes. Those files won’t be deleted because, even though a backup may be older than 7 days, there are incremental backups that are newer than 7 days.

So, after two weeks have passed, those files will be deleted, since the full backup and the incremental backups are now 14 days old, and there exists a full backup newer than those backups.

Restoring backup

You can list the available backups:

$ duplicity collection-status /path/to/backup
Sun Sep 11 05:00:19 2012         4.25 GB           4.25 GB   (current mirror)
Sun Sep  4 00:00:18 2012         13.3 MB           4.26 GB
Sun Aug 28 00:00:13 2012          674 MB           4.92 GB
Sun Aug 21 00:00:14 2012         5.50 MB           4.93 GB
Sun Aug 14 00:00:16 2012         1.75 MB           4.93 GB
Sun Aug  7 00:00:12 2012          288 KB           4.93 GB
Sun Jul 31 00:00:13 2012         43.0 KB           4.93 GB
Fri Jul 29 10:36:39 2012         5.56 KB           4.93 GB

Then decide to restore one of the backups, for example the one made on Sun Aug 28 00:00:13 2012:

$ duplicity restore --restore-time 2012-08-28 /path/to/backup /path/to/destination

Note

The --restore-time parameter accepts several formats. See duplicity documentation and Duplicity Backup Howto for more info.

Note

Inside /path/to/destination, you should see a directory /home/openkm, and inside it a couple of directories:

• db: The backup of the database.
• tomcat-9.0.76: The backup of the Tomcat installation and OpenKM repository into.

Additional information

• Using Duplicity to Backup Your Data
• Ubuntu/Debian - Encrypted Incremental Backups With Duplicity on Amazon S3
• Duplicity Backup Howto
• Duplicity URL Format
• Installation and configuration of duplicity for encrypted SFTP remote backup
• Playing with Duplicity backup and restore tool and Amazon S3
• Bash wrapper script for automated backups with duplicity