Linux Sample Backup with Duplicity
Duplicity backs directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. Because duplicity uses GnuPG to encrypt and/or sign these archives, they will be safe from spying and/or modification by the server.
Installation
Duplicity can be installed in Debian / Ubuntu as simple as:
$ sudo apt-get install duplicity ncftp python-boto python-paramiko
To avoid the Import of duplicity.backends.giobackend Failed: No module named gio warning, you need to install the python-gobject-2 package:
$ apt-get install python-gobject-2
But it not in the CentOS / RedHat default repositories, so you need to install from another source. This script will help in this installation process:
#!/bin/bash
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/duplicity-0.6.18-1.el6.x86_64.rpm
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/ncftp-3.2.4-1.el6.x86_64.rpm
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/librsync-0.9.7-15.el6.x86_64.rpm
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/python-GnuPGInterface-0.3.2-6.el6.noarch.rpm
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/python-boto-2.5.2-1.el6.noarch.rpm
rpm -Uvh duplicity-0.6.18-1.el6.x86_64.rpm ncftp-3.2.4-1.el6.x86_64.rpm librsync-0.9.7-15.el6.x86_64.rpm python-GnuPGInterface-0.3.2-6.el6.noarch.rpm python-boto-2.5.2-1.el6.noarch.rpm
Command line basics
To make a backup you can do:
$ duplicity --no-encryption /home/openkm file:///path/to/backup
To verify it:
$ duplicity --no-encryption verify file:///path/to/backup /home/openkm
To list backed files:
$ duplicity --no-encryption list-current-files file:///path/to/backup
To restore a single file:
$ duplicity --no-encryption --file-to-restore tomcat/OpenKM.cfg file:///path/to/backup RestoredBackup
The path to the file that is to be restored is related to the directory on which the backup set is based. So in the command above, tomcat/OpenKM.cfg plus the directory on which we based our backup (/home/openkm) equals /home/openkm/tomcat/OpenKM.cfg
Preliminaries
In these scripts are assumed these default values:
- $TOMCAT_HOME value is "/home/openkm/tomcat"
- OpenKM database is named "okmdb".
- OpenKM database password value is "*secret*".
- OpenKM start & stop service script location is "/etc/init.d/tomcat"
Sample script
As a good practice the backup should be done by root user.
The script below does a backup to USB and the application database is MySQL.
USB disk mount point can be defined in /etc/fstab as:
/dev/sdb1 /mnt/backup ext4 defaults 0 0
These are the global script sections:
- Global var configuration under ## BEGIN CONFIG ## section.
- Checking user root under # Check root user section.
- Deleting older database backup # Delete older local database backup section.
- Mount disk ( optional section ) under # Mount disk section.
- Stop application ( optional section ) under # Stop tomcat section.
- Clean logs ( optional section ) under # Clean logs section.
- Backup the database to the file system under #Backup database section.
- Backup repository and database dump to backup destination under #Backup and purge old backups section.
- Start application ( optional section ) # Start tomcat section.
- Show statistics ( optional section ) # Status section.
- Un mount disk ( optional section ) under # Umount disk section.
Configuration parameters explanation:
Parameter | Description |
---|---|
HOST |
The server host name. |
DATABASE_PASS |
The database password. The data base user to do the backup is always root user. |
OPENKM_DB |
The application database name. Usually will be named "okmdb". |
OPENKM_HOME |
The openkm home folder. |
TOMCAT_HOME |
The tomcat home folder. Usually into OPENKM_HOME folder. |
DATABASE_EXP |
The database dump folder. Usually into the same server. |
BACKUP_DIR |
Backup destination. |
Create a backup script /root/backup.sh
$ sudo su
$ vim /root/backup.sh
#!/bin/bash
#
## BEGIN CONFIG ##
HOST=$(uname -n)
DATABASE_PASS="*secret*"
OPENKM_DB="okmdb"
OPENKM_HOME="/home/openkm"
TOMCAT_HOME="$OPENKM_HOME/tomcat"
DATABASE_EXP="$OPENKM_HOME/db"
BACKUP_DIR="file:///mnt/backup"
## END CONFIG ##
# Check root user
if [ $(id -u) != 0 ]; then echo "You should run this script as root"; exit; fi
# Delete older local database backup
echo -e "### BEGIN: $(date +"%x %X") ###\n"
rm -rf $DATABASE_EXP
mkdir -p $DATABASE_EXP
# Mount disk
if mount | grep "$BACKUP_DIR type" > /dev/null; then
echo "$BACKUP_DIR already mounted";
else
mount "$BACKUP_DIR";
if mount | grep "$BACKUP_DIR type" > /dev/null; then
echo "$BACKUP_DIR mounted";
else
echo "$BACKUP_DIR error mounting";
exit -1;
fi
fi
# Stop Tomcat
/etc/init.d/tomcat stop
# Clean logs
#echo "Clean Tomcat temporal files."
#rm -rf $TOMCAT_HOME/logs/*
#rm -rf $TOMCAT_HOME/temp/*
#rm -rf $TOMCAT_HOME/work/Catalina/localhost
# Backup database
if [ -n "$DATABASE_PASS" ]; then
echo "* Backuping MySQL data from $OPENKM_DB..."
mysqldump -h localhost -u root -p$DATABASE_PASS $OPENKM_DB > $DATABASE_EXP/mysql_$OPENKM_DB.sql
echo "-------------------------------------";
fi
# Backup and purge old backups
duplicity remove-older-than 1M --force $BACKUP_DIR/$HOST
if [ $(date +%u) -eq 7 ]; then
echo "*** Full Backup ***"
duplicity full --no-encryption $OPENKM_HOME $BACKUP_DIR/$HOST
RETVAL=$?
else
echo "*** Incremental Backup ***"
duplicity --no-encryption $OPENKM_HOME $BACKUP_DIR/$HOST
RETVAL=$?
fi
[ $RETVAL -eq 0 ] && echo "*** SUCCESS ***"
[ $RETVAL -ne 0 ] && echo "*** FAILURE ***"
# Start Tomcat
/etc/init.d/tomcat start
echo -e "\n### END: $(date +"%x %X") ###"
# Status
echo "=================================";
duplicity collection-status $BACKUP_DIR/$HOST
echo "*********************************";
df -h | grep "$BACKUP_DIR"
echo "=================================";
# Umount disk
sync
umount "$BACKUP_DIR"
Increase the backup period
Modify the # Backup and purge old backups section increasing the parameter value:
The "1M" value indicate will be removed backups older one month. Changing to "3M" or "1Y" indicate older 3 Month or 1 Year, etc..
--remove-older-than 3M
Perform the remote backup
Modify the ## Config section section.
BACKUP_DIR="ftp://user@ftp.domain.es/backup"
export FTP_PASSWORD="WhateverPasswordYouSetUp"
Modify the # Backup and purge old backups section.
# Backup and purge old backups
duplicity remove-older-than 1M --force $BACKUP_DIR/$HOST
if [ $(date +%u) -eq 7 ]; then
echo "*** Full Backup ***"
duplicity full --no-encryption $OPENKM_HOME $BACKUP_DIR/$HOST
RETVAL=$?
else
echo "*** Incremental Backup ***"
duplicity --no-encryption $OPENKM_HOME $BACKUP_DIR/$HOST
RETVAL=$?
fi
[ $RETVAL -eq 0 ] && echo "*** SUCCESS ***"
[ $RETVAL -ne 0 ] && echo "*** FAILURE ***"
Modify the # Status section.
duplicity collection-status $BACKUP_DIR/$HOST
unset FTP_PASSWORD
Perform the remote backup to SMB or CIFS
Modify the # Mount disk section.
# Mount disk
if mount | grep "$BACKUP_DIR type" > /dev/null; then
echo "$BACKUP_DIR already mounted";
else
echo "Mounting $BACKUP_DIR ...";
mount -t cifs //REMOTE_SERVER/REMOTE_DIR $BACKUP_DIR -o username=REMOTE_USER,password=REMOTE_PASSWORD,iocharset=utf8,file_mode=0777,dir_mode=0777
if mount | grep "$BACKUP_DIR type" > /dev/null; then
echo "$BACKUP_DIR mounted";
else
echo "Error mounting $BACKUP_DIR.";
exit -1;
fi
fi
Do the PostgreSQL backup
Modify the # Backup database section
# Backup de PostgreSQL
echo "* Backuping PostgreSQL data from $OPENKM_DB..."
su postgres -c "pg_dump $OPENKM_DB" > $DATABASE_EXP/pg_$OPENKM_DB.sql
# Databases optimizations
su postgres -c "vacuumdb -a -z" > /dev/null
su postgres -c "reindexdb -a -q" 2> /dev/null
Do the remote backup to Amazon s3
Modify the ## Config section section.
BACKUP_DIR="s3+http://somebucket/somedirectory"
export AWS_ACCESS_KEY_ID=""
export AWS_SECRET_ACCESS_KEY=""
export PASSPHRASE=""
Modify the # Backup and purge old backups section.
# Backup and purge old backups
if [ $(date +%u) -eq 7 ]; then
echo "*** Full Backup ***"
duplicity full --no-encryption --s3-use-new-style $OPENKM_HOME $BACKUP_DIR/$HOST
RETVAL=$?
else
echo "*** Incremental Backup ***"
duplicity --no-encryption --s3-use-new-style $OPENKM_HOME $BACKUP_DIR/$HOST
RETVAL=$?
fi
[ $RETVAL -eq 0 ] && echo "*** SUCCESS ***"
Modify the # Status section.
# Status
duplicity collection-status --s3-use-new-style $BACKUP_DIR/$HOST
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
unset PASSPHRASE
Configure crontab
To install the cron job, run:
$ sudo mkdir /root/logs
$ sudo crontab -e
Add these lines according to your personal configuration:
MAILTO=nomail@openkm.com
@daily /root/backup.sh | tee /root/logs/backup.$(date +\%Y.\%m.\%d_\%H.\%M.\%S).log
More information at Crontab quick reference
If you want to be notified by mail you should install "postfix" service in your server.
Troubleshooting deleting old backups
You have configured Duplicity to make a full backup every 7 days, and deleting backups older than 14 days to save space. What happen if you see a message like this?
Tue May 1 00:00:00 2012
Wed May 2 00:00:00 2012
Which can't be deleted because newer sets depend on them
Duplicity uses rsync, which contains incremental changes. Those files won't be deleted because, even though the backup maybe older than 7 days, there are backups which are incremental and younger than 7 days.
So, after 2 weeks have passed, those files will be deleted, since the full backup and the incremental backups are now 14 days old, and there exists a full backup newer than the full and incremental backups.
Restoring backup
You can list the available backups:
$ duplicity collection-status /path/to/backup
Sun Sep 11 05:00:19 2012 4.25 GB 4.25 GB (current mirror)
Sun Sep 4 00:00:18 2012 13.3 MB 4.26 GB
Sun Aug 28 00:00:13 2012 674 MB 4.92 GB
Sun Aug 21 00:00:14 2012 5.50 MB 4.93 GB
Sun Aug 14 00:00:16 2012 1.75 MB 4.93 GB
Sun Aug 7 00:00:12 2012 288 KB 4.93 GB
Sun Jul 31 00:00:13 2012 43.0 KB 4.93 GB
Fri Jul 29 10:36:39 2012 5.56 KB 4.93 GB
Then decide to restore some of the backups, for example the one made on Sun Aug 28 00:00:13 2012:
$ duplicity restore --restore-time 2012-08-28 /path/to/backup /path/to/destination
The --restore-time parameter accepts several formats. See duplicity documentation and Duplicity Backup Howto for more info.
Inside of /path/to/destination you should see a directory /home/openkm, and inside it a couple of directories:
- db: The backup of the database.
- tomcat: The backup of the Tomcat installation and OpenKM repository into.
Additional information
- Using Duplicity to Backup Your Data
- Ubuntu/Debian - Encrypted Incremental Backups With Duplicity on Amazon S3
- Duplicity Backup Howto
- Duplicity URL Format
- Installation and configuration of duplicity for encrypted SFTP remote backup
- Playing with Duplicity backup and restore tool and Amazon S3
- Bash wrapper script for automated backups with duplicity