Linux backup and restore with rsync

rsync is a software application and network protocol for Unix, Linux and Windows systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. An important feature of rsync not found in similar programs/protocols is that the mirroring takes place with only one transmission in each direction. rsync can copy or display directory contents and copy files, optionally using compression and recursion.

Preliminaries

In these scripts, the following default values are assumed:

• $TOMCAT_HOME value is “/home/openkm/tomcat-9.0.76”
• OpenKM database is named “okmdb”.
• The OpenKM database password is “*secret*”.
• The OpenKM start & stop service script location is “/etc/init.d/tomcat”

Sample script

Caution

As a good practice, the backup should be done by the root user.

Note

The script below performs a backup to a USB drive and the application’s database is MySQL.

The USB disk mount point can be defined in /etc/fstab as:

/dev/sdb1    /mnt/backup    ext4    defaults    0    0

These are the global script sections:

• Global variable configuration under ## BEGIN CONFIG ## section.
• Checking for the root user under # Check root section.
• Deleting older database dump under # Delete older local database backup section.
• Mount disk (optional section) under # Mount disk section.
• Stop application (optional section) under # Stop tomcat section.
• Clean logs (optional section) under # Clean logs section.
• Backup the database to the filesystem under # Backup database section.
• Backup repository and database dump to the backup destination under # Create backup section.
• Start application (optional section) under # Start tomcat section.
• Umount disk (optional section) under # Umount disk section.

Explanation of configuration parameters:

Parameter	Description
HOST	The server hostname.
DATABASE_PASS	The database password. The database user that performs the backup is always the root user.
OPENKM_DB	The application’s database name. It is usually named “okmdb”.
OPENKM_HOME	The OpenKM home folder.
TOMCAT_HOME	The Tomcat home folder. Usually inside the OPENKM_HOME folder.
DATABASE_EXP	The database dump folder. Usually on the same server.
BACKUP_DIR	Backup destination.
RSYNC_OPTS	rsync parameters.

Create a backup script /root/backup.sh

$ sudo su
$ vim /root/backup.sh

#!/bin/bash
#
## BEGIN CONFIG ##
HOST=$(uname -n)
DATABASE_PASS="*secret*"
OPENKM_DB="okmdb"
OPENKM_HOME="/home/openkm"
TOMCAT_HOME="$OPENKM_HOME/tomcat-9.0.76"
DATABASE_EXP="$OPENKM_HOME/db"
BACKUP_DIR="/mnt/backup"
RSYNC_OPTS="-apzhR --stats --delete --exclude=*~ --delete-excluded"
## END CONFIG ##

# Check root user
if [ $(id -u) != 0 ]; then echo "You should run this script as root"; exit; fi

# Delete older local database backup
echo -e "### BEGIN: $(date +"%x %X") ###\n"
rm -rf $DATABASE_EXP
mkdir -p $DATABASE_EXP

# Mount disk
if mount | grep "$BACKUP_DIR type" > /dev/null; then
  echo "$BACKUP_DIR already mounted";
else
  mount "$BACKUP_DIR";
  if mount | grep "$BACKUP_DIR type" > /dev/null; then
    echo "$BACKUP_DIR mounted";
  else
    echo "$BACKUP_DIR error mounting";
    exit -1;
  fi
fi

# Stop Tomcat
/etc/init.d/tomcat stop

# Clean logs
#echo "Clean Tomcat temporal files."
#rm -rf $TOMCAT_HOME/logs/*
#rm -rf $TOMCAT_HOME/temp/*
#rm -rf $TOMCAT_HOME/work/Catalina/localhost

# Backup database
if [ -n "$DATABASE_PASS" ]; then
  echo "* Backuping MySQL data from $OPENKM_DB..."
  mysqldump -h localhost -u root -p$DATABASE_PASS $OPENKM_DB > $DATABASE_EXP/mysql_$OPENKM_DB.sql
  echo "-------------------------------------";
fi

# Create backup
rsync $RSYNC_OPTS $OPENKM_HOME $BACKUP_DIR/$HOST

# Start Tomcat
/etc/init.d/tomcat start
echo -e "\n### END: $(date +"%x %X") ###"

# Umount disk
sync
umount "$BACKUP_DIR"

Do backup with rotation

Modify the # Create backup section.

# Create backup
# Calculate snapshot
LAST_SNAPSHOT=`ls -ltr $BACKUP_DIR | tail -1 | awk {'print $8'} | cut -d . -f 2`
NEW_SNAPSHOT=$((LAST_SNAPSHOT+1))

rsync $RSYNC_OPTS --link-dest="$BACKUP_DIR/$HOST/backup.$LAST_SNAPSHOT" $FILES "$BACKUP_DIR/$HOST/backup.$NEW_SNAPSHOT"

Note

Other configuration:

# Create backup
"cd $BACKUP_DIR/$HOST; rm -rf backup.3; mv backup.2 backup.3; mv backup.1 backup.2; mv backup.0 backup.1"
rsync $RSYNC_OPTS --link-dest="$BACKUP_DIR/$HOST/backup.1" "OPENKM_HOME" "$BACKUP_DIR/$HOST/backup.0"

Do remote backup

Modify the ## BEGIN CONFIG ## section.

BACKUP_SRV="user@server.com"

Modify the # Create backup section.

# Create backup
# Create missing backup folder ( optional )
ssh $BACKUP_SRV "mkdir -p $BACKUP_DIR"

# Calculate snapshot
NEW_SNAPSHOT=$(date "+%Y-%m-%dT%H:%M:%S")
LAST_SNAPSHOT=$(ssh $BACKUP_SRV ls -tr $BACKUP_DIR/$HOST | tail -1)

if [ $LAST_SNAPSHOT ]; then
  echo "Incremental backup $NEW_SNAPSHOT based on $LAST_SNAPSHOT... "
  rsync $RSYNC_OPTS --link-dest="$BACKUP_DIR/$HOST/$LAST_SNAPSHOT" "$OPENKM_HOME" "$BACKUP_SRV:$BACKUP_DIR/$HOST/$NEW_SNAPSHOT"
else
  echo "Initial full backup $NEW_SNAPSHOT..."
  rsync $RSYNC_OPTS "$OPENKM_HOME" "$BACKUP_SRV:$BACKUP_DIR/$HOST/$NEW_SNAPSHOT"
fi

Note

More simple configuration:

rsync $RSYNC_OPTS "$OPENKM_HOME" "$BACKUP_SRV:$BACKUP_DIR/$HOST"

Do incremental backup with backup-dir

Modify the ## BEGIN CONFIG ## section.

RSYNC_OPTS="-ahR --partial --stats --delete --exclude=*~ --exclude=tomcat-9.0.76/temp/* --exclude=tomcat-9.0.76/work/* --exclude=tomcat-9.0.76/logs/* --delete-excluded"

Modify the # Create backup section.

# Create backup
NEW_SNAPSHOT=`date +%Y-%m-%d`
echo -e "\n### RSYNC - BEGIN : $(date +"%x %X") ###\n"
echo "* New Snapshot: $NEW_SNAPSHOT"
rsync $RSYNC_OPTS --backup --backup-dir="$BACKUP_DIR/$HOST/INCREMENTAL/$NEW_SNAPSHOT" $OPENKM_HOME/ "$BACKUP_DIR/$HOST/LAST_BACKUP"

Do incremental backup with link-dest

Modify the ## BEGIN CONFIG ## section.

RSYNC_OPTS="-ahR --partial --stats --delete --exclude=*~ --exclude=tomcat-9.0.76/temp/* --exclude=tomcat-9.0.76/work/* --exclude=tomcat-9.0.76/logs/* --delete-exclude"

Modify the # Create backup section.

# Create backup
# Calculate snapshot
LAST_SNAPSHOT=`ssh $BACKUP_SRV "cd $BACKUP_DIR && stat --format='%n' * | tail -1"`
NEW_SNAPSHOT=`date +%Y-%m-%d`
echo -e "\n### RSYNC - BEGIN : $(date +"%x %X") ###\n"
echo "* Last Snapshot: $LAST_SNAPSHOT"
echo "* New Snapshot: $NEW_SNAPSHOT"

if [ ! $LAST_SNAPSHOT ]; then
  rsync $RSYNC_OPTS $OPENKM_HOME/ "$BACKUP_DIR/$HOST/$NEW_SNAPSHOT"
else
  rsync $RSYNC_OPTS --link-dest="$BACKUP_DIR/$HOST/$LAST_SNAPSHOT" $OPENKM_HOME/ "$BACKUP_DIR/$HOST/$NEW_SNAPSHOT"
fi

Do remote backup to SMB or CIFS

Modify the # Mount disk section.

# Mount disk
if mount | grep "$BACKUP_DIR type" > /dev/null; then
  echo "$BACKUP_DIR already mounted";
else
  echo "Mounting $BACKUP_DIR ...";
  mount -t cifs //REMOTE_SERVER/REMOTE_DIR $BACKUP_DIR -o username=REMOTE_USER,password=REMOTE_PASSWORD,iocharset=utf8,file_mode=0777,dir_mode=0777
  if mount | grep "$BACKUP_DIR type" > /dev/null; then
    echo "$BACKUP_DIR mounted";
  else
    echo "Error mounting $BACKUP_DIR.";
    exit -1;
  fi
fi

Do PostgreSQL backup

Modify the # Backup database section.

# Backup de PostgreSQL
echo "* Backuping PostgreSQL data from $OPENKM_DB..."
su postgres -c "pg_dump $OPENKM_DB" > $DATABASE_EXP/pg_$OPENKM_DB.sql

# Databases optimizations
su postgres -c "vacuumdb -a -z" > /dev/null
su postgres -c "reindexdb -a -q" 2> /dev/null

Configure crontab

To install the cron job, run:

$ sudo mkdir /root/logs
$ sudo crontab -e

And add these lines according to your personal configuration:

MAILTO=nomail@openkm.com
@daily /root/backup.sh | tee /root/logs/backup.$(date +\%Y.\%m.\%d_\%H.\%M.\%S).log

Note

More information at Crontab quick reference

If you want to be notified by mail, you should install the “postfix” service on your server.

Rsync and FTP

rsync needs SSH access to work, but you can also use it with an FTP server if you mount it locally with curlftpfs:

Caution

Backup performance may degrade when using this method.

Install curlftpfs and rsync:

$ sudo apt-get install curlftpfs rsync

Create mount point:

$ mkdir /path/to/mount

Mount remote directory:

$ curlftpfs ftp.example.com /path/to/mount

Create local directory:

$ mkdir /local/path

Rsync the mounted directory to the local directory:

$ rsync -r -t -v --progress --bwlimit=500 /path/to/mount/ /local/path/

Restoring backup

You can list the available backups:

$ ls -ld /path/to/backup/*
drwxr-xr-x 4 root root 4096 sep 11 05:00:19 2012 2012-09-11T05:00:19
drwxr-xr-x 4 root root 4096 sep  4 00:00:18 2012 2012-09-04T00:00:18
drwxr-xr-x 4 root root 4096 aug 28 00:00:13 2012 2012-08-28T00:00:13
drwxr-xr-x 4 root root 4096 aug 21 00:00:14 2012 2012-08-21T00:00:14
drwxr-xr-x 4 root root 4096 aug 14 00:00:16 2012 2012-08-14T00:00:16
drwxr-xr-x 4 root root 4096 aug  7 00:00:12 2012 2012-08-07T00:00:12
drwxr-xr-x 4 root root 4096 jul 31 00:00:13 2012 2012-07-31T00:00:13
drwxr-xr-x 4 root root 4096 jul 29 10:36:39 2012 2012-07-29T10:36:39

Then decide to restore some of the backups; for example, the one made on Sun Aug 28 00:00:13 2012:

$ cp /path/to/backup/2012-08-28T00:00:13 /path/to/destination

Note

See rsync documentation for more information.

Note

Inside /path/to/destination, you should see a directory /home/openkm, and inside it a couple of directories:

• db: The backup of the database.
• tomcat-9.0.76: The backup of the Tomcat installation and OpenKM repository.

Additional information

• Easy Automated Snapshot-Style Backups with Linux and Rsync