Linux backup and restore with rsync

Rsync is a software application and network protocol for Unix, Linux and Windows systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. An important feature of rsync not found in similar programs/protocols is that the mirroring takes place with only one transmission in each direction. rsync can copy or display directory contents and copy files, optionally using compression and recursion.

Preliminaries

In these scripts are assumed these default values:

• $TOMCAT_HOME value is "/home/openkm/tomcat"
• OpenKM database is named "okmdb".
• OpenKM database password value is "*secret*".
• OpenKM start & stop service script location is "/etc/init.d/tomcat"

Sample script

USB disk mount point can be defined in /etc/fstab as:

/dev/sdb1    /mnt/backup    ext4    defaults    0    0

These are the global script sections:

• Global var configuration under ## BEGIN CONFIG ## section.
• Checking user root under # Check root user section.
• Deleting older database dump # Delete older local database backup section.
• Mount disk  ( optional section ) under # Mount disk section.
• Stop application ( optional section ) under  # Stop tomcat section.
• Clean logs ( optional section ) under # Clean logs section.
• Backup the database to the filesystem under #Backup database section.
• Backup repository and database dump to backup destination under # Create backup section.
• Start application ( optional section ) # Start tomcat section.
• Umount disk ( optional section ) under # Umount disk section.

Configuration parameters explanation:

Create a backup script /root/backup.sh

#!/bin/bash
#
## BEGIN CONFIG ##
HOST=$(uname -n)
DATABASE_PASS="*secret*"
OPENKM_DB="okmdb"
OPENKM_HOME="/home/openkm"
TOMCAT_HOME="$OPENKM_HOME/tomcat"
DATABASE_EXP="$OPENKM_HOME/db"
BACKUP_DIR="/mnt/backup"
RSYNC_OPTS="-apzhR --stats --delete --exclude=*~ --delete-excluded"
## END CONFIG ##

# Check root user 
if [ $(id -u) != 0 ]; then echo "You should run this script as root"; exit; fi

# Delete older local database backup 
echo -e "### BEGIN: $(date +"%x %X") ###\n"
rm -rf $DATABASE_EXP
mkdir -p $DATABASE_EXP
 
# Mount disk
if mount | grep "$BACKUP_DIR type" > /dev/null; then
  echo "$BACKUP_DIR already mounted";
else
  mount "$BACKUP_DIR";
  
  if mount | grep "$BACKUP_DIR type" > /dev/null; then
    echo "$BACKUP_DIR mounted";
  else
    echo "$BACKUP_DIR error mounting";
    exit -1;
  fi
fi
 
# Stop Tomcat
/etc/init.d/tomcat stop
 
# Clean logs
#echo "Clean Tomcat temporal files."
#rm -rf $TOMCAT_HOME/logs/*
#rm -rf $TOMCAT_HOME/temp/*
#rm -rf $TOMCAT_HOME/work/Catalina/localhost
 
# Backup database
if [ -n "$DATABASE_PASS" ]; then
  echo "* Backuping MySQL data from $OPENKM_DB..."
  mysqldump -h localhost -u root -p$DATABASE_PASS $OPENKM_DB > $DATABASE_EXP/mysql_$OPENKM_DB.sql
  echo "-------------------------------------";
fi
 
# Create backup
rsync $RSYNC_OPTS $OPENKM_HOME $BACKUP_DIR/$HOST
 
# Start Tomcat
/etc/init.d/tomcat start
echo -e "\n### END: $(date +"%x %X") ###"
 
# Umount disk
sync
umount "$BACKUP_DIR"

Do backup with rotation

Modify the # Create backup section.

# Create backup
# Calculate snapshot
LAST_SNAPSHOT=`ls -ltr $BACKUP_DIR | tail -1 | awk {'print $8'} | cut -d . -f 2`
NEW_SNAPSHOT=$((LAST_SNAPSHOT+1))
 
rsync $RSYNC_OPTS --link-dest="$BACKUP_DIR/$HOST/backup.$LAST_SNAPSHOT" $FILES "$BACKUP_DIR/$HOST/backup.$NEW_SNAPSHOT"

# Create backup
"cd $BACKUP_DIR/$HOST; rm -rf backup.3; mv backup.2 backup.3; mv backup.1 backup.2; mv backup.0 backup.1"
rsync $RSYNC_OPTS --link-dest="$BACKUP_DIR/$HOST/backup.1" "OPENKM_HOME" "$BACKUP_DIR/$HOST/backup.0" 

Do remote backup

Modify the  ## BEGIN CONFIG ## section.

BACKUP_SRV="user@server.com"

Modify the # Create backup section.

# Create backup
# Create missing backup folder ( optional )
ssh $BACKUP_SRV "mkdir -p $BACKUP_DIR"

# Calculate snapshot
NEW_SNAPSHOT=$(date "+%Y-%m-%dT%H:%M:%S")
LAST_SNAPSHOT=$(ssh $BACKUP_SRV ls -tr $BACKUP_DIR/$HOST | tail -1)
 
if [ $LAST_SNAPSHOT ]; then
  echo "Incremental backup $NEW_SNAPSHOT based on $LAST_SNAPSHOT... "
  rsync $RSYNC_OPTS --link-dest="$BACKUP_DIR/$HOST/$LAST_SNAPSHOT" "$OPENKM_HOME" "$BACKUP_SRV:$BACKUP_DIR/$HOST/$NEW_SNAPSHOT"
else
  echo "Initial full backup $NEW_SNAPSHOT..."
  rsync $RSYNC_OPTS "$OPENKM_HOME" "$BACKUP_SRV:$BACKUP_DIR/$HOST/$NEW_SNAPSHOT"
fi

rsync $RSYNC_OPTS "$OPENKM_HOME" "$BACKUP_SRV:$BACKUP_DIR/$HOST" 

Do incremental backup with backup-dir

Modify the  ## BEGIN CONFIG ## section.

RSYNC_OPTS="-ahR --partial --stats --delete --exclude=*~ --exclude=tomcat/temp/* --exclude=tomcat/work/* --exclude=tomcat/logs/* --delete-excluded"

Modify the # Create backup section.

# Create backup
NEW_SNAPSHOT=`date +%Y-%m-%d`
echo -e "\n### RSYNC - BEGIN : $(date +"%x %X") ###\n"
echo "* New Snapshot: $NEW_SNAPSHOT"
rsync $RSYNC_OPTS --backup --backup-dir="$BACKUP_DIR/$HOST/INCREMENTAL/$NEW_SNAPSHOT" $OPENKM_HOME/ "$BACKUP_DIR/$HOST/LAST_BACKUP"

Do incremental backup with link-dest

Modify the  ## BEGIN CONFIG ## section.

RSYNC_OPTS="-ahR --partial --stats --delete --exclude=*~ --exclude=tomcat/temp/* --exclude=tomcat/work/* --exclude=tomcat/logs/* --delete-exclude"

Modify the # Create backup section.

# Create backup
# Calculate snapshot
LAST_SNAPSHOT=`ssh $BACKUP_SRV "cd $BACKUP_DIR && stat --format='%n' * | tail -1"`
NEW_SNAPSHOT=`date +%Y-%m-%d`
echo -e "\n### RSYNC - BEGIN : $(date +"%x %X") ###\n"
echo "* Last Snapshot: $LAST_SNAPSHOT"
echo "* New Snapshot: $NEW_SNAPSHOT"
 
if [ ! $LAST_SNAPSHOT ]; then
  rsync $RSYNC_OPTS $OPENKM_HOME/ "$BACKUP_DIR/$HOST/$NEW_SNAPSHOT"
else 
  rsync $RSYNC_OPTS --link-dest="$BACKUP_DIR/$HOST/$LAST_SNAPSHOT" $OPENKM_HOME/ "$BACKUP_DIR/$HOST/$NEW_SNAPSHOT"
fi

Do remote backup to SMB or CIFS

Modify the # Mount disk section.

# Mount disk
if mount | grep "$BACKUP_DIR type" > /dev/null; then
  echo "$BACKUP_DIR already mounted";
else
  echo "Mounting $BACKUP_DIR ...";
  mount -t cifs //REMOTE_SERVER/REMOTE_DIR $BACKUP_DIR -o username=REMOTE_USER,password=REMOTE_PASSWORD,iocharset=utf8,file_mode=0777,dir_mode=0777
  if mount | grep "$BACKUP_DIR type" > /dev/null; then
    echo "$BACKUP_DIR mounted";
  else
    echo "Error mounting $BACKUP_DIR.";
    exit -1;
  fi
fi

Do PostgreSQL backup

Modify the # Backup database section

# Backup de PostgreSQL
echo "* Backuping PostgreSQL data from $OPENKM_DB..."
su postgres -c "pg_dump $OPENKM_DB" > $DATABASE_EXP/pg_$OPENKM_DB.sql
 
# Databases optimizations
su postgres -c "vacuumdb -a -z" > /dev/null
su postgres -c "reindexdb -a -q" 2> /dev/null

Configure crontab

To install the cron job, run:

And add these lines according to your personal configuration:

MAILTO=nomail@openkm.com
@daily /root/backup.sh | tee /root/logs/backup.$(date +\%Y.\%m.\%d_\%H.\%M.\%S).log

Rsync and FTP

Rsync need SSH access to work, but you can also use it with an FTP server if you mount it locally with curlftpfs:

Install curlftpfs and rsync:

Create mountpoin:

Mount remote director:

Create local directory:

Rsync mounted directory to local director:

Restoring backup

You can list the available backups:

Then decide to restore some of the backups, for example the one made on Sun Aug 28 00:00:13 2012:

Additional information

• Easy Automated Snapshot-Style Backups with Linux and Rsync