SystemAdministration/Backups - Revision history

Paulej72: /* rsnapshot Backups */

2015-03-22T02:31:30Z

rsnapshot Backups

← Older revision		Revision as of 02:31, 22 March 2015
Line 4:		Line 4:

	== rsnapshot Backups ==		== rsnapshot Backups ==
	~~<a href=~~http://www.rsnapshot.org/howto/1.2/rsnapshot-HOWTO.en.html#intro">rsnapshot~~</a>~~ is a filesystem backup utility based on rsync. Using rsnapshot, it is possible to take snapshots of filesystems at different points in time. Using hard links, rsnapshot creates the illusion of multiple full backups, while only taking up the space of one full backup plus differences.		[http://www.rsnapshot.org/howto/1.2/rsnapshot-HOWTO.en.html#intro rsnapshot] is a filesystem backup utility based on rsync. Using rsnapshot, it is possible to take snapshots of filesystems at different points in time. Using hard links, rsnapshot creates the illusion of multiple full backups, while only taking up the space of one full backup plus differences.

	Currently rsnapshot is configured to keep 7 daily backups, 4 weekly backups, and 3 monthly backups. The backups are scheduled using cron with the script /etc/cron.d/backups. Since rsnapshot uses rsync over ssh and needs to run as root to copy all files easily, a root Kerberos principal has been created with a corresponding keytab. A wrapper script, /home/backups/rsnapshot/krsnapshot, setups the kinit before running rsnapshot. The backup cron script also rsyncs /etc on Oxygen to /root/oxygen-backups/ on Boron as the only live backup of the Oxygen system (static copies of the different scripts in /home/backups/ was also made).		Currently rsnapshot is configured to keep 7 daily backups, 4 weekly backups, and 3 monthly backups. The backups are scheduled using cron with the script /etc/cron.d/backups. Since rsnapshot uses rsync over ssh and needs to run as root to copy all files easily, a root Kerberos principal has been created with a corresponding keytab. A wrapper script, /home/backups/rsnapshot/krsnapshot, setups the kinit before running rsnapshot. The backup cron script also rsyncs /etc on Oxygen to /root/oxygen-backups/ on Boron as the only live backup of the Oxygen system (static copies of the different scripts in /home/backups/ was also made).

Paulej72: /* rsnapshot Backups */

2015-03-22T02:30:53Z

rsnapshot Backups

← Older revision		Revision as of 02:30, 22 March 2015
Line 4:		Line 4:

	== rsnapshot Backups ==		== rsnapshot Backups ==
	rsnapshot is a filesystem backup utility based on rsync. Using rsnapshot, it is possible to take snapshots of filesystems at different points in time. Using hard links, rsnapshot creates the illusion of multiple full backups, while only taking up the space of one full backup plus differences.		<a href=http://www.rsnapshot.org/howto/1.2/rsnapshot-HOWTO.en.html#intro">rsnapshot</a> is a filesystem backup utility based on rsync. Using rsnapshot, it is possible to take snapshots of filesystems at different points in time. Using hard links, rsnapshot creates the illusion of multiple full backups, while only taking up the space of one full backup plus differences.

	Currently rsnapshot is configured to keep 7 daily backups, 4 weekly backups, and 3 monthly backups. The backups are scheduled using cron with the script /etc/cron.d/backups. Since rsnapshot uses rsync over ssh and needs to run as root to copy all files easily, a root Kerberos principal has been created with a corresponding keytab. A wrapper script, /home/backups/rsnapshot/krsnapshot, setups the kinit before running rsnapshot. The backup cron script also rsyncs /etc on Oxygen to /root/oxygen-backups/ on Boron as the only live backup of the Oxygen system (static copies of the different scripts in /home/backups/ was also made).		Currently rsnapshot is configured to keep 7 daily backups, 4 weekly backups, and 3 monthly backups. The backups are scheduled using cron with the script /etc/cron.d/backups. Since rsnapshot uses rsync over ssh and needs to run as root to copy all files easily, a root Kerberos principal has been created with a corresponding keytab. A wrapper script, /home/backups/rsnapshot/krsnapshot, setups the kinit before running rsnapshot. The backup cron script also rsyncs /etc on Oxygen to /root/oxygen-backups/ on Boron as the only live backup of the Oxygen system (static copies of the different scripts in /home/backups/ was also made).

Paulej72: /* Service VMs */

2015-03-21T13:37:17Z

Service VMs

← Older revision		Revision as of 13:37, 21 March 2015
Line 10:		Line 10:
	== MySQL Backups ==		== MySQL Backups ==
	=== Service VMs ===		=== Service VMs ===
	Full MySQL backups on each service vm (Boron and Beryllium) is handled by a simple cron job the does a mysqldump of all databases on the server. The file is gziped and saved to /root/db_dumps/. It runs daily and overwrites the backup each day. The backup file will be stored by rsnapshot on Oxygen, so no old backups on local server needs to be kept.		Full MySQL backups on each service vm (Boron and Beryllium) is handled by a simple cron job the does a mysqldump of all databases on the server. The file is gziped and saved to /root/db_dumps/. It runs daily and overwrites the backup each day. The backup file will be stored by rsnapshot on Oxygen, so no old backups on local server needs to be kept. The cron job is /etc/cron.d/mysql_backup and calls /root/db_dumps/mysql_backup.sh.

	=== Production Server ===		=== Production Server ===
	<<<This will be updated soon when the new MySQL Cluster takes over as the database server for both Production and Dev.>>>		<<<This will be updated soon when the new MySQL Cluster takes over as the database server for both Production and Dev.>>>

Paulej72: /* rsnapshot Backups */

2015-03-21T13:29:35Z

rsnapshot Backups

← Older revision		Revision as of 13:29, 21 March 2015
Line 4:		Line 4:

	== rsnapshot Backups ==		== rsnapshot Backups ==
	rsnapshot is a filesystem backup utility based on rsync. Using rsnapshot, it is possible to take snapshots of ~~your~~ filesystems at different points in time. Using hard links, rsnapshot creates the illusion of multiple full backups, while only taking up the space of one full backup plus differences.		rsnapshot is a filesystem backup utility based on rsync. Using rsnapshot, it is possible to take snapshots of filesystems at different points in time. Using hard links, rsnapshot creates the illusion of multiple full backups, while only taking up the space of one full backup plus differences.

	Currently rsnapshot is configured to keep 7 daily backups, 4 weekly backups, and 3 monthly backups. The backups are scheduled using cron with the script /etc/cron.d/backups. Since rsnapshot uses rsync over ssh and needs to run as root to copy all files easily, a root Kerberos principal has been created with a corresponding keytab. A wrapper script, /home/backups/rsnapshot/krsnapshot, setups the kinit before running rsnapshot. The backup cron script also rsyncs /etc on Oxygen to /root/oxygen-backups/ on Boron as the only live backup of the Oxygen system (static copies of the different scripts in /home/backups/ was also made).		Currently rsnapshot is configured to keep 7 daily backups, 4 weekly backups, and 3 monthly backups. The backups are scheduled using cron with the script /etc/cron.d/backups. Since rsnapshot uses rsync over ssh and needs to run as root to copy all files easily, a root Kerberos principal has been created with a corresponding keytab. A wrapper script, /home/backups/rsnapshot/krsnapshot, setups the kinit before running rsnapshot. The backup cron script also rsyncs /etc on Oxygen to /root/oxygen-backups/ on Boron as the only live backup of the Oxygen system (static copies of the different scripts in /home/backups/ was also made).

Paulej72: /* Production Server */

2015-03-21T13:23:14Z

Production Server

← Older revision		Revision as of 13:23, 21 March 2015
Line 15:		Line 15:
	<<<This will be updated soon when the new MySQL Cluster takes over as the database server for both Production and Dev.>>>		<<<This will be updated soon when the new MySQL Cluster takes over as the database server for both Production and Dev.>>>

	The primary MySQL database for the production slash server is now backed up to Oxygen via a script that runs from a crontab job on Neon. The jobs are run by the user slash from the /srv/soylentnews.org directory. The scripts ~~are similar for each, but production~~ is run daily~~, while dev and staff run weekly~~. A local copy is generated, gziped, then stored in /srv/soylentnews.org/db_dumps. Old copies are kept for two weeks and deleted by the script.		The primary MySQL database for the production slash server is now backed up to Oxygen via a script that runs from a crontab job on Neon. The jobs are run by the user slash from the /srv/soylentnews.org directory. The scripts is run daily. A local copy is generated, gziped, then stored in /srv/soylentnews.org/db_dumps. Old copies are kept for two weeks and deleted by the script.

	Files are permanently stored in /home/backups/db_dumps on ~~oxygen~~. These files are not automatically deleted so there may be a need in the future to delete the old files.		Files are permanently stored in /home/backups/db_dumps on Oxygen. These files are not automatically deleted so there may be a need in the future to delete the old files.

	Here is the mysql_backup.sh script for ~~neon~~:		Here is the mysql_backup.sh script for Neon:

	<pre>		<pre>

Paulej72 at 13:21, 21 March 2015

2015-03-21T13:21:39Z

← Older revision		Revision as of 13:21, 21 March 2015
Line 1:		Line 1:
			== Oxygen Backup Server ==
			Oxygen is our backup server. It is a Ramnode 1024MB CVZ - OpenVZ SSD-Cached system with 200GB of hard drive space, that is located in the New York data center. It connects to the Linode servers over IPv6 and looks to the other systems like it is just another Linode VM. Backup are stored in /home/backups. There are two types of backups. /home/backups/tarballs stores the full tar backups of the various servers at certain points in time with the date of the archive in the archive name. /home/backups/rsnapshot stores full system snapshots of the servers as regular files that can easily be browsed and copied.


			== rsnapshot Backups ==
			rsnapshot is a filesystem backup utility based on rsync. Using rsnapshot, it is possible to take snapshots of your filesystems at different points in time. Using hard links, rsnapshot creates the illusion of multiple full backups, while only taking up the space of one full backup plus differences.

			Currently rsnapshot is configured to keep 7 daily backups, 4 weekly backups, and 3 monthly backups. The backups are scheduled using cron with the script /etc/cron.d/backups. Since rsnapshot uses rsync over ssh and needs to run as root to copy all files easily, a root Kerberos principal has been created with a corresponding keytab. A wrapper script, /home/backups/rsnapshot/krsnapshot, setups the kinit before running rsnapshot. The backup cron script also rsyncs /etc on Oxygen to /root/oxygen-backups/ on Boron as the only live backup of the Oxygen system (static copies of the different scripts in /home/backups/ was also made).

	== MySQL Backups ==		== MySQL Backups ==
	The primary ~~MySql~~ database for ~~each~~ slash server is now backed up to ~~oxygen~~ via a script that runs from a crontab job on ~~neon, lithium, and nitrogen, to back up production, dev, and staff respectively~~. The jobs are run by the user slash from the /srv/soylentnews.org directory. The scripts are similar for each, but production is run daily, while dev and staff run weekly. A local copy is generated, gziped, then stored in /srv/soylentnews.org/db_dumps. Old copies are kept for two weeks and deleted by the script.		=== Service VMs ===
			Full MySQL backups on each service vm (Boron and Beryllium) is handled by a simple cron job the does a mysqldump of all databases on the server. The file is gziped and saved to /root/db_dumps/. It runs daily and overwrites the backup each day. The backup file will be stored by rsnapshot on Oxygen, so no old backups on local server needs to be kept.

			=== Production Server ===
			<<<This will be updated soon when the new MySQL Cluster takes over as the database server for both Production and Dev.>>>

			The primary MySQL database for the production slash server is now backed up to Oxygen via a script that runs from a crontab job on Neon. The jobs are run by the user slash from the /srv/soylentnews.org directory. The scripts are similar for each, but production is run daily, while dev and staff run weekly. A local copy is generated, gziped, then stored in /srv/soylentnews.org/db_dumps. Old copies are kept for two weeks and deleted by the script.

	Files are permanently stored in /home/backups/db_dumps on oxygen. These files are not automatically deleted so there may be a need in the future to delete the old files.		Files are permanently stored in /home/backups/db_dumps on oxygen. These files are not automatically deleted so there may be a need in the future to delete the old files.
Line 35:		Line 50:
	</pre>		</pre>

	For now production mysqldump takes only a few seconds and the transfer to oxygen takes actually more time. Current dump size is approximately ~~250Mbi uncompressed and 40Mbi~~ compressed for the production database.		For now production mysqldump takes only a few seconds and the transfer to oxygen takes actually more time. Current dump size is approximately 100Mbi compressed for the production database.

	[[Category:System administration]]		[[Category:System administration]]

Paulej72 at 03:27, 22 May 2014

2014-05-22T03:27:33Z

← Older revision		Revision as of 03:27, 22 May 2014
Line 36:		Line 36:

	For now production mysqldump takes only a few seconds and the transfer to oxygen takes actually more time. Current dump size is approximately 250Mbi uncompressed and 40Mbi compressed for the production database.		For now production mysqldump takes only a few seconds and the transfer to oxygen takes actually more time. Current dump size is approximately 250Mbi uncompressed and 40Mbi compressed for the production database.

			[[Category:System administration]]

Paulej72: Created page with "== MySQL Backups == The primary MySql database for each slash server is now backed up to oxygen via a script that runs from a crontab job on neon, lithium, and nitrogen, to ba..."

2014-05-22T03:26:14Z

Created page with "== MySQL Backups == The primary MySql database for each slash server is now backed up to oxygen via a script that runs from a crontab job on neon, lithium, and nitrogen, to ba..."

New page

== MySQL Backups ==
The primary MySql database for each slash server is now backed up to oxygen via a script that runs from a crontab job on neon, lithium, and nitrogen, to back up production, dev, and staff respectively. The jobs are run by the user slash from the /srv/soylentnews.org directory. The scripts are similar for each, but production is run daily, while dev and staff run weekly. A local copy is generated, gziped, then stored in /srv/soylentnews.org/db_dumps. Old copies are kept for two weeks and deleted by the script.

Files are permanently stored in /home/backups/db_dumps on oxygen. These files are not automatically deleted so there may be a need in the future to delete the old files.

Here is the mysql_backup.sh script for neon:

<pre>
#! /bin/bash
HOME=/srv/soylentnews.org

backup_dir=$HOME/db_dumps
today=`date +%F`
week=`date --date="-14 days" +%F`

if [ -e $backup_dir/soylentnews-$week.sql.gz ]
then
rm -rf $backup_dir/soylentnews-$week.sql.gz
fi

/opt/mysql-5.6.17-linux-glibc2.5-x86_64/bin/mysqldump -h 127.0.0.1 --single-transaction soylentnews > $backup_dir/soylentnews-$today.sql

gzip $backup_dir/soylentnews-$today.sql

kinit -k -t $HOME/slash.keytab slash

scp $backup_dir/soylentnews-$today.sql.gz oxygen:/home/backups/db_dumps/soylentnews-$today.sql.gz

</pre>

Here is the assoicated crontab entry:

<pre>
4 5 * * * /srv/soylentnews.org/mysql_backup.sh
</pre>

For now production mysqldump takes only a few seconds and the transfer to oxygen takes actually more time. Current dump size is approximately 250Mbi uncompressed and 40Mbi compressed for the production database.