cron script to backup and delete old

[splat]

so I've made a cron job to do nightly backups of my website, but that could potentially lead to filling up the disk. So, I was hoping there might be a way of deleting old files, like maybe keep 14 days worth and once a file is 15 days old it is deleted.

right now i do:

DATE=$(date +%s)

at the start of my script so I can saves files as backup_$DATE.tar.gz if that helps at all.

 

[smokie mcpott]

have you ever looked into running backuppc? it has excellent compression and tons of configurable options

 

[splat]

my hosting provider does tape backups, but I wanted to have my own for quicker access just in case. but no i've never looked in to backuppc. I figured some one would know how to add a couple lines to a bash script to check a files date and compare it to todays date

 

[TalRW]

Here is my script that I use. I still have some work to do. I want to make the mount point in two places and read only for the users when it is mounted but write for root (to add security) but I haven't implemented this yet. Either way though it gets the job done.

Note: oh it also has one other bug that I haven't fixed. I nuke the /mnt/rsyncbackup/ directory at the end. This is kinda a low priority bug but still possible. If the directory exists and there are files there before you mount and you nuke it then you will lose your data. I need to put an if statement to check to see if the directory was there before running the script and if it does exist don't delete it. Either way it's a minor bug so I haven't gotten around to fixing it, just be careful though when you pick your mount location that there isn't data already there.

#!/bin/sh
# -------------------------------------------------------------------------------
# Grant's rsync script which backs up your home directory to a backup hard drive
#--------------------------------------------------------------------------------

# ---------------------------IMPORTANT!!!!--------------------------------------
# ---------------------------IMPORTANT!!!!--------------------------------------
#THESE VARIABLES NEED TO BE CHANGED TO FIT YOUR SYSTEM FOR THIS SCRIPT TO WORK!

#Locations:
DATA_BACKUP=/home/; #This is the data you want to be backed up, in my case it is my home directory
MOUNT_DEVICE=/dev/hdb2; #This is the device or folder where you want to back up your data.  This is usually another hard drive or network location.

#USERS:
USER=sirgrant; #Change this to the user you want to own the log file and directory backed up.

#THESE VARIABLES NEED TO BE CHANGED TO FIT YOUR SYSTEM FOR THIS SCRIPT TO WORK!
# ---------------------------IMPORTANT!!!!--------------------------------------
# ---------------------------IMPORTANT!!!!--------------------------------------

unset PATH         # suggestion from H. Milz: avoid accidental use of $PATH
# ------------- system commands used by this script --------------------

CHOWN=/bin/chown;
ECHO=/bin/echo;
ID=/usr/bin/id;
MKDIR=/bin/mkdir;
MOUNT=/bin/mount;
MV=/bin/mv;
RM=/bin/rm;
RSYNC=/usr/bin/rsync;
TOUCH=/usr/bin/touch;
UMOUNT=/bin/umount;

# ------------------------ file locations -----------------------------------------
MOUNT_LOCATION=/mnt/rsyncbackup; #This is the location where the mount device is mounted to on your system

#Ignore this, it is going to be used in a later version of this script
#SNAPSHOT_RW=/root/snapshot;

# ------------- variables used by this script -------------------- 
DATE=`/bin/date "+%m_%d_%y"`; #Used to append the date to the created log file

# ---------------------- the actual script ----------------------------------
# make sure we're running as root
if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root.  Exiting..."; exit; } fi

$ECHO ""
$ECHO "~*~*~*~*~*~*~*~*~*~*~*~*~*"
$ECHO "Backing up home directory, this may take some time depending on how long 
it was since you last backed up..."
$ECHO "~*~*~*~*~*~*~*~*~*~*~*~*~*"

#Creates the mount point if it doesn't exist, if it does -p causes no error if existing
$MKDIR -p $MOUNT_LOCATION/
$MOUNT  $MOUNT_DEVICE $MOUNT_LOCATION

#Creates directory for log files and data to be backed up if they don't exist
$MKDIR -p $MOUNT_LOCATION/Rsync\ Logs/
$MKDIR -p $MOUNT_LOCATION/Data\ Backup/

#Actual rsync backup command.  Uses > to create a log file of what is copied.  
#-v, -- verbose - increase verbosity
#-a, -- archive - archive mode (equals -rlptgoD (no -H, -A, -X)
#-h, -- human-readable - output numbers in a human-readable format
#-u, -- update - skip files that are newer on the receiver
#-r, -- recursive - recurse into directories
#-l, -- links - copy symlinks as symlinks
#-p, -- perms - perserve permissions
#-t, -- times - preserve modification times
#-g, -- group - preserve group
#-o, -- owner - preserve owner (super-user only)
#-D, -- same as --devices --specials
#-H, -- hard links - preserve hard links
#-A, -- acls - preserve ACLs (implies -p)
#-X, -- xattrs - preserve extended attributes
# --progress - show progress during transfer
# --delete - delete extraneous files from dest dirs
# --ignore-errors - delete even if there are i/O errors
# --devices - preserve device files (super-user only)
# --specials - preserve special files
$RSYNC -vuahH --progress --delete --ignore-errors  $DATA_BACKUP $MOUNT_LOCATION/Data\ Backup/ > $MOUNT_LOCATION/Rsync\ Logs/"$DATE"_rsync.log

#Since we are running this script as root the log file that is created is owned by root, have to change it back to the user
$CHOWN $USER\: $MOUNT_LOCATION/Rsync\ Logs/
$CHOWN $USER\: $MOUNT_LOCATION/Rsync\ Logs/"$DATE"_rsync.log
$CHOWN $USER\: $MOUNT_LOCATION/Data\ Backup/

#Unmount devices
$UMOUNT $MOUNT_DEVICE

#Delete the folder created at the mount point after script completion
$RM -r $MOUNT_LOCATION 

$ECHO ""
$ECHO "~*~*~*~*~*~*~*~*~*~*~*~*~*"
$ECHO "All done!"

 

[TalRW]

I looked at your first post this might be what you need.

http://www.mikerubel.org/computers/rsync_snapshots/

It creates a rotating backup. While the page is a tiny bit out of date it still should work for you.

 

[splat]

i need to 1) get today's date, 2) get old file date, 3) if date older than 14 days [arbitrary amount of time picked], then rm 4) rinse, repeat for next file in dir.

 

[TalRW]

Ok well getting today's date is easy it is just this command

/bin/date "+%m_%d_%y"

and you can adjust the format to however you want it. I mean I use underscores but you could use -'s or whatever. To do the other stuff I can probably help I'll just have to do it later cause I gotta go to work right now.

 

[Shelnutt2]

A slightly different approach would be simply to as part of your script have to make a file that simply list the files that it backs up. After the file has 14 lines have it take the first line (being the name of the file that is 14 days old), have it delete the line and delete the file's name. Finally add the new file to the end of the list in a new line.

The file would simply list

backup_10132008
backup_10142008
backup_10152008
backup_10162008
backup_10172008
backup_10182008
backup_10192008
backup_10202008
backup_10212008
backup_10222008
backup_10123008
backup_10124008
backup_10125008
backup_10126008

You script would check the file and see how many lines are in it. I know in Java all I use to do was in a loop readline until it == null, and had x++ (x+1=x) so that when readline == null x = number of lines. Then if x = 14 have it readline #1 again and then delete that file. Then also delete that line from the file and add backup_10272008 to the end of the file.

 

[splat]

after searching around i found out i can use 'date -r $file' to get the modified date. so i can 'date -r $file +%s' to get the same format as my 'date +%s', then subtract to get the difference, and compare that to the number of seconds old i want the file to be (%s seconds since 1970-01-01 00:00:00 UTC).

just to give you guys a better idea of exactly what i have so far, here is my backup dir:

html_1223020961.tar.gz                sqlbackup_1223020961.tar.gz
html_1223107362.tar.gz                sqlbackup_1223107362.tar.gz
html_1223193759.tar.gz                sqlbackup_1223193759.tar.gz
html_1223280186.tar.gz                sqlbackup_1223280186.tar.gz
html_1223366573.tar.gz                sqlbackup_1223366573.tar.gz
html_1223452972.tar.gz                sqlbackup_1223452972.tar.gz
html_1223539392.tar.gz                sqlbackup_1223539392.tar.gz
html_1223625760.tar.gz                sqlbackup_1223625760.tar.gz
html_1223712160.tar.gz                sqlbackup_1223712160.tar.gz
html_1223798556.tar.gz                sqlbackup_1223798556.tar.gz
html_1223884966.tar.gz                sqlbackup_1223884966.tar.gz
html_1223971375.tar.gz                sqlbackup_1223971375.tar.gz

shelnutt, i like your idea too, just counting the number of files and making sure there are only X number of files. Now I just need to figure out how to get a file list and loop thru it.

 

[Shelnutt2]

shelnutt, i like your idea too, just counting the number of files and making sure there are only X number of files. Now I just need to figure out how to get a file list and loop thru it.

On other way I just thought of, was you don't even need a file, just ls the directory everytime, and you can count the number of lines ls returns. Then just delete the first line and the 15th line ls returns. I'm not positive on how to read lines with bash or I'd post up code.

 

[{PMS}fishy]

after searching around i found out i can use 'date -r $file' to get the modified date. so i can 'date -r $file +%s' to get the same format as my 'date +%s', then subtract to get the difference, and compare that to the number of seconds old i want the file to be (%s seconds since 1970-01-01 00:00:00 UTC).

Why not just use "find -mtime +15 ......."

find can do lots of things for you.

http://linuxcommand.org/man_pages/find1.html

You could also probably get logrotate to do it for you too.

 

[splat]

logrotate...interesting idea. I just set up a config file, we'll see what it does.

# nano -w /etc/logrotate.d/backup
/var/backups/html* {
weekly
rotate 0
}