Dealing with SFTP (push-pull bash scripts)

[Stratus_ss]

HowTo: Dealing with SFTP (push-pull bash scripts)

Well I took some time to deal with a Windows 2k8 SFTP server with linux/unix clients and I ended up writing a couple of scripts to help syncing the two.

Keep in mind that dealing with Windows as the sftp server forced me to use some less than conventional scripting methods. Anyways here they are incase they are of use to anyone (or even the Overclockix distro... who knows?)

Currently they are set to syncing xml files. As usual you will have to adjust the variables to suit your environment

The Pull Script

#!/bin/bash
#Sets the IP address of the ftp server
SERVER=192.168.56.50

#this grabs the list of files available on the server
FILE_LIST=`sftp $SERVER << EOF
cd sftp
ls
quit
EOF`
echo $FILE_LIST > temp_file_list

#this is used to remove everything except xml files
for DOWNLOAD_FILE in `cat temp_file_list |sort`
do
        echo $DOWNLOAD_FILE |grep .xml >> download
done
#ensure that the files are sorted properly
sort download > download_list

#check to see if each file in the server's list exists 
for SERVER_FILE in `cat download_list`
do
        if [ -f "$SERVER_FILE" ];
                then 
                        #if the file exists on the local machine
                        echo $SERVER_FILE Exists on local  machine
                else
                       #if the file does not exist, grab it 
                        sftp $SERVER << EOF
                        get sftp/$SERVER_FILE
                exit
EOF
        fi
done

#remove the temporary file lists
rm temp_file_list
rm download
rm download_list

The Push Script

#!/bin/bash

#Sets the IP address of the ftp server
SERVER=192.168.56.50

FOLDER=/root/sftp

#this grabs the list of files available on the server
FILE_LIST=`sftp $SERVER << EOF
cd sftp
ls
quit
EOF`
echo $FILE_LIST > temp_file_list

#this is used to remove everything except xml files
for DOWNLOAD_FILE in `cat temp_file_list`
do
        echo $DOWNLOAD_FILE |grep .xml >> download
done

#ensure that the files are sorted properly
sort download > download_list

#get the local list
ls |grep .xml | sort > local_list

# checks to see which files are on the server that are not
# on the local machine
DIFF=`diff local_list download_list | cut -c3- |grep .xml`

#check to see if each file in the server's list exists on the local maching
if [ `echo $DIFF` =="" ]

# If there are no differences in the folder assume there is nothing
# new to transfer
then
echo No New Files to transfer
else
for SERVER_FILE in $DIFF
        do
                        #if the file does not exist, grab it
                        sftp $SERVER << EOF
                        cd sftp
                        put $SERVER_FILE
                exit
fi 2>/dev/null

#remove the temporary file lists
rm local_list
rm temp_file_list
rm download
rm download_list

NOTE the push script expects to be run within the folder that you want to sync. If you are running it in a crontab you will have to adjust the 'ls' command to reflect your proper path

 

[Stratus_ss]

I re-wrote the put script a bit I figured I would append instead of get rid of the first one

#!/bin/bash
# This script takes an argument from the CLI
# If it does not receive one, it assumes you want to sync the servers
# If the file exists on the server, no files will be transfered

#Sets the IP address of the ftp server
SERVER=192.168.56.50

#this grabs the list of files available on the server
FILE_LIST=`sftp $SERVER << EOF
cd sftp
ls
quit
EOF`
echo $FILE_LIST > temp_file_list

#this is used to remove everything except xml files
for DOWNLOAD_FILE in `cat temp_file_list`
do
	echo $DOWNLOAD_FILE |grep .xml >> download
done

#ensure that the files are sorted properly
sort download > download_list

# checks to see if the file passed to the script exists on the server
EXISTS=`cat download_list| grep "$1"`
##########################################
# Begin Function
##########################################
sftpFunction () {
#check to see if each file in the server's list exists on the local machine
if [ `echo $DIFF` == "" ]
# If there are no differences in the folder assume there is nothing
# new to transfer
then
echo No New Files to transfer
else
for SERVER_FILE in $DIFF
	do
			date +%F-%T
	       		#if the file does not exist, push it
		        sftp $SERVER << EOF
			cd sftp
        		put $SERVER_FILE
   	  	exit
EOF
#		mv $SERVER_FILE /root
	done
fi 2>/dev/null

#remove the temporary file lists
rm local_list 2>/dev/null
rm temp_file_list
rm download
rm download_list
}
############################################
# End Function
#############################################
# Begin Main Program
##########################################
case "$1" in
	$EXISTS)
		echo "That file already exists of the SFTP server"
		echo "Exiting without further action"
	;;
	"" )
		echo "you did not enter a file to transfer"
		echo "assuming syncronization"
		ls |grep .xml | sort > local_list
		# checks to see which files are on the server that are n$
		# on the local machine
		DIFF=`diff local_list download_list | cut -c3- |grep .xml`
		#begin the file syncing
		sftpFunction
	;;
	*)
		echo "Copying " $1 " to the SFTP server"
		DIFF="$1"
		# Copy over the selected file
		sftpFunction
	;;
esac

I also implemented file sorting by date (for things such as logs) for the sftp_get script

#!/bin/bash
# sftp_get.sh  for the purpose of pulling down files off the ftp
# server. Logging is handled by pipping the stout to a file

#Sets the IP address of the ftp server
SERVER=192.168.56.50

#This is the pattern that will be searched for in the file names
PATTERN=PPF_XRS_IN

#this grabs the list of files available on the server
#It runs the commands on from within the sftp shell
FILE_LIST=`sftp $SERVER << EOF
cd sftp
ls
quit
EOF`
echo $FILE_LIST > temp_file_list

#put each entry of the file into a variable
LIST=`cat temp_file_list |sort`

#this is used to remove everything except for the specified pattern

for DOWNLOAD_FILE in $LIST
do
	echo $DOWNLOAD_FILE |grep $PATTERN >> download
done

#ensure that the files are sorted properly
sort download > download_list

#check to see if each file in the server's list exists on the local machine
for SERVER_FILE in `cat download_list |grep PPF_XRS_`
do
	if [ -f /root/sftp/"$SERVER_FILE" ];
		then 
		        #if the file exists on the local machine move to the next file
	       		echo $SERVER_FILE Exists on local  machine
		else
	       		#if the file does not exist, grab it and put it in default direct$
		        sftp $SERVER << EOF
        		get sftp/$SERVER_FILE
      	  	exit
EOF
###############################################################
# This is to handle parsing file dates from the format *_*_*_mmddyyyy
# It will eventually sort the files into the correct folders
##############################################################
FOLDER=`ls $SERVER_FILE | awk -F "_" '{print $4}' |cut -c1,2`
case "$FOLDER" in
	01)
		echo "moving " $SERVER_FILE " to /home/jan"
	;;
	02)
		echo "moving " $SERVER_FILE " to /home/feb"
	;;
	03)
		echo "moving " $SERVER_FILE " to /home/mar"
	;;
	04)
		echo "moving " $SERVER_FILE " to /home/apr"
	;;
	05)
		echo "moving " $SERVER_FILE " to /home/may"
	;;
	06)
		echo "moving " $SERVER_FILE " to /home/jun"
	;;
	07)
		echo "moving " $SERVER_FILE " to /home/jul"
	;;
	08)
		echo "moving " $SERVER_FILE " to /home/aug"
	;;
	09)
		echo "moving " $SERVER_FILE " to /home/sept"
	;;
	10)
		echo "moving " $SERVER_FILE " to /home/oct"
	;;
	11)	
		echo "moving " $SERVER_FILE " to /home/nov"
	;;
	12)
		echo "moving " $SERVER_FILE " to /home/dec"
	;;
esac


#remove the temporary file lists
rm temp_file_list
rm download
rm download_list
	fi
done

 

[dropadrop]

You could actually do this with less code like this

#put each entry of the file into a variable
LIST=`cat temp_file_list`

#this is used to remove everything except xml files
for DOWNLOAD_FILE in $LIST
do
	echo $DOWNLOAD_FILE |grep .xml >> download
done

for DOWNLOAD_FILE in `cat temp_file_list `; do
	echo $DOWNLOAD_FILE |grep .xml >> download
done

edit:

One thing the script does not take into consideration is that a files contents would have changed... Don't know how easy it would be to do something like take the md5 hash of each file and compare if they match?

 

[Stratus_ss]

You could actually do this with less code like this

for DOWNLOAD_FILE in `cat temp_file_list `; do
	echo $DOWNLOAD_FILE |grep .xml >> download
done

edit:

One thing the script does not take into consideration is that a files contents would have changed... Don't know how easy it would be to do something like take the md5 hash of each file and compare if they match?

I dont want it to take that into consideration. My instructions were to not step on the old file. Specifically in our case the time stamp is being appended as well as the date therefore contents are really unlikely to have changed.

As for the loop, good catch, had caught that myself but I guess the changes never made it to the edits here on OCForums
*Edited to reflect dropadrop's catch*

 

[dropadrop]

Oh, in my example (and yours) the output should be piped to download_list, not download since download_list is what will be expected.

 

[HankB]

I'm curious if you looked at rsync or unison to sync files. I use rsync over ssh between two Linux hosts. I suppose I'm presuming that an ssh server exists for Windows. (Google says OpenSSH which I use on Linux is available for Windows.)

rsync has a myriad of options to tailor operations including '-backup' to cause it to backup files that would otherwise be replaced. I suspect that the entire script could be replaced by a single command.

 

[Stratus_ss]

The problem is that I am dealing with a client site and thus i was confined to dealing with their restrictions. Also, rsync from windows to linux doesnt work very well in general especially over the internet. In this solution nothing needs to be added to network configurations in order to have things work

 

[${freak}]

this will also break for any filename with a space in it. for foo in $(command) is generally not a good idea, it's much better to use a while read loop.

while read -r line; do
  [[ $line = *.xml ]] && printf "%s\n" "$line" >>"output file"
done < "input file"

 

[dropadrop]

The problem is that I am dealing with a client site and thus i was confined to dealing with their restrictions. Also, rsync from windows to linux doesnt work very well in general especially over the internet. In this solution nothing needs to be added to network configurations in order to have things work

We have a windows backup script for our workstations that uses cygwin and rsync to back up to a linux server. It's worked fine for the last 4-5 years... I did not write it and actually have not even read it (since I don't run Windows) but it's definitely doable.

It could even be taken one step further and rsnapshot could be used instead, but if the backups are going to tape (which are rotated) then that would just add unneeded complexity.