I came across a real life-saver today…
All of my backup scripts are based on using RSYNC. This is because I am used to working between linux systems on which I always have root/sudo access and also have SSH. However I was recently tasked with a “fun” project that involved me needing to script the syncing of some directories between a linux server and a windows server running third-party SFTP software… that doesn’t allow SSH, SCP, etc.
Now, IMHO, the built-in SFTP program for Ubuntu has a syntax that is a royal PIA for any kind of complex or sophisticated file copy operations. Therefore, getting RSYNC like “mirroring” capability using plain SFTP was going to be a scripting nightmare (perhaps not, maybe I didn’t dig deep enough but that is certainly what it looked like to me…).
After much searching of the Google (thankfully Donald Trump hasn’t turned the internet off yet…) I came across a small program called LFTP referenced on several sites linked below in my “references” section.
First I installed LFTP which is in the ubuntu repositories (love Ubuntu…)
apt-get install lftp
Once installed, here is the scriptable syntax:
/usr/bin/lftp -c "open -p 9922 -u jdoe,passw0rd sftp://sftpsiteurl.com; mirror -c -e -R -L /source-folder /destination-folder/"
The above tells LFTP to execute two commands separated by a semi-colon. The first opens up a connection to the SFTP server using the username “jdoe” and password “passw0rd”. Notice you need to specify the protocol by pre-appending “sftp://” to the url. Also I am using a non default port with the -p flag of “9922”. If you are using key authentication just put some random text where the password should be.
The next command after the semi-colon says I want to “mirror” the two directories and the flags do the following:
-c, –continue – continue a mirror job if possible
-e, –delete delete files not present at remote site
-R, –reverse reverse mirror (put files, vs. a normal mirror used for pulling files I guess??)
-L, –dereference download symbolic links as files
One final note, I think RSYNC is still preferable as I believe (not sure but I think this is correct) it can transfer byte-level changes. So if you have large files and something small changes it will only transfer the changes and not the entire file again. I also don’t know how “intelligent” LFTP is… i.e. does it just check for file names or does it do more in-depth checks like making sure file-size matches or even MD5 sums? I don’t know. So I will continue to prefer RSYNC where I can use it but I will keep LFTP handy going forward when RSYNC isn’t an option.