How to rsync multiple
From arrizza.org wikiJump to navigation Jump to search
|Previous ⇦ Main Page||Main Page ⇫ Up||Main Page ⇨ Next|
At times it is necessary to transfer multiple files or directories using rsync over an ssh session. Normally each rsync session establishes a new ssh connection which can take time.
This shows how to establish a single ssh master session and then use that ssh session in multiple rsync calls.
This is outlined in several stack exchange discussions and other sites, but none give specific instructions and explanations of what's going on.
- this works under Ubuntu. It has not been tested on cygwin, Mac or other operating systems.
- tested with OpenSSL version 1.1.1g, it may or may not work with other or older versions.
#! /bin/bash # 1) replace "your_ip_address" remote=your_ip_address # 2) create config for common ssh session # Note this assumes User is root, change as needed echo "HostName $remote" > ssh_config echo "User root" >> ssh_config echo "ControlMaster auto" >> ssh_config echo "ControlPath ~/.ssh/%C" >> ssh_config master_ssh='ssh -F ssh_config' # 3) start common ssh session $master_ssh -MNf $remote opts="-h -stats" echo retrieving filee from $remote mkdir -p local_directory # 4) transfer using rsync rsync -e "$master_ssh" $opts $remote:/etc/some_file1 local_directory/ rsync -e "$master_ssh" $opts $remote:/etc/some_file2 local_directory/ rsync -e "$master_ssh" $opts $remote:/etc/some_file3 local_directory/ # 5) close the common ssh session $master_ssh -O exit $remote
1) Use your IP address or URL to your host/server here
2) This sets up a local file called ssh_config that ssh uses to set up the master session. Note I've set up ssh to login to root account without a password, so I don't need to specify one here.
- HostName $remote specifies the host/server address
- "User root" specifies the user id to use for the ssh connection
- ControlMaster auto tells ssh to set up a master session
- ControlPath etc. tells ssh where to save the master session info so multiple processes can find the same one
- master_ssh='ssh -F ssh_config' sets up a variable for convenience
3) start the common ssh session. Note this is done before any calls to rsync.
- -M sets ssh in Master mode
- -N do not execute a remote command at this time. Subsequent calls to rsync will do those commands
- -f go into background before executing any commands. Again subsequent calls to rsync will be those commands.
4) transfer files using rsync.
- -e "$master_ssh" tells rsync to use the ssh master session. The details and configuration info needed by ssh is in the ssh_config file set up in step 2)
5) when all transfers are complete, close the ssh session
- -O exit closes the master session