How to rsync multiple

From arrizza.org wiki

Jump to navigation Jump to search


Previous ⇦ Main Page Main Page ⇫ Up Main Page ⇨ Next

At times it is necessary to transfer multiple files or directories using rsync over an ssh session. Normally each rsync session establishes a new ssh connection which can take time.

This shows how to establish a single ssh master session and then use that ssh session in multiple rsync calls.

This is outlined in several stack exchange discussions and other sites, but none give specific instructions and explanations of what's going on.

Notes:

  • this works under Ubuntu. It has not been tested on cygwin, Mac or other operating systems.
  • tested with OpenSSL version 1.1.1g, it may or may not work with other or older versions.
#! /bin/bash

# 1) replace "your_ip_address"
remote=your_ip_address

# 2) create config for common ssh session
# Note this assumes User is root, change as needed
echo "HostName $remote" > ssh_config
echo "User root" >> ssh_config
echo "ControlMaster auto" >> ssh_config
echo "ControlPath ~/.ssh/%C" >> ssh_config
master_ssh='ssh -F ssh_config'

# 3) start common ssh session
$master_ssh -MNf $remote

opts="-h -stats"
echo retrieving filee from $remote
mkdir -p local_directory

# 4) transfer using rsync
rsync -e "$master_ssh" $opts $remote:/etc/some_file1  local_directory/

rsync -e "$master_ssh" $opts $remote:/etc/some_file2    local_directory/

rsync -e "$master_ssh" $opts $remote:/etc/some_file3    local_directory/

# 5) close the common ssh session
$master_ssh -O exit $remote

1) Use your IP address or URL to your host/server here

2) This sets up a local file called ssh_config that ssh uses to set up the master session. Note I've set up ssh to login to root account without a password, so I don't need to specify one here.

  • HostName $remote specifies the host/server address
  • "User root" specifies the user id to use for the ssh connection
  • ControlMaster auto tells ssh to set up a master session
  • ControlPath etc. tells ssh where to save the master session info so multiple processes can find the same one
  • master_ssh='ssh -F ssh_config' sets up a variable for convenience


3) start the common ssh session. Note this is done before any calls to rsync.

  • -M sets ssh in Master mode
  • -N do not execute a remote command at this time. Subsequent calls to rsync will do those commands
  • -f go into background before executing any commands. Again subsequent calls to rsync will be those commands.

4) transfer files using rsync.

  • -e "$master_ssh" tells rsync to use the ssh master session. The details and configuration info needed by ssh is in the ssh_config file set up in step 2)

5) when all transfers are complete, close the ssh session

  • -O exit closes the master session
Personal tools