Imagine you are trying to copy a large file to a remote server and the operation fails due to network connectivity. So you restart the process from scratch, but face the same issue again and again. rsync will come to your rescue here, as it uses the delta transfer algorithm to perform the copy operation effectively. In this article, we will see how to use rsync to create a backup of a system.
rsync stands for ‘remote sync’. This protocol has been designed and implemented by Andrew Tridgell, and can be used to synchronise files from local or remote systems. It uses the delta transfer algorithm to copy only the modified portion from the source to the destination file. This saves network bandwidth and effectively reduces the time taken to complete the operation.
Salient features and installation
a) Efficiency: rsync is efficient as it transfers only ‘delta’ instead of copying the whole file.
b) Performance: It performs better than scp because of the efficient delta transfer algorithm.
c) Security: It allows encryption of data using the SSH protocol during transfer.
d) Ease of use: It is simple to use. Also, no special privileges are required to use it.
To install rsync on RPM based GNU/Linux distros, execute the command given below with root privileges:
# yum install rsync
Next, execute the following command as a non-root user to verify its installation:
$ rsync –version
If you see an output similar to the following, then rsync has been installed successfully:
rsync version 3.1.3 protocol version 31 Copyright (C) 1996-2018 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/
Primitive operations
The syntax of rsync command is as follows:
rsync [options] <source> <destination>
In rsync, the source and destination can be any combination of local or remote systems. Please note that the syntax will vary when a remote system comes into the picture.
a) Sync a single file on the local server: Let us use rsync to synchronise a 1GB file on a local server:
[root]# ls -lrth total 1.0G -rw-r--r--. 1 root root 1.0G Dec 6 17:46 data.iso [root]# rsync -vh --no-W data.iso /opt/backup/
When the above command is executed, it generates the following output:
data.iso sent 1.07G bytes received 35 bytes 126.35M bytes/sec # This indicated entire file is copied total size is 1.07G speedup is 1.00
The above output indicates that the entire file was copied, as this copying was done for the first time. Now let us try to copy the same file:
[root]# rsync -vh --no-W data.iso /opt/backup/
When the above command is executed, it generates the following output:
data.iso sent 131.16K bytes received 229.42K bytes 24.87K bytes/sec # This indicated only 131.16K bytes are copied total size is 1.07G speedup is 2,977.76
If you compare the above output with the previous output, then you will observe that this time only 131.16K bytes of data was transferred.
Now, let us demystify the command. In the above command:
- The -v flag is used to show verbose output.
- The -h flag is used to show output in human readable format, e.g., in KB, MB, GB, and so on.
- The –no-W flag is used to enable the delta transfer algorithm.
b) Enable compression: rsync supports various compression configurations. We can enable the default compression level using the -z option. It will compress file data during the transfer. We can use it as follows:
[root]# rsync -zvh --no-W data.iso /opt/backup/
We can also set the compression level explicitly using the –compress-level option. This option accepts compression level as a digit from 0-9, where 1 signifies the least compressed and 9 the most compressed. We can use this option as follows:
[root]# rsync -zvh --no-W --compress-level=5 data.iso /opt/backup/
In addition, we can disable compression for certain file types. We can use the –skip-compress option for this. This option accepts the suffix list for which compression is disabled. We can use this option as follows:
[root]# rsync -zvh --no-W --skip-compress=7z/bz2/gz data.iso /opt/backup/
c) Show progress: The rsync operation will take some time depending upon the file size. In such cases, showing progress information is very helpful. It supports the —progress option, which shows progress during the file transfer. We can use this option as follows:
[root]# rsync -rvh data/data-1.iso /opt/backup/ --progress
When the above command is executed, it generates the following output:
sending incremental file list data-1.iso 1.07G 100% 120.27MB/s 0:00:08 (xfr#1, to-chk=0/1) sent 1.07G bytes received 35 bytes 126.35M bytes/sec total size is 1.07G speedup is 1.00
d) Recursive mode: So far we have synchronised a single file. However, rsync supports the recursive mode, which allows synchronisation of all files from the directory. We can use the -r option for this as follows:
[root]# rsync -rvh data/ /opt/backup/
When the above command is executed, it generates the following output:
sending incremental file list data-1.iso data-2.iso data-3.iso sent 3.22G bytes received 73 bytes 121.59M bytes/sec total size is 3.22G speedup is 1.00
e) Preserve timestamp: Let us compare timestamps of the source and the destination files:
[root]# ls -lrth total 1.0G -rw-r--r--. 1 root root 1.0G Dec 6 17:46 data.iso [root]# ls -lrth /opt/backup/data.iso -rw-r--r--. 1 root root 1.0G Dec 6 18:21 /opt/backup/data.iso
If you observe carefully, you will notice that rsync is not preserving timestamp during the copy operation. However, we can do so by using the -a option as follows:
[root]# rsync -avh --no-W data.iso /opt/backup/
Now let us compare timestamps of the source and the destination files again:
[root]# ls -lrth total 1.0G -rw-r--r--. 1 root root 1.0G Dec 6 17:46 data.iso [root]# ls -lrth /opt/backup/data.iso -rw-r--r--. 1 root root 1.0G Dec 6 17:46 /opt/backup/data.iso
This time, rsync has preserved the timestamps. The -a option enables the archive mode. It performs the following things:
- Enables recursive mode
- Preserves symbolic links
- Preserves permissions
- Preserves timestamp
- Preserves owner and group
f) Synchronise directory structure: Sometimes we need to create only a directory structure at the destination, which is the same as the source. rsync provides the -d option for this. We can use this option as follows.
Let us create a directory structure with some files:
[root]# mkdir -p src/java src/cpp src/php [root]# touch src/java/Main.java [root]# touch src/cpp/main.cpp [root]# touch src/php/main.php
Now verify that the directory structure is hierarchical:
[root]# tree src/ src/ ├── cpp │ └── main.cpp ├── java │ └── Main.java └── php └── main.php 3 directories, 3 files
Let us create a directory structure at the destination using the -d option as follows:
[root]# rsync -dv src/ /opt/backup/
Now let us check the directory structure at the destination:
[root]# tree /opt/backup/ opt/backup/ ├── cpp ├── java └── php 3 directories, 0 files
Remote operations
So far we ran all commands on the local machine only. In this section, we will use rsync with a remote system. We can use the syntax given below when the destination is a remote system:
rsync [options] <source> <user>@<remote-ip>:<destination>
Similarly, we can use the syntax given below when the source is remote:
rsync [options] <user>@<remote-ip>:<destination> <source>
Now, let us get our hands dirty with some practical examples.
a) Dry run: It is always a good practice to do a trial run before performing actual operations. rsync has a -n option, which helps to do so. We can use this option as follows:
[root]# rsync -nazv data/ root@172.16.208.11:/opt/backup/ root@172.16.208.11’s password:
When you execute the above command, it will generate the following output:
sending incremental file list created directory /opt/backup ./ data-1.iso data-2.iso data-3.iso sent 133 bytes received 62 bytes 43.33 bytes/sec total size is 3,221,225,472 speedup is 16,519,104.98 (DRY RUN)
b) Synchronise files from the local to the remote server: In the previous example we verified server connectivity using a dry run. Now let us perform an actual file transfer without the -n option. We can do this as follows:
[root]# rsync -azv data/ root@172.16.208.11:/opt/backup/ root@172.16.208.11’s password:
When you execute the above command, it will generate the following output:
sending incremental file list created directory /opt/backup ./ data-1.iso data-2.iso data-3.iso sent 3,223,059,976 bytes received 110 bytes 17,374,986.99 bytes/sec total size is 3,221,225,472 speedup is 1.00
c) Synchronise files from the remote to the local server: To synchronise files from the remote to the local machine, we need to specify a remote path in the source and local path in the destination, as shown below:
[root]# rsync -azv root@172.16.208.11:/opt/backup/ /opt/backup/ root@172.16.208.11’s password:
When you execute the above command, it will generate the following output:
receiving incremental file list created directory /opt/backup ./ data-1.iso data-2.iso data-3.iso sent 84 bytes received 3,223,059,976 bytes 17,374,986.85 bytes/sec total size is 3,221,225,472 speedup is 1.00
d) Use remote shell during transfer: rsync allows us to specify remote shell during file transfer. For example, we can use SSH for secure file transfer from the source to the destination. We can use the -e option for this, as follows:
[root]# rsync -azv -e test data/data-2.iso root@172.16.208.11:/opt/backup root@172.16.208.11’s password:
When you execute the above command, it will generate the following output:
sending incremental file list sent 51 bytes received 12 bytes 18.00 bytes/sec total size is 1,073,741,824 speedup is 17,043,521.02
e) Resume file transfer: Let us take a real world situation —you are copying a large file and the operation failed due to a network connectivity issue. One of the options is to start the copy operation all over again, but this is not an efficient method. Here, we can take advantage of rsync’s delta transfer algorithm. We can use the -P option, as follows.
First, partially copy the file from the source to the destination, as destination:
[root]# scp data/data-2.iso root@172.16.208.11:/opt/backup/ root@172.16.208.11’s password: data-2.iso 73% 752MB 64.8MB/s 00:04 ETA^C # Press ctrl + c to simulate failure scenario
Now let us resume the file transfer using the -P option:
[root]# rsync -azv -P --progress data/data-2.iso root@172.16.208.11:/opt/backup/
When you execute the above command, it will generate the following output:
sending incremental file list data-2.iso 1,073,741,824 100% 23.83MB/s 0:00:42 (xfr#1, to-chk=0/1) sent 222,542,063 bytes received 204,307 bytes 4,499,926.67 bytes/sec total size is 1,073,741,824 speedup is 4.82
In this beginner-friendly article we have discussed some practical examples for using rsync. However, rsync has many more capabilities. You can use this article as a starting point, and implement it in many different and useful ways.
