rsync
rsync is a utility for transferring and synchronizing files between a computer and a storage drive and across networked computers by comparing the modification times and sizes of files.[8] It is commonly found on Unix-like operating systems and is under the GPL-3.0-or-later license.[4][5][9][10][11][12]
Original author(s)
Andrew Tridgell, Paul Mackerras[1]
Wayne Davison[2]
June 19, 1996[1]
/ 6 April 2024
2007: GPL-3.0-or-later[a][4][5][6]
2007: GPL-3.0-only[b]
2007: GPL-2.0-only[c]
1996: GPL-2.0-or-later[d][7]
rsync is written in C as a single threaded application.[13] The rsync algorithm is a type of delta encoding, and is used for minimizing network usage. Zstandard, LZ4, or Zlib may be used for additional data compression,[8] and SSH or stunnel can be used for security.
rsync is typically used for synchronizing files and directories between two different systems. For example, if the command rsync local-file user@remote-host:remote-file is run, rsync will use SSH to connect as user to remote-host.[14] Once connected, it will invoke the remote host's rsync and then the two programs will determine what parts of the local file need to be transferred so that the remote file matches the local one. One application of rsync is the synchronization of software repositories on mirror sites used by package management systems.[15][16]
rsync can also operate in a daemon mode (rsyncd), serving and receiving files in the native rsync protocol (using the rsync:// syntax).
History[edit]
Andrew Tridgell and Paul Mackerras wrote the original rsync, which was first announced on 19 June 1996.[1] It is similar in function and invocation to rdist (rdist -c), created by Ralph Campbell in 1983 and released as part of 4.3BSD.[17] Tridgell discusses the design, implementation, and performance of rsync in chapters 3 through 5 of his 1999 Ph.D. thesis.[18] As of 2023, it is maintained by Wayne Davison.[2]
Because of the flexibility, speed, and scriptability of rsync, it has become a standard Linux utility, included in all popular Linux distributions. It has been ported to Windows (via Cygwin, Grsync, or SFU[19]), FreeBSD,[20] NetBSD,[21] OpenBSD,[22] and macOS.
Connection[edit]
An rsync process operates by communicating with another rsync process, a sender and a receiver. At startup, an rsync client connects to a peer process. If the transfer is local (that is, between file systems mounted on the same host) the peer can be created with fork, after setting up suitable pipes for the connection. If a remote host is involved, rsync starts a process to handle the connection, typically Secure Shell. Upon connection, a command is issued to start an rsync process on the remote host, which uses the connection thus established. As an alternative, if the remote host runs an rsync daemon, rsync clients can connect by opening a socket on TCP port 873, possibly using a proxy.[29]
Rsync has numerous command line options and configuration files to specify alternative shells, options, commands, possibly with full path, and port numbers. Besides using remote shells, tunnelling can be used to have remote ports appear as local on the server where an rsync daemon runs. Those possibilities allow adjusting security levels to the state of the art, while a naive rsync daemon can be enough for a local network.
One solution is the --dry-run option, which allows users to validate their command-line arguments and to simulate what would happen when copying the data without actually making any changes or transferring any data.
Variations[edit]
The rdiff utility uses the rsync algorithm to generate delta files with the difference from file A to file B (like the utility diff, but in a different delta format). The delta file can then be applied to file A, turning it into file B (similar to the patch utility). rdiff works well with binary files.
The rdiff-backup script maintains a backup mirror of a file or directory either locally or remotely over the network on another server. rdiff-backup stores incremental rdiff deltas with the backup, with which it is possible to recreate any backup point.[33]
The librsync library used by rdiff is an independent implementation of the rsync algorithm. It does not use the rsync network protocol and does not share any code with the rsync application.[34] It is used by Dropbox, rdiff-backup, duplicity, and other utilities.[34]
The acrosync library is an independent, cross-platform implementation of the rsync network protocol.[35] Unlike librsync, it is wire-compatible with rsync (protocol version 29 or 30). It is released under the Reciprocal Public License and used by the commercial rsync software Acrosync.[36]
The duplicity backup software written in python allows for incremental backups with simple storage backend services like local file system, sftp, Amazon S3 and many others. It utilizes librsync to generate delta data against signatures of the previous file versions, encrypting them using gpg, and storing them on the backend. For performance reasons a local archive-dir is used to cache backup chain signatures, but can be re-downloaded from the backend if needed.
As of macOS 10.5 and later, there is a special -E or --extended-attributes switch which allows retaining much of the HFS+ file metadata when syncing between two machines supporting this feature. This is achieved by transmitting the Resource Fork along with the Data Fork.[37]
zsync is an rsync-like tool optimized for many downloads per file version. zsync is used by Linux distributions such as Ubuntu[38] for distributing fast changing beta ISO image files. zsync uses the HTTP protocol and .zsync files with pre-calculated rolling hash to minimize server load yet permit diff transfer for network optimization.[39]
Rclone is an open-source tool inspired by rsync that focuses on cloud and other high latency storage. It supports more than 50 different providers and provides an rsync-like interface for cloud storage.[40] However, Rclone does not support rolling checksums for partial file syncing (binary diffs) because cloud storage providers do not usually offer the feature and Rclone avoids storing additional metadata.[41]