Need to transfer files between servers? Sure, you could boot up your favorite [S]FTP client. Maybe you’re familiar with
scp so you reach for that. While both are valid solutions to the problem, there’s still one transfer program that leads the pack in terms of speed and that’s
Most of the speed comes from the “quick check” algorithm that is used by default. Since it checks for differences, it won’t move files that haven’t changed. More so, files that it does detect as changed will only send the differences, not the entire file.
You may not necessarily notice this speed boost on your initial transfer. But on subsequent transfers, the diffing will limit how many files are transferred. If you’re working with gigs and gigs of data across a ton of files, this will be an absolute life saver.
To get started, it’s highly recommended that you have a remote server available to send files to and from.
As mentioned, this guide will cover using the
rsync command which is pretty standard issue these days on macOS and Linux. If you don’t happen to have it installed, please consult your favorite package manager to get it installed.
In addition to having
rsync installed on your local machine, you will also need to have it installed on your remote machine as well. Obviously, if it’s not installed, you’ll need to do so to be able to follow along at home.
— Sorry to interrupt this program! 📺
Given the current situation with COVID-19, times are hard for most of us, but if you're stuck at home you can perhaps use that extra time to work on your React skills. Wes put all of his courses at 50% off to help out with the current challenge we're facing.
Plus, these are affiliate links, so if you purchase a course you help Alligator.io continue to exist at the same time! 🙏
A small disclaimer
Out of the box,
rsync will try to synchronize the destination directory to look exactly like the source one. This means that there is a destructive element that you need to be aware of.
When synchronizing two directories, if the destination has a file that the source does not, the file will be removed.
This very well may be desirable, but in scenarios where you’d like to preserve the files in the destination as well as copy over new files from the source, you’ll want to pass in the
This argument is an alias for a handful of other arguments (7 or so) that implies recursion as well as forcing more file preservation. It does omit the preservation of hard links, but there are speed implications to calculating those particular differences.
Of course there’s an argument for that,
-H will, while slower, does handle the preservation of hard links.
Between you and me, I do about 90% of my
rsyncing with the arguments
-avz. This allows me to copy directories, thanks to the recursion and gives me the safety of the aforementioned preservation.
--verbose adds a bit more output and thus clarity while
--compress adds an additional speed boost to transfers by compressing the files.
The following examples will use these arguments.
Copying files locally
It’s worth mentioning that
rsync allows you transfer files locally as well as between remote machines.
Because of how
rsync handles differences, only moving what’s changed, it can be a huge time saver in comparison to using
cp, while will copy all of the files over.
To transfer a file between 2 directories, you can do this:
$ rsync -avz /path/to/source/file.txt /path/to/destination # Or to also rename it: $ rsync -avz /path/to/source/file.txt /path/to/destination/file.log
If you omit the name of a specific file, you can copy over an entire directory instead:
$ rsync -avz /path/to/source/directory /path/to/destination/directory
Uploading files from your local machine to a remote server
Similar to the example when working locally, the order of things stays the same.
The destination this time will be that of a remote system. You can specify this as just the host name or IP address or you can combine that with a user account if it’s different than that of your local user.
The full format goes
@ symbol, the
: and then the path on the remote server we’re transferring files to.
Put it all together and it looks like this:
$ rsync -avz /path/to/source username@hostname:/path/to/destination
Of course, the user you’re referencing on the server will need to have the correct permissions to write to the directory.
Downloading files from a remote server to your local machine
To pull files from a remote server down to our local machine, all we need to do is flip the source and destination in our arguments list from our previous example:
$ rsync -avz username@hostname:/path/to/source /path/to/destination
As mentioned, the aforementioned examples make up the bulk of my
rsync usage both locally as well as between remote servers.
Like most of the simple, yet powerful command-line tools,
rsync has a ton more options that can help get you out of even the stickiest file transferring jams.
Ready to learn more? Kick your feet up, run
man rsync, and spend some time reading all of the ins and outs of