Command-line Basics: Downloading Files with cURL

joshtronic

Client URL, or simple cURL is a library and command-line utility for transferring data between systems. It supports a myriad of different protocols and tends to be installed by default on many Unix-like operating systems. Because of it’s general availability, it is a great choice for when you need to quickly download a file to your local system.

Getting started

To follow along at home, you will need to have the curl utility installed. As mentioned, it’s pretty standard issue on Unix-like operating systems such as Linux and macOS.

If you don’t have the curl command available, please consult your favorite package manager. Even if it’s not installed, your package manager more than likely has it available to install.

The commands we will be issuing will be pretty safe as they will be downloading files from the Internet and non-destructive in nature. Obviously, downloading files off of the Internet can be sketchy, so be sure you are downloading from reputable sources.

Also, if you plan to run any scripts you have downloaded, it’s good practice to check their contents before making them executable and running them. A quick cat and looking over the code is often sufficient depending on the size of the file and your knowledge of the code you’re reviewing.

Recommended Node.js video course

Fetching remote files

Out of the box, without any command-line arguments, the curl command will fetch a file and display it’s contents to the standard output.

Let’s give it a try by downloading the robots.txt file from your favorite development blog:

$ curl https://alligator.io/robots.txt
User-agent: *
Sitemap: https://alligator.io/sitemap.xml

Not much to it! Give curl a URL and it will fetch the resource and display it’s contents.

Saving remote files

Fetching a file and display it’s contents is all well and good, but what if you want to actually save the file to your system?

To save the remote file to your local system, with the same filename as the server you’re downloading from, add the --remote-name argument, or simply, -O:

$ curl -O https://alligator.io/robots.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    56  100    56    0     0    251      0 --:--:-- --:--:-- --:--:--   251

Instead of displaying the contents of the file, curl displays a nice little text-based progress meter and saves the file to the same name as the remote file’s name. You can check on things with the cat command:

$ cat robots.txt
User-agent: *
Sitemap: https://alligator.io/sitemap.xml

Saving remote files to a specific filename

What if you already had a local file with the same name as the file on the remote server?

Unless you are okay with overwriting your local file of the same name, you will want to add the -o or --output argument followed by the name of the local file you’d like to save the contents to:

$ curl -o gator-bots.txt https://alligator.io/robots.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    56  100    56    0     0    221      0 --:--:-- --:--:-- --:--:--   221

Which downloads the remote robots.txt file to the locally named gator-bots.txt file:

$ cat gator-bots.txt
User-agent: *
Sitemap: https://alligator.io/sitemap.xml

Following redirects

Thus far all of the examples have included fully qualified URLs that include the https:// protocol. If you happened to try to fetch the robots.txt file and only specified alligator.io, you would be presented with an error about a redirect as we redirect requests from http:// to https:// (as one should):

$ curl alligator.io/robots.txt
Redirecting to https://alligator.io/robots.txt

No big deal though. curl has a flag you can pass in. The --location or -L argument tells curl to redo the request to the new location when a 3xx response code is encountered:

$ curl -L alligator.io/robots.txt
User-agent: *
Sitemap: https://alligator.io/sitemap.xml

Of course if so desired, you can combine the -L argument with some of the aforementioned arguments to download the file to your local system.

Conclusion

curl is a great utility for quickly and easily downloading files from a remote system. While it’s similar to rsync in functionality, I find it to be a bit easier to work with since there’s less to remember in terms of arguments for some of the more common / basic tasks.

Like most of the simple yet powerful command-line utilities we discuss, this post really only covers the tip of the iceberg. With support for many different protocols and the added upload capabilities, curl has a ton to offer.

Ready to learn more? Check out the manual page for curl by running man curl.

  Tweet It

🕵 Search Results

🔎 Searching...

Sponsored by #native_company# — Learn More
#native_title# #native_desc#
#native_cta#