CURL — Everything You Need To Know
curl is a command-line utility for transferring data from or to a server designed to work without user interaction. With curl, you can download or upload data using various supported protocols including HTTP, HTTPS, SCP, SFTP, and FTP. curl provides a number of options allowing you to resume transfers, limit the bandwidth, proxy support, user authentication, and much more.
In this tutorial, we will show you how to use the curl tool through practical examples and detailed explanations of the most common curl options.
Basic Usage
GET Request
The default HTTP requests made by curl are GET requests.. In order to retrieve the example.com homepage, you would run:
{kiran@parrot} ~$ curl example.com
<!doctype html>
<html>
<head>
<title>Example Domain</title>
<meta charset="utf-8" />
<meta http-equiv="Content-type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<style type="text/css">
body {
<SNIP>If no protocol is specified, curl tries to guess the protocol you want to use, and it will default to HTTP.
GET Request Verbose
It’s possible to view the raw HTTP requests using the “-v” switch to increase verbosity.
{kiran@parrot} ~$ curl -v example.com
* Trying 93.184.216.34:80...
* TCP_NODELAY set
* Connected to example.com (93.184.216.34) port 80 (#0)
> GET / HTTP/1.1
> Host: example.com
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Age: 339480
< Cache-Control: max-age=604800
< Content-Type: text/html; charset=UTF-8
<SNIP>Saving the output to a file
To save the result of the curl command, use either the -o or -O option.
Lowercase -o saves the file with a predefined filename, which in the example below is my_file.js.
curl -o my_file.js https://test-site/vue.jsUppercase -O saves the file with its original filename.
curl -O https://test-site/vue.jsDownload multiple files at once
curl -O https://test-site/file1.zip -O https://test-site/file2.zipResume a download
You can continue/resume a previous file transfer at the given offset by using the -C - option. This is useful if your connection drops during the download of a large file, and instead of starting the download from scratch, you can continue the previous one.
curl -C - -O http://releases.ubuntu.com/18.04/ubuntu-18.04-live-server-amd64.isoFollow redirects
By default, curl doesn’t follow the HTTP Location headers.
If you try to retrieve the non-www version of google.com, you will notice that instead of getting the source of the page you’ll be redirected to the www version.
{kiran@parrot} ~$ curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
The -L option instructs curl to follow any redirect until it reaches the final destination.
curl -L google.comProgress Meter
curl normally displays a progress meter during operations, indicating the amount of transferred data, transfer speeds and estimated time left, etc. The progress meter displays number of bytes and the speeds are in bytes per second.
{kiran@parrot} ~$ curl -O localhost:8000/shell.php
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5494 100 5494 0 0 2682k 0 --:--:-- --:--:-- --:--:-- 5365k
If you prefer a progress “bar” instead of the regular meter, -#, –progress-bar is your friend.
{kiran@parrot} ~$ curl -O localhost:8000/shell.php -#
####################################################################### 100.0%
You can also completely get rid off the progress-meter with the -s, –silent option. Remember this mode also turns off the error messages.
curl -O localhost:8000/shell.php --silentHTTP Methods
-X, --request flag specifies a custom request method to use when communicating with the HTTP server. The specified request method will be used instead of the method otherwise used (which defaults to GET).
Passing data using POST
Let’s try to login to the page from the POST section using cURL. The default “Content-Type” used by cURL is “application/x-www-form-urlencoded“, the data for which can be passed using the “-d” flag.
curl -d 'username=admin&password=password' -L http://example.com/login.phpcURL automatically sends a POST request when the “-d” flag is used.
File Upload using PUT
curl -X PUT -d @test.txt http://inlanefreight.com/test.txt -vvThe “@” symbol is used by curl to read the file and send it’s contents as the data.
Deleting a remote file using DELETE
curl -X DELETE http://inlanefreight.com/test.txt -vvHTTP Header Options
Fetch the headers only
Use the -I option to fetch only the HTTP headers of the specified resource.
{kiran@parrot} ~$ curl -I example.com
HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 490767
Cache-Control: max-age=604800
Content-Type: text/html; charset=UTF-8
Date: Fri, 16 Apr 2021 17:44:05 GMT
Etag: "3147526947"
Expires: Fri, 23 Apr 2021 17:44:05 GMT
Last-Modified: Thu, 17 Oct 2019 07:18:26 GMT
Server: ECS (dcb/7F84)
X-Cache: HIT
Content-Length: 1256
Extra Headers
Any number of extra headers can be sent to the server utilizing the -H option.
curl -H "X-First-Name: Joe" http://example.com/Change the User-Agent
Sometimes when downloading a file, the remote server may be set to block the Curl User-Agent or to return different contents depending on the visitor device and browser.
In situations like this to emulate a different browser, use the -A option.
For example to emulates Firefox 60 you would use:
curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" https://getfedora.org/Content-Type header
It’s also possible to send JSON data using cURL. This can be done by specifying the “application/json” header with the “-H” flag.
curl -H 'Content-Type: application/json' -d '{ "username" : "admin", "password" : "password" }' --cookie-jar /dev/null -L http://example.com/login.phpAuthentication
Basic AUTH Login
curl http://admin:password@example.com/ -vIt parsed the credentials and added the Authorization header with the encoded data. The server then responded with 302 Found, which means authentication was successful.
Alternately, the “-u” flag can be used to specify credentials as well.
curl -u admin:password http://example.com/ -vUnlike browsers, cURL doesn’t redirect us to the specified location by default. That’s why it responded with 302 HTTP code. The “-L” flag instructs curl to follow redirections.
curl -u admin:password -L http://example.com/Cookie Authentication
In order to get authenticated, we need to pass the data to the HTTP server in the Cookie header. It is supposedly the data previously received from the server in a “Set-Cookie:” line. The data should be in the format “NAME1=VALUE1; NAME2=VALUE2”.
For example, to download the Oracle Java JDK rpm file jdk-10.0.2_linux-x64_bin.rpm you’ll need to pass a cookie named oraclelicense with value a. -b,--cookie flag is used to send the cookie.
curl -L -b "oraclelicense=a" -O http://download.oracle.com/jdk-10.0.2_linux-x64_bin.rpmYou can also mention the cookie file instead of key-value pair.
curl -L -b /path/to/cookie.txt -O http://download.oracle.com/jdk-10.0.2_linux-x64_bin.rpmThe file specified with -b, –cookie is only used as input. No cookies will be written to the file. To store cookies, use the -c, –cookie-jar option.
Often times when we log in using the basic AUTH method i.e, using credentials, a cookie is set by the server and sent via HTTP response headers. This cookie must be saved somewhere so that it can be sent in subsequent requests to the server. This is when comes the cookie-jar concept.
curl -d 'username=admin&password=password' -L --cookie-jar cookies.txt http://example.com/login.phpWrite-Out Options
We can make curl display information on stdout after a completed transfer. The format is a string that may contain plain text mixed with any number of variables. All variables are specified as %{variable_name}.
{kiran@parrot} ~$ curl -O localhost:8000/shell.php -w "File Size: %{size_download} bytes\nTime taken: %{time_total} seconds" -s
File Size: 5494 bytes
Time taken: 0.001441 secondsDifferent variables are available. Some of them are:
- http_code
- scheme
- speed_upload
- size_header and many more.
Using Proxies
curl supports different types of proxies, including HTTP, HTTPS, and SOCKS. To transfer data through a proxy server, use the -x (--proxy) option, followed by the proxy URL.
The following command downloads the specified resource using a proxy on 192.168.44.1 port 8888:
curl -x 192.168.44.1:8888 http://linux.com/If the proxy server requires authentication, use the -U (--proxy-user) option followed by the user name and password separated by a colon (user:password):
curl -U username:password -x 192.168.44.1:8888 http://linux.com/There are still various features of curl that I haven’t mentioned in this post. Details on everything can be found on their official website https://curl.se/docs/manpage.html.
Founder of cybersecnerds.com. Electronics Engineer by profession, Security Engineer by passion.
I am a Linux Enthusiast and highly interested in the offensive side of the CyberSec industry. You will find me reading InfoSec blogs most of the time.
