CURL — Everything You Need To Know - CyberSec Nerds

CURL — Everything You Need To Know

curl is a command-line utility for transferring data from or to a server designed to work without user interaction. With curl, you can download or upload data using various supported protocols including HTTP, HTTPS, SCP, SFTP, and FTP. curl provides a number of options allowing you to resume transfers, limit the bandwidth, proxy support, user authentication, and much more.

In this tutorial, we will show you how to use the curl tool through practical examples and detailed explanations of the most common curl options.

Basic Usage

GET Request

The default HTTP requests made by curl are GET requests.. In order to retrieve the example.com homepage, you would run:

{kiran@parrot} ~$ curl example.com
<!doctype html>
<html>
<head>
   <title>Example Domain</title>

   <meta charset="utf-8" />
   <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
   <meta name="viewport" content="width=device-width, initial-scale=1" />
   <style type="text/css">
   body {
<SNIP>

If no protocol is specified, curl tries to guess the protocol you want to use, and it will default to HTTP.

GET Request Verbose

It’s possible to view the raw HTTP requests using the “-v” switch to increase verbosity.

{kiran@parrot} ~$ curl -v example.com
*   Trying 93.184.216.34:80...
* TCP_NODELAY set
* Connected to example.com (93.184.216.34) port 80 (#0)
> GET / HTTP/1.1
> Host: example.com
> User-Agent: curl/7.68.0
> Accept: */*
>  
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Age: 339480
< Cache-Control: max-age=604800
< Content-Type: text/html; charset=UTF-8
<SNIP>

Saving the output to a file

To save the result of the curl command, use either the -o or -O option.

Lowercase -o saves the file with a predefined filename, which in the example below is my_file.js.

curl -o my_file.js https://test-site/vue.js

Uppercase -O saves the file with its original filename.

curl -O https://test-site/vue.js

Download multiple files at once

curl -O https://test-site/file1.zip -O https://test-site/file2.zip

Resume a download

You can continue/resume a previous file transfer at the given offset by using the -C - option. This is useful if your connection drops during the download of a large file, and instead of starting the download from scratch, you can continue the previous one.

curl -C - -O http://releases.ubuntu.com/18.04/ubuntu-18.04-live-server-amd64.iso

Follow redirects

By default, curl doesn’t follow the HTTP Location headers.

If you try to retrieve the non-www version of google.com, you will notice that instead of getting the source of the page you’ll be redirected to the www version.

{kiran@parrot} ~$ curl google.com     
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

The -L option instructs curl to follow any redirect until it reaches the final destination.

curl -L google.com

Progress Meter

curl normally displays a progress meter during operations, indicating the amount of transferred data, transfer speeds and estimated time left, etc. The progress meter displays number of bytes and the speeds are in bytes per second.

{kiran@parrot} ~$ curl -O localhost:8000/shell.php
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100  5494  100  5494    0     0  2682k      0 --:--:-- --:--:-- --:--:-- 5365k

If you prefer a progress “bar” instead of the regular meter, -#, –progress-bar is your friend. 

{kiran@parrot} ~$ curl -O localhost:8000/shell.php -#
####################################################################### 100.0%

You can also completely get rid off the progress-meter with the -s, –silent option. Remember this mode also turns off the error messages.

curl -O localhost:8000/shell.php --silent

HTTP Methods

-X, --request flag specifies a custom request method to use when communicating with the HTTP server. The specified request method will be used instead of the method otherwise used (which defaults to GET).

Passing data using POST

Let’s try to login to the page from the POST section using cURL. The default “Content-Type” used by cURL is “application/x-www-form-urlencoded“, the data for which can be passed using the “-d” flag.

curl -d 'username=admin&password=password' -L http://example.com/login.php

cURL automatically sends a POST request when the “-d” flag is used.

File Upload using PUT

curl -X PUT -d @test.txt http://inlanefreight.com/test.txt -vv

The “@” symbol is used by curl to read the file and send it’s contents as the data. 

Deleting a remote file using DELETE

curl -X DELETE http://inlanefreight.com/test.txt -vv

HTTP Header Options

Fetch the headers only

Use the -I option to fetch only the HTTP headers of the specified resource.

{kiran@parrot} ~$ curl -I example.com
HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 490767
Cache-Control: max-age=604800
Content-Type: text/html; charset=UTF-8
Date: Fri, 16 Apr 2021 17:44:05 GMT
Etag: "3147526947"
Expires: Fri, 23 Apr 2021 17:44:05 GMT
Last-Modified: Thu, 17 Oct 2019 07:18:26 GMT
Server: ECS (dcb/7F84)
X-Cache: HIT
Content-Length: 1256

Extra Headers

Any number of extra headers can be sent to the server utilizing the -H option.

  curl -H "X-First-Name: Joe" http://example.com/

Change the User-Agent

Sometimes when downloading a file, the remote server may be set to block the Curl User-Agent or to return different contents depending on the visitor device and browser.

In situations like this to emulate a different browser, use the -A option.

For example to emulates Firefox 60 you would use:

curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" https://getfedora.org/

Content-Type header

It’s also possible to send JSON data using cURL. This can be done by specifying the “application/json” header with the “-H” flag.

curl -H 'Content-Type: application/json' -d '{ "username" : "admin", "password" : "password" }' --cookie-jar /dev/null -L  http://example.com/login.php

Authentication

Basic AUTH Login

curl http://admin:password@example.com/ -v

It parsed the credentials and added the Authorization header with the encoded data. The server then responded with 302 Found, which means authentication was successful.

Alternately, the “-u” flag can be used to specify credentials as well.

curl -u admin:password  http://example.com/ -v

Unlike browsers, cURL doesn’t redirect us to the specified location by default. That’s why it responded with 302 HTTP code. The “-L” flag instructs curl to follow redirections.

curl -u admin:password -L http://example.com/

Cookie Authentication

In order to get authenticated, we need to pass the data to the HTTP server in the Cookie header. It is supposedly the data previously received from the server in a “Set-Cookie:” line. The data should be in the format “NAME1=VALUE1; NAME2=VALUE2”.

For example, to download the Oracle Java JDK rpm file jdk-10.0.2_linux-x64_bin.rpm you’ll need to pass a cookie named oraclelicense with value a. -b,--cookie flag is used to send the cookie.

curl -L -b "oraclelicense=a" -O http://download.oracle.com/jdk-10.0.2_linux-x64_bin.rpm

You can also mention the cookie file instead of key-value pair.

curl -L -b /path/to/cookie.txt -O http://download.oracle.com/jdk-10.0.2_linux-x64_bin.rpm

The file specified with -b, –cookie is only used as input. No cookies will be written to the file. To store cookies, use the -c, –cookie-jar option.

Often times when we log in using the basic AUTH method i.e, using credentials, a cookie is set by the server and sent via HTTP response headers. This cookie must be saved somewhere so that it can be sent in subsequent requests to the server. This is when comes the cookie-jar concept.

curl -d 'username=admin&password=password' -L --cookie-jar cookies.txt  http://example.com/login.php

Write-Out Options

We can make curl display information on stdout after a completed transfer. The format is a string that may contain plain text mixed with any number of variables. All variables are specified as %{variable_name}.

{kiran@parrot} ~$ curl -O localhost:8000/shell.php -w "File Size: %{size_download} bytes\nTime taken: %{time_total} seconds" -s
File Size: 5494 bytes
Time taken: 0.001441 seconds

Different variables are available. Some of them are:

  • http_code
  • scheme
  • speed_upload
  • size_header and many more.

Using Proxies

curl supports different types of proxies, including HTTP, HTTPS, and SOCKS. To transfer data through a proxy server, use the -x (--proxy) option, followed by the proxy URL.

The following command downloads the specified resource using a proxy on 192.168.44.1 port 8888:

curl -x 192.168.44.1:8888 http://linux.com/

If the proxy server requires authentication, use the -U (--proxy-user) option followed by the user name and password separated by a colon (user:password):

curl -U username:password -x 192.168.44.1:8888 http://linux.com/

There are still various features of curl that I haven’t mentioned in this post. Details on everything can be found on their official website https://curl.se/docs/manpage.html.

Kiran Dawadi

Founder of cybersecnerds.com. Electronics Engineer by profession, Security Practitioner by passion. I am a Linux Enthusiast and highly interested in the offensive side of the CyberSec industry. You will find me reading InfoSec blogs most of the time.

Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments