What is cURL?

The word cURL means “Client for URLs.” cURL is defined as a simple and flexible tool to perform complicated tasks.

cURL comes with a command-line tool and a URL syntax for data transfer to and from servers, and this command-line tool can pass HTTP web URLs through the terminal without the help of a browser. cURL acts as a bridge between web servers to help network communications.

cURL is open-source, free software that exists due to the help of code enthusiasts!

Two Products of cURL are :

1. libcurl – It is a ready-to-use client sider library that is used by several applications. libcurl is said to be the most used and most powerful network transfer library used in the world. libcurl is portable and can be used in any operating system such as windows, Mac os, Linux, Solaris, RISC OS, DOS, OpenBSD, OpenVMS, HPUX, AIX, HURD, etc. 
2. curl– It is a command-line tool that provides more than 200 command options. curl is readily available in most operating systems.

The default protocol of cURL is HTTP. Below is the list of cURL supported Protocols. 

What is the Latest cURL Version?

According to the site curl.se, the latest stable version(curl-7.791) was released on 22 September 2021.

What are the Advantages of cURL?

1. Availability

When we think about using software, the first concern will be how to install it. Some software requires deep technical knowledge for installation.

Most people leave such software installations halfway and move to the next one in line. Curl, on the other hand, is compatible with almost all operating systems.

It comes preloaded in different versions of mac, windows, and Linux Os. If it is not present, all you need is a couple of codes to run in your console. 

2. Curl Is Good at Explaining What Went Wrong!

It is pretty natural to write error codes while at the terminal. Typical codes end up in errors that give no clue to the programmer about what went wrong and where lies the actual issue.

If the software points out the most probable issue and explains the error, it would be helpful and time-saving.

Curl has excellent error logging support that will help you identify the error at ease. Curl is also good at describing the actions that occurred between the sender and receiver.

3. Curl Is Popular Among Developers.

Basic code commands, effortless structure, and easy availability of plenty of online support pages related to curl queries make it a hotshot in the developer’s world!

4. Easy Testing

Since the code structure of the curl is meaningful and straightforward, it is easy to test the code during compilation.

How to Install cURL in Windows?

The curl comes preinstalled if the latest Windows 10 version build of your OS is 17063 or the later models.

For previous versions, please follow the steps below:

1. You can find the latest curl.exe files in http://curl.haxx.se/download.html
2. Make sure you are downloading the correct version of curl that is compatible with your windows version.
3. To check the version of your computer.

a. press Windows + i together.

b. System – About

c. From the system type, you will understand the windows version.

4. From http://curl.haxx.se/download.html, download the appropriate windows version of curl
5. Create a folder in your C: Drive and name it “curl.”
6. After downloading the zip file, save the unzipped file in the “curl” folder.
7. Download and install the digital certificate “cacert.pem”. You can get it from 

http://curl.haxx.se/docs/caextract.html.

8. Now from the properties menu,click Advanced Settings
9. Click Environment Variables => System variable => Path => Edit
10. Edit => New => enter the curl.exe file location path
11. Press Ok until the dialog box closes.

How to Install cURL in Mac OSX?

Usually, curl comes preinstalled with mac by default. In case if it is not present or you want to reinstall it, follow the steps mentioned here.

1. Open the terminal by pressing Command and Space keys together and type terminal.
2. Type or copy/paste the code given below:

ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)” < /dev/null 2> /dev/null (source: https://help.ubidots.com/en/articles/2165289-learn-how-to-install-run-curl-on-windows-macosx-linux)

If you are asked to enter the password, please enter your Mac password. If you are using the terminal for the first time, don’t worry about the password not displaying while typing. It’s for security purposes!

3. After the installation, type “brew install curl,” and you are in!

How to Install cURL in Linux OS?

1. Open the terminal- Ctrl + Alt + T
2. Sudo apt-get install curl, Type and run the command
3. Enter the computer password if asked.
4. Now type curl on the console. If the curl is installed successfully, you can see certain options like 

curl: try ‘curl –help’ or ‘curl –manual.’

Is it Possible to use cURL Online?

Using ReqBin, a popular Online Curl Client, it is possible to us curl online without the help of any desktop applications, plugins, etc.

ReqBin can perform actions from the URLs such as location, cookie, data, head, header, proxy, request, user-agent, etc.

The curl command line builder is an opposite online application compared to ReqBin. We can describe the actions in a beautiful UI, and the application will generate the corresponding curl code.

(Source: installmd)

How to Write the Curl Syntax?

The Curl Progress Chart

While using the curl command, you can see the terminal displaying various values. These values include, 

1. The amount of data transferred sofar
2. The Estimated Time left to complete the operation
3. Current data transfer speed

Suffixes on the terminal: k, M, G, T, P(bytes are calculated based on 1k=1024bytes)

For HTTP POST, PUT requests, you should redirect the output file with the help of the shell redirect option.

The Structure of Options in Curl

Options are used with a ‘-‘ or ‘–‘ before the keyword. For example,’ -a’ and ‘–append’ have the same operation, but the difference is that space is not mandatory with the’-‘ but when you use ‘–‘ options, there must be a space between them the value.

• -O
  This option saves the downloaded file with the name mentioned in the code.

Syntax: curl -o file_name FILE_ADDRESS

For example, consider the screenshot shared.

The file will be saved as hello.zip

• -C –

This option is required when you perform operations using large-size files. There are chances that the upload/download process of large files gets interrupted due to network issues. -C – options resume the operation.

Syntax: curl -C – -O file_name FILE_ADDRESS

• -limit-rate

If you are looking for an option to keep a boundary for the maximum data size of the file to be downloaded, you can use the limit rate.

Syntax: curl –limit-rate size_value FILE_ADDRESS

• -u

If the FTP file is protected with a user name and password, you can use this option.

Syntax: curl -u username:password FTP_FILE_ADDRESS

• -T

You can use this option to upload files to FTP

Syntax: curl-u username:password -T filename FILE_ADDRESS

Now let us check out some of the curl syntaxes.

1. Basic Syntax

curl [option] [url]

Syntax: curl http://www.google.com

This code on the console will display all the HTML content of the page.

2. How to Download a File from FTP Server using curl?

Syntax: curl -u username: password -O ftp://ftpurl/mydownloadfile.pdf

3. How to check the response header of the website?

Syntax: curl site_url -i

Example:

curl https://curl.se/ -i

Result:

Connection: keep-alive

Content-Length: 3079

Server: nginx/1.21.1

Content-Type: text/html

X-Frame-Options: SAMEORIGIN

Last-Modified: Tue, 05 Oct 2021 02:05:04 GMT

ETag: “21a7-5cd91753d3879-gzip”

Cache-Control: max-age=60

Expires: Tue, 05 Oct 2021 09:08:42 GMT

Content-Encoding: gzip

X-Content-Type-Options: nosniff

Content-Security-Policy: default-src ‘self’ curl.haxx.se www.curl.se curl.se www.fastly-insights.com fastly-insights.com; style-src ‘unsafe-inline’ ‘self’ curl.haxx.se www.curl.se curl.se

Strict-Transport-Security: max-age=31536000

Via: 1.1 varnish, 1.1 varnish

Accept-Ranges: bytes

Date: Tue, 05 Oct 2021 09:10:53 GMT

Age: 51

X-Served-By: cache-bma1620-BMA, cache-ewr18153-EWR

X-Cache: HIT, MISS

X-Cache-Hits: 5, 0

X-Timer: S1633425054.654700,VS0,VE256

Vary: Accept-Encoding

4. Curl and HTTP methods.

The HTTP methods are GET, POST, PUT, DELETE. The GET HTTP method is used by default while submitting a request to the website.

Since Curl uses it by default, there is no need to mention it with the code. But all the rest of the HTTP methods have to be specified.

(Image Source)

Syntax for Curl GET request: curl -X https://curl.se/

            Syntax for Curl POST request: curl -X POST https://curl.se/

            Here -X denotes the existence of a HTTP request method used.

5. How to Upload a File with Curl?

@ – This sign ensures the content part as a file

            < – This sign grabs the text part mentioned in the file

            Syntax:  curl -F [email protected] UPLOAD_ADDRESS

Here, data is the form field into which curl will upload the example.doc file. 

Syntax for < :  curl -F data=<example.doc UPLOAD_ADDRESS

Here, curl will collect the content in the example.doc file.

For multiple files upload:

Syntax: curl -F [email protected] -F [email protected],etc. UPLOAD_ADDRESS

6. What is the Verbose option in Curl?

The verbose operation helps debug. The curl becomes descriptive when the verbose command(-v, –verbose) is passed along with the code. It explains the actions that curl performed in silence previously.

>: Header data sent by Curl

<: Header data received by Curl

Syntax : curl -v https://curl.se/

7. How to restart the download using curl?

It is pretty natural to fail the download when the network is weak or has site-related issues. The curl has an option to resume the download from where it is halted.

  Syntax: curl -C -O site_url             

8. Maximum Operation Time

The curl allows its users to set a specific code running time. Using this function, the user can fix a maximum time for specific actions to complete.

Once the maximum time allotted is reached in a slow network server, the code gets terminated even if not completed.

Syntax: curl –max-time 10 site_url

9. How to connect a proxy in curl?

It is effortless to connect a proxy to your curl. Mainly it comes handly for people working in DMZ servers.

Syntax: curl –proxy yourproxy:port site_url

10. Control the network speed and bandwidth

There is a specific option in curl to control the data usage. If you are working in a slow network or low data quota, perform the following curl code.

Syntax : curl –limit-rate 100K site_url. Here K means Kilobyte, M(or m) means Megabyte, G(or g) means Gigabyte, etc.

11. Connection time out in Curl

–connect-timeout option lets you set a specific connection timeout deadline in seconds. For example, if the time out is set to be 20 seconds, curl will try to accomplish the connection for 20 seconds and automatically terminate after that.

Syntax: curl –connect-timeout 20 UPLOAD_ADDRESS

12. Exit user name –disallow-username-in-url

This option comes in handy when URLs are provided during run time. When a URL is passed with a user name, the curl will terminate the operation.

Syntax: curl –disallow-username-in-url FILE_ADDRESS

13. IPv4 DNS –dns-ipv4-addr

This option is used for a single IPv4 address so that the DNS requests will begin from the specified address. 

Syntax: curl –dns-ipv4-addr <address> FILE_ADDRESS

14. IPv6 DNS –dns-ipv6-addr

This option is used for a single IPv6 address so that the DNS requests will begin from the specified address. 

Syntax: curl –dns-ipv6-addr <address> FILE_ADDRESS

15. Fail and Exit

There are chances for in-between fails during multiple file transfers. When curl is used for this process, it tends to skip the errors between moving from one file to the next.

Finally, if the last file transfer returns errorless, the curl won’t show the errors between the transfers. 

When this function is used, the curl command will identify the presence of the error and will display it on the first file transfer fail.

Syntax: curl –fail-early FILE_ADDRESS_1 FILE_ADDRESS_2

16. Fail silently  -f, –fail

When an HTTP server cannot transfer a file, it returns an HTML file containing the failure. If this option is used, the HTTP server will not return any response on server errors. The error code is 22.

Syntax: curl –fail FILE_ADDRESS

17. IPv4 and IPv6 Parallel Connection Try

“–happy-eyeballs-timeout-ms” is used to send the connection request to IPv4 and IPv6 with a small gap of milliseconds.

If the connection were unsuccessful with IPv6, the next attempt would begin in the mentioned microseconds to connect with IPv4. The first connection is retained.

According to Happy Eyeballs RFC 6555,

• Timeout milliseconds to balance human factors against network load is 150-250ms.
• For libcurl: 200ms
• For Chrome and Firefox: 300ms

           Syntax: curl –happy-eyeballs-timeout-ms 500 FILE_ADDRESS

18. Header files -H, –header

This option allows you to add extra header values during HTTP transfer to a server. If the externally specified header name is the same as the internal header, the curl will ignore the internal header and proceed with the external header.

It is better not to do this unless you are looking to overwrite the internal header.

Syntax: curl -H “User-Agent: value” FILE_ADDRESS

What are the Applications of cURL?

(Image Source)

Web Data Extraction

The cURL software is mainly used to transfer data from server to server, and it has plenty of other applications. Data extraction from websites is one of the significant applications.

A simple curl command can extract the complete website HTML source code. The required data can be taken from this HTML content using some cleaning software and codes. 

Manage day-to-day applications and testing API modules

Curl can be coupled with API modules that APIs can use to operate various online applications such as social media.

The Redhat website has mentioned an interesting case study using curl with an open-source social network known as Mastodon.