A Step-by-Step Guide to Performing a GET Request with cURL

cURL is a library and command-line tool that lets you exchange data with servers using various protocols. It’s a handy tool for fetching public data like HTML, text, JSON, images, and more. Plus, it helps manage sessions and cookies. cURL can interact with APIs and can be easily integrated with shell commands for automation, making it a great tool for web scraping.

In this guide, we’ll show you how to GET Request with cURL to pull public data.

What is a GET Request?

A GET request in HTTP is a method used to retrieve information from a specific website. When you type a URL and press enter, your browser sends a GET request to the server, asking it to return the associated HTML code.

There are other HTTP methods that can also receive HTML from the server, but they are used for different purposes. For example, HTTP POST is used to submit data to the server. When you click the Login button, your username and password are sent to the server via a POST request. You can still view the webpage, but POST is only used to submit form data. To learn how to send POST requests, you can refer to our blog post on cURL.

On the other hand, a GET request does not send any data to the server. It is only used to request a webpage or other resources such as images. Here is a simple example of a GET request using cURL:

curl http://twistory.net

The result returned will be the HTML code from the web server.

Procedure for Implementing GET Request with cURL

Now that we have covered the basics, let’s delve deeper into how to send GET requests using cURL. In this guide, we will focus on using httpbin.org, a simple service that allows us to interact with and receive HTTP responses. The aspects of cURL that we will explore include:

• Executing a basic cURL GET request
• Sending a GET request with parameters
• Retrieving HTTP headers from a GET request
• Receiving responses in JSON format
• Tracking redirections
• Sending cookies along with a GET request

Step 1 – Implementing a Basic cURL GET Request

If you’re using the default GET request method, there’s no need to use the --request option. Instead, you can simply send a GET request as follows:

curl http://httpbin.org/get

Step 2 – Implementing GET Requests with Accompanying Parameters

To send additional data to the server in your request URL, you can use a GET request with parameters. cURL supports this through two powerful options -d and -G.

If you use -d without -G, the request will become a POST request. If you use the -X option with the GET value, the data you want to send with -d will be ignored.

Therefore, if you want to send parameters with a GET request, you need to use -G along with -d.

For example:

curl -G -d "param1=value1" -d "param2=value2" http://httpbin.org/get

In the above example, ‘param1’ and ‘param2’ are the keys, while ‘value1’ and ‘value2’ are their corresponding values. You can use the -d option multiple times to send different parameters.

You can also include GET parameters in the URL:

curl 'http://httpbin.org/get?param1=value1&param2=value2'

Step 3 – Receiving and Processing HTTP Headers

HTTP headers are an integral part of an HTTP request, allowing for the exchange of additional information between the client and the server. To view the HTTP headers along with the response, you can use the -i or --include option in your cURL GET command:

curl -i http://httpbin.org/headers

This command will return the HTTP response headers, which include information such as the server, date, content type, and content length. This information can provide a better understanding of the response data.

If you want to view only the response headers without the content, you can use the --head option:

curl --head http://httpbin.org/headers

Step 4 – Retrieving Data in JSON Format

JSON has become a standard for data exchange in the modern web development ecosystem. When interacting with APIs, it’s often crucial to request data in JSON format. You can instruct cURL to accept responses in JSON format by using the -H option followed by “Accept: application/json”:

curl -H "Accept: application/json" http://httpbin.org/get

However, it’s important to note that sending “Accept: application/json” doesn’t guarantee that the response will be in JSON format. This largely depends on whether the website supports JSON responses or not.

Step 5 – Handling Redirects

In some situations, the URL you’re requesting might redirect to another URL. By default, cURL doesn’t follow such redirects. However, you can explicitly instruct it to do so by using the -L or --location option:

curl -L http://httpbin.org/redirect-to?url=http://httpbin.org/get

This command tells cURL to follow any redirects, ensuring that you reach the final destination URL. This can be particularly useful when the URL you’re initially requesting is set up to redirect to another URL.

Step 6 – Sending Cookies Along with the GET Request

There may be times when you want to send a cookie along with your GET request, especially when interacting with websites that require user sessions or track user activity. To accomplish this, you can use the -b or --cookie option in cURL, accompanied by the name and value of the cookie:

curl -b "username=John" http://httpbin.org/cookies

The above command sends a GET request to http://httpbin.org/cookies with a cookie named “username” and a value of “John”.

Parameters for GET Requests in cURL

To help you better understand how to use cURL to send GET requests, here’s a summary table of key options, including short options, long options, and descriptions. Please note that you should only use either the short or long option, not both.

Conclusion

We believe this guide will help you grasp the basics of sending GET requests using cURL. But remember, like any skill, practice is key to becoming proficient. We encourage you to spend time practicing sending requests to quickly familiarize yourself with cURL.

If you’re interested in learning more about using proxies with cURL, consider checking out our blog post on the topic.