cURL is a library and command
-line tool that lets you exchange data with servers using various protocols. It’s a handy tool for fetching public data like HTML, text, JSON, images, and more. Plus, it helps manage sessions and cookies. cURL can interact with APIs and can be easily integrated with shell commands for automation, making it a great tool for web scraping.
In this guide, we’ll show you how to GET Request with cURL to pull public data.
What is a GET Request?
A GET request in HTTP is a method used to retrieve information from a specific website. When you type a URL and press enter, your browser sends a GET request to the server, asking it to return the associated HTML code.
There are other HTTP methods that can also receive HTML from the server, but they are used for different purposes. For example, HTTP POST is used to submit data to the server. When you click the Login button, your username and password are sent to the server via a POST request. You can still view the webpage, but POST is only used to submit form data. To learn how to send POST requests, you can refer to our blog post on cURL.
On the other hand, a GET request does not send any data to the server. It is only used to request a webpage or other resources such as images. Here is a simple example of a GET request using cURL:
The result returned will be the HTML code from the web server.
Procedure for Implementing GET Request with cURL
Now that we have covered the basics, let’s delve deeper into how to send GET requests using cURL. In this guide, we will focus on using
httpbin.org, a simple service that allows us to interact with and receive HTTP responses. The aspects of cURL that we will explore include:
- Executing a basic cURL GET request
- Sending a GET request with parameters
- Retrieving HTTP headers from a GET request
- Receiving responses in JSON format
- Tracking redirections
- Sending cookies along with a GET request
Step 1 – Implementing a Basic cURL GET Request
If you’re using the default GET request method, there’s no need to use the
--request option. Instead, you can simply send a GET request as follows:
Step 2 – Implementing GET Requests with Accompanying Parameters
To send additional data to the server in your request URL, you can use a GET request with parameters. cURL supports this through two powerful options
If you use
-G, the request will become a POST request. If you use the -X option with the GET value, the data you want to send with
-d will be ignored.
Therefore, if you want to send parameters with a GET request, you need to use -G along with -d.
curl -G -d "param1=value1" -d "param2=value2" http://httpbin.org/get
In the above example, ‘param1’ and ‘param2’ are the keys, while ‘value1’ and ‘value2’ are their corresponding values. You can use the
-d option multiple times to send different parameters.
You can also include GET parameters in the URL:
Step 3 – Receiving and Processing HTTP Headers
HTTP headers are an integral part of an HTTP request, allowing for the exchange of additional information between the client and the server. To view the HTTP headers along with the response, you can use the
--include option in your cURL GET command:
curl -i http://httpbin.org/headers
This command will return the HTTP response headers, which include information such as the server, date, content type, and content length. This information can provide a better understanding of the response data.
If you want to view only the response headers without the content, you can use the
curl --head http://httpbin.org/headers
Step 4 – Retrieving Data in JSON Format
JSON has become a standard for data exchange in the modern web development ecosystem. When interacting with APIs, it’s often crucial to request data in JSON format. You can instruct cURL to accept responses in JSON format by using the -H option followed by “Accept: application/json”:
curl -H "Accept: application/json" http://httpbin.org/get
However, it’s important to note that sending “Accept: application/json” doesn’t guarantee that the response will be in JSON format. This largely depends on whether the website supports JSON responses or not.
Step 5 – Handling Redirects
In some situations, the URL you’re requesting might redirect to another URL. By default, cURL doesn’t follow such redirects. However, you can explicitly instruct it to do so by using the -L or
curl -L http://httpbin.org/redirect-to?url=http://httpbin.org/get
This command tells cURL to follow any redirects, ensuring that you reach the final destination URL. This can be particularly useful when the URL you’re initially requesting is set up to redirect to another URL.
Step 6 – Sending Cookies Along with the GET Request
There may be times when you want to send a cookie along with your GET request, especially when interacting with websites that require user sessions or track user activity. To accomplish this, you can use the
--cookie option in cURL, accompanied by the name and value of the cookie:
curl -b "username=John" http://httpbin.org/cookies
The above command sends a GET request to
http://httpbin.org/cookies with a cookie named “username” and a value of “John”.
Parameters for GET Requests in cURL
To help you better understand how to use cURL to send GET requests, here’s a summary table of key options, including short options, long options, and descriptions. Please note that you should only use either the short or long option, not both.
|Retrieves only the HTTP headers.
|Includes the HTTP response headers in the output.
|Specifies the User-Agent string to send to the server.
|Specifies the type of request to use (GET, POST, PUT, DELETE, etc.)
|Follows redirects in the server’s response.
|Sends a cookie from a string or file. The format of the string must be NAME=VALUE; another=another.
|Provides additional information (debug info).
|Silent mode. Don’t output anything.
|Output to File
|Writes output to a file instead of standard output.
|Pass Custom Header
|Passes a custom header to the server. To receive JSON, use -H “Accept: application/json”.
We believe this guide will help you grasp the basics of sending GET requests using cURL. But remember, like any skill, practice is key to becoming proficient. We encourage you to spend time practicing sending requests to quickly familiarize yourself with cURL.
If you’re interested in learning more about using proxies with cURL, consider checking out our blog post on the topic.