cURL is one of the most used command line tools for transferring data over the internet. It supports many protocols including HTTP and HTTPS and is used for web scraping, testing APIs and debugging network connections. Whether you need to fetch data from a public website or interact with an API, cURL is the solution.

This post will walk you through sending GET requests with cURL. We’ll break it down into simple steps with examples of common scenarios like passing parameters, headers, and JSON responses.

Table of Contents

  1. What is a GET Request?
  2. Sending a Simple GET Request with cURL
  3. GET Request with Parameters
  4. Retrieving HTTP Headers with GET Requests
  5. Fetching JSON Data Using cURL
  6. Handling Redirects in cURL GET Requests
  7. Sending Cookies with a GET Request
  8. cURL GET Request Options Overview
  9. Final Thoughts
  10. Frequently Asked Questions

What is a GET Request?

A GET request is the most simple and most used HTTP method to retrieve data from a server. When you type a URL in your browser and hit enter, your browser sends a GET request to the server hosting the website. The server responds by sending back the requested data, like HTML content, images or JSON data and your browser displays it.

Unlike other HTTP methods like POST which sends data to a server, GET requests are only for fetching resources. This makes GET requests perfect for:

  • Retrieving web pages.
  • Accessing API endpoints to get data.
  • Fetching static resources like images or stylesheets.

Here’s a simple example of a GET request using cURL:

1
curl https://crawlbase.com

This command retrieves the HTML of the specified webpage and displays it in your terminal.

Sending a Simple GET Request with cURL

A GET request with cURL is the simplest way to get data from a server. cURL defaults to GET so you don’t need to specify it.

Here’s how you can make a basic GET request:

1
curl http://httpbin.org/get

This will request the content from https://example.com. The server will respond with the resource, usually HTML or other data, which will be displayed in your terminal.

To save the output to a file instead of printing to the terminal use the -o option:

1
curl -o output.html https://example.com

Key Points:

  • Use curl followed by the URL to make a basic GET request.
  • Add the -o flag to save the response to a file.

This is your starting point for getting data from websites or APIs. Now we’ll look at how to send GET requests with parameters.

GET Request with Parameters

Sometimes you need to pass parameters in your GET request to give more data to the server. There are two ways to pass parameters:

Using the -G and -d Options

The -G option allows you to send data with a GET request, and the -d option specifies the parameters. Here’s an example:

1
curl -G -d "param1=value1" -d "param2=value2" http://httpbin.org/get

In this case, param1=value1 and param2=value2 are the parameters sent to the server as query strings.

Appending Parameters to the URL

You can also append parameters directly to the URL. Here’s how you do it:

1
curl 'http://httpbin.org/get?param1=value1&param2=value2'

In this case parameters param1 and param2 are added to the URL after the ? and separated by &.

Both methods will process the parameters and return the data. The choice is up to you but both will give the same result.

Retrieving HTTP Headers with GET Requests

When you do a GET with cURL you can also get the HTTP headers along with the response body. HTTP headers are very useful to know about the response, server type, content type and more.

To include HTTP headers in the response, use the -i or --include option:

1
curl -i http://httpbin.org/headers

This command will return both the body of the response and the HTTP headers. Alternatively, if you just want the headers without the body use the --head option:

1
2
curl --head http://httpbin.org/headers

This is useful when you only need metadata about the request and not the actual content.

Fetching JSON Data Using cURL

To fetch JSON data from a server with cURL, you can tell cURL you want the response in JSON. This is handy when dealing with APIs that return data in JSON.

To request JSON data use the -H option to set the Accept header to application/json:

1
curl -H "Accept: application/json" http://httpbin.org/get

This tells the server that you expect the response in JSON format. If the server supports it, you will receive a response in JSON, which you can use for further processing.

Handling Redirects in cURL GET Requests

When making GET requests with cURL, the server may return a redirect to another URL. cURL doesn’t follow redirects by default. To follow the redirect, use the -L or --location option.

Here’s how to use it:

1
curl -L http://httpbin.org/redirect-to?url=http://httpbin.org/get

This will follow the redirect and fetch the content from the new URL. Useful when working with websites or APIs that send redirects (301 or 302 status codes).

Sending Cookies with a GET Request

Some websites require you to send cookies with your GET request to keep sessions or track user activity. You can include cookies using the -b or --cookie option in cURL.

Here’s how to send cookies with a GET request:

1
curl -b "username=JohnDoe; sessionId=12345" http://httpbin.org/cookies

In this example, the cookies username and sessionId are sent along with the GET request. You can also provide cookies from a file:

1
curl -b cookies.txt http://httpbin.org/cookies

This allows you to interact with websites that require user sessions, logins, or other cookie-based data.

cURL GET Request Options Overview

Here are some key cURL options you can use with GET requests:

Image of cURL GET Request Options

This table summarizes the key cURL options that allow you to customize GET requests to fit your needs.

Final Thoughts

cURL is a great tool for sending GET requests and getting data from web servers. You can fetch web content, handle params, manage cookies, follow redirects, and work with APIs. Whether you’re scraping data or automating tasks cURL makes it easy and flexible.

Once you master cURL GET requests and options you can interact with web resources and get the data you need. With practice you’ll be able to use cURL for all your web scraping and automation projects, it’s a must-have in your toolbox.

Frequently Asked Questions

Q. What is a cURL GET request?

A cURL GET request is used to get data from a server. It sends an HTTP request to the specified URL and gets the server’s response which can be a web page, JSON data or any other resource. This is used for web scraping, API interaction, and testing web resources.

Q. How do I send parameters with a GET request in cURL?

You can send params in a GET request by either appending them to the URL or using the -G and -d options.

  • Example with parameters in URL:
    curl 'http://example.com/data?param1=value1&param2=value2'
  • Example with -G and -d:
    curl -G -d "param1=value1" -d "param2=value2" http://example.com/data

Q. Can I retrieve JSON data using cURL?

Yes, you can fetch JSON data by setting the Accept: application/json header in your cURL request.
Example:
curl -H "Accept: application/json" http://example.com/data
This tells the server to return data in JSON format if it’s supported.