curl

curl Notes

2024/03/22

Client URL (cURL is a handy tool for transferring data specified with URLs. I mostly use it for pulling down files when I’m working on a server remotely. I also use it to check servers I’m configuring or test APIs.

Basic cURL Usage

File Downloads

Giving curl the URL of a webpage will display it in your console window.

curl https://curl.se/

Use -o to specify a filename to save the output to a file.

curl https://curl.se/ -o out.html

Using the -O option will download a file with its original name

curl -O https://repo.almalinux.org/almalinux/9.3/isos/x86_64/AlmaLinux-9.3-x86_64-minimal.iso

Download a script and run with bash uwing the pipe |. ( -o- redirects out put to stdout, which is the default. So it’s not necessary here)

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash

Sometimes a download gets interrupted. When this happens, you can resume a download with the -C option. ( -L here handles redirects )

curl -L -O -C -  https://repo.almalinux.org/almalinux/9.3/isos/x86_64/AlmaLinux-9.3-x86_64-minimal.iso

The -u options lets you specify a username. cURL will prompt you for the password

curl -u username http://example.com/somefile

You can also specify a password by following the username with a colon and the password ( This is not recommended as the command can be viewed while its running )

curl -u username:password http://example.com/somefile

Testing With cURL

Getting information on a server

Use -v to get verbose output about the connection. This can be helpful when debugging a problem.

 curl -v https://www.example.com/robots.txt

Sometimes it’s helpful to see the headers returned from the server. Use -I or -i to view Headers.

curl https://www.example.com/ -i  # Headers with Body
curl https://www.example.com/ -I  # Headers Only

-H is used to set headers. This can be use to specify a website If your sever is handling multiple websites.

curl -H 'Host: example.com' http://10.10.0.1
curl -H 'Host: codepala.com' http://10.10.0.1

API Testing

When I’m testing APIs I don’t really want to see the progress info. I use -s to hide it

curl -s https://www.example.com/v1/endpoint

I often use test servers with self signed certificates. Use -k to bypass SSL checks

curl -k https://test.example.com

I use -w to view HTTP codes when I test APIs. The following command just shows the http code by redirecting stdout to /dev/null

curl -o /dev/null -s -w "%{http_code}\n" https://www.example.com/somepage

My APIs tend to use JWT for authentication. I specify the bearer token in a header with -H.

curl -s -w "%{http_code}\n" -H 'Accept: application/json' \
    -H "Authorization: Bearer ${TOK}"  \
    https://www.example.com/api/create-article \
    -H "Content-Type: application/x-www-form-urlencoded"  \
    -d "title=The Title&summary=blah blah blah&body=blah blah blah blah blah&publish=true"

This assumes ${TOK} has been set to the bearer token value.

POST

The following example uses -H to set the Content-Type as URL encoded.

curl -s -w "%{http_code}\n" https://www.example.com/articles \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -d "title=The Title&summary=Article Summary Here&body=Lorem ipsum dolor sit amet, consectetur adipiscing elit,&publish=false"

This specifies the Content-Type as JSON.

curl  -s -w "%{http_code}\n"  -H 'Accept: application/json' \
   -X POST http://www.example.com/articles  \
   -H "Content-Type: application/json"  \
   -d '{"title":"The Title","summary":"Article Summary Here","body":"Lorem ipsum dolor sit amet","publish":true,"keywords":["text"]}'

-X can be used to set the HTTP Method. It’s not strictly necesary, because curl can infer the method from other options. -d implies a POST, so specifying -X POST is not necessary here

PUT

Using -T uploads a file as a PUT operation

curl -H "Authorization: Bearer ${TOK}" -T /path/to/local/file.txt http://example.com/destination/

You can also use -X PUT to override -d and specify the data in the command instead of using a file.

curl -s -w "%{http_code}\n"   \ 
    -H 'Accept: application/json'  \ 
    -H "Authorization: Bearer ${TOK}"  \ 
    -H "Content-Type: application/x-www-form-urlencoded"  \ 
    -d "name=flat color" -X PUT https://www.example.com/entity/cidh2i8"
DELETE

Send a delete with -X DELETE

curl -s -w "%{http_code}\n"  \
    -H 'Accept: application/json' \
    -H "Authorization: Bearer ${TOK}" \
    -X DELETE https://www.example.com/entity/hp38fh"

Use -c, --cookie-jar and -b, --cookie to manage cookies.

The following command saves HTTP cookies to a file called cookie.jar

curl -o -  -s -w "%{http_code}\n" \
    -X POST https://www.example.com:3000/auth/signin \
    -H "Content-Type: application/x-www-form-urlencoded"\
    -d "username=someuser&password=somepass" \
    --cookie-jar cookie.jar

The following uses the cookie.jar file to send the cookies with the HTTP request.

curl -o -  -s -w "%{http_code}\n" \
    -X POST https://www.example.com/some/action  \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -d "param1=5043&param2&abbybabble" \
    -b cookie.jar

About Me

Greg Gallardo

I'm a software developer and sys-admin in Iowa. I use C++, C#, Java, Swift, Python, JavaScript and TypeScript in various projects. I also maintain Windows and Linux systems on-premise and in the cloud ( Linode, AWS, and Azure )

Github

Mastodon

YouTube

About you

IP Address: 34.236.191.0

User Agent: CCBot/2.0 (https://commoncrawl.org/faq/)

Language: en-US,en;q=0.5

Latest Posts

Iowa City Weather

Today

-- ˚F / 61 ˚F

Sunday

71 ˚F / 54 ˚F

Monday

64 ˚F / 46 ˚F

Tuesday

76 ˚F / 54 ˚F

Wednesday

76 ˚F / 56 ˚F

Thursday

72 ˚F / 51 ˚F

Friday

67 ˚F / 47 ˚F