Download as PDF, PPTX
![Under the hood
GET / HTTP/1.1
Host: www.example.com
Accept: */*
User-Agent: HTTP-eats-the-world/2019
HTTP/1.1 200 OK
Date: Thu, 09 Nov 2018 14:49:00 GMT
Server: my-favorite v3
Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT
Content-Length: 12345
Set-Cookie: this-is-simple=yeah-really;
Content-Type: text/html
[content]
@bagder@bagder](https://image.slidesharecdn.com/curl-nordicapis-2019-191022204611/85/Just-curl-it-13-320.jpg)
![$ curl https://example.com/ --libcurl sourcecode.c
$ gcc sourcecode.c -lcurl -o ./myapp
$ ./myapp
$ cat sourcecode.c
@bagder@bagder
Convert it into an application?
/********* Sample code generated by the curl command line tool **********
* All curl_easy_setopt() options are documented at:
* https://curl.haxx.se/libcurl/c/curl_easy_setopt.html
************************************************************************/
#include <curl/curl.h>
int main(int argc, char *argv[])
{
CURLcode ret;
CURL *hnd;
hnd = curl_easy_init();
curl_easy_setopt(hnd, CURLOPT_BUFFERSIZE, 102400L);
curl_easy_setopt(hnd, CURLOPT_URL, "https://example.com");
curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.66.0");
curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L);
curl_easy_setopt(hnd, CURLOPT_HTTP_VERSION, (long)CURL_HTTP_VERSION_2TLS);
curl_easy_setopt(hnd, CURLOPT_SSH_KNOWNHOSTS, "/home/user/.ssh/known_hosts");
curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L);
...](https://image.slidesharecdn.com/curl-nordicapis-2019-191022204611/85/Just-curl-it-39-320.jpg)
![Under the hood
GET / HTTP/1.1
Host: www.example.com
Accept: */*
User-Agent: HTTP-eats-the-world/2019
HTTP/1.1 200 OK
Date: Thu, 09 Nov 2018 14:49:00 GMT
Server: my-favorite v3
Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT
Content-Length: 12345
Set-Cookie: this-is-simple=yeah-really;
Content-Type: text/html
[content]
@bagder@bagder](https://image.slidesharecdn.com/curl-nordicapis-2019-191022204611/75/Just-curl-it-13-2048.jpg)
![$ curl https://example.com/ --libcurl sourcecode.c
$ gcc sourcecode.c -lcurl -o ./myapp
$ ./myapp
$ cat sourcecode.c
@bagder@bagder
Convert it into an application?
/********* Sample code generated by the curl command line tool **********
* All curl_easy_setopt() options are documented at:
* https://curl.haxx.se/libcurl/c/curl_easy_setopt.html
************************************************************************/
#include <curl/curl.h>
int main(int argc, char *argv[])
{
CURLcode ret;
CURL *hnd;
hnd = curl_easy_init();
curl_easy_setopt(hnd, CURLOPT_BUFFERSIZE, 102400L);
curl_easy_setopt(hnd, CURLOPT_URL, "https://example.com");
curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.66.0");
curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L);
curl_easy_setopt(hnd, CURLOPT_HTTP_VERSION, (long)CURL_HTTP_VERSION_2TLS);
curl_easy_setopt(hnd, CURLOPT_SSH_KNOWNHOSTS, "/home/user/.ssh/known_hosts");
curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L);
...](https://image.slidesharecdn.com/curl-nordicapis-2019-191022204611/75/Just-curl-it-39-2048.jpg)
Uploaded byDaniel Stenberg
Just curl it!
Curl is an open source command line tool and library for transferring data with various internet protocols. It supports many protocols including HTTP, HTTPS, FTP, FTPS, SFTP, SCP, SMTP, IMAP, POP3, and more. Curl has over 2,000 contributors and is widely used across operating systems like Linux, Windows, macOS, and others. The presentation discussed curl's history, features, usage examples, and how it can be used to mimic browser behavior and inspect HTTPS traffic.
More Related Content
Just curl it!
- 1. Just curl itJustcurl it October 22, Platform Summit 2019October 22, Platform Summit 2019
- 2.
- 3.
- 4. An open sourceproject that makes a command line tool and a library for transferring data using Internet protocols @bagder@bagder
- 5. Features! DICT, FILE, FTP,FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, Telnet and TFTP TLS certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, HTTP/HTTPS/SOCKS proxy, cookies, authentication (Basic, Digest, NTLM, Negotiate, Kerberos), HTTP/2, HTTP/3, alt-svc:, happy eyeballs, file transfer resume, proxy tunneling, DNS-over-HTTPS, HTTP compression and much more @bagder@bagder
- 6.
- 7. 2,000 contributors Who makescurl curl 730 authors 150 authors per year 12 regulars Daniel @bagder@bagder (The boxes are not drawn to scale)
- 8. @bagder@bagder Conquered the world Righttime Open Source Portable Backwards compatible Stable
- 9. Many places curl hasbeen around since spring 1998 MacOS (since 2001) Windows 10 (since 2018) Linux distros (since 1999) BSDs, other Unixes, MS-DOS, older systems Side-loaded - from the official site or elsewhere Built from source @bagder@bagder
- 10. Many flavors Millions ofbuild combinations Thirty different dependencies in combinations Frequent releases Rarely the same in two places! @bagder@bagder
- 11. Many protocols andoptions Today we only talk about the tool named curl curl knows many protocols curl features 226 command line options We stick to HTTP(S) in this presentation Using a small subset of flags @bagder@bagder
- 12. HTTP RequestRequest - method +path- method + path - headers- headers - body- body ResponseResponse - response code- response code - headers- headers - body- body @bagder@bagder
- 13. Under the hood GET/ HTTP/1.1 Host: www.example.com Accept: */* User-Agent: HTTP-eats-the-world/2019 HTTP/1.1 200 OK Date: Thu, 09 Nov 2018 14:49:00 GMT Server: my-favorite v3 Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT Content-Length: 12345 Set-Cookie: this-is-simple=yeah-really; Content-Type: text/html [content] @bagder@bagder
- 14. Minimal behavior bydefault Basic set of headers No fancy functionality Let users enable more when wanted Toggle features on/off one-by-one @bagder@bagder
- 15. Long and shortoptions All short options have a long alternative -s can also be set with –-silent I will use the short options today Many options only exist as long options @bagder@bagder
- 16. $ curl example.com <html> <body> <div> <h1>ExampleDomain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </div> </body> </html> @bagder@bagder Plain curl HTTP GET
- 17. $ curl -ihttps://example.com/ HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder GET and show me the response headers
- 18. $ curl https://example.com/json| jq { "id": "kr09ddfgbfsf", "name": "Issues, PRs, Dashboard, Projects", "status": "operational", "created_at": "2017-01-31T20:01:46.638Z", "updated_at": "2019-09-26T14:00:45.562Z", "position": 3, "description": "Fake JSON for an example", "group": false, "only_show_if_degraded": false }, @bagder@bagder GET and show JSON nicely
- 19. $ curl -Ihttps://example.com/ HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 @bagder@bagder HEAD only shows the response headers HEAD returns no body!
- 20. $ curl -Ihttps://example.com/redirected HTTP/1.1 302 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html location: /a/separate/place content-length: 306 server: server/3000 @bagder@bagder Curl doesn’t follow redirects by default HEAD returns no body!
- 21. $ curl -I-L https://example.com/redirected HTTP/1.1 200 OK date: Wed, 19 Oct 2019 11:23:06 GMT content-type: text/html content-length: 4676 server: server/3000 @bagder@bagder Please follow redirects HEAD returns no body!
- 22. $ curl -vhttps://example.com/ -o /dev/null * Trying 93.184.216.34:443... * Connected to example.com (93.184.216.34) port 443 (#0) … * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: C=US; ST=California; L=Los Angeles; O=Corporation; CN=www.example.org … > GET / HTTP/2 > Host: example.com > User-Agent: curl/7.66.0 > Accept: */* > < HTTP/2 200 < date: Wed, 09 Oct 2019 11:16:06 GMT < content-type: text/html < content-length: 306 < server: server/3000 @bagder@bagder Verbose shows more from under the hood
- 23. $ curl https://example.com/-H "Magic: disc0" $ curl https://example.com/ -H "User-agent: disc0 1n th3 n1gh7" $ curl https://example.com/ -H "User-agent:" $ curl https://example.com/ -H "User-agent;" @bagder@bagder Pass in custom HTTP headers
- 24. $ curl -dname=Daniel https://example.com/receiver HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder POST some basic data to the remote
- 25. $ curl -d@file https://example.com/receiver -o saved $ ls -l | curl -d @- https://example.com/receiver -o saved $ ls -l | curl --data-binary @- https://example.com/receiver -o saved $ curl --data-binary @file.json -H "Content-Type: application/json" https://example.com @bagder@bagder POST a file
- 26. $ curl -Tlocalfile https://example.com/remote_name HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder PUT a file
- 27. $ curl -Tlocalfile -X SWOOSH https://example.com/remote_name -o save @bagder@bagder With -X we change the method string
- 28. curl -X invain leads to pain https://xkcd.com/386/ @bagder@bagder
- 29. $ curl -dpostdata -X POST https://example.com/remote_name -o save $ curl -d postdata -X POST https://example.com/remote_name -o save -L @bagder@bagder Bad -X leads to pain Probably not what you wanted!
- 30. @bagder@bagder Bad -X -X meansuse this method in all requests -L means follow redirects Redirects instruct the client what method to use in the next request -X overrides the method the server wants the client to use Use -X if you want a different method than curl would use
- 31.
- 32. @bagder@bagder Cookies are namevalue pairs Cookies are name=value pairs sent by servers Sent back by clients when the URL matches curl doesn’t know cookies unless you tell it to curl has separate options for reading and writing cookie files
- 33. $ curl -ccookiejar.txt https://example.com/ $ curl -b cookiejar.txt https://example.com/ $ cat cookiejar.txt # Netscape HTTP Cookie File # https://curl.haxx.se/docs/http-cookies.html # This file was generated by libcurl! Edit at your own risk. example.com TRUE / FALSE 1602699857 duidd182180ebab48 @bagder@bagder Cookies!
- 34. $ curl -ccookiejar.txt https://example.com/login_form $ curl -b cookiejar.txt -c cookiejar.txt https://example.com/login -d user=daniel -d password=1234 $ curl -b cookiejar.txt -c cookiejar.txt https://example.com/profile @bagder@bagder Cookies in a login procedure
- 35. $ curl https://127.0.0.1/ $curl -k https://127.0.0.1/ $ curl -k https://127.0.0.1/ -H "Host: example.com" $ curl https://example.com/ --resolve example.com:443:127.0.0.1 $ curl https://example.com/ --connect-to example.com:443:host.tld:8443 @bagder@bagder HTTPS obstacles, locally hosted named site Causes certificate problems Ducks for the cert problem Doesn’t work with cookies Works with cookies Doesn’t work with virtual servers Does TLS and certs correctly Works with cookies Works with virtual servers
- 36.
- 37.
- 38. $ curl https://example.com/-d sendthisdata --trace-ascii - … => Send header, 143 bytes (0x8f) 0000: POST / HTTP/2 000f: Host: example.com 0022: User-Agent: curl/7.66.0 003b: Accept: */* 0048: Content-Length: 12 005c: Content-Type: application/x-www-form-urlencoded 008d: => Send SSL data, 5 bytes (0x5) 0000: ....& => Send SSL data, 1 bytes (0x1) 0000: . => Send data, 12 bytes (0xc) 0000: sendthisdata == Info: We are completely uploaded and fine … @bagder@bagder More details!
- 39. $ curl https://example.com/--libcurl sourcecode.c $ gcc sourcecode.c -lcurl -o ./myapp $ ./myapp $ cat sourcecode.c @bagder@bagder Convert it into an application? /********* Sample code generated by the curl command line tool ********** * All curl_easy_setopt() options are documented at: * https://curl.haxx.se/libcurl/c/curl_easy_setopt.html ************************************************************************/ #include <curl/curl.h> int main(int argc, char *argv[]) { CURLcode ret; CURL *hnd; hnd = curl_easy_init(); curl_easy_setopt(hnd, CURLOPT_BUFFERSIZE, 102400L); curl_easy_setopt(hnd, CURLOPT_URL, "https://example.com"); curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L); curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.66.0"); curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L); curl_easy_setopt(hnd, CURLOPT_HTTP_VERSION, (long)CURL_HTTP_VERSION_2TLS); curl_easy_setopt(hnd, CURLOPT_SSH_KNOWNHOSTS, "/home/user/.ssh/known_hosts"); curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L); ...
- 40. @bagder@bagder Mimic that browser– copy as curl
- 41. $ curl 'https://curl.haxx.se/'-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:69.0) Gecko/ 20100101 Firefox/69.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en- US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Upgrade-Insecure- Requests: 1' -H 'If-Modified-Since: Thu, 17 Oct 2019 02:05:02 GMT' -H 'If-None-Match: "21eb- 59511a53fa694-gzip"' -H 'Cache-Control: max-age=0' -H 'TE: Trailers' $ curl 'https://curl.haxx.se/' -H 'authority: curl.haxx.se' -H 'cache-control: max-age=0' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/ OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Mobile Safari/537.36' -H 'sec-fetch-mode: navigate' -H 'sec-fetch-user: ?1' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/ *;q=0.8,application/signed-exchange;v=b3' -H 'sec-fetch-site: none' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: sv,en-US;q=0.9,en;q=0.8' -H 'if-none-match: "21eb- 59511a53fa694-gzip"' -H 'if-modified-since: Thu, 17 Oct 2019 02:05:02 GMT' --compressed @bagder@bagder Mimic that browser – copy as curl
- 42. SSLKEYLOGFILE Why trust yourtools when you can snoop on them Wireshark is your friend Tell curl or your browser to store its secrets in SSLKEYLOGFILE Tell Wireshark where the secrets are located Run curl or your browser @bagder@bagder
- 43. $ export SSLKEYLOGFILE=$HOME/tmp/tlskey $curl https://example.com @bagder@bagder Snoop on curl
- 44.
- 45. Many HTTP versions– but they look the same HTTP/1.0 shipped in 1996 We’ve learned how HTTP headers look like HTTP/2 came in 2015 HTTP/3 is “coming soon” curl makes HTTP/2 and HTTP/3 headers look and work like HTTP/1 @bagder@bagder
- 46. $ curl –-http1.0https://example.com $ curl –-http1.1 https://example.com $ curl –-http2 https://example.com $ curl –-http3 https://example.com @bagder@bagder HTTP versions HTTP/3 support is experimental!
- 47. $ curl -duser=daniel https://example.com https://another.example.com $ curl -d user=daniel https://example.com –-next https://another.example.com $ curl https://example.com –-next -d user=daniel https://another.example.com @bagder@bagder Different requests on the same command line!
- 48. All this, andmuch much more... @bagder@bagder
- 49.
- 50. 51 You can help!Youcan help! @bagder@bagder
- 51. Daniel Stenberg @bagder https://daniel.haxx.se/ Thank you!Thankyou! Questions?Questions? @bagder@bagder
- 52. License This presentation andits contents are licensed under the Creative Commons Attribution 4.0 license: http://creativecommons.org/licenses/by/4.0/ @bagder@bagder