{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
]
}
,
{
"cell_type": "code",
"metadata": {
"dotnet_interactive": {
"language": "fsharp"
},
"polyglot_notebook": {
"kernelName": "fsharp"
}
},
"execution_count": null, "outputs": [],
"source": [
"#r \"nuget: FSharp.Data,6.6.0\"\n",
"\n",
"Formatter.SetPreferredMimeTypesFor(typeof\u003cobj\u003e, \"text/plain\")\n",
"Formatter.Register(fun (x: obj) (writer: TextWriter) -\u003e fprintfn writer \"%120A\" x)\n",
"#endif\n"
]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[](https://mybinder.org/v2/gh/fsprojects/FSharp.Data/gh-pages?filepath=library/Http.ipynb)\u0026emsp;\n",
"[](https://fsprojects.github.io/FSharp.Data//library/Http.fsx)\u0026emsp;\n",
"[](https://fsprojects.github.io/FSharp.Data//library/Http.ipynb)\n",
"\n",
"# HTTP Utilities\n",
"\n",
"The .NET library provides a powerful API for creating and sending HTTP web requests.\n",
"There is a simple `WebClient` type (see [MSDN](http://msdn.microsoft.com/en-us/library/system.net.webclient.aspx)) and a more flexible `HttpWebRequest`\n",
"type (see [MSDN](http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx)). However, these two types are quite difficult to use if you\n",
"want to quickly run a simple HTTP request and specify parameters such as method,\n",
"HTTP POST data, or additional headers.\n",
"\n",
"The FSharp.Data package provides a simple [Http](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-http.html) type with four methods:\n",
"[Http.RequestString](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-http.html#RequestString) and [Http.AsyncRequestString](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-http.html#AsyncRequestString), that can be used to create a simple request and\n",
"perform it synchronously or asynchronously, and [Http.Request](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-http.html#Request) and it\u0027s async companion [Http.AsyncRequest](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-http.html#AsyncRequest) if\n",
"you want to request binary files or you want to know more about the response like the status code,\n",
"the response URL, or the returned headers and cookies.\n",
"\n",
"The type is located in `FSharp.Data` namespace:\n",
"\n"
]
}
,
{
"cell_type": "code",
"metadata": {
"dotnet_interactive": {
"language": "fsharp"
},
"polyglot_notebook": {
"kernelName": "fsharp"
}
},
"execution_count": 2, "outputs": [],
"source": [
"open FSharp.Data\n"
]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Sending simple requests\n",
"\n",
"To send a simple HTTP (GET) request that downloads a specified web page, you\n",
"can use [Http.RequestString](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-http.html#RequestString) and [Http.AsyncRequestString](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-http.html#AsyncRequestString) with just a single parameter:\n",
"\n"
]
}
,
{
"cell_type": "code",
"metadata": {
"dotnet_interactive": {
"language": "fsharp"
},
"polyglot_notebook": {
"kernelName": "fsharp"
}
},
"execution_count": 3, "outputs": [
{
"data": {
"text/plain": ["val it: unit = ()"]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}],
"source": [
"// Download the content of a web site\n",
"Http.RequestString(\"http://tomasp.net\")\n",
"\n",
"// Download web site asynchronously\n",
"async {\n",
" let! html = Http.AsyncRequestString(\"http://tomasp.net\")\n",
" printfn \"%d\" html.Length\n",
"}\n",
"|\u003e Async.Start\n"
]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the rest of the documentation, we focus on the `RequestString` method, because\n",
"the use of `AsyncRequestString` is exactly the same.\n",
"\n",
"## Query parameters and headers\n",
"\n",
"You can specify query parameters either by constructing\n",
"an URL that includes the parameters (e.g. `http://...?test=foo\u0026more=bar`) or you\n",
"can pass them using the optional parameter `query`. The following example also explicitly\n",
"specifies the GET method, but it will be set automatically for you if you omit it:\n",
"\n"
]
}
,
{
"cell_type": "code",
"metadata": {
"dotnet_interactive": {
"language": "fsharp"
},
"polyglot_notebook": {
"kernelName": "fsharp"
}
},
"execution_count": 4, "outputs": [
{
"data": {
"text/plain": ["38968",
"",
"val it: string =",
"",
" \"{",
" \"args\": {",
" \"test\": \"foo\"",
" }, ",
" \"headers\": {",
" \"Accept-Encoding\": \"gzip, deflate\", ",
" \"Host\": \"httpbin.org\", ",
" \"X-Amzn-Trace-Id\": \"Root=1-68b5b3e7-2ca967250dbad7047ec75a0b\"",
" }, ",
" \"origin\": \"64.236.161.17\", ",
" \"url\": \"http://httpbin.org/get?test=foo\"",
"}",
"\""]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}],
"source": [
"Http.RequestString(\"http://httpbin.org/get\", query = [ \"test\", \"foo\" ], httpMethod = \"GET\")\n"
]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Additional headers are specified similarly - using an optional parameter `headers`.\n",
"The collection can contain custom headers, but also standard headers such as the\n",
"Accept header (which has to be set using a specific property when using `HttpWebRequest`).\n",
"\n",
"The following example uses [The Movie Database](http://www.themoviedb.org) API\n",
"to search for the word \"batman\". To run the sample, you\u0027ll need to register and\n",
"provide your API key:\n",
"\n"
]
}
,
{
"cell_type": "code",
"metadata": {
"dotnet_interactive": {
"language": "fsharp"
},
"polyglot_notebook": {
"kernelName": "fsharp"
}
},
"execution_count": null, "outputs": [],
"source": [
"// API key for http://www.themoviedb.org\n",
"let apiKey = \"\u003cplease register to get a key\u003e\"\n",
"\n",
"// Run the HTTP web request\n",
"Http.RequestString(\n",
" \"http://api.themoviedb.org/3/search/movie\",\n",
" httpMethod = \"GET\",\n",
" query = [ \"api_key\", apiKey; \"query\", \"batman\" ],\n",
" headers = [ \"Accept\", \"application/json\" ]\n",
")\n"
]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The library supports a simple and unchecked string based API (used in the previous example),\n",
"but you can also use pre-defined header names to avoid spelling mistakes. The named headers\n",
"are available in [HttpRequestHeaders](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-httprequestheaders.html) (and [HttpResponseHeaders](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-httpresponseheaders.html)) modules, so you can either\n",
"use the full name `HttpRequestHeaders.Accept`, or open the module and use just the short name\n",
"`Accept` as in the following example. Similarly, the `HttpContentTypes` enumeration provides\n",
"well known content types:\n",
"\n"
]
}
,
{
"cell_type": "code",
"metadata": {
"dotnet_interactive": {
"language": "fsharp"
},
"polyglot_notebook": {
"kernelName": "fsharp"
}
},
"execution_count": 5, "outputs": [],
"source": [
"open FSharp.Data.HttpRequestHeaders\n"
]
}
,
{
"cell_type": "code",
"metadata": {
"dotnet_interactive": {
"language": "fsharp"
},
"polyglot_notebook": {
"kernelName": "fsharp"
}
},
"execution_count": null, "outputs": [],
"source": [
"// Run the HTTP web request\n",
"Http.RequestString(\n",
" \"http://api.themoviedb.org/3/search/movie\",\n",
" query = [ \"api_key\", apiKey; \"query\", \"batman\" ],\n",
" headers = [ Accept HttpContentTypes.Json ]\n",
")\n"
]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Getting extra information\n",
"\n",
"Note that in the previous snippet, if you don\u0027t specify a valid API key, you\u0027ll get a (401) Unauthorized error,\n",
"and that will throw an exception. Unlike when using `WebRequest` directly, the exception message will still include\n",
"the response content, so it\u0027s easier to debug in F# interactive when the server returns extra info.\n",
"\n",
"You can also opt out of the exception by specifying the `silentHttpErrors` parameter:\n",
"\n"
]
}
,
{
"cell_type": "code",
"metadata": {
"dotnet_interactive": {
"language": "fsharp"
},
"polyglot_notebook": {
"kernelName": "fsharp"
}
},
"execution_count": 6, "outputs": [
{
"data": {
"text/plain": ["val it: string =",
"",
" \"{\"status_code\":7,\"status_message\":\"Invalid API key: You must be granted a valid key.\",\"success\":false}",
"\""]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}],
"source": [
"Http.RequestString(\"http://api.themoviedb.org/3/search/movie\", silentHttpErrors = true)\n"
]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this case, you might want to look at the HTTP status code so you don\u0027t confuse an error message for an actual response.\n",
"If you want to see more information about the response, including the status code, the response\n",
"headers, the returned cookies, and the response url (which might be different to\n",
"the url you passed when there are redirects), you can use the [Http.Request](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-http.html#Request) method\n",
"instead of the [Http.RequestString](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-http.html#RequestString) method:\n",
"\n"
]
}
,
{
"cell_type": "code",
"metadata": {
"dotnet_interactive": {
"language": "fsharp"
},
"polyglot_notebook": {
"kernelName": "fsharp"
}
},
"execution_count": 7, "outputs": [],
"source": [
"let response =\n",
" Http.Request(\"http://api.themoviedb.org/3/search/movie\", silentHttpErrors = true)\n",
"\n",
"// Examine information about the response\n",
"response.Headers\n",
"response.Cookies\n",
"response.ResponseUrl\n",
"response.StatusCode\n"
]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Sending request data\n",
"\n",
"If you want to create a POST request with HTTP POST data, you can specify the\n",
"additional data in the `body` optional parameter. This parameter is of type [HttpRequestBody](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-httprequestbody.html), which\n",
"is a discriminated union with three cases:\n",
"\n",
"* `TextRequest` for sending a string in the request body.\n",
"\n",
"* `BinaryUpload` for sending binary content in the request.\n",
"\n",
"* `FormValues` for sending a set of name-value pairs correspondent to form values.\n",
"\n",
"If you specify a body, you do not need to set the `httpMethod` parameter, it will be set to `Post` automatically.\n",
"\n",
"The following example uses the [httpbin.org](http://httpbin.org) service which\n",
"returns the request details:\n",
"\n"
]
}
,
{
"cell_type": "code",
"metadata": {
"dotnet_interactive": {
"language": "fsharp"
},
"polyglot_notebook": {
"kernelName": "fsharp"
}
},
"execution_count": 8, "outputs": [
{
"data": {
"text/plain": ["val it: string =",
"",
" \"{",
" \"args\": {}, ",
" \"data\": \"\", ",
" \"files\": {}, ",
" \"form\": {",
" \"test\": \"foo\"",
" }, ",
" \"headers\": {",
" \"Accept-Encoding\": \"gzip, deflate\", ",
" \"Content-Length\": \"8\", ",
" \"Content-Type\": \"application/x-www-form-urlencoded\", ",
" \"Host\": \"httpbin.org\", ",
" \"X-Amzn-Trace-Id\": \"Root=1-68b5b3e8-183a998f4c59e2d67b98ce05\"",
" }, ",
" \"json\": null, ",
" \"origin\": \"64.236.161.17\", ",
" \"url\": \"http://httpbin.org/post\"",
"}",
"\""]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}],
"source": [
"Http.RequestString(\"http://httpbin.org/post\", body = FormValues [ \"test\", \"foo\" ])\n"
]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": [
"By default, the `Content-Type` header is set to `text/plain`, `application/x-www-form-urlencoded`,\n",
"or `application/octet-stream`, depending on which kind of `HttpRequestBody` you specify, but you can change\n",
"this behaviour by adding `content-type` to the list of headers using the optional argument `headers`:\n",
"\n"
]
}
,
{
"cell_type": "code",
"metadata": {
"dotnet_interactive": {
"language": "fsharp"
},
"polyglot_notebook": {
"kernelName": "fsharp"
}
},
"execution_count": 9, "outputs": [
{
"data": {
"text/plain": ["val it: string =",
"",
" \"{",
" \"args\": {}, ",
" \"data\": \" {\\\"test\\\": 42} \", ",
" \"files\": {}, ",
" \"form\": {}, ",
" \"headers\": {",
" \"Accept-Encoding\": \"gzip, deflate\", ",
" \"Content-Length\": \"14\", ",
" \"Content-Type\": \"application/json\", ",
" \"Host\": \"httpbin.org\", ",
" \"X-Amzn-Trace-Id\": \"Root=1-68b5b3e9-7598315134612840144ad1d0\"",
" }, ",
" \"json\": {",
" \"test\": 42",
" }, ",
" \"origin\": \"64.236.161.17\", ",
" \"url\": \"http://httpbin.org/post\"",
"}",
"\""]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}],
"source": [
"Http.RequestString(\n",
" \"http://httpbin.org/post\",\n",
" headers = [ ContentType HttpContentTypes.Json ],\n",
" body = TextRequest \"\"\" {\"test\": 42} \"\"\"\n",
")\n"
]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Maintaining cookies across requests\n",
"\n",
"If you want to maintain cookies between requests, you can specify the `cookieContainer`\n",
"parameter.\n",
"\n",
"The following is an old sample showing how this is set.\n",
"\n"
]
}
,
{
"cell_type": "code",
"metadata": {
"dotnet_interactive": {
"language": "fsharp"
},
"polyglot_notebook": {
"kernelName": "fsharp"
}
},
"execution_count": null, "outputs": [],
"source": [
"// Build URL with documentation for a given class\n",
"let msdnUrl className =\n",
" let root = \"http://msdn.microsoft.com\"\n",
" sprintf \"%s/en-gb/library/%s.aspx\" root className\n",
"\n",
"// Get the page and search for F# code\n",
"let docInCSharp = Http.RequestString(msdnUrl \"system.web.httprequest\")\n",
"docInCSharp.Contains \"\u003ca\u003eF#\u003c/a\u003e\"\n",
"\n",
"open System.Net\n",
"let cc = CookieContainer()\n",
"\n",
"// Send a request to switch the language\n",
"Http.RequestString(\n",
" msdnUrl \"system.datetime\",\n",
" query = [ \"cs-save-lang\", \"1\"; \"cs-lang\", \"fsharp\" ],\n",
" cookieContainer = cc\n",
")\n",
"|\u003e ignore\n",
"\n",
"// Request the documentation again \u0026 search for F#\n",
"let docInFSharp =\n",
" Http.RequestString(msdnUrl \"system.web.httprequest\", cookieContainer = cc)\n",
"\n",
"docInFSharp.Contains \"\u003ca\u003eF#\u003c/a\u003e\"\n"
]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Requesting binary data\n",
"\n",
"The [Http.RequestString](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-http.html#RequestString) method will always return the response as a `string`, but if you use the\n",
"[Http.Request](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-http.html#Request) method, it will return a `HttpResponseBody.Text` or a\n",
"`HttpResponseBody.Binary` depending on the response `content-type` header:\n",
"\n"
]
}
,
{
"cell_type": "code",
"metadata": {
"dotnet_interactive": {
"language": "fsharp"
},
"polyglot_notebook": {
"kernelName": "fsharp"
}
},
"execution_count": 10, "outputs": [],
"source": [
"let logoUrl = \"https://raw.github.com/fsharp/FSharp.Data/master/misc/logo.png\"\n",
"\n",
"match Http.Request(logoUrl).Body with\n",
"| Text text -\u003e printfn \"Got text content: %s\" text\n",
"| Binary bytes -\u003e printfn \"Got %d bytes of binary content\" bytes.Length\n"
]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Customizing the HTTP request\n",
"\n",
"For the cases where you need something not natively provided by the library, you can use the\n",
"`customizeHttpRequest` parameter, which expects a function that transforms an `HttpWebRequest`.\n",
"\n",
"As an example, let\u0027s say you want to add a client certificate to your request. To do that,\n",
"you need to open the `X509Certificates` namespace from `System.Security.Cryptography`,\n",
"create a `X509ClientCertificate2` value, and add it to the `ClientCertificates` list of the request.\n",
"\n",
"Assuming the certificate is stored in `myCertificate.pfx`:\n",
"\n"
]
}
,
{
"cell_type": "code",
"metadata": {
"dotnet_interactive": {
"language": "fsharp"
},
"polyglot_notebook": {
"kernelName": "fsharp"
}
},
"execution_count": null, "outputs": [],
"source": [
"open System.Security.Cryptography.X509Certificates\n",
"\n",
"// Load the certificate from local file\n",
"let clientCert = new X509Certificate2(\".\\myCertificate.pfx\", \"password\")\n",
"\n",
"// Send the request with certificate\n",
"Http.Request(\n",
" \"http://yourprotectedresouce.com/data\",\n",
" customizeHttpRequest =\n",
" fun req -\u003e\n",
" req.ClientCertificates.Add(clientCert) |\u003e ignore\n",
" req\n",
")\n"
]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Handling multipart form data\n",
"\n",
"You can also send http multipart form data via the `Multipart` [HttpRequestBody](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-httprequestbody.html) case.\n",
"Data sent in this way is streamed instead of being read into memory in its entirety, allowing for\n",
"uploads of arbitrary size.\n",
"\n"
]
}
,
{
"cell_type": "code",
"metadata": {
"dotnet_interactive": {
"language": "fsharp"
},
"polyglot_notebook": {
"kernelName": "fsharp"
}
},
"execution_count": null, "outputs": [],
"source": [
"let largeFilePath = \"//path/to/large/file.mp4\"\n",
"let data = System.IO.File.OpenRead(largeFilePath) :\u003e System.IO.Stream\n",
"\n",
"Http.Request(\n",
" \"http://endpoint/for/multipart/data\",\n",
" body =\n",
" Multipart(\n",
" boundary = \"define a custom boundary here\", // this is used to separate the items you\u0027re streaming\n",
" parts = [ MultipartItem(\"formFieldName\", System.IO.Path.GetFileName(largeFilePath), data) ]\n",
" )\n",
")\n"
]
}
,
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Related articles\n",
"\n",
"* API Reference: [Http](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-http.html)\n",
"\n",
"* API Reference: [HttpContentTypes](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-httpcontenttypes.html)\n",
"\n",
"* API Reference: [HttpEncodings](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-httpencodings.html)\n",
"\n",
"* API Reference: [HttpMethod](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-httpmethod.html)\n",
"\n",
"* API Reference: [HttpRequestBody](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-httprequestbody.html)\n",
"\n",
"* API Reference: [HttpRequestHeaders](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-httprequestheaders.html)\n",
"\n",
"* API Reference: [HttpResponse](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-httpresponse.html)\n",
"\n",
"* API Reference: [HttpResponseBody](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-httpresponsebody.html)\n",
"\n",
"* API Reference: [HttpResponseHeaders](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-httpresponseheaders.html)\n",
"\n",
"* API Reference: [HttpResponseWithStream](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-httpresponsewithstream.html)\n",
"\n",
"* API Reference: [HttpStatusCodes](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-httpstatuscodes.html)\n",
"\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".NET (F#)",
"language": "F#",
"name": ".net-fsharp"
},
"language_info": {
"file_extension": ".fs",
"mimetype": "text/x-fsharp",
"name": "polyglot-notebook",
"pygments_lexer": "fsharp"
},
"polyglot_notebook": {
"kernelInfo": {
"defaultKernelName": "fsharp",
"items": [
{
"aliases": [],
"languageName": "fsharp",
"name": "fsharp"
}
]
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}