HTTP requests with PowerShell’s Invoke-WebRequest – by Example

If you ever find yourself on a Windows system needing to make a HTTP request, the Invoke-WebRequest cmdlet will be your friend.

Let’s have a look on how to send various things with iwr (legit alias!) and how to get around common issues. We will be focussing on (manually) sending/requesting data, not so much on reading/parsing it.

In case it’s the first time you’re using Invoke-WebRequest or doing stuff with PowerShell in general, I recommend reading this post sequentially from top to bottom.

I will be using PowerShell 5.1 for this article. You can find your version with $PSVersionTable. As destination we will use several HTTP endpoints from httpbin.org.

A simple first request

Invoke-WebRequest http://httpbin.org/json

Staying with the defaults, this command will translate to the following request:

GET /json HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT; Windows NT 10.0; de-DE) WindowsPowerShell/5.1.17763.316
Host: httpbin.org

What we get back is a HtmlWebResponseObject in a nicely formatted way, displaying everything from (parts) of the body, response headers, length, etc.

Let’s store the response in a variable to be able to access the individual parts:

Accessing parts of the response

$r = Invoke-WebRequest http://httpbin.org/json

$r.StatusCode
200

$r.Headers
Key                              Value
---                              -----
Access-Control-Allow-Credentials true
Access-Control-Allow-Origin      *
Connection                       keep-alive
Content-Length                   429
Content-Type                     application/json
Date                             Wed, 10 Apr 2019 20:33:00 GMT
Server                           nginx

The content can be accessed with $r.Content. If we want to see what actually came back and was being parsed, we can use $r.RawContent. And, as we can redirect outputs just like in any other shell, we could store the response like this:

$r.RawContent > C:\My\Path\to\file.txt

Setting request headers

Custom request headers can be set by passing a hash table to Invoke-WebRequest’s -Headers option. The syntax for creating a hash table is as follows:

@{ <name> = <value>; [<name> = <value> ] ...}

Let’s make a new request and add some custom headers. We’ll also start using the alias iwr from now on to safe some typing.

Tip: A list of aliases for a cmdlet can be retrieved with Get-Alias -Definition <cmdlet>, in our case Get-Alias -Definition Invoke-WebRequest.

$r = iwr http://httpbin.org/headers `
-Headers @{'Accept' = 'application/json'; 'X-My-Header' = 'Hello World'}

This will produce something like:

GET /headers HTTP/1.1
X-My-Header: Hello World
Accept: application/json
...

Note that if you want to set cookies, you should do so with Invoke-WebRequest’s -WebSession option (see below). Manually including a Cookie HTTP header will not work. The same applies, according to the docs, to the user agent, which should only be set via the -UserAgent option, not via -Headers (in practice, I had no issues setting it via -Headers, though).

Debugging request headers

Debugging the request headers can be done with a service like httpbin.org (httpbin.org/headers) or simply by sniffing the traffic (e.g. with Wireshark) while making the request. Unfortunately, I am not aware of any way inside PowerShell to retrieve the headers that were actually sent.

Sending data and setting the content type

To give our request a body, we can either use the -Body option, the -InFile option or use a pipeline. For these examples we will do a POST request, so use -Method 'POST'.

Before actually sending data, let’s talk about the content type. If you do a POST request, but neither specify a Content-Type header nor use the -ContentType option, Invoke-WebRequest will automatically set the content type to application/x-www-form-urlencoded.

More gotchas: when you do set a Content-Type header via -Headers, say application/json; charset=utf8 and then pipe a utf8 file to iwr like so…

$r = Get-Content test.txt -ReadCount 0 | `
iwr http://httpbin.org/post `
    -Method 'POST' `
    -Headers @{'Content-Type' = 'application/json; charset=utf-8'}

… you may not actually send what you expect, as iwr will not read the piped data as utf8.

Depending on the encoding, you may send something like this:

{ "umlauts": "äüö" }

As something like this (ISO-8859-1):

7b 20 22 75 6d 6c 61 75 74 73 22 3a 20 22 e4 fc f6 22 20 7d

Instead of like this (UTF-8):

7b 20 22 75 6d 6c 61 75 74 73 22 3a 20 22 c3 a4 c3 bc c3 b6 22 20 7d

To be on the safe side, make sure to either use the -InFile option (and specify the -Headers) and/or use a pipeline, but set the -ContentType option (instead of the Content-Type header in the -Headers) the Invoke-WebRequest cmdlet provides:

$r = Get-Content test.txt -ReadCount 0 | `
iwr http://httpbin.org/post `
-Method 'POST' `
-ContentType 'application/json; charset=utf-8'

… will properly send the UTF-8 data.

Using -Body

If you want to build your body manually in the command, you can use the -Body option:

iwr http://httpbin.org/post `
-Method 'POST' `
-ContentType 'application/json; charset=utf-8' `
-Body '{"hello": "world"}'

For posting form data, you can use a hash table (going with the default application/x-www-form-urlencoded here):

iwr http://httpbin.org/post `
-Method 'POST' `
-Body @{ 'hello' = 'world'}

Sending Cookies, building sessions

When having multiple interactions with an endpoint, you might want to use a session object, for example to capture/send cookies. The Invoke-WebRequest cmdlet provides the option -SessionVariable, which you can give a target variable name to be used later for subsequent requests with the -WebSession option. Since we’re focussing on just manually sending data, let’s rather see, how we can manually create a .NET CookieContainer, add a cookie, and then pass the whole thing to iwr:

$s = New-Object Microsoft.PowerShell.Commands.WebRequestSession
$c = New-Object System.Net.Cookie('Hello','World','/','httpbin.org')
$s.Cookies.Add($c)

$r = iwr 'http://httpbin.org/cookies' -WebSession $s

This basically just translates to a Cookie: Hello=World header.

The arguments to .Net.Cookie are referenced here at the MS docs; in short, we’re using Name, Value, Path, Domain. Inspect $c to see other attributes to set (like http only, secure, expiry, etc.)

Authentication

Let’s look at three ways to authenticate against a web service: using basic auth, using client certificates and using Windows authentication via NTLM or Kerberos.

Using Basic Auth

In this case we will not wait for a server challenge, but build the Authorization header ourselves (don’t do this with sensitive creds as they will go right into your history file!):

$creds = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes('AzureDiamond:hunter2'))
iwr 'http://httpbin.org/basic-auth/AzureDiamond/hunter2' `
-Headers @{ 'Authorization' = 'Basic ' + $creds }

If you want to use your Windows user’s credentials for the request, you can just use the -Credential option, as in -Credential domain\you. In this case, there’s no need for you to create the Authorization header yourself. Not though, that this will make two requests; one that the server will answer with a 401, and another one with your credentials.

Using a client certificate

Using a client-certificate-based authentication is easiest when you access the certificate directly from the Windows cert store. Make sure to have your client certificate and private key installed, then use the -CertificateThumbprint option to pass the thumbprint of the cert you want to use. For example:

Get-ChildItem Cert:\CurrentUser\My # Check installed certs

Thumbprint                                Subject
----------                                -------
28D8AB79A7976FEED36D1DF0B9AC3D3F36B7C4DF  CN=BadSSL Client Certificate, O=BadSSL, L=San Francisco, S=California, C=US

$r = iwr 'https://client.badssl.com' `
-CertificateThumbprint 28D8AB79A7976FEED36D1DF0B9AC3D3F36B7C4DF

Using Windows authentication / HTTP Negotiate

You can also instruct iwr to use the domain credentials of the current user (for example for an intranet service). This is helpful if you want to send requests to an endpoint that wants you to connect via a Windows Authentication provider like NTLM or Kerberos.

Just adding -DefaultCredentials to your iwr will handle the negotiation for you:

iwr http://my-domain.local -UseDefaultCredentials

iwr will make an unauthenticated request which will result in a 401 error, then make another request (or more) for the NTLMSSP (NTLM Secure Service Provider) or SPNEGO (Simple Protected Negotiation) negotiation.

Note that -DefaultCredentials will not work for Basic Auth!

Catching exceptions

Combining all the options from above can lead to errors, so let’s see how we can catch these exceptions.

Something like the following will give an error (invalid option to iwr):

$r = iwr http://httpbin.org/get -SomethingInvalid

However, this will also give an error (syntactically correct command, but server returns a 404):

$r = iwr http://httpbin.org/doesnt-exist

You can catch all kinds of exceptions by wrapping the request into a try-catch block:

try {
    $r = iwr http://httpbin.org/doesnt-exist
} catch {
    $r = $_.Exception
}

This will catch all exceptions. If you want to handle certain exceptions differently, use multiple catch statements.

We can access the exception though the pipeline variable $_. The server response object (obviously only if it is a WebException and not something like a Command exception, ParameterBindException, etc.) can then be accessed via $r.Response.

More information

There are naturally many more options, more scenarios and the whole topic about response parsing I didn’t mention here. Have a look at the official docs for a first overview, then start tinkering.

Have fun!