rest - PowerShell Invoke-RestMethod Umlauts issues with UTF-8 and Windows-1252

Question

Upon executing Confluence REST API calls I get back a response encoded in UTF-8. However, when I export the results with either Out-File or Export-CSV even with the -Encoding utf8 parameter German Umlauts are not correctly represented. For example, 'ü' is still '??'.

From what I could gather it's due to the fact that PowerShell 5.1 natively relies on Windows-1252. I verified that Umlauts are preserved when using PowerShell Core by executing

[psobject].Assembly.GetTypes() | Where-Object { $_.Name -eq 'ClrFacade'} | ForEach-Object { $_.GetMethod('GetDefaultEncoding', [System.Reflection.BindingFlags]'nonpublic,static').Invoke($null, @()) }

Even changing the script file itself to use the encoding UTF-8 with BOM or Windows-1252 does not preserve Umlauts, neither in the PowerShell nor exportet output.

Do you know of any way to tell PowerShell 5.1 to preserve Umlauts while executing the REST call?

I cannot use PowerShell core as further operations require cmdlets which do net yet exist for PowerShell Core.

Thanks!

Answer 1

As discussed in the comments, it looks like the Confluence API encodes http responses using UTF8, but does not include the "Content-Type" header to indicate that.

The HTTP specification for the charset parameter says that in the absence of this header, the client should assume it's encoded with ISO-8859-1 character set, so what is happening in your request is something like this:

# server (Confluence API) encodes response text using utf8
PS> $text = "ü";
PS> $bytes = [System.Text.Encoding]::UTF8.GetBytes($text);
PS> write-host $bytes;
195 188

# client (Invoke-RestMethod) decodes bytes as ISO-8859-1
PS> $text = [System.Text.Encoding]::GetEncoding("ISO-8859-1").GetString($bytes);
PS> write-host $text;
??

Given that you can't control what the server sends, you'll either need to capture the raw bytes yourself (e.g. using System.Net.Http.HttpClient) and decode them using UTF8, or modify the existing response to compensate for the encoding mismatch (e.g. below).

PS> $text = "??"
PS> $bytes = [System.Text.Encoding]::GetEncoding("ISO-8859-1").GetBytes($text)
PS> $text = [System.Text.Encoding]::UTF8.GetString($bytes)
PS> write-host $text
ü

Note that if you use the -Outfile parameter of Invoke-RestMethod it presumably streams the response bytes directly to disk without decoding or encoding them, so the resultant file already contains utf8 $bytes rather than utf8 $bytes -> string decoded using ISO-8859-1 -> file bytes encoded using utf8