unicode - PowerShell script runs when pasted into the PowerShell window, but not when run from shortcut

Question

I am trying to put together a script that will convert several Excel files into PDF files. I found a link to one online that works.

$path = Read-Host -Prompt 'Input Directory Path and Press Enter'
$xlFixedFormat = “Microsoft.Office.Interop.Excel.xlFixedFormatType” -as [type]
$excelFiles = Get-ChildItem -Path $path -include *.xls, *.xlsx -recurse
$objExcel = New-Object -ComObject excel.application
$objExcel.visible = $false
foreach($wb in $excelFiles)
{
    $filepath = Join-Path -Path $path -ChildPath ($wb.BaseName + “.pdf”)
    $workbook = $objExcel.workbooks.open($wb.fullname, 3)
    $workbook.Saved = $true
    “saving $filepath”
    $workbook.ExportAsFixedFormat($xlFixedFormat::xlTypePDF, $filepath)
    $objExcel.Workbooks.close()
}
$objExcel.Quit()

If I copy and paste this into a PowerShell window, the program runs as intended. However, when I attempted to make a shortcut to run the program, I get several errors (the file is saved as a .ps1).

This is the path and arguments I made when setting up the shortcut:

C:WindowsSystem32WindowsPowerShellv1.0powershell.exe -noexit -ExecutionPolicy Bypass -File C:[File Path]

This is the error message I get:

At C:Userscbeals.ENVIROTECHDocumentsTestConvertExcelToPDF.ps1:8 char:62
+  $filepath = Join-Path -Path $path -ChildPath ($wb.BaseName + a€?.pdf ...
+                                                              ~
You must provide a value expression following the '+' operator.
At C:Userscbeals.ENVIROTECHDocumentsTestConvertExcelToPDF.ps1:8 char:63
+ ... lepath = Join-Path -Path $path -ChildPath ($wb.BaseName + a€?.pdfa€)
+                                                               ~~~~~~~~~~
Unexpected token 'a€?.pdfa€' in expression or statement.
At C:Userscbeals.ENVIROTECHDocumentsTestConvertExcelToPDF.ps1:8 char:62
+  $filepath = Join-Path -Path $path -ChildPath ($wb.BaseName + a€?.pdf ...
+                                                              ~
Missing closing ')' in expression.
At C:Userscbeals.ENVIROTECHDocumentsTestConvertExcelToPDF.ps1:7 char:1
+ {
+ ~
Missing closing '}' in statement block or type definition.
At C:Userscbeals.ENVIROTECHDocumentsTestConvertExcelToPDF.ps1:8 char:73
+ ... lepath = Join-Path -Path $path -ChildPath ($wb.BaseName + a€?.pdfa€)
+                                                                         ~
Unexpected token ')' in expression or statement.
At C:Userscbeals.ENVIROTECHDocumentsTestConvertExcelToPDF.ps1:14 char:1
+ }
+ ~
Unexpected token '}' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : ExpectedValueExpression

Why would this be failing?

Answer 1

To clarify:

• It is perfectly fine to use Unicode (non-ASCII-range) quotation marks such as “ in PowerShell - see the bottom section.

• However, in order to use such characters in script files, these files must use a Unicode character encoding such as UTF-8 or UTF-16LE ("Unicode").

Your problem was that your script file was saved as UTF-8 without a BOM, which causes Windows PowerShell (but not PowerShell Core) to misinterpret it, because it defaults to "ANSI" encoding, i.e., the single-byte legacy encoding associated with the legacy system locale (e.g., Windows-1252 in the US and Western Europe), which PowerShell calls Default.

While replacing the Unicode quotation marks with their ASCII counterparts solves the immediate problem, any other non-ASCII-range characters in the script would continue to be misinterpreted.

• The proper solution is to re-save the file as UTF-8 with BOM.
• It is a good habit to form to routinely save all PowerShell scripts (source code) as UTF-8 with BOM, because that ensures that they're interpreted the same, irrespective of any given computer's system locale.
  • See this related answer, which shows how to configure Visual Studio Code accordingly.

---

To demonstrate the specific problem:

• “, the LEFT DOUBLE QUOTATION MARK (U+201C) Unicode character, is encoded as three bytes in UTF-8 format: 0xE2 0x80 0x9C.

• You can verify this via the output from '“' | Format-Hex -Encoding Utf8 (only the byte sequence matters here; the printed characters. on the right are not representative in this case).

• When Windows PowerShell reads this sequence as "ANSI"-encoded, it considers each byte a character in its own right, which is why you saw three characters for the single “ in your output, namely a€?.

• You can verify this with [Text.Encoding]::Default.GetString([byte[]] (0xE2, 0x80, 0x9C)) (from PowerShell Core, use [Text.Encoding]::GetEncoding([cultureinfo]::CurrentCulture.TextInfo.ANSICodePage).GetString([byte[]] (0xE2, 0x80, 0x9C))).

---

Interchangeable use of ASCII-range and Unicode quotation marks, dashes, and whitespace in PowerShell:

In a properly encoded input file, PowerShell allows interchangeable use of the following quotation and punctuation characters; e.g., "hi", ”hi” and even "hi? are equivalent.

• Double quotes:

  • " (ASCII-range) - QUOTATION MARK (U+0022)

  • “ - LEFT DOUBLE QUOTATION MARK (U+201C)

  • ” - RIGHT DOUBLE QUOTATION MARK (U+201D)

  • ? - DOUBLE LOW-9 QUOTATION MARK (U+201E)

  • But not: ? - DOUBLE HIGH-REVERSED-9 QUOTATION MARK (U+201F), even though its single-quote counterpart is recognized - see this GitHub issue.

• Single quotes:

  • ' - (ASCII-range) APOSTROPHE (U+0027)
  • ‘ - LEFT SINGLE QUOTATION MARK (U+2018)
  • ’ - RIGHT SINGLE QUOTATION MARK (U+2019)
  • ? - SINGLE LOW-9 QUOTATION MARK (U+201A)
  • ? - SINGLE HIGH-REVERSED-9 QUOTATION MARK (U+201B)

• Dashes (strictly speaking, the ASCII-range "dash" is a hyphen):

  • - (ASCII-range) - HYPHEN-MINUS (U+002D)
  • – - EN DASH (U+2013)
  • — - EM DASH (U+2014)
  • ― - HORIZONTAL BAR (U+2015)

• Whitespace:

  • Note: The source-code location linked to below doesn't define equivalent whitespace characters explicitly (unlike quotation marks and dashes). The following was gleaned from experiments based on Unicode character descriptions and may be incomplete. Characters outside the Unicode BMP (basic multilingual plane), i.e. those with code point that won't fit into the 16-bit code units .NET uses to represent characters, were excluded.

  • Intra-line whitespace:

    • Note: Space-character and tab-character variations can interchangeably serve as syntactic word separators. Among the space-character variations, only the U+200B (ZERO WIDTH SPACE) character is not considered a space syntactically.

    • Spaces:

      • (ASCII-range space character) U+0020 (SPACE)
      • U+00A0 (NO-BREAK SPACE)
      • ? U+2002 (EN SPACE)
      • ? U+2003 (EM SPACE)
      • ? U+2004 (THREE-PER-EM SPACE)
      • ? U+2005 (FOUR-PER-EM SPACE))
      • ? U+2006 (SIX-PER-EM SPACE)
      • ? U+2007 (FIGURE SPACE)
      • ? U+2008 (PUNCTUATION SPACE)
      • ? U+2009 (THIN SPACE)
      • ? U+200A (HAIR SPACE)
      • ? U+202F (NARROW NO-BREAK SPACE)
      • ? U+205F (MEDIUM MATHEMATICAL SPACE)
      • 　 U+3000 (IDEOGRAPHIC SPACE)

    • Tabulators (shown as escape sequences due to not being directly printable here):

      • "`t" (ASCII-range tab character) - U+0009 (CHARACTER TABULATION)
      • "`v" (ASCII-range vertical-tab character) - U+000B (LINE TABULATION))

  • Line-separating whitespace:

    • (ASCII-range LF) U+000A (LINE FEED)
    • (ASCII-range CR) U+000D (CARRIAGE RETURN)

Note:

• Important: The above describes interchangeable syntactic use of these characters; if you use such characters in identifiers (which you shouldn't) or in strings^[1], they are not treated the same.

• The above was in part gleaned from the source code on GitHub (class SpecialCharacters in file parserutils.cs).

---

^{[1] There are limited exceptions: given that PowerShell's -eq operator compares string using the invariant culture rather than performing ordinal comparison, space-character variations may be treated the same in string comparisons, depending on the host platform; e.g., "foo bar" -eq "foo`u{a0}bar" yields $true on macOS and Linux (but not Windows!), because the regular ASCII-range space is considered equal to the no-break space (U+00A0) there.}