printing UTF-8 in Python 3 using Sublime Text 3

Question

I have this Python3 code to attempt to read and print from a utf-8 encoded file:

f = open('mybook.txt', encoding='utf-8')
for line in f:
    print(line)

When I build using Sublime Text 3 I get the following error:

UnicodeEncodeError: 'ascii' codec can't encode character 'u2019' in position 18: ordinal not in range(128)

However, it works file when I just execute my code in the terminal with python3.

My build configuration is

{
"cmd": ["/usr/local/bin/python3", "$file"]
, "selector": "source.python"
, "file_regex": "file "(...*?)", line ([0-9]+)"
}

If I change it to:

f = open('mybook.txt', encoding='utf-8')
for line in f:
    print(line.encode('utf-8'))

Then it does print the utf-8 encoded byte string (I think that's what's happening).

b'Hello
'
b'xc2xabxe2x80xa2
'
b'Goodbye'

But I also don't know how to go from this to printing the unicode characters on the screen...

Also, if I try changing this env variable as per A python program fails to execute in sublime text 3, but success in bash it still doesn't fix it.

Answer 1

The answer was actually in the question linked in your question - PYTHONIOENCODING needs to be set to "utf-8". However, since OS X is silly and doesn't pick up on environment variables set in Terminal or via .bashrc or similar files, this won't work in the way indicated in the answer to the other question. Instead, you need to pass that environment variable to Sublime.

Luckily, ST3 build systems (I don't know about ST2) have the "env" option. This is a dictionary of keys and values passed to exec.py, which is responsible for running build systems without the "target" option set. As discussed in our comments above, I indicated that your sample program worked fine on a UTF-8-encoded text file containing non-ASCII characters when run with ST3 (Build 3122) on Linux, but not with the same version run on OS X. All that was necessary to get it to run was to change the build system to enclude this line:

"env": {"PYTHONIOENCODING": "utf8"},

I saved the build system, hit ?B, and the program ran fine.

BTW, if you'd like to read exec.py, or Packages/Python/Python.sublime-build, or any other file packed up in a .sublime-package archive, install PackageResourceViewer via Package Control. Use the "Open Resource" option in the Command Palette to pick individual files, or "Extract Package" (both are preceded by "PackageResourceViewer:", or prv using fuzzy search) to extract an entire package to your Packages folder, which is accessed by selecting Sublime Text → Preferences → Browse Packages… (just Preferences → Browse Packages… on other operating systems). It is located on your hard drive in the following location:

• Linux: ~/.config/sublime-text-3/Packages
• OS X: ~/Library/Application Support/Sublime Text 3/Packages
• Windows Regular Install: C:UsersYourUserNameAppDataRoamingSublime Text 3Packages
• Windows Portable Install: InstallationFolderSublime Text 3DataPackages

Once files are saved to your Packages folder (if you just view them via the "Open Resource" option and close without changing or saving them, they won't be), they will override the identically-named file contained within the .sublime-package archive. So, for instance, if you want to edit the default Python.sublime-build file in the Python package, your changes will be saved as Packages/Python/Python.sublime-build, and when you choose the Python build system from the menu, it will only use your version.