console - How to do proper Unicode and ANSI output redirection on cmd.exe?

Question

If you are doing automation on windows and you are redirecting the output of different commands (internal cmd.exe or external, you'll discover that your log files contains combined Unicode and ANSI output (meaning that they are invalid and will not load well in viewers/editors).

Is it is possible to make cmd.exe work with UTF-8? This question is not about display, s about stdin/stdout/stderr redirection and Unicode.

I am looking for a solution that would allow you to:

• redirect the output of the internal commands to a file using UTF-8
• redirect output of external commands supporting Unicode to the files but encoded as UTF-8.

If it is impossible to obtain this kind of consistence using batch files, is there another way of solving this problem, like using python scripting for this? In this case, I would like to know if it is possible to do the Unicode detection alone (user using the scripting should not remember if the called tools will output Unicode or not, it will just expect to convert the output to UTF-8.

For simplicity we'll assume that if the tool output is not-Unicode it will be considered as UTF-8 (no codepage conversion).

Answer 1

You can use chcp to change the active code page. This will be used for redirecting text as well:

chcp 65001

Keep in mind, though, that this will have no effect if cmd was started with the /u switch which forces Unicode (UTF-16 in this case) redirection output. If that switch is active then all output will be in UTF-16LE, regardless of the codepage set with chcp.

Also note that the console will be unusable for interactive output when set to Raster Fonts. I'm getting fun error messages in that case:

C:UsersJohannes R?sselDocuments>x
Active code page: 65001

The system cannot write to the specified device.

So either use a sane setup (TrueType font for the console) or don't pull this stunt when using the console interactively and having a path that contains non-ASCII characters.