encoding - PHP output showing little black diamonds with a question mark

Question

I'm writing a php program that pulls from a database source. Some of the varchars have quotes that are displaying as black diamonds with a question mark in them (?, REPLACEMENT CHARACTER, I assume from Microsoft Word text).

How can I use php to strip these characters out?

Answer 1

If you see that character (? U+FFFD "REPLACEMENT CHARACTER") it usually means that the text itself is encoded in some form of single byte encoding but interpreted in one of the unicode encodings (UTF8 or UTF16).

If it were the other way around it would (usually) look something like this: ?¤.

Probably the original encoding is ISO-8859-1, also known as Latin-1. You can check this without having to change your script: Browsers give you the option to re-interpret a page in a different encoding -- in Firefox use "View" -> "Character Encoding".

To make the browser use the correct encoding, add an HTTP header like this:

header("Content-Type: text/html; charset=ISO-8859-1");

or put the encoding in a meta tag:

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">

Alternatively you could try to read from the database in another encoding (UTF-8, preferably) or convert the text with iconv().