parsing - python UnicodeEncodeError > How can I simply remove troubling unicode characters?

Question

Heres what I did..

>>> soup = BeautifulSoup (html)
>>> soup
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'xae' in position 96953: ordinal not in range(128)
>>> 
>>> soup.find('div')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'xae' in position 11035: ordinal not in range(128)
>>> 
>>> soup.find('span')
<span id="navLogoPrimary" class="navSprite"><span>amazon.com</span></span>
>>> 

How can I simply remove troubling unicode characters from html ?
Or is there any cleaner solution ?

Answer 1

Try this way: soup = BeautifulSoup (html.decode('utf-8', 'ignore'))