python - How to preserve punctuation marks in Scikit-Learn text CountVectorizer or TfidfVectorizer?

Question

Is there any way for me to preserve punctuation marks of !, ?, " and ' from my text documents using text CountVectorizer or TfidfVectorizer parameters in scikit-learn?

Answer 1

You should customize the token_pattern parameter when you instantiate the vectorizer. For example:

vent = CountVectorizer(token_pattern=r"(?u)ww+|!|?|"|'")