nlp - Stanford coreNLP - split words ignoring apostrophe

Question

I'm trying to split a sentence into words using Stanford coreNLP . I'm having problem with words that contains apostrophe.

For example, the sentence: I'm 24 years old.

Splits like this: [I] ['m] [24] [years] [old]

Is it possible to split it like this using Stanford coreNLP?: [I'm] [24] [years] [old]

I've tried using tokenize.whitespace, but it doesn't split on other punctuation marks like: '?' and ','

Answer 1

How about if you just re-concatenate tokens that are split by an apostrophe?

Here's an implementation in Java:

public static List<String> tokenize(String s) {
    PTBTokenizer<CoreLabel> ptbt = new PTBTokenizer<CoreLabel>(
            new StringReader(s), new CoreLabelTokenFactory(), "");
    List<String> sentence = new ArrayList<String>();
    StringBuilder sb = new StringBuilder();
    for (CoreLabel label; ptbt.hasNext();) {
        label = ptbt.next();
        String word = label.word();
        if (word.startsWith("'")) {
            sb.append(word);
        } else {
            if (sb.length() > 0)
                sentence.add(sb.toString());
            sb = new StringBuilder();
            sb.append(word);
        }
    }
    if (sb.length() > 0)
        sentence.add(sb.toString());
    return sentence;
}

public static void main(String[] args) {
    System.out.println(tokenize("I'm 24 years old."));  // [I'm, 24, years, old, .]
}