Chapter 1. Tokenizing Text and WordNet Basics

In this chapter, we will cover the following recipes:

• Tokenizing text into sentences
• Tokenizing sentences into words
• Tokenizing sentences using regular expressions
• Training a sentence tokenizer
• Filtering stopwords in a tokenized sentence
• Looking up Synsets for a word in WordNet
• Looking up lemmas and synonyms in WordNet
• Calculating WordNet Synset similarity
• Discovering word collocations