nltk python how to tokenize text

GelatinousMustard answered on October 29, 2021 Popularity 7/10 Helpfulness 2/10

answer nltk python how to tokenize text

related python nltk tokenize

nltk python how to tokenize text

Comment

>>> tokens = word_tokenize(raw)
>>> type(tokens)
<class 'list'>
>>> len(tokens)
254354
>>> tokens[:10]
['The', 'Project', 'Gutenberg', 'EBook', 'of', 'Crime', 'and', 'Punishment', ',', 'by']

Popularity 7/10 Helpfulness 2/10 Language python

Source: www.nltk.org

Tags: nltk python text tokenize

Link to this answer
Share Copy Link

Contributed on Feb 21 2022

GelatinousMustard

0 Answers Avg Quality 2/10

Closely Related Answers

python nltk tokenize

Comment

>>> import nltk
>>> sentence = """At eight o'clock on Thursday morning
... Arthur didn't feel very good."""
>>> tokens = nltk.word_tokenize(sentence)
>>> tokens
['At', 'eight', "o'clock", 'on', 'Thursday', 'morning',
'Arthur', 'did', "n't", 'feel', 'very', 'good', '.']

Popularity 10/10 Helpfulness 7/10 Language python

Source: stackoverflow.com

Tags: nltk

Link to this answer
Share Copy Link