NLTK - Revision history

Admin: /* All The Words */

2017-08-07T00:22:50Z

All The Words

← Older revision		Revision as of 00:22, 7 August 2017
Line 44:		Line 44:
	Analyze all the words in this poem:		Analyze all the words in this poem:

	<~~source lang="~~pre">		<pre>

	In [4]: words = [w.lower() for w in tokens]		In [4]: words = [w.lower() for w in tokens]
Line 119:		Line 119:
	We can compile a vocabulary by sorting our unique words:		We can compile a vocabulary by sorting our unique words:

	<~~source lang="python"~~>		<pre>
	In [8]: vocab=sorted(set(words))		In [8]: vocab=sorted(set(words))

Line 567:		Line 567:
	'yourself']		'yourself']

	</~~source~~>		</pre>

	==Vowel/Consonant Analysis==		==Vowel/Consonant Analysis==

Admin: /* Vowel/Consonant Analysis */

2017-08-07T00:03:41Z

Vowel/Consonant Analysis

← Older revision		Revision as of 00:03, 7 August 2017
Line 571:		Line 571:
	==Vowel/Consonant Analysis==		==Vowel/Consonant Analysis==

	<~~source lang="python"~~>		<pre>
	In [12]: cvs = [cv for w in words for cv in re.findall(r'[ptksvr][aeiou]', w)]		In [12]: cvs = [cv for w in words for cv in re.findall(r'[ptksvr][aeiou]', w)]

Line 590:		Line 590:
	In [17]: cv_index = nltk.Index(cv_word_pairs)		In [17]: cv_index = nltk.Index(cv_word_pairs)

	</~~source~~>		</pre>

	Now we can print out some words associated with various		Now we can print out some words associated with various
	vowel-consonant combinations:		vowel-consonant combinations:

	<~~source lang="python"~~>		<pre>

	In [18]: cv_index['to']		In [18]: cv_index['to']
Line 759:		Line 759:
	'correct',		'correct',
	'precision']		'precision']
	</~~source~~>		</pre>


	=References=		=References=

Admin: /* Related Software */

2017-08-07T00:03:11Z

Related Software

← Older revision		Revision as of 00:03, 7 August 2017
Line 779:		Line 779:

	Spacy: cython + python + ntlk: https://spacy.io/		Spacy: cython + python + ntlk: https://spacy.io/

			prodi.gy for annotated machine learning models (like Tinder for data): https://prodi.gy/



	[[Category:NLTK]]		[[Category:NLTK]]

Admin: /* References */

2017-08-06T23:54:58Z

References

← Older revision		Revision as of 23:54, 6 August 2017
Line 775:		Line 775:

	lousy but ok: http://www.gilesthomas.com/2010/05/generating-political-news-using-nltk/		lousy but ok: http://www.gilesthomas.com/2010/05/generating-political-news-using-nltk/

			==Related Software==

			Spacy: cython + python + ntlk: https://spacy.io/


	[[Category:NLTK]]		[[Category:NLTK]]

Admin: /* All The Words */

2016-07-15T05:48:47Z

All The Words

← Older revision		Revision as of 05:48, 15 July 2016
Line 44:		Line 44:
	Analyze all the words in this poem:		Analyze all the words in this poem:

	<source lang="~~python~~">		<source lang="pre">

	In [4]: words = [w.lower() for w in tokens]		In [4]: words = [w.lower() for w in tokens]
Line 113:		Line 113:
	',',		',',
	'1956']		'1956']
	</~~source~~>		</pre>

	==Vocabulary==		==Vocabulary==

Admin: /* All The Words */

2016-07-15T05:48:33Z

All The Words

Show changes

Admin at 03:30, 25 September 2015

2015-09-25T03:30:53Z

← Older revision		Revision as of 03:30, 25 September 2015
Line 1,601:		Line 1,601:
	'precision']		'precision']
	</source>		</source>


			=References=

			NLTK book, baby steps: http://www.nltk.org/book/ch01.html

			The Language Archives (resources): https://tla.mpi.nl/resources/data-archive/ {{,}} https://tla.mpi.nl/tools2/tooltype/analysis/page/3/

			NLTK corpora readers list: https://en.wikipedia.org/wiki/User:Alvations/NLTK_cheatsheet/CorporaReaders

			text generator class: https://github.com/kopertop/nltkfun

			multi-part blog series: http://textminingonline.com/dive-into-nltk-part-ii-sentence-tokenize-and-word-tokenize

			lousy but ok: http://www.gilesthomas.com/2010/05/generating-political-news-using-nltk/

			[[Category:NLTK]]

Admin at 20:03, 9 January 2015

2015-01-09T20:03:07Z

← Older revision		Revision as of 20:03, 9 January 2015
Line 1,601:		Line 1,601:
	'precision']		'precision']
	</source>		</source>

	~~=Cool Stuff=~~

	~~Check this out. It's a bit like word association:~~

	~~<pre>~~
	~~In [25]: Word('tasting').get_synsets()[0].common_hypernyms(Word('tasting').get_synsets()[0])~~
	~~Out[25]:~~
	~~[Synset('information.n.02'),~~
	~~Synset('example.n.01'),~~
	~~Synset('abstraction.n.06'),~~
	~~Synset('entity.n.01'),~~
	~~Synset('psychological_feature.n.01'),~~
	~~Synset('sample.n.01'),~~
	~~Synset('cognition.n.01'),~~
	~~Synset('tasting.n.01')]~~

	~~In [27]: z='python';Word(z).get_synsets()[0].common_hypernyms(Word(z).get_synsets()[0])~~
	~~Out[27]:~~
	~~[Synset('chordate.n.01'),~~
	~~Synset('python.n.01'),~~
	~~Synset('living_thing.n.01'),~~
	~~Synset('physical_entity.n.01'),~~
	~~Synset('animal.n.01'),~~
	~~Synset('snake.n.01'),~~
	~~Synset('vertebrate.n.01'),~~
	~~Synset('entity.n.01'),~~
	~~Synset('object.n.01'),~~
	~~Synset('boa.n.02'),~~
	~~Synset('constrictor.n.01'),~~
	~~Synset('organism.n.01'),~~
	~~Synset('reptile.n.01'),~~
	~~Synset('whole.n.02'),~~
	~~Synset('diapsid.n.01')]~~

	~~In [32]: z='bee';Word(z).get_synsets()[0].common_hypernyms(Word(z).get_synsets()[0])~~
	~~Out[32]:~~
	~~[Synset('living_thing.n.01'),~~
	~~Synset('physical_entity.n.01'),~~
	~~Synset('invertebrate.n.01'),~~
	~~Synset('animal.n.01'),~~
	~~Synset('bee.n.01'),~~
	~~Synset('entity.n.01'),~~
	~~Synset('hymenopterous_insect.n.01'),~~
	~~Synset('object.n.01'),~~
	~~Synset('organism.n.01'),~~
	~~Synset('arthropod.n.01'),~~
	~~Synset('insect.n.01'),~~
	~~Synset('whole.n.02')]~~

	~~In [33]: z='thirst';Word(z).get_synsets()[0].common_hypernyms(Word(z).get_synsets()[0])~~
	~~Out[33]:~~
	~~[Synset('physical_condition.n.01'),~~
	~~Synset('abstraction.n.06'),~~
	~~Synset('entity.n.01'),~~
	~~Synset('thirst.n.01'),~~
	~~Synset('state.n.02'),~~
	~~Synset('condition.n.01'),~~
	~~Synset('drive.n.09'),~~
	~~Synset('attribute.n.02')]~~

	~~In [35]: z='death';Word(z).get_synsets()[0].common_hypernyms(Word(z).get_synsets()[0])~~
	~~Out[35]:~~
	~~[Synset('change.n.01'),~~
	~~Synset('happening.n.01'),~~
	~~Synset('event.n.01'),~~
	~~Synset('death.n.01'),~~
	~~Synset('abstraction.n.06'),~~
	~~Synset('psychological_feature.n.01'),~~
	~~Synset('entity.n.01')]~~

	~~</pre>~~

Admin at 07:02, 9 January 2015

2015-01-09T07:02:50Z

← Older revision		Revision as of 07:02, 9 January 2015
Line 1,601:		Line 1,601:
	'precision']		'precision']
	</source>		</source>

			=Cool Stuff=

			Check this out. It's a bit like word association:

			<pre>
			In [25]: Word('tasting').get_synsets()[0].common_hypernyms(Word('tasting').get_synsets()[0])
			Out[25]:
			[Synset('information.n.02'),
			Synset('example.n.01'),
			Synset('abstraction.n.06'),
			Synset('entity.n.01'),
			Synset('psychological_feature.n.01'),
			Synset('sample.n.01'),
			Synset('cognition.n.01'),
			Synset('tasting.n.01')]

			In [27]: z='python';Word(z).get_synsets()[0].common_hypernyms(Word(z).get_synsets()[0])
			Out[27]:
			[Synset('chordate.n.01'),
			Synset('python.n.01'),
			Synset('living_thing.n.01'),
			Synset('physical_entity.n.01'),
			Synset('animal.n.01'),
			Synset('snake.n.01'),
			Synset('vertebrate.n.01'),
			Synset('entity.n.01'),
			Synset('object.n.01'),
			Synset('boa.n.02'),
			Synset('constrictor.n.01'),
			Synset('organism.n.01'),
			Synset('reptile.n.01'),
			Synset('whole.n.02'),
			Synset('diapsid.n.01')]

			In [32]: z='bee';Word(z).get_synsets()[0].common_hypernyms(Word(z).get_synsets()[0])
			Out[32]:
			[Synset('living_thing.n.01'),
			Synset('physical_entity.n.01'),
			Synset('invertebrate.n.01'),
			Synset('animal.n.01'),
			Synset('bee.n.01'),
			Synset('entity.n.01'),
			Synset('hymenopterous_insect.n.01'),
			Synset('object.n.01'),
			Synset('organism.n.01'),
			Synset('arthropod.n.01'),
			Synset('insect.n.01'),
			Synset('whole.n.02')]

			In [33]: z='thirst';Word(z).get_synsets()[0].common_hypernyms(Word(z).get_synsets()[0])
			Out[33]:
			[Synset('physical_condition.n.01'),
			Synset('abstraction.n.06'),
			Synset('entity.n.01'),
			Synset('thirst.n.01'),
			Synset('state.n.02'),
			Synset('condition.n.01'),
			Synset('drive.n.09'),
			Synset('attribute.n.02')]

			In [35]: z='death';Word(z).get_synsets()[0].common_hypernyms(Word(z).get_synsets()[0])
			Out[35]:
			[Synset('change.n.01'),
			Synset('happening.n.01'),
			Synset('event.n.01'),
			Synset('death.n.01'),
			Synset('abstraction.n.06'),
			Synset('psychological_feature.n.01'),
			Synset('entity.n.01')]

			</pre>

Admin at 17:24, 14 May 2014

2014-05-14T17:24:59Z

← Older revision		Revision as of 17:24, 14 May 2014
Line 7:		Line 7:
	Applying it to Allen Ginsberg's poem "America". Mind is blown.		Applying it to Allen Ginsberg's poem "America". Mind is blown.

	=~~Script~~=		=NLTK Basics=

			You will need to always include the following in your NLTK scripts:

	<source lang="python">		<source lang="python">
	from __future__ import division		from __future__ import division
	import nltk, re, pprint		import nltk, re, pprint
			</source>

			==Reading Text==

			===Opening File from URL===

			<source lang="python">
			from urllib import urlopen
			url = "http://www.gutenberg.org/files/2554/2554.txt"
			raw = urlopen(url).read()
			type(raw)
			</source>

	#from ~~urllib import urlopen~~		===Opening File from Disk===
	~~#url~~ = ~~"http://www.gutenberg.org/files/2554/2554.txt"~~
	~~#raw~~ = ~~urlopen(url).read()~~
	~~#type(raw)~~
	~~#len(raw)~~
	~~#raw[:75]~~

			<source lang="python">
	f = open('poems/america.txt')		f = open('poems/america.txt')
	raw = f.read()		raw = f.read()

	tokens = nltk.word_tokenize(raw)		tokens = nltk.word_tokenize(raw)
	~~print type(tokens)~~
	~~print len(tokens)~~
	~~print tokens[:12]~~

	text = nltk.Text(tokens)		text = nltk.Text(tokens)
	~~print type(text)~~
	~~print text[1020:1060]~~
	~~print text.collocations()~~

	words = [w.lower() for w in tokens]		words = [w.lower() for w in tokens]
	~~print type(words)~~
	~~print len(words)~~
	~~print words~~
	vocab = sorted(set(words))		vocab = sorted(set(words))
	~~print type(vocab)~~
	~~print len(vocab)~~
	~~print vocab~~
	</source>		</source>

	=~~The Jump~~-~~Off~~=		=First-Pass Analysis: NLTK Book Chapter 3=

	==~~Word-level Analysis: Consonant/Vowel Stuff~~==		==All The Words==

			Analyze all the words in this poem:

	<source lang="python">		<source lang="python">
Line 957:		Line 954:
	',',		',',
	'1956']		'1956']
			</source>

			==Vocabulary==

			We can compile a vocabulary by sorting our unique words:

			<source lang="python">
	In [8]: vocab=sorted(set(words))		In [8]: vocab=sorted(set(words))

Line 1,405:		Line 1,408:
	'yourself']		'yourself']

	~~In [12]: words = [w.lower() for w in tokens]~~		</source>

	~~KeyboardInterrupt~~		==Vowel/Consonant Analysis==

	~~In [12]:~~

	~~In [12]:~~

			<source lang="python">
	In [12]: cvs = [cv for w in words for cv in re.findall(r'[ptksvr][aeiou]', w)]		In [12]: cvs = [cv for w in words for cv in re.findall(r'[ptksvr][aeiou]', w)]

Line 1,430:		Line 1,430:

	In [17]: cv_index = nltk.Index(cv_word_pairs)		In [17]: cv_index = nltk.Index(cv_word_pairs)

			</source>

			Now we can print out some words associated with various
			vowel-consonant combinations:

			<source lang="python">

	In [18]: cv_index['to']		In [18]: cv_index['to']
Line 1,489:		Line 1,496:
	In [21]: cv_index['po']		In [21]: cv_index['po']
	Out[21]: ['poem', 'point.', 'responsibility', 'flowerpots', 'power']		Out[21]: ['poem', 'point.', 'responsibility', 'flowerpots', 'power']

	~~In [22]: cv_index['ir']~~
	~~Out[22]: []~~

	In [23]: cv_index['ri']		In [23]: cv_index['ri']
Line 1,542:		Line 1,546:
	'factories',		'factories',
	'america']		'america']

	~~In [24]: cv_index['']~~

	~~KeyboardInterrupt~~

	In [24]: cfd.tabulate()		In [24]: cfd.tabulate()