Ofn\SrSSKrSSKrSSKrSSKJrJrJr SSKJ r SSK J r SSK J r SSKJr SSKJr SS KJrJr SS KJr SS KJr SS KJr SS KJrJrJr \"S/SQ5r"SS5r "SS5r!"SS5r""SS5r#"SS\#5r$Sr%\&S:Xa\%"5 /SQr'g)a  This module brings together a variety of NLTK functionality for text analysis, and provides simple, interactive interfaces. Functionality includes: concordancing, collocation discovery, regular expression search over tokenized strings, and distributional similarity. N)Counter defaultdict namedtuple)reduce)log)BigramCollocationFinder)MLE)padded_everygram_pipeline)BigramAssocMeasures f_measure)ConditionalFreqDist)FreqDist) sent_tokenize)LazyConcatenation cut_string tokenwrapConcordanceLine)leftqueryrightoffset left_print right_printlinecZ\rSrSrSr\S5rSSS4SjrSrSr S S jr S S jr S r g) ContextIndex'z A bidirectional index between words and their 'contexts' in a text. The context of a word is usually defined to be the words that occur in a fixed window around the word; but other definitions may also be used by providing a custom context function. cUS:waXS- R5OSnU[U5S- :waXS-R5OSnX#4$)z;One left token and one right token, normalized to lowercaser*START**END*)lowerlen)tokensirrs +/usr/lib/python3/dist-packages/nltk/text.py_default_contextContextIndex._default_context/sM)*Qv!e}""$I)*c&kAo)=1u ##%7}NcU$Nxs r&ContextIndex.6sQr)cN^^UTlTTlU(aUTlOTRTlU(a!TVs/sHoS"U5(dMUPM snm[ UU4Sj[ T555Tl[ UU4Sj[ T555Tlgs snf)Nc3n># UH*upTRU5TRTU54v M, g7fr+)_key _context_func.0r%wselfr$s r& (ContextIndex.__init__..?s1% FWdaTYYq\4--fa8 9FW25c3n># UH*upTRTU5TRU54v M, g7fr+)r4r3r5s r&r9r:Bs1% FWdaT   *DIIaL 9FWr;)r3_tokensr4r'CFD enumerate_word_to_contexts_context_to_words)r8r$ context_funcfilterkeyts`` r&__init__ContextIndex.__init__6s  !-D !%!6!6D  !'5A6!9a5F!$% FOPVFW% " "%% FOPVFW% "  6s B"B"cUR$)zW :rtype: list(str) :return: The document that this context index was created from. r=r8s r&r$ContextIndex.tokensF ||r)cURU5n[URU5n0nURR5HupE[ U[U55X4'M U$)z Return a dictionary mapping from words to 'similarity scores,' indicating how often these two words occur in the same context. )r3setr@itemsr )r8word word_contextsscoresr7 w_contextss r&word_similarity_dict!ContextIndex.word_similarity_dictNs_ yyD22489 !3399;MA!-ZAFI< r)c0[[5nURURU5HOnURUH9nXQ:wdM X5==URUUURUU-- ss'M; MQ [ X3R SS9SU$)NT)rDreverse)rintr@r3rAsortedget)r8rPnrRcr7s r& similar_wordsContextIndex.similar_words]sS!'' $8A++A.9I..q1$7$:P:PQR:STU:VVI/9 f**d;BQ??r)c^^UVs/sHnTRU5PM nnUVs/sHn[TRU5PM nn[[ U55Vs/sHoTU(aMXPM nn[ [R U5mU(a"U(a[SSRU55eT(d [5$[UU4SjU55nU$s snfs snfs snf)ag Find contexts where the specified words can all appear; and return a frequency distribution mapping each context to the number of times that context was used. :param words: The words used to seed the similarity search :type words: str :param fail_on_unknown: If true, then raise a value error if any of the given words do not occur at all in the index. z%The following word(s) were not found: c3d># UH%nTRUHo"T;dM Uv M M' g7fr+)r@)r6r7r\commonr8s r&r9/ContextIndex.common_contexts..|s. a$*@*@*CQF{*C5s0 0) r3rNr@ranger#r intersection ValueErrorjoinr) r8wordsfail_on_unknownr7contextsr%emptyfdrbs ` @r&common_contextsContextIndex.common_contextsgs(--u!1u- -1;  @r)rcL\rSrSrSrS4SjrSrSrSrS Sjr S S jr S r g )ConcordanceIndexzg An index that can be used to look up the offset locations at which a given word occurs in a document. cU$r+r,r-s r&r/ConcordanceIndex.sQr)cXlX l[[5Ul[ U5H4up4URU5nURUR U5 M6 g)a Construct a new concordance index. :param tokens: The document (list of tokens) that this concordance index was created from. This list can be used to access the context of a given word occurrence. :param key: A function that maps each token to a normalized version that will be used as a key in the index. E.g., if you use ``key=lambda s:s.lower()``, then the index will be case-insensitive. N)r=r3rlist_offsetsr?append)r8r$rDindexrPs r&rFConcordanceIndex.__init__s\   D#D) L$V,KE99T?D MM$  & &u --r)cUR$)z[ :rtype: list(str) :return: The document that this concordance index was created from. rIrJs r&r$ConcordanceIndex.tokensrLr)cBURU5nURU$)z :rtype: list(int) :return: A list of the offset positions at which the given word occurs. If a key function was specified for the index, then given word's key will be looked up. )r3rr8rPs r&offsetsConcordanceIndex.offsetss yy}}T""r)c\S[UR5[UR54-$)Nz+)r#r=rrJs r&__repr__ConcordanceIndex.__repr__s-<    @   r)c ^[U[5(aUnOU/nSRU5n[SU55nX%- S- S-nUS-n/nUR US5n [ USS5HEupUR U5V s1sH oU - S- iM n n [ U RU 55n MG U (aU Hn SRURX[U5-5n UR[SX- 5U nURU [U5-X-n[SRU5U*5RU5n[SRU5U5nSRUU U/5n[UU UU UUU5nURU5 M U$s sn f)zs Find all concordance lines given the query word. Provided with a list of words, these will be found as a phrase. r`c3`# UH$n[R"U5(aM Sv M& g7f)rN) unicodedata combining)r6chars r&r94ConcordanceIndex.find_concordance..sUzt9N9Nt9Tzs. .rrN) isinstancer~rgsumrr?rYrer=r#maxrrjustrr)r8rPwidthphrase phrase_str phrase_len half_widthcontextconcordance_listrr%r word_offsets query_word left_context right_contextrr line_printconcordance_lines r&find_concordance!ConcordanceIndex.find_concordances dD ! !FVFXXf% UzUU (1,2 1*,,vay) ,GA9=d9KL9KvQJN9KLL\66w?@G-  XXdll13v;&GH #||C1;,?!D $ QV_q{ K '(> LRR )-)@*M  XXz:{&KL #2 !$ !''(89-. 5Ms F*cURXS9nU(d [S5 g[U[U55n[SUS[U5S35 [ USU5HupV[UR 5 M g)aa Print concordance lines given the query word. :param word: The target word or phrase (a list of strings) :type word: str or list :param lines: The number of lines to display (default=25) :type lines: int :param width: The width of each line, in characters (default=80) :type width: int :param save: The option to save the concordance. :type save: bool )rz no matchesz Displaying z of z matches:N)rprintminr#r?r)r8rPrlinesrr%rs r&print_concordance"ConcordanceIndex.print_concordances} 000C , s#345E Kwd3/?+@*AK L'01A&51I'J#&++,(Kr))r3rr=N)P)r) rqrrrsrtrurFr$rrrrrwr,r)r&ryrys+ $/.4# . `-r)ryc$\rSrSrSrSrSrSrg) TokenSearcheria A class that makes it easier to use regular expressions to search over tokenized strings. The tokenized string is converted to a string where tokens are marked with angle brackets -- e.g., ``''``. The regular expression passed to the ``findall()`` method is modified to treat angle brackets as non-capturing parentheses, in addition to matching the token boundaries; and to have ``'.'`` not match the angle brackets. c>SRSU55Ulg)Nc32# UH nSU-S-v M g7f)<>Nr,)r6r7s r&r9)TokenSearcher.__init__.. s:6aC!GcM6s)rg_raw)r8r$s r&rFTokenSearcher.__init__ sGG:6:: r)c[R"SSU5n[R"SSU5n[R"SSU5n[R"SSU5n[R"XR5nUH<nUR S5(aMUR S5(dM3[ S 5e UVs/sHo3S S RS 5PM nnU$s snf) ak Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> from nltk.text import TokenSearcher >>> from nltk.book import text1, text5, text9 >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str z\srrz(?:<(?:rz)>)z (?]z$Bad regexp for TokenSearcher.findallrz><)resubfindallr startswithendswithrfsplitr8regexphitshs r&rTokenSearcher.findalls0r6*i0eV, ff5zz&)),A<<$$C !GHH .22T!B d#T2 3sC()rN)rqrrrsrtrurFrrwr,r)r&rrs;'r)rc\rSrSrSrSrSSjrSrSrSSjr SS jr SS jr SS jr S r S rSrSSjrSSjrSrS SjrS!SjrSrSrSr\R2"S5rSrSrSrSrg)"Texti9a A wrapper around a sequence of simple (string) tokens, which is intended to support initial exploration of texts (via the interactive console). Its methods perform a variety of analyses on the text's contexts (e.g., counting, concordancing, collocation discovery), and display the results. If you wish to write a program which makes use of these analyses, then you should bypass the ``Text`` class, and use the appropriate analysis function or class directly instead. A ``Text`` is typically initialized from a given document or corpus. E.g.: >>> import nltk.corpus >>> from nltk.text import Text >>> moby = Text(nltk.corpus.gutenberg.words('melville-moby_dick.txt')) TNc&UR(a [U5nXlU(aX lgSUSS;a5USSR S5nSR SUSU55UlgSR SUSS55S -Ulg) zV Create a Text object. :param tokens: The source text. :type tokens: sequence of str ]Nrpr`c38# UHn[U5v M g7fr+strr6toks r&r9 Text.__init__..bs C]cS]rc38# UHn[U5v M g7fr+rrs r&r9rds @ZcSZrz...) _COPY_TOKENSr~r$namerrg)r8r$rends r&rF Text.__init__Ss   &\F I F3BK "+##C(C CVAc] CCDI @VBQZ @@5HDIr)c URU$r+)r$)r8r%s r& __getitem__Text.__getitem__js{{1~r)c,[UR5$r+)r#r$rJs r&__len__ Text.__len__ms4;;r)cSUR;a[URSS9UlURR XU5$)a| Prints a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word or phrase (a list of strings) :type word: str or list :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` _concordance_indexc"UR5$r+r"ss r&r/"Text.concordance.. 1779r)rD)__dict__ryr$rrr8rPrrs r& concordanceText.concordancetsB t}} 4&6 !4'D #&&88eLLr)cSUR;a[URSS9UlURR X5SU$)a~ Generate a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word or phrase (a list of strings) :type word: str or list :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` rc"UR5$r+rrs r&r/'Text.concordance_list..rr)rN)rryr$rrrs r&rText.concordance_listsG t}} 4&6 !4'D #&&77DVeLLr)c^SUR;a URU:XaURU:XdXlX lSSKJn UR S5m[ R"URU5nURS5 URU4Sj5 [5n[URURU55UlUR$)a Return collocations derived from the text, ignoring stopwords. >>> from nltk.book import text4 >>> text4.collocation_list()[:2] [('United', 'States'), ('fellow', 'citizens')] :param num: The maximum number of collocations to return. :type num: int :param window_size: The number of tokens spanned by a collocation (default=2) :type window_size: int :rtype: list(tuple(str, str)) _collocationsr) stopwordsenglishrcT>[U5S:=(d UR5T;$)N)r#r")r7 ignored_wordss r&r/'Text.collocation_list..s s1vz/WQWWY-=W/Wr))r_num _window_size nltk.corpusrrhr from_wordsr$apply_freq_filterapply_word_filterr r~nbestlikelihood_ratior)r8num window_sizerfinderbigram_measuresrs @r&collocation_listText.collocation_lists t}} , S !![0I +  .%OOI6M,77 [QF  $ $Q '  $ $%W X13O!% _==sC"D !!!r)cURX5VVs/sH up4US-U-PM nnn[[USS95 gs snnf)a Print collocations derived from the text, ignoring stopwords. >>> from nltk.book import text4 >>> text4.collocations() # doctest: +NORMALIZE_WHITESPACE United States; fellow citizens; years ago; four years; Federal Government; General Government; American people; Vice President; God bless; Chief Justice; one another; fellow Americans; Old World; Almighty God; Fellow citizens; Chief Magistrate; every citizen; Indian tribes; public debt; foreign nations :param num: The maximum number of collocations to print. :type num: int :param window_size: The number of tokens spanned by a collocation (default=2) :type window_size: int r`; ) separatorN)rrr)r8rrw1w2collocation_stringss r& collocationsText.collocationssM()-(=(=c(O (OfbBHrM(O   i+t<= sAc8URRU5$)z: Count the number of times this word appears in the text. )r$countrs r&r  Text.count{{  &&r)c8URRU5$)zA Find the index of the first occurrence of the word in the text. )r$rrs r&r Text.indexrr)c[er+)NotImplementedError)r8methods r& readabilityText.readabilitys!!r)c^^^SUR;a[URSSS9UlTR 5mURR mTTR 5;am[TT5m[UUU4SjTR 555nURU5VVs/sHupEUPM nnn[[U55 g[S5 gs snnf)a> Distributional similarity: find other words which appear in the same contexts as the specified word; list most similar words first. :param word: The word used to seed the similarity search :type word: str :param num: The number of words to generate (default=20) :type num: int :seealso: ContextIndex.similar_words() _word_context_indexc"UR5$r+)isalphar-s r&r/Text.similar..s aiikr)c"UR5$r+rrs r&r/rs r))rCrDc3b># UH$nTUHnUT;dM UT:XaMUv M M& g7fr+r,)r6r7r\rjwcirPs r&r9Text.similar..s@)AQA=*+d)s // /z No matchesN) rrr$rr"r@ conditionsrNr most_commonrr) r8rPrrlr7_rhrjrs ` @@r&similar Text.similars ! 5'3 $9?R(D $ zz|&&88 3>># #3t9~H)B $&>>##67#641Q#6E7 )E" # , 8s1C#cSUR;a[URSS9UlURR US5nU(d [ S5 gUR U5VVs/sHupEUPM nnn[ [SU555 gs snnf![an[ U5 SnAgSnAff=f)a Find contexts where the specified words appear; list most frequent common contexts first. :param words: The words used to seed the similarity search :type words: str :param num: The number of words to generate (default=20) :type num: int :seealso: ContextIndex.common_contexts() rc"UR5$r+rrs r&r/&Text.common_contexts..rr)rTzNo common contexts were foundc36# UHupUS-U-v M g7f)r"Nr,)r6rrs r&r9'Text.common_contexts..!sLO&"S2 OsN) rrr$rrmrr!rrf)r8rhrrlr7r"ranked_contextses r&rmText.common_contexts s ! 5'3 !4(D $ ))99%FB56131D"E1D11D"EiLOLLM#F  !HH s/.B"B"/B=B"B"" C, B<<Cc SSKJn U"X5 g)z Produce a plot showing the distribution of the words through the text. Requires pylab to be installed. :param words: The words to be plotted :type words: list(str) :seealso: nltk.draw.dispersion_plot() r)dispersion_plotN) nltk.drawr.)r8rhr.s r&r.Text.dispersion_plot&s .$r)cT[X!5up4[US9nURX45 U$)N)order)r r fit)r8tokenized_sentsr[ train_data padded_sentsmodels r&_train_default_ngram_lmText._train_default_ngram_lm3s)#zrrN)rrgr$r_tokenized_sentshasattrrsysstderrr8r;r#r?generaterr) r8lengthr=r>sentgenerated_tokensidxtokenprefix output_strs r&rC Text.generate9sU")6chht{{6K(L! (LJJsO(L! t-.. +#** ="&">">%%#?#D z>>>z"#f,'##,,[-  E>F? ''. 1 K"#f,/8)$s*Ri(8&(ABB  j9! sE c<UR5R"U6$)zK See documentation for FreqDist.plot() :seealso: nltk.prob.FreqDist.plot() )vocabplot)r8argss r&rN Text.plotgs zz|  $''r)cZSUR;a[U5UlUR$)z :seealso: nltk.prob.FreqDist _vocab)rrrRrJs r&rM Text.vocabns% 4== ("4.DK{{r)cSUR;a[U5UlURRU5nUVs/sHnSR U5PM nn[ [ US55 gs snf)aC Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> from nltk.book import text1, text5, text9 >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str _token_searcherr`rN)rrrUrrgrrrs r&r Text.findallwsc. DMM 1#0#6D ##++F3%)*T T* id#$+sA3z \w+|[\.\!\?]cUS- nUS:aQURRX5(d/US-nUS:a$URRX5(dM/US:waXOSnUS-nU[U5:aZURRX5(d8US- nU[U5:a$URRX5(dM8U[U5:waXOSnXE4$)z One left & one right token, both case-normalized. Skip over non-sentence-final punctuation. Used by the ``ContextIndex`` that is created for ``similar()`` and ``common_contexts()``. rrr r!) _CONTEXT_REmatchr#)r8r$r%jrrs r&_context Text._contexts E1fT--33FI>> FA1fT--33FI>>Fvy  E#f+od&6&6&<&rrJs r&__str__ Text.__str__dii''r)c SUR-$r^r_rJs r&r Text.__repr__rbr)) rrrrUr?r;rRrrrr$r+)Or)rprro)r)dN*)rqrrrsrtrurrFrrrrrr r rrr#rmr.r8rCrNrMrrcompilerXr[r`rrwr,r)r&rr9s.LI. M*M(!"F>0' ' "  D8 % ,\(%D**_-K0((r)rc0\rSrSrSrSrSrSrSrSr g) TextCollectioniaA collection of texts, which can be loaded with list of texts, or with a corpus consisting of one or more texts, and which supports counting, concordancing, collocation discovery, etc. Initialize a TextCollection as follows: >>> import nltk.corpus >>> from nltk.text import TextCollection >>> from nltk.book import text1, text2, text3 >>> gutenberg = TextCollection(nltk.corpus.gutenberg) >>> mytexts = TextCollection([text1, text2, text3]) Iterating over a TextCollection produces all the tokens of all the texts in order. c[US5(a.UR5Vs/sHo!RU5PM nnXl[R U[ U55 0Ulgs snf)Nrh)r@fileidsrh_textsrrFr _idf_cache)r8sourcefs r&rFTextCollection.__init__sX 67 # #/5~~/?@/?!ll1o/?F@  d-f56 AsA-c<URU5[U5- $)z"The frequency of the term in text.)r r#r8termtexts r&tfTextCollection.tfszz$#d)++r)c^URRU5nUc[URVs/sH o1U;dM SPM sn5n[UR5S:Xa [ S5eU(a![ [UR5U- 5OSnX RU'U$s snf)zThe number of texts in the corpus divided by the number of texts that the term appears in. If a term does not appear in the corpus, 0.0 is returned.Trz+IDF undefined for empty document collectiong)rnrZr#rmrfr)r8rtidfrumatchess r&ryTextCollection.idfs oo!!$' ;DKKHKD4<4KHIG4;;1$ !NOO5<#c$++&01#C$'OOD ! Is B*B*cHURX5URU5-$r+)rvryrss r&tf_idfTextCollection.tf_idfswwt"TXXd^33r))rnrmN) rqrrrsrtrurFrvryr}rwr,r)r&rjrjs , 4r)rjctSSKJn [URSS95n[ U5 [ 5 [ S5 UR S5 [ 5 [ S5 UR S5 [ 5 [ S5 UR5 [ 5 [ S5 UR/S Q5 [ 5 [ S 5 URS 5 [ 5 [ S 5 [ S US5 [ SUSS5 [ SUR5S5 g)Nr)brownnews) categoriesz Concordance:zDistributionally similar words:z Collocations:zDispersion plot:)rreportsaid announcedzVocabulary plot:2z Indexing:ztext[3]:rz text[3:5]:ztext.vocab()['news']:) rrrrhrrr#r r.rNrM)rrus r&demors!  v . /D $K G .V G +,LL G / G @A G IIbM G + *d1g ,Qq " !4::<#78r)__main__)rryrrrj)(rurrAr collectionsrrr functoolsrmathrnltk.collocationsrnltk.lmr nltk.lm.preprocessingr nltk.metricsr r nltk.probabilityr r>r nltk.tokenizer nltk.utilrrrrrryrrrjrrq__all__r,r)r&rs 885;77%'>>M XXv|-|-~55p~(~(D +4T+4\9< zF r)