OfA SrSSKrSSK7 SSK7 SSK7 "SS\5r"SS\\5r "SS \ 5r "S S \5r g) z; A reader for corpora that consist of plaintext documents. N)*cj\rSrSrSr\r\"5S\S4Sjr S Sjr S Sjr S Sjr S r S rS rS rg)PlaintextCorpusReadera Reader for corpora that consist of plaintext documents. Paragraphs are assumed to be split using blank lines. Sentences and words can be tokenized using the default tokenizers, or by custom tokenizers specified as parameters to the constructor. This corpus reader can be customized (e.g., to skip preface sections of specific document formats) by creating a subclass and overriding the ``CorpusView`` class variable. Nutf8cT[RXX&5 X0lX@lXPlg)au Construct a new plaintext corpus reader for a set of documents located at the given root directory. Example usage: >>> root = '/usr/local/share/nltk_data/corpora/webtext/' >>> reader = PlaintextCorpusReader(root, '.*\.txt') # doctest: +SKIP :param root: The root directory for this corpus. :param fileids: A list or regexp specifying the fileids in this corpus. :param word_tokenizer: Tokenizer for breaking sentences or paragraphs into words. :param sent_tokenizer: Tokenizer for breaking paragraphs into words. :param para_block_reader: The block reader used to divide the corpus into paragraph blocks. N) CorpusReader__init___word_tokenizer_sent_tokenizer_para_block_reader)selfrootfileidsword_tokenizersent_tokenizerpara_block_readerencodings >/usr/lib/python3/dist-packages/nltk/corpus/reader/plaintext.pyr PlaintextCorpusReader.__init__%s&2 d'<--"3c [URUSS5VVVs/sH up#nURX RUS9PM" snnn5$s snnnf)z^ :return: the given file(s) as a list of words and punctuation symbols. :rtype: list(str) Tr)concatabspaths CorpusView_read_word_blockrrpathencfileids rwordsPlaintextCorpusReader.wordsCsX ,0==$+M +M'T&;&;cJ+M    s'A c URc[5Ul[UR USS5VVVs/sH up#nUR X R US9PM" snnn5$! [S5e=fs snnnf)z :return: the given file(s) as a list of sentences or utterances, each encoded as a list of word strings. :rtype: list(list(str)) %No sentence tokenizer for this corpusTr)r PunktTokenizer ValueErrorrrr_read_sent_blockrs rsentsPlaintextCorpusReader.sentsP    ' J'5'7$,0==$+M +M'T&;&;cJ+M    J !HII A*'A: * A7c URc[5Ul[UR USS5VVVs/sH up#nUR X R US9PM" snnn5$! [S5e=fs snnnf)z :return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as lists of word strings. :rtype: list(list(list(str))) r%Tr)r r&r'rrr_read_para_blockrs rparasPlaintextCorpusReader.parasdr+r,c/n[S5H;nURURRUR 555 M= U$Nr)rangeextendr tokenizereadlinerstreamr"is rr&PlaintextCorpusReader._read_word_blockxs>rA LL--66v7HI J rc /nURU5HWnURURRU5Vs/sHnURRU5PM sn5 MY U$s snfN)r r4r r5r rr8r)parasents rr(&PlaintextCorpusReader._read_sent_block~s++F3D LL!% 4 4 = =d C C((11$7 C 4 %A1 c /nURU5HWnURURRU5Vs/sHnURRU5PM sn5 MY U$s snfr<)r appendr r5r rr8r/r>r?s rr.&PlaintextCorpusReader._read_para_blockrArB)r r r r<)__name__ __module__ __qualname____firstlineno____doc__StreamBackedCorpusViewrWordPunctTokenizerread_blankline_blockr r"r)r/rr(r.__static_attributes__rrrrsH (JE*+.4<   ( (  rrc\rSrSrSrSrSrg) CategorizedPlaintextCorpusReaderzm A reader for plaintext corpora whose documents are divided into categories based on their file identifiers. cb[RX5 [R"U/UQ70UD6 g)z Initialize the corpus reader. Categorization arguments (``cat_pattern``, ``cat_map``, and ``cat_file``) are passed to the ``CategorizedCorpusReader`` constructor. The remaining arguments are passed to the ``PlaintextCorpusReader`` constructor. N)CategorizedCorpusReaderr rrargskwargss rr )CategorizedPlaintextCorpusReader.__init__s* ((6&&t=d=f=rrPN)rGrHrIrJrKr rOrPrrrRrRs  >rrRc\rSrSrSrSrg)*PortugueseCategorizedPlaintextCorpusReadercJ[RX5 [S5US'g)N portugueser)rUr r&rVs rr 3PortugueseCategorizedPlaintextCorpusReader.__init__s ((6#1,#? rrPN)rGrHrIrJr rOrPrrr[r[s@rr[c>\rSrSrSrSrSrSrS SjrS Sjr S r g) EuroparlCorpusReadera Reader for Europarl corpora that consist of plaintext documents. Documents are divided into chapters instead of paragraphs as for regular plaintext documents. Chapters are separated using blank lines. Everything is inherited from ``PlaintextCorpusReader`` except that: - Since the corpus is pre-processed and pre-tokenized, the word tokenizer should just split the line at whitespaces. - For the same reason, the sentence tokenizer should just split the paragraph at line breaks. - There is a new 'chapters()' method that returns chapters instead instead of paragraphs. - The 'paras()' method inherited from PlaintextCorpusReader is made non-functional to remove any confusion between chapters and paragraphs for Europarl. c/n[S5H0nURUR5R55 M2 U$r2)r3r4r6splitr7s rr%EuroparlCorpusReader._read_word_blocks6rA LL*002 3 rc/nURU5H@nURUR5Vs/sHoDR5PM sn5 MB U$s snfr<)r r4 splitlinesrdr=s rr(%EuroparlCorpusReader._read_sent_blockO++F3D LL4??3DE3D4**,3DE F4 FA c/nURU5H@nURUR5Vs/sHoDR5PM sn5 MB U$s snfr<)r rDrgrdrEs rr.%EuroparlCorpusReader._read_para_blockrirjNc [URUS5VVs/sHup#URX RUS9PM! snn5$s snnf)z :return: the given file(s) as a list of chapters, each encoded as a list of sentences, which are in turn encoded as lists of word strings. :rtype: list(list(list(str))) Tr)rrrr.)rrr!r s rchaptersEuroparlCorpusReader.chapterssS&*]]7D%A %AMV(=(=L%A    s&A c[S5e)NzVThe Europarl corpus reader does not support paragraphs. Please use chapters() instead.)NotImplementedError)rrs rr/EuroparlCorpusReader.parass! d  rrPr<) rGrHrIrJrKrr(r.rnr/rOrPrrraras $     rra) rK nltk.datanltknltk.corpus.reader.apinltk.corpus.reader.util nltk.tokenizer rrUrRr[rarPrrrxsS$%~L~B>'>@U>(@1Q@ 6 06 r