Of6SrSSKrSSKrSSKJr SSKJrJr SSKJ r Sr "SS\ 5r "S S 5r "S S 5r "S S\5rg)z9 A reader for corpora whose documents are in MTE format. N)reduce)TaggedCorpusReaderconcat) XMLCorpusViewc$URX5$N)findall)rootpathnss 8/usr/lib/python3/dist-packages/nltk/corpus/reader/mte.pyxpathr s << !!c,\rSrSrSrSSjrSSjrSrg) MTECorpusViewz( Class for lazy viewing the MTE Corpus. Nc2[R"XX#5 gr)r__init__)selffileidtagspec elt_handlers r rMTECorpusView.__init__stWBrc X[[S[R"XX#555$)Nc USL$rxs r *MTECorpusView.read_block..!4-r)listfilterr read_block)rstreamrrs r r$MTECorpusView.read_blocks* '((wL   rrr)NN)__name__ __module__ __qualname____firstlineno____doc__rr$__static_attributes__rrr rrsC rrc\rSrSrSrSSS.rSrSrSrS r S r S r \ S 5r \ S 5r\ S5r\ S5r\ S5r\ S5r\ S5r\ S5r\ S5rSrSrSrSrSrSrSrSrSrSrg) MTEFileReader"z Class for loading the content of the multext-east corpus. It parses the xml files and does some tag-filtering depending on the given method parameters. zhttps://www.tei-c.org/ns/1.0z%https://www.w3.org/XML/1998/namespace)teixmlz{https://www.tei-c.org/ns/1.0}z'{https://www.w3.org/XML/1998/namespace}zTEI/text/body/div/div/p/s/(w|c)zTEI/text/body/div/div/p/szTEI/text/body/div/div/pcXlgr)_MTEFileReader__file_path)r file_paths r rMTEFileReader.__init__3s$rcUR$r)textclseltcontexts r _word_eltMTEFileReader._word_elt6s xxrcz[USUR5Vs/sHo0RUS5PM sn$s snfN*)rr r<r9r:r;ws r _sent_eltMTEFileReader._sent_elt:205c30GH0G1 a&0GHHH8cz[USUR5Vs/sHo0RUS5PM sn$s snfr?)rr rCr9r:r;ss r _para_eltMTEFileReader._para_elt>rErFcSUR;aURS4$URS:Xa+URS:XaURURS4$URS:Xa>URS:Xa.UR[R URS54$[ R"S[ R"SSUR5-S-5nURURS5(aYURS:XaURURS4$UR[R URS54$g) Nanamsd universal^-.z.*$) attribr7_MTEFileReader__tags_MTEFileReader__tagsetMTETagConvertermsd_to_universalrecompilesubmatch)r9r:r;tagss r _tagged_word_eltMTEFileReader._tagged_word_eltBs  "HHb> ! ::   5HHcjj/0 0 ZZ2 #,,+"=HHo>>szz%?PQR R::cBFF3SZZ$@@5HIDzz#**U+,,<<5(HHcjj&788'88E9JK rc [[S[USUR5Vs/sHo0R US5PM sn55$s snf)Nc USL$rrrs r r0MTEFileReader._tagged_sent_elt..\r!rr@)r"r#rr r^rAs r _tagged_sent_eltMTEFileReader._tagged_sent_eltXL '8=c38OP8O1%%a.8OP   QA c [[S[USUR5Vs/sHo0R US5PM sn55$s snf)Nc USL$rrrs r r0MTEFileReader._tagged_para_elt..er!rr@)r"r#rr rcrHs r _tagged_para_eltMTEFileReader._tagged_para_eltarerfctSUR;aURS4$URURS4$)NlemmarN)rTr7r8s r _lemma_word_eltMTEFileReader._lemma_word_eltjs4 #** $HHb> !HHcjj12 2rcz[USUR5Vs/sHo0RUS5PM sn$s snfr?)rr rnrAs r _lemma_sent_eltMTEFileReader._lemma_sent_eltq46;Ccff6MN6M##At,6MNNNrFcz[USUR5Vs/sHo0RUS5PM sn$s snfr?)rr rqrHs r _lemma_para_eltMTEFileReader._lemma_para_eltursrFch[UR[R[R5$r)rr3r. word_pathr<rs r wordsMTEFileReader.wordsy(   m55}7N7N  rch[UR[R[R5$r)rr3r. sent_pathrCrys r sentsMTEFileReader.sents~r|rch[UR[R[R5$r)rr3r. para_pathrJrys r parasMTEFileReader.parasr|rch[UR[R[R5$r)rr3r.rxrnrys r lemma_wordsMTEFileReader.lemma_words(   m55}7T7T  rcU[lU[l[UR[R [R 5$r)r.rVrUrr3rxr^rtagsetr]s r tagged_wordsMTEFileReader.tagged_words8!' #    m55}7U7U  rch[UR[R[R5$r)rr3r.r~rqrys r lemma_sentsMTEFileReader.lemma_sentsrrcU[lU[l[UR[R [R 5$r)r.rVrUrr3r~rcrs r tagged_sentsMTEFileReader.tagged_sentsrrch[UR[R[R5$r)rr3r.rrurys r lemma_parasMTEFileReader.lemma_parasrrcU[lU[l[UR[R [R 5$r)r.rVrUrr3rrjrs r tagged_parasMTEFileReader.tagged_parasrr) __file_pathN) r'r(r)r*r+r tag_nsxml_nsrxr~rr classmethodr<rCrJr^rcrjrnrqrurzrrrrrrrrr,rrr r.r."s.6 B.F 6F1I+I)I%IIII*    33 OOOO         rr.c F\rSrSrSrSSSSSSS S S S S SS. r\S5rSrg)rWzi Class for converting msd tags to universal tags, more conversion options are currently not implemented. ADJADPADVCONJDETNOUNNUMPRTPRONVERBrSX) ASRCDNMQPVrSrRczUSS:XdUSOUSnU[R;aSn[RU$)z This function converts the annotation from the Multex-East to the universal tagset as described in Chapter 5 of the NLTK-Book Unknown Tags will be mapped to X. Punctuation marks are not supported in MSD tags, so r#rR)rWmapping_msd_universal)tag indicators r rX MTETagConverter.msd_to_universalsA#&a&C-CFSV OAAAI44Y??rrN) r'r(r)r*r+r staticmethodrXr,rrr rWrWsK              @ @rrWc\rSrSrSrSSjrSrSSjrSSjrSSjr SS jr SS jr SS jr SS jr SS jrSSjrSrg)MTECorpusReaderz Reader for corpora following the TEI-p5 xml scheme, such as MULTEXT-East. MULTEXT-East contains part-of-speech-tagged words with a quite precise tagging scheme. These tags can be converted to the Universal tagset Nc@[R"XX#5 SUlg)a Construct a new MTECorpusreader for a set of documents located at the given root directory. Example usage: >>> root = '/...path to corpus.../' >>> reader = MTECorpusReader(root, 'oana-*.xml', 'utf8') # doctest: +SKIP :param root: The root directory for this corpus. (default points to location in multext config file) :param fileids: A list or regexp specifying the fileids in this corpus. (default is oana-en.xml) :param encoding: The encoding of the given files (default is utf8) z 00README.txtN)rr_readme)rr fileidsencodings r rMTECorpusReader.__init__s ##DB% rc^Uc TRnO[U[5(aU/n[U4SjU5n[SU5nU(d [ S5 U$)Nc">UTR;$r)_fileids)rrs r r+MTECorpusReader.__fileids..s1 #5rc US;$)N)z oana-bg.xmlz oana-mk.xmlrrs r rrs 1,J#Jrz$No valid multext-east file specified)r isinstancestrr#print)rrs` r __fileidsMTECorpusReader.__fileidssP ?mmG  % %iG5w?JGT 8 9rc [URU5Vs/sHDn[[RR UR U55R5PMF sn5$s snf)z :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of words and punctuation symbols. :rtype: list(str) )r_MTECorpusReader__fileidsr.osr join_rootrzrrfs r rzMTECorpusReader.wordss] 0 0Abggll4::q9:@@B0    A A+c [URU5Vs/sHDn[[RR UR U55R5PMF sn5$s snf)z :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of sentences or utterances, each encoded as a list of word strings :rtype: list(list(str)) )rrr.rr rrrrs r rMTECorpusReader.sents]0 0Abggll4::q9:@@B0    rc [URU5Vs/sHDn[[RR UR U55R5PMF sn5$s snf)z :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as lists of word string :rtype: list(list(list(str))) )rrr.rr rrrrs r rMTECorpusReader.parasrrc [URU5Vs/sHDn[[RR UR U55R5PMF sn5$s snf)z :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of words, the corresponding lemmas and punctuation symbols, encoded as tuples (word, lemma) :rtype: list(tuple(str,str)) )rrr.rr rrrrs r rMTECorpusReader.lemma_wordss]0 0Abggll4::q9:FFH0    rc US:XdUS:Xak[URU5Vs/sHEn[[RR UR U55RX#5PMG sn5$[S5 gs snf)a :param fileids: A list specifying the fileids that should be used. :param tagset: The tagset that should be used in the returned object, either "universal" or "msd", "msd" is the default :param tags: An MSD Tag that is used to filter all parts of the used corpus that are not more precise or at least equal to the given tag :return: the given file(s) as a list of tagged words and punctuation symbols encoded as tuples (word, tag) :rtype: list(tuple(str, str)) rPrOUnknown tagset specified.N) rrr.rr rrrrrrrr]rs r rMTECorpusReader.tagged_words, [ FeO "^^G4 5""'',,tzz1"=>KK5   - .A Bc [URU5Vs/sHDn[[RR UR U55R5PMF sn5$s snf)a :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of sentences or utterances, each encoded as a list of tuples of the word and the corresponding lemma (word, lemma) :rtype: list(list(tuple(str, str))) )rrr.rr rrrrs r rMTECorpusReader.lemma_sentsC]0 0Abggll4::q9:FFH0    rc US:XdUS:Xak[URU5Vs/sHEn[[RR UR U55RX#5PMG sn5$[S5 gs snf)a :param fileids: A list specifying the fileids that should be used. :param tagset: The tagset that should be used in the returned object, either "universal" or "msd", "msd" is the default :param tags: An MSD Tag that is used to filter all parts of the used corpus that are not more precise or at least equal to the given tag :return: the given file(s) as a list of sentences or utterances, each each encoded as a list of (word,tag) tuples :rtype: list(list(tuple(str, str))) rPrOrN) rrr.rr rrrrrs r rMTECorpusReader.tagged_sentsRrrc [URU5Vs/sHDn[[RR UR U55R5PMF sn5$s snf)a= :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as a list of tuples of the word and the corresponding lemma (word, lemma) :rtype: list(List(List(tuple(str, str)))) )rrr.rr rrrrs r rMTECorpusReader.lemma_parasirrc US:XdUS:Xak[URU5Vs/sHEn[[RR UR U55RX#5PMG sn5$[S5 gs snf)a/ :param fileids: A list specifying the fileids that should be used. :param tagset: The tagset that should be used in the returned object, either "universal" or "msd", "msd" is the default :param tags: An MSD Tag that is used to filter all parts of the used corpus that are not more precise or at least equal to the given tag :return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as a list of (word,tag) tuples :rtype: list(list(list(tuple(str, str)))) rPrOrN) rrr.rr rrrrrs r rMTECorpusReader.tagged_parasxs [ FeO "^^G4 5""'',,tzz1"=>KK5   - .r)r)NNutf8r)NrOrN)r'r(r)r*r+rrrzrrrrrrrrr,rrr rrs> &         /.  /.  /rr)r+rrY functoolsrnltk.corpus.readerrrnltk.corpus.reader.xmldocsrrrr.rWrrrr rsU 94" M "H H V"@"@J|/(|/r