Of>SrSSKrSSKJr SSKJr SSK7 SSKJr SSK J r SSK J r "S S \5r "S S \5rg) z Corpus reader for corpora whose documents are xml files. (note -- not named 'xml' to avoid conflicting w/ standard xml package) N) ElementTree) CorpusReader)*)SeekableUnicodeStreamReader)ElementWrapper)WordPunctTokenizerc6\rSrSrSrSSjrS SjrS SjrSrg) XMLCorpusReaderz Corpus reader for corpora whose documents are xml files. Note that the ``XMLCorpusReader`` constructor does not take an ``encoding`` argument, because the unicode encoding is specified by the XML files themselves. See the XML specs for more info. c>X0l[R"XU5 gN) _wrap_etreer__init__)selfrootfileids wrap_etrees c$--0A5]]1%F&#&&FG G \\& ! & & (B##B'//1C)    %C ) (s +%B66 CcvURU5nURU5n[5nUR5n/nUH\nUR nUcM[ U[5(aURU5nURU5n URU 5 M^ U$! UR 5nNz=f)a Returns all of the words and punctuation symbols in the specified file that were in text nodes -- ie, tags are ignored. Like the xml() method, fileid can only specify one file. :return: the given file's text nodes as a list of words and punctuation symbols :rtype: list(str) ) r%encodingr getiteratoritertextrbytesdecodetokenizeextend) rr"r$r(word_tokenizeriteratoroutnoder+tokss rwordsXMLCorpusReader.words4shhv==(+- "(HD99DdE**;;x0D%..t4 4   "xxzHs B$$B8)r)Fr ) __name__ __module__ __qualname____firstlineno____doc__rr%r5__static_attributes__rrr r s3 rr c \rSrSrSrSrSrSSjrSrSr \ RS \ R\ R-5r\ RS 5r\ RS \ R\ R-5rS rSS jrSrg) XMLCorpusViewQa A corpus view that selects out specified elements from an XML file, and provides a flat list-like interface for accessing them. (Note: ``XMLCorpusView`` is not used by ``XMLCorpusReader`` itself, but may be used by subclasses of ``XMLCorpusReader``.) Every XML corpus view has a "tag specification", indicating what XML elements should be included in the view; and each (non-nested) element that matches this specification corresponds to one item in the view. Tag specifications are regular expressions over tag paths, where a tag path is a list of element tag names, separated by '/', indicating the ancestry of the element. Some examples: - ``'foo'``: A top-level element whose tag is ``foo``. - ``'foo/bar'``: An element whose tag is ``bar`` and whose parent is a top-level element whose tag is ``foo``. - ``'.*/foo'``: An element whose tag is ``foo``, appearing anywhere in the xml tree. - ``'.*/(foo|bar)'``: An wlement whose tag is ``foo`` or ``bar``, appearing anywhere in the xml tree. The view items are generated from the selected XML elements via the method ``handle_elt()``. By default, this method returns the element as-is (i.e., as an ElementTree object); but it can be overridden, either via subclassing or via the ``elt_handler`` constructor parameter. FiNcU(aX0l[RUS-5UlSS0UlUR U5n[ RXUS9 g)a Create a new corpus view based on a specified XML file. Note that the ``XMLCorpusView`` constructor does not take an ``encoding`` argument, because the unicode encoding is specified by the XML files themselves. :type tagspec: str :param tagspec: A tag specification, indicating what XML elements should be included in the view. Each non-nested element that matches this specification corresponds to one item in the view. :param elt_handler: A function used to transform each element to a value for the view. If no handler is specified, then ``self.handle_elt()`` is called, which returns the element as an ElementTree object. The signature of elt_handler is:: elt_handler(elt, tagspec) -> value z\Zrr=)r(N) handle_eltrecompile_tagspec _tag_context_detect_encodingStreamBackedCorpusViewr)rr"tagspec elt_handlerr(s rrXMLCorpusView.__init__usY* )O 7U?3 9G  ((0''x'Hrc[U[5(a2UR5nUR5nUR 5 O%[US5nUR5nSSS5 WR [ R5(agUR [ R5(agUR [ R5(agUR [ R5(agUR [ R5(ag[RSU5nU(aURS5R5$[RS U5nU(aURS5R5$g!WR 5 f=f!,(df  GNV=f) Nrbz utf-16-bez utf-16-lez utf-32-bez utf-32-lezutf-8s!\s*<\?xml\b.*\bencoding="([^"]+)"rs!\s*<\?xml\b.*\bencoding='([^']+)')r PathPointerrreadlineclose startswithcodecs BOM_UTF16_BE BOM_UTF16_LE BOM_UTF32_BE BOM_UTF32_LEBOM_UTF8rCmatchgroupr-)rr"infilesms rrGXMLCorpusView._detect_encodings4 fk * * OO% fd#vOO%$ <<++ , , <<++ , , <<++ , , <<++ , , << ( ( HH:A > 771:$$& & HH:A > 771:$$& &+ ##s FF4F14 GcU$)a Convert an element into an appropriate value for inclusion in the view. Unless overridden by a subclass or by the ``elt_handler`` constructor argument, this method simply returns ``elt``. :return: The view value corresponding to ``elt``. :type elt: ElementTree :param elt: The element that should be converted. :type context: str :param context: A string composed of element tags separated by forward slashes, indicating the XML context of the given element. For example, the string ``'foo/bar/baz'`` indicates that the element is a ``baz`` element whose parent is a ``bar`` element and whose grandparent is a top-level ``foo`` element. r=)rr$contexts rrBXMLCorpusView.handle_elts ( ra; [^<]* ( (() | # comment () | # doctype decl (<[^!>][^>]*>)) # tag or PI [^<]*)* \Zz<\s*(?:/\s*)?([^\s>]+)a6 # Include these so we can skip them: (?P )| (?P )| (?P <\?.*?\?> )| (?P ]*(\[[^\]]*])?\s*>)| # These are the ones we actually care about: (?P <\s*[^>/\?!\s][^>]*/\s*> )| (?P <\s*[^>/\?!\s][^>]*> )| (?P <\s*/[^>/\?!\s][^>]*> )cSn[U[5(aUR5nURUR5nX$- nUR R U5(aU$[RSU5RS5S:XaOUR5[U5[RSU5R5- - n[SU-5eU(d [S5eURS5nUS:aUR R USU5(a\[U[5(a#URW5 URU5 OUR[U5U- *S 5 USU$GMd) aC Read a string from the given stream that does not contain any un-closed tags. In particular, this function first reads a block from the stream of size ``self._BLOCK_SIZE``. It then checks if that block contains an un-closed tag. If it does, then this function either backtracks to the last '<', or reads another block. z[<>]r>zUnexpected ">" near char %sz&Unexpected end of file: tag not closed!DEE !IJJ !)s 3  1$%%++H5G6G,HII!&*EFF H-001BC c(m6G&G$H!L#$6%677?rc d Uc URnUc URn[URR UR 555nUce/nSnSnSnU/:XdUGb[ U[5(aUR 5n URU5n U (dUcGOj[S5eURRU 5GHn UR(a;[SRSRU5SSU R!555 U R!S5(aUR"R%U R!55R!S5n UR'U 5 UcI[(R%USRU55(aU R+5n[-U5nMMMU R!S 5(aUR"R%U R!55R!S5n U(d[S U -5eXS :wa[S US S U S35eUbJU[-U5:Xa;XXkR/5- nUR'USRU545 S=pgSnUR15 GMU R!S5(dGMUR"R%U R!55R!S5n UbGM4[(R%USRU5S-U -5(dGMgUR'U R!5SRU5S-U -45 GM UbU/:Xa XUS- nSnOUR(a [S5 [ U[5(a#UR3W 5 UR5U5 OUR3[-U 5U- *S5 USUS- nS=pgSnU/:XaGMUbGMUR 5n XR;a[7U5URU :XdeO[7U5URU 'UVVs/sH2upU"[8R:"UR=SS55U5PM4 snn$s snnf)z Read from ``stream`` until we find at least one element that matches ``tagspec``, and return the result of applying ``elt_handler`` to each element found. NrbzUnexpected end of filez {:>25} {}/i START_TAGrEND_TAGzUnmatched tag zUnmatched tag ...>!;;=226:L $$%=>>11,?;;+,,SXXg->st-DekkmTU;;{++--33EKKMBHHKDNN4( (88GSXXg->??(- I(+G I@) [[++--33EKKMBHHKD"()>)EFFr{*(?72;-vdVST)UVV ,c'l1J YY[$II Xsxx/@$AB044 #%KKM[[11--33EKKMBHHKD (88GSXXg->-Dt-KLL KK8IC8ORV8V(WXE@H$2:YZ 88H !I{{67!&*EFF H-00; c,&7)&C$DaH%o A 6G,00I!HObjI1Tkkm ## #>T%6%6s%;; ;;%*7^D  c "#'  #' &&szz';N'OP #'    s/9R,)rFrErBr )NN)r7r8r9r:r;rrgrrGrBrCrDDOTALLVERBOSErhrrrurr<r=rrr?r?Qs<FK"IH:0JJ  BJJ MJJ89M  E BJJ J,8bk rr?)r;rR xml.etreernltk.corpus.reader.apirnltk.corpus.reader.util nltk.datarnltk.internalsr nltk.tokenizerr rHr?r=rrrs= !/%1),6l6r| *| r