[g$"SS5rg)cz\rSrSrSrSrSrSrSr\ S5r \ S5r \ S5r \ S 5r S rS rS rg )TokenSetaG A token set is used to store the unique list of all tokens within an index. Token sets are also used to represent an incoming query to the index, this query token set and index token set are then intersected to find which tokens to look up in the inverted index. A token set can hold multiple tokens, as in the case of the index token set, or it can hold a single token as in the case of a simple query token set. Additionally token sets are used to perform wildcard matching. Leading, contained and trailing wildcards are supported, and from this edit distance matching can also be provided. Token sets are implemented as a minimal finite state automata, where both common prefixes and suffixes are shared between tokens. This helps to reduce the space used for storing the token set. TODO: consider https://github.com/glyph/automat cSUl0UlURUlUR=RS- slg)NFr)finaledges_next_idid __class__selfs 0/usr/lib/python3/dist-packages/lunr/token_set.py__init__TokenSet.__init__s0  -- 1$cNUR$![a Of=fUR(aSOSn[[ UR R 555HBnUR Un[UR5nO![a SnOf=fX-U-nMD U$)N10) _stringAttributeErrorrsortedlistrkeysstrr )r stringlabelnodenode_ids r __str__TokenSet.__str__ s <<     D!234E::e$D dgg,!  ^g-F5 s  2B BBc6SR[U55$)Nz)formatrr s r __repr__TokenSet.__repr__3s ''D 22rc[5nUn[U5HRupEU[U5S- :HnUS:XaX"RU'XblM2[5nXglXrRU'UnMT U$)zCreates a TokenSet from a string. The string may contain one or more wildcard characters (*) that will allow wildcard matching when intersecting with another TokenSet r*)r enumeratelenrr)r rrrooticharrnext_s r from_stringTokenSet.from_string6sqz !(GAVq(Es{#' 4 "  # #( 4 ) rc[5nX2US./nU(GaUR5n[US5S:aUSSnSnXeSR;aUSRUnO[5nXuSRU'[US5S:XaSUlUR UUSUSSSS.5 USS:XaMS USR;aUSRS nO[5nXSRS '[US5S:XaSUlUR UUSS- USS.5 [US5S:a$UR USUSS- USSSS.5 [US5S:Xa SUSl[US5S:a{S USR;aUSRS n O[5n XSRS '[US5S:XaSU lUR U USS- USSSS.5 US(a[US5S:aUSSn USSn Sn XSR;aUSRU n O[5n XSRU '[US5S:XaSU lUR U USS- XSS S-S.5 U(aGMU$) arCreates a token set representing a single string with a specified edit distance. Insertions, deletions, substitutions and transpositions are each treated as an edit distance of 1. Increasing the allowed edit distance will have a dramatic impact on the performance of both creating and intersecting these TokenSets. It is advised to keep the edit distance less than 3. )redits_remainingrrNrrTr0r&)rpopr(rrappend) clsr edit_distancer)stackframer+ no_edit_nodeinsertion_nodesubstitution_nodechar_achar_btranspose_nodes r from_fuzzy_stringTokenSet.from_fuzzy_stringQs:zFSTIIKE5?#a'Xq)# =...#(=#6#6t#'?!'C#Ho 5?#a' %f +01B+Ca+G"'/!""55?#q(&*f #5?#q(%----(-f (;(;C(@%(0 %/@&M'',uX'1,.2%+ 1+01B+Ca+G"'/!""5&'Ch,@1,Dx+x+!%6]000%*6]%8%8%@N%-ZN2@&M''/uX'1,+/N( .+01B+Ca+G"(?12+>">eN rcSSKJn U"5nUHnURU5 M UR5 UR$)Nr1)TokenSetBuilder)lunr.token_set_builderrBinsertfinishr))r5 list_of_wordsrBbuilderwords r from_listTokenSet.from_lists9:!#!D NN4 " ||rcUR(a&URURUR5$URUR5$)N)r6r?termr-)r5clauses r from_clauseTokenSet.from_clauses;   ((f6J6JK K??6;;/ /rcT/nSUS./nU(aUR5nUSR(aURUS5 USRR 5H6nURUS[ U5-USRUS.5 M8 U(aMU$)Nr)prefixrrrQ)r3rr4rrr)r wordsr7r8edges r to_listTokenSet.to_lists-.IIKEV}"" U8_-f ++002 "'/CI"= %f 3 3D 93 e rc[5nXUS./nU(Ga(UR5nUSRR5HnUSRR5HnXe:XdUS:XdMUSRUnUSRUnUR=(a URn Sn XdSR;a-USRUn U R=(d U U lO![5n XlXSRU'UR XxU S.5 M M U(aGM(U$)zReturns a new TokenSet that is the intersection of this TokenSet and the passed TokenSet. This intersection will take into account any wildcards contained within the TokenSet. )rq_nodeoutputrWrr&NrX)rr3rrrr4) r otherrXr7r8q_edgen_edgerrWrr,s r intersectTokenSet.intersects6BCIIKE///446#Fm11668F'6S=$V}226:!&x!6!6v!> $ ;v|| $!8_%:%::$)(O$9$9&$AE*/++*>EK$,JE*/KrfsLLr