[gN<SSKJr SSKJr SSKJr SrSrSSjrg) )deepcopy)Token) as_stringu  -c&U=(a U[;$N)SEPARATOR_CHARS)chars 0/usr/lib/python3/dist-packages/lunr/tokenizer.pydefault_separatorr s  +DO++Nc ^Uc/$U=(d 0n[U[[45(a=UVs/sH/n[[ U5R 5[ U55PM1 sn$Tc[nO[T5(aTnOU4Sjn[U5R 5n[U5n/nSn[US-5Hn X:waXYOSn X- n U"U 5(dX:XdM&U S:aO0n X/U S'[U5U S'U RU5 [X5n UR[X]U 55 U S-nM U$s snf)akSplits a string into tokens ready to be inserted into the search index. Args: metadata (dict): Optional metadata can be passed to the tokenizer, this metadata will be cloned and added as metadata to every token that is created from the object to be tokenized. separator (callable or compiled regex): This tokenizer will convert its parameter to a string by calling `str` and then will split this string on characters for which `separator` is True. Lists will have their elements converted to strings and wrapped in a lunr `Token`. Returns: List of Token instances. c&>TRU5$r)match)c separators r Tokenizer..+s!3r rpositionindex) isinstancelisttuplerrlowerrr callablestrlenrangeupdatesliceappend)objmetadatarelement is_separatorstringlengthtokens slice_start slice_endr slice_lengthtoken_metadatasls ` r Tokenizerr/ sI { ~2H#e}%%QT QTgE)G$**,hx.@ AQT  ( )   3 X^^ F [F FK6A:& $-$7v R .   !4a!#.9-Hz**-f+w'%%h/;2 eFJ?@#a-K' M= s6E )NN) copyr lunr.tokenr lunr.utilsrrr r/r r r4s $,3r