o Dh@szddlmZddlmZmZmZddlZddlZddlZddlZe dej Z ddZ ddZ d d Zd d Zd dZdS))absolute_import)divisionprint_functionunicode_literalsNz\s+cCs tt|S)z Translates multiple whitespace into single space character. If there is at least one new line character chunk is replaced by single LF (Unix new line) character. )MULTIPLE_WHITESPACE_PATTERNsub_replace_whitespace)textr N/home/air/segue/gemini/back/venv/lib/python3.10/site-packages/justext/utils.pynormalize_whitespaces r cCs |}d|vs d|vrdSdS)zBNormalize all spacing characters that aren't a newline to a space.   )group)matchr r r r rsrcCs| p|S)zw Returns `True` if string contains only white-space characters or is empty. Otherwise `False` is returned. )isspace)stringr r r is_blanksrcCs`tjtjdj}tj|d}g}t|D]}tj|\}}|dkr+| |qt |S)z,Returns a collection of built-in stop-lists.justext stoplistsz.txt) ospathdirnamesysmodules__file__joinlistdirsplitextappend frozenset)path_to_stoplistsstoplist_namesfilenamename extensionr r r get_stoplists%s r'cCsTtjdd|}ztd|}Wn tytd|wtdd|DS)zAReturns an built-in stop-list for the language as a set of words.rz%s.txtrzStoplist for language '%s' is missing. Please use function 'get_stoplists' for complete list of stoplists and feel free to contribute by your own stoplist.css|] }|dVqdS)utf8N)decodelower).0wr r r ?szget_stoplist..) rrrpkgutilget_dataIOError ValueErrorr! splitlines)language file_path stopwordsr r r get_stoplist3s r6) __future__rrrrrerrr.compileUNICODErr rrr'r6r r r r s