o Dh81@sddlmZddlmZmZmZddlZddlZddlZddlZddl m Z m Z m Z m Z ddlTddlmZmZdd Zejd fd d Zejfd dZejfddZddZedkr_edSdS))absolute_import)divisionprint_functionunicode_literalsN)escapePY3URLErrorurllib)*) get_stoplist get_stoplistsc Cs2dtjtjtjdttttt t t t d S)NaUsage: %(progname)s -s STOPLIST [OPTIONS] [HTML_FILE] Convert HTML to plain text and remove boilerplate. -o OUTPUT_FILE if not specified, output is written to stdout --encoding=... default character encoding to be used if not specified in the HTML meta tags (default: %(default_encoding)s) --enc-force force specified encoding, ignore HTML meta tags --enc-errors=... errors handling for character encoding conversion: strict: fail on error ignore: ignore characters which can't be converted replace: replace characters which can't be converted with U+FFFD unicode replacement characters (default: %(default_enc_errors)s) --format=... output format; possible values: default: one paragraph per line, each preceded with

or (headings) boilerplate: same as default, except for boilerplate paragraphs are included, too, preceded with detailed: one paragraph per line, each preceded with

tag containing detailed information about classification as attributes krdwrd: KrdWrd compatible format --no-headings disable special handling of headings --list-stoplists print a list of inbuilt stoplists and exit -V, --version print version information and exit -h, --help display this help and exit If no HTML_FILE specified, input is read from stdin. STOPLIST must be one of the following: - one of the inbuilt stoplists; see: %(progname)s --list-stoplists - path to a file with the most frequent words for given language, one per line, in UTF-8 encoding - None - this activates a language-independent mode Advanced options: --length-low=INT (default %(length_low)i) --length-high=INT (default %(length_high)i) --stopwords-low=FLOAT (default %(stopwords_low)f) --stopwords-high=FLOAT (default %(stopwords_high)f) --max-link-density=FLOAT (default %(max_link_density)f) --max-heading-distance=INT (default %(max_heading_distance)i) r) progname length_low length_high stopwords_lowstopwords_highmax_link_densitymax_heading_distancedefault_encodingdefault_enc_errors) ospathbasenamesysargvLENGTH_LOW_DEFAULTLENGTH_HIGH_DEFAULTSTOPWORDS_LOW_DEFAULTSTOPWORDS_HIGH_DEFAULTMAX_LINK_DENSITY_DEFAULTMAX_HEADING_DISTANCE_DEFAULTDEFAULT_ENCODINGDEFAULT_ENC_ERRORSr$r$Q/home/air/segue/gemini/back/venv/lib/python3.10/site-packages/justext/__main__.pyusages-r&TcCsR|D]$}|jdkr|jrd}nd}n|rqd}td|t|jddf|dqd S) z Outputs the paragraphs as: text of the first paragraph text of the second paragraph ... where is

, or which indicates standard paragraph, heading or boilerplate respecitvely. goodhpbz<%s> %sFquotefileN) class_typeheadingprintrtext) paragraphsfpno_boilerplate paragraphtagr$r$r%output_defaultJs  r8c CsB|D]}d|j|jt|j|jt|jddf}t||dqdS)z Same as output_default, but only

tags are used and the following attributes are added: class, cfclass and heading. z6

%sFr+r-N)r/cf_classintr0xpathrr2r1)r3r4r6outputr$r$r%output_detailedas r=cCsR|D]$}|jdvr|jrd}nd}nd}|jD]}td||f|dqqdS)a Outputs the paragraphs in a KrdWrd compatible format: classfirst text node classsecond text node ... where class is 1, 2 or 3 which means boilerplate, undecided or good respectively. Headings are output as undecided. )r'neargoodrz%i %sr-N)r/r0 text_nodesr1strip)r3r4r6cls text_noder$r$r% output_krdwrdqs  rEc CsLddl}ddlm}z|tjdddgd\}}Wn'|jyC}zt|tjdtttjdt dWYd}~nd}~wwt dd}tj }t rW|tjj}n|tj}d}d } d } t} t} t} t}t}t}d}t}d }t}z|D]\}}|d vrttt d|d vrtd tjtjd|ft dqz|dkrtdttt dqz|dkrz t |dd}Wqzty}zt d||fd}~ww|dkrV|!dkrt"}qztj#|r0zt |dd}t"dd|D}|$Wqzty}zt d||fd}~wt%y/}zt d|d}~ww|tvr;t&|}qzt'(d|rPt d|dttft d||dkrsz |}d)|Wqzt*yrt d|w|d kr{d!}qz|d"kr|!d#vr|!}qzt d$||d%kr|d&vr|} qzt d'||d(krd!} qz|d)krzt+|} Wqzt,yt d*||fw|d+krzt+|} Wqzt,yt d*||fw|d,krzt-|} Wqzt,yt d-||fw|d.krzt-|}Wqzt,yt d-||fw|d/kr9zt-|}Wqzt,y8t d-||fw|d0krUzt+|}Wqzt,yTt d*||fwqz|r[|}|durdt d1|skd}d} |rzt'(d2|drt./|d}nt|dd}Wntt0fy}z t d|d|fd}~ww|1}|tj ur|$t||| | | |||| ||| }| d krt2||WdS| d3krt2||d d4WdS| d5krt3||WdS| d6krt4||WdSt5d7| t6y%}ztd8tjtjd|ftjdt dWYd}~dSd}~ww)9Nr) __version__rzo:s:hV)z encoding=z enc-forcez enc-errors=zformat=z no-headingshelpversionz length-low=z length-high=zstopwords-low=zstopwords-high=zmax-link-density=zmax-heading-distance=zlist-stoplistsr-utf8defaultF)z-hz--help)z-Vz --versionzK%s: jusText v%s Copyright (c) 2011 Jan Pomikalek z--list-stoplists z-owzCan't open %s for writing: %sz-snonercSsg|]}|qSr$)rB).0lr$r$r% szmain..zCan't open %s for reading: %szLUnicode decoding error when reading the stoplist (probably not in UTF-8): %sz^\w*$z,Unknown stoplist: %s Available stoplists: %szFile not found: %sz --encodingzUknown character encoding: %sz --enc-forceTz --enc-errors)strictignorereplacezInvalid --enc-errors value: %sz--format)rK boilerplatedetailedkrdwrdzUknown output format: %sz --no-headingsz --length-lowz-Invalid value for %s: '%s'. Integer expected.z --length-highz--stopwords-lowz+Invalid value for %s: '%s'. Float expected.z--stopwords-highz--max-link-densityz--max-heading-distancezNo stoplist specified.z [^:/]+://rW)r5rXrYzUnknown format: %sz%s: %s)7getoptjustextrFrr GetoptErrorr1stderrr&exitcodecslookupstdinrstdoutbufferrrrrr r!r"r#rrrjoinsortedr openIOErrorJustextInvalidOptionslowersetisfilecloseUnicodeDecodeErrorr rematchencode LookupErrorr: ValueErrorfloatr urlopenr readr8r=rEAssertionError JustextError)rZVERSIONoptsargse stream_writerfp_infp_outstoplistformat no_headingsrrrrrrencodingrforce_default_encoding enc_errorsoa fp_stoplist html_textr3r$r$r%mainsx $                                                 $r__main__) __future__rrrrr_rrnr_compatrrr r coreutilsr r r&rbr8r=rEr__name__r$r$r$r%s$ :8