o Dh|"@s,UdZddlZddlZddlmZddlmZmZmZmZm Z ddl Z zddl m Z Wn ey6dZ YnwddlmZddlmZmZmZdd lmZeeZd d hZeeed <e e jje j j!d dgddZ"e j#e"dZ$eddd ddZ%e&dej'Z(e&dej'Z)GdddZ*dede+fddZ,de-de+fddZ.de-deefdd Z/d!e e-efdefd"d#Z0d$edefd%d&Z1d'edeefd(d)Z2d*ede+fd+d,Z3d-ed*edefd.d/Z4d0edeefd1d2Z5d0e e-eefdeefd3d4Z6d5ed6eedefd7d8Z7d9edefd:d;Z8dS)z(c @s:eZdZdZgdZdedededededd f d d Zd S) Extractorz0Defines a class to store all extraction options. extensiveformatmaxminoriginalextensive_searchmax_datemin_date original_date outputformatreturnNcCs"||_||_||_||_||_dS)Nr)selfr"r#r$r%r&r)O/home/air/segue/gemini/back/venv/lib/python3.10/site-packages/htmldate/utils.py__init__4s  zExtractor.__init__) __name__ __module__ __qualname____doc__ __slots__boolrstrr+r)r)r)r*r/s rdatar'cCs|rt|tkr dSdS)z6Check if the input object is suitable to be processed.TF)lenrr3r)r)r*is_wrong_documentCsr6cCs&z|dWdStyYdSw)zLSimple heuristic to determine if a bytestring uses standard unicode encodingzUTF-8FT)decodeUnicodeDecodeErrorr5r)r)r*isutf8Js   r9 bytesobjectcCsvt|rdgSg}tdurt|d}|dur||t|ddp)t|}|dd|Ddd|DS)zcsz#detect_encoding..cSsg|]}|tvr|qSr))r)r;gr)r)r*r=fs)r9cchardet_detectappendlowerr extend)r:guessescchardet_guessdetection_resultsr)r)r*detect_encodingSs rF filecontentc Csht|tr|Sd}t|D]}z||}Wnttfy)td|d}Yq w|p3t|dddS)znGuess bytestring encoding and try to decode to Unicode string. Resort to destructive conversion otherwise.Nzwrong encoding detected: %srreplace)rerrors) isinstancer2rFr7 LookupErrorr8LOGGERwarning)rGhtmltextguessed_encodingr)r)r* decode_fileis   rPresponsecCs2t|tjjs t|dr|j}t|S|}t|S)zRead the urllib3 object corresponding to the server response, then try to guess its encoding and decode it to return a unicode stringr3)rJurllib3rQ HTTPResponsehasattrr3rP)rQ resp_contentr)r)r*decode_response}s rVurlc Csz tjd|dd}Wnty$}ztd||WYd}~dSd}~ww|jdkr4td|j|dSt|jrAtd|dSt|jS) a,Fetches page using urllib3 and decodes the response. Args: url: URL of the page to fetch. Returns: HTML code as string, or Urllib3 response object (headers + body), or empty string in case the result is invalid, or None if there was a problem with the network. GET)timeoutzdownload error: %s %sNz!not a 200 response: %s for URL %szincorrect input data for URL %s) HTTP_POOLrequest ExceptionrLerrorstatusr6r3rV)rWrQerrr)r)r* fetch_urls     rb beginningcCsd|vS)zOAssess if the object is proper HTML (awith a corresponding tag or declaration).htmlr))rcr)r)r*is_dubious_htmlsre htmlstringcCsd|vr|d\}}}tjd|ddd|}tt|D]\}}d|vr8|dr8tjd|dd}|S|d kr?|Sq |S) z>Repair faulty HTML strings to make then palatable for libxml2.doctype r )countzz\1>) partition DOCTYPE_TAGsub enumerateiter splitlinesendswith FAULTY_HTML)rfrc firstline_restiliner)r)r*repair_faulty_htmlsry htmlobjectc CsPd}z t|dtd}W|Sty'}z td|WYd}~|Sd}~ww)z!Try to pass bytes to LXML parser.Nutf8parserzlxml parser bytestring %s)r encode HTML_PARSERr^rLr_)rztreerar)r)r*fromstring_bytessrc CsHt|tr|St|ttfstdt|t|tr6|dr6d|vr6td|t |}|dur6t d|d}t |}|dd }t ||}d}zt|td }Wn&t ybd }t|}Yntyy}z td |WYd}~nd}~ww|dust|d kr|st|}|durt|rt|d krtdt|d}|S)zkLoad object given as input and validate its type (accepted: lxml.html tree, bytestring and string) zincompatible input type: %shttp zURL detected, downloading: %sNzURL couldn't be processed: %s2Fr|Tzlxml parsing failed: %sr rkz9parsed tree length: %s, wrong data type or not valid HTML)rJr bytesr2 TypeErrortype startswithrLdebugrb ValueErrorrPrAryr rrr^r_r4re)rzrrcfallback_parserar)r)r* load_htmlsF      rrelemlistcCs.||D]}|}|dur||q|S)zDelete selected elements.N)rp getparentremove)rrelementparentr)r)r* clean_htmls  rstringcCsd|S)z7Remove superfluous space and normalize remaining space.r)joinsplitstrip)rr)r)r* trim_textsr)9r/loggingrertypingrrrrrrRcchardetrr? ImportErrorcharset_normalizerr lxml.htmlr r r settingsr getLoggerr,rLrr2__annotations__disable_warnings exceptionsInsecureRequestWarningutilRetryRETRY_STRATEGY PoolManagerr\rcompileIrmrsrr1r6rr9rFrPrVrbreryrrrrr)r)r)r*sR         4