g$ ddlZddlZddlZdgZej ddZGddZGddZGdd Z y) NRobotFileParser RequestRatezrequests secondscXeZdZ ddZdZdZdZdZdZdZ dZ d Z d Z d Z d Zy )rczg|_g|_d|_d|_d|_|j |d|_y)NFr)entriessitemaps default_entry disallow_all allow_allset_url last_checkedselfurls 9/opt/alt/python312/lib64/python3.12/urllib/robotparser.py__init__zRobotFileParser.__init__s;  !! Sc |jSN)r rs rmtimezRobotFileParser.mtime%s    rc8 ddl}|j|_y)Nr)timer )rrs rmodifiedzRobotFileParser.modified.s   IIKrcr ||_tjj|dd\|_|_y)N)rurllibparseurlparsehostpathrs rr zRobotFileParser.set_url6s0:%||44S9!A> 49rc tjj|j}|j }|j |j djy#tjj$rT}|jdvrd|_ n4|jdk\r |jdkr d|_ Yd}~yYd}~yYd}~yYd}~yd}~wwxYw)Nzutf-8)iiTii) rrequesturlopenrreadrdecode splitlineserror HTTPErrorcoder r )rfrawerrs rr&zRobotFileParser.read;sB 9&&txx0A&&(C JJszz'*557 8||%% &xx:%$(!SSXX^!%&4" &s)A++C;CCcd|jvr|j||_yy|jj|yN*) useragentsr rappend)rentrys r _add_entryzRobotFileParser._add_entryHs= %"" "!!)%*"* LL   &rc d}t}|j|D]}|s4|dk(r t}d}n"|dk(r|j|t}d}|jd}|dk\r|d|}|j }|sh|j dd}t |dk(s|dj j|d<tjj|dj |d<|ddk(rB|dk(r|j|t}|jj|dd}*|ddk(r3|dk7s9|jjt|ddd}e|dd k(r3|dk7st|jjt|dd d}|dd k(r?|dk7s|dj jrt!|d|_d}|dd k(r|dk7s|dj d }t |dk(rk|dj jrJ|dj jr)t%t!|dt!|d|_d}|ddk(s|j(j|d|dk(r|j|yy)Nrr#:z user-agentdisallowFallowTz crawl-delayz request-rate/sitemap)Entryrr5findstripsplitlenlowerrrunquoter2r3 rulelinesRuleLineisdigitintdelayrreq_rater)rlinesstater4lineinumberss rrzRobotFileParser.parseQs  DA:!GEEaZOOE*!GEE #AAvBQx::>   \\**6<<+?+?+DE ll%%r"Z__   j.. 0C0C'EFll  %C\\E *s++"   %%//4 4rc|jsy|jD]!}|j|s|jcS|jr|jjSyr)rrrWrIr rrYr4s r crawl_delayzRobotFileParser.crawl_delaysTzz|\\E *{{""   %%++ +rc|jsy|jD]!}|j|s|jcS|jr|jjSyr)rrrWrJr r]s r request_ratezRobotFileParser.request_ratesTzz|\\E *~~%"   %%.. .rc4|jsy|jSr)rrs r site_mapszRobotFileParser.site_mapss}}}}rc|j}|j||jgz}djtt|S)Nz )rr joinmapstr)rrs r__str__zRobotFileParser.__str__s@,,    )!3!3 44G{{3sG,--rN)rQ)__name__ __module__ __qualname__rrrr r&r5rr[r^r`rbrgrrrrsE !(? 9'G#R: .rc eZdZ dZdZdZy)rFc|dk(r|sd}tjjtjj|}tjj ||_||_y)NrQT)rrrRr rVr"rX)rr"rXs rrzRuleLine.__init__sP 2:iI||&&v||'<'>zTADIIMMrN)rhrirjrrWrgrkrrrFrFs1#BNrrFc&eZdZ dZdZdZdZy)r>c<g|_g|_d|_d|_yr)r2rErIrJrs rrzEntry.__init__s  rcg}|jD]}|jd||j|jd|j|j7|j}|jd|jd|j |j tt|jdj|S)Nz User-agent: z Crawl-delay: zRequest-rate: r< ) r2r3rIrJrequestssecondsextendrerfrErd)rretagentrates rrgz Entry.__str__s__E JJeW- .% :: ! JJtzzl3 4 == $==D JJ a ~F G 3sDNN+,yy~rc |jddj}|jD]}|dk(ry|j}||vsyy)Nr<rr1TF)rArCr2)rrYr|s rrWzEntry.applies_tosT@OOC(+113 __E|KKME ! %rcf |jD]!}|j|s|jcSy)NT)rErWrX)rrprMs rrXzEntry.allowance s2 %NNDx(~~%#rN)rhrirjrrgrWrXrkrrr>r>sI  rr>) collections urllib.parserurllib.request__all__ namedtuplerrrFr>rkrrrsU   $k$$]4FG ~.~.BNN$((r