]4@s^ddlZddlZdgZGdddZGdddZGdddZdS)NRobotFileParserc@seZdZdddZddZddZdd Zd d Zd d ZddZ ddZ ddZ dS)rcCs>g|_d|_d|_d|_|j|d|_dS)NFr)entries default_entry disallow_all allow_allset_url last_checked)selfurlr 0/opt/alt/python35/lib64/python3.5/robotparser.py__init__s      zRobotFileParser.__init__cCs|jS)N)r )r r r r mtimeszRobotFileParser.mtimecCsddl}|j|_dS)Nr)timer )r rr r r modified(s zRobotFileParser.modifiedcCs5||_tjj|dd\|_|_dS)N)r urllibparseurlparsehostpath)r r r r r r0s zRobotFileParser.set_urlcCsytjj|j}Wnmtjjk r}zG|jdkrOd|_n'|jdkrv|jdkrvd|_WYdd}~Xn)X|j }|j |j dj dS)NTiizutf-8)rr) rZrequesturlopenr error HTTPErrorcoderrreadrdecode splitlines)r ferrrawr r r r5s  zRobotFileParser.readcCs>d|jkr*|jdkr:||_n|jj|dS)N*) useragentsrrappend)r entryr r r _add_entryBs zRobotFileParser._add_entrycCsd}t}|jx|D]}|sr|dkrJt}d}n(|dkrr|j|t}d}|jd}|dkr|d|}|j}|sq |jdd}t|dkr |djj|ds z+RobotFileParser.__str__..)joinr)r r r r __str__szRobotFileParser.__str__N) __name__ __module__ __qualname__rrrrrr)rrArGr r r r rs    4 c@s4eZdZddZddZddZdS)r5cCsY|dkr| rd}tjjtjj|}tjj||_||_dS)NrT)rrr;rr=rr?)r rr?r r r rs zRuleLine.__init__cCs|jdkp|j|jS)Nr%)r startswith)r filenamer r r r>szRuleLine.applies_tocCs|jrdndd|jS)NZAllowZDisallowz: )r?r)r r r r rGszRuleLine.__str__N)rHrIrJrr>rGr r r r r5s   r5c@s@eZdZddZddZddZddZd S) r-cCsg|_g|_dS)N)r&r4)r r r r rs zEntry.__init__cCsjg}x'|jD]}|jd|dgqWx*|jD]}|jt|dgq:Wdj|S)Nz User-agent: rBr)r&extendr4rCrF)r retagentr8r r r rGs z Entry.__str__cCs]|jddj}x=|jD]2}|dkr9dS|j}||kr#dSq#WdS)Nr:rr%TF)r0r2r&)r r@rOr r r r>s   zEntry.applies_tocCs.x'|jD]}|j|r |jSq WdS)NT)r4r>r?)r rLr8r r r r?s zEntry.allowanceN)rHrIrJrrGr>r?r r r r r-s    r-)Z urllib.parserZurllib.request__all__rr5r-r r r r  s