Yf{)@sdZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlZddlZddlZddlmZmZmZddlmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)ddl*m+Z+m,Z,yddl-Z-Wne.k rdZ/YnXdZ/dd d d d d ddddddddddddddddddd d!d"d#d$d%d&d'd(g!Z0e j1dd)Z2da3de j4d*dd+dd,dd-dd.dZ5d/d Z6gZ7dddd0d%Z8d1d&Z9e j:d2e j;Z<d3d4Z=Gd5ddZ>Gd6d d Z?d7d!Z@Gd8d d ZAGd9ddeAZBGd:d d eAZCGd;d d eAZDd<d=ZEGd>ddeAZFGd?ddZGGd@ddeGZHGdAddeHZIGdBddZJGdCddeJeAZKGdDddeJeAZLejMZNGdEddZOGdFddeAeOZPGdGddeAeOZQGdHdIdIeAZRGdJddeRZSeTejUdKrGdLdMdMeRZVe0jWdMGdNd d eAZXGdOddeAZYdPdQZZdRdSZ[GdTddeAZ\dUdVZ]GdWddeAZ^GdXdde^Z_GdYddeAZ`dZZaejbd[kr`dd\lcmdZdmeZend]d#Zdd^d"ZeiZfGd_d'd'ZgGd`d(d(egZhdaidadbZjdakdcddZldamdedfZndaodgdhZpGdidjdjZqdkdlZrddmdnZsdodpZte judqkrddrlvmwZwmxZxdsdtZydudvZzdwdxZ{dyd$Z|nNejbd[krdzd{Z}d|d$Z|d}d~Z~ddxZ{n erZ|esZ{dS)a An extensible library for opening URLs using a variety of protocols The simplest way to use this module is to call the urlopen function, which accepts a string containing a URL or a Request object (described below). It opens the URL and returns the results as file-like object; the returned object has some extra methods described below. The OpenerDirector manages a collection of Handler objects that do all the actual work. Each Handler implements a particular protocol or option. The OpenerDirector is a composite object that invokes the Handlers needed to open the requested URL. For example, the HTTPHandler performs HTTP GET and POST requests and deals with non-error returns. The HTTPRedirectHandler automatically deals with HTTP 301, 302, 303 and 307 redirect errors, and the HTTPDigestAuthHandler deals with digest authentication. urlopen(url, data=None) -- Basic usage is the same as original urllib. pass the url and optionally data to post to an HTTP URL, and get a file-like object back. One difference is that you can also pass a Request instance instead of URL. Raises a URLError (subclass of OSError); for HTTP errors, raises an HTTPError, which can also be treated as a valid response. build_opener -- Function that creates a new OpenerDirector instance. Will install the default handlers. Accepts one or more Handlers as arguments, either instances or Handler classes that it will instantiate. If one of the argument is a subclass of the default handler, the argument will be installed instead of the default. install_opener -- Installs a new opener as the default opener. objects of interest: OpenerDirector -- Sets up the User Agent as the Python-urllib client and manages the Handler classes, while dealing with requests and responses. Request -- An object that encapsulates the state of a request. The state can be as simple as the URL. It can also include extra HTTP headers, e.g. a User-Agent. BaseHandler -- internals: BaseHandler and parent _call_chain conventions Example usage: import urllib.request # set up authentication info authinfo = urllib.request.HTTPBasicAuthHandler() authinfo.add_password(realm='PDQ Application', uri='https://mahler:8092/site-updates.py', user='klem', passwd='geheim$parole') proxy_support = urllib.request.ProxyHandler({"http" : "http://ahad-haam:3128"}) # build a new opener that adds authentication and caching FTP handlers opener = urllib.request.build_opener(proxy_support, authinfo, urllib.request.CacheFTPHandler) # install it urllib.request.install_opener(opener) f = urllib.request.urlopen('http://www.python.org/') N)URLError HTTPErrorContentTooShortError)urlparseurlspliturljoinunwrapquoteunquote splittype splithost splitport splituser splitpasswd splitattr splitquery splitvaluesplittagto_bytesunquote_to_bytes urlunparse) addinfourl addclosehookFTRequestOpenerDirector BaseHandlerHTTPDefaultErrorHandlerHTTPRedirectHandlerHTTPCookieProcessor ProxyHandlerHTTPPasswordMgrHTTPPasswordMgrWithDefaultRealmHTTPPasswordMgrWithPriorAuthAbstractBasicAuthHandlerHTTPBasicAuthHandlerProxyBasicAuthHandlerAbstractDigestAuthHandlerHTTPDigestAuthHandlerProxyDigestAuthHandler HTTPHandler FileHandler FTPHandlerCacheFTPHandler DataHandlerUnknownHandlerHTTPErrorProcessorurlopeninstall_opener build_opener pathname2url url2pathname getproxies urlretrieve urlcleanup URLopenerFancyURLopenercafilecapath cadefaultcontextc Cs|s|s|r{|dk r*tdts<tdtjtjjd|d|}td|}t|}nF|rtd|}t|}n"tdkrta}nt}|j |||S)NzDYou can't pass both context and any of cafile, capath, and cadefaultzSSL support not availabler;r<r>) ValueError _have_sslsslZcreate_default_contextZPurposeZ SERVER_AUTH HTTPSHandlerr2_openeropen) urldatatimeoutr;r<r=r>Z https_handleropenerrI3/opt/alt/python35/lib64/python3.5/urllib/request.pyr0s$     cCs |adS)N)rC)rHrIrIrJr1scCst|\}}tjt||:}|j}|dkrb| rbtjj||fS|rzt|d}n(t j dd}|j }t j ||||f} d } d} d} d} d |krt|d } |r|| | | xW|j| }|sP| t|7} |j|| d7} |r|| | | qWWd QRXWd QRX| dkr| | krtd | | f| | S)aW Retrieve a URL into a temporary location on disk. Requires a URL argument. If a filename is passed, it is used as the temporary file location. The reporthook argument should be a callable that accepts a block number, a read size, and the total file size of the URL target. The data argument should be valid URL encoded data. If a filename is passed and the URL points to a local resource, the result is a copy from local file to new file. Returns a tuple containing the path to the newly created data file as well as the resulting HTTPMessage object. filewbZdeleteFirzcontent-lengthzContent-LengthNz1retrieval incomplete: got only %i out of %i bytesi )r contextlibclosingr0infoospathnormpathrDtempfileZNamedTemporaryFilename_url_tempfilesappendintreadlenwriter)rEfilename reporthookrFZurl_typerTfpheaderstfpresultbssizer[blocknumblockrIrIrJr6sD       "c CsTx4tD],}ytj|Wqtk r2YqXqWtdd=trPdadS)z0Clean up temporary files from urlretrieve calls.N)rXrSunlinkOSErrorrC)Z temp_filerIrIrJr7s    z:\d+$cCsV|j}t|d}|dkr7|jdd}tjd|d}|jS)zReturn request-host, as defined by RFC 2965. Variation from RFC: returned value is lowercased, for convenient comparison. rNHost)full_urlr get_header _cut_port_resublower)requestrEhostrIrIrJ request_hosts   rsc@s*eZdZdidddddZeddZejddZejddZed d Zejd d Zejd d Zd dZ ddZ ddZ ddZ ddZ ddZddZddZdddZdd Zd!d"ZdS)#rNFc Cs||_i|_i|_d|_||_d|_x*|jD]\}}|j||qCW|dkr{t|}||_ ||_ |r||_ dS)N) rlraunredirected_hdrs_datarF _tunnel_hostitems add_headerrsorigin_req_host unverifiablemethod) selfrErFraryrzr{keyvaluerIrIrJ__init__ s          zRequest.__init__cCs&|jrdj|j|jS|jS)Nz{}#{})fragmentformat _full_url)r|rIrIrJrls zRequest.full_urlcCs8t||_t|j\|_|_|jdS)N)rrrr_parse)r|rErIrIrJrl"scCsd|_d|_d|_dS)Nrj)rrselector)r|rIrIrJrl)s  cCs|jS)N)ru)r|rIrIrJrF/sz Request.datacCs8||jkr4||_|jdr4|jddS)NzContent-length)ru has_header remove_header)r|rFrIrIrJrF3s cCs d|_dS)N)rF)r|rIrIrJrF=scCsqt|j\|_}|jdkr:td|jt|\|_|_|jrmt|j|_dS)Nzunknown url type: %r) r rtyper?rlr rrrr )r|restrIrIrJrAs  zRequest._parsecCs+|jdk rdnd}t|d|S)z3Return a string indicating the HTTP request method.NPOSTGETr{)rFgetattr)r|Zdefault_methodrIrIrJ get_methodIszRequest.get_methodcCs|jS)N)rl)r|rIrIrJ get_full_urlNszRequest.get_full_urlcCsJ|jdkr(|j r(|j|_n||_|j|_||_dS)Nhttps)rrvrrrlr)r|rrrrIrIrJ set_proxyQs   zRequest.set_proxycCs|j|jkS)N)rrl)r|rIrIrJ has_proxyYszRequest.has_proxycCs||j|jss z7HTTPRedirectHandler.redirect_request..raryrzT)rrrr)rr)rrr)rr) rrrlreplacerrarwrry) r|rr`rrranewurlmZ newheadersrI)rrJrXs  (  z$HTTPRedirectHandler.redirect_requestc Csd|kr|d}nd|kr2|d}ndSt|}|jdkrst||d||f|||j r|jrt|}d|d sz'ProxyHandler.__init__..)r5rAssertionErrorproxiesrwsetattrr)r|rrrErIrIrJrs    zProxyHandler.__init__c Cs|j}t|\}}}}|dkr3|}|jrOt|jrOdS|r|rdt|t|f} tj| jjd} |j dd| t|}|j ||||ks|dkrdS|j j |d|j SdS)Nz%s:%sasciizProxy-authorizationzBasic rrG)rrrr proxy_bypassr base64 b64encodeencodedecoderxrrrDrG) r|rr rZ orig_typeZ proxy_typerrrZ user_passZcredsrIrIrJrs      zProxyHandler.proxy_open)rrrrrrrIrIrIrJrs  c@sOeZdZddZddZddZddd Zd d Zd S) r cCs i|_dS)N)passwd)r|rIrIrJr szHTTPPasswordMgr.__init__cst|tr|g}|jkr4ij|s z0HTTPPasswordMgr.add_password..)TF)rrrtuple)r|realmrrr reduced_urirI)r"r|rJ add_password s   zHTTPPasswordMgr.add_passwordc Cs|jj|i}xddD]\}|j||}xA|jD]3\}}x$|D]}|j||rT|SqTWqAWqWdS)NTF)TF)NN)rrr rw is_suburi) r|r%authuriZdomainsr"reduced_authuriZurisZauthinforrIrIrJfind_user_passwords  z"HTTPPasswordMgr.find_user_passwordTc Cst|}|dr=|d}|d}|dp7d}nd}|}d}t|\}}|r|dkr|dk rdddd ij|} | dk rd || f}||fS) z@Accept authority or URI and extract only the authority and path.rNrrrNrPriz%s:%d)rr r) r|rr"partsrr rTrrportZdportrIrIrJr "s        zHTTPPasswordMgr.reduce_uricCsi||krdS|d|dkr(dStj|d|df}t|t|dkredSdS)zcCheck if test is below base in a URI tree Both args must be URIs in reduced form. TrFrN) posixpath commonprefixr\)r|basetestcommonrIrIrJr(9s zHTTPPasswordMgr.is_suburiN)rrrrr'r+r r(rIrIrIrJr s   c@seZdZddZdS)r!cCsDtj|||\}}|dk r1||fStj|d|S)N)r r+)r|r%r)rrrIrIrJr+Js    z2HTTPPasswordMgrWithDefaultRealm.find_user_passwordN)rrrr+rIrIrIrJr!Hs csReZdZfddZdfddZdddZdd ZS) r"cs i|_tj||dS)N) authenticatedsuperr)r|rkwargs) __class__rIrJrTs z%HTTPPasswordMgrWithPriorAuth.__init__FcsR|j|||dk r5tjd|||tj||||dS)N)update_authenticatedr5r')r|r%rrris_authenticated)r7rIrJr'Xs z)HTTPPasswordMgrWithPriorAuth.add_passwordcCs]t|tr|g}x>dD]6}x-|D]%}|j||}||j|rrFrDrMr https_requestrrIrIrIrJr#qs     c@s"eZdZdZddZdS)r$rKcCs%|j}|jd|||}|S)Nzwww-authenticate)rlrF)r|rr`rrrarErrIrIrJhttp_error_401s  z#HTTPBasicAuthHandler.http_error_401N)rrrrGrRrIrIrIrJr$s c@s"eZdZdZddZdS)r%zProxy-authorizationcCs%|j}|jd|||}|S)Nzproxy-authenticate)rrrF)r|rr`rrrar rrIrIrJhttp_error_407s  z$ProxyBasicAuthHandler.http_error_407N)rrrrGrSrIrIrIrJr%s c@sseZdZdddZddZddZdd Zd d Zd d ZddZ ddZ dS)r&NcCsL|dkrt}||_|jj|_d|_d|_d|_dS)Nr)r rr'retried nonce_count last_nonce)r|rrIrIrJrs     z"AbstractDigestAuthHandler.__init__cCs d|_dS)Nr)rT)r|rIrIrJreset_retry_countsz+AbstractDigestAuthHandler.reset_retry_countcCs|j|d}|jdkr?t|jdd|dn|jd7_|r|jd}|jdkr|j||S|jdkrtd|dS) Nizdigest auth failedrNrZdigestr:zEAbstractDigestAuthHandler does not support the following scheme: '%s')rrTrrlr=rpretry_http_digest_authr?)r|rGrrrrarErrIrIrJrFsz/AbstractDigestAuthHandler.http_error_auth_reqedcCs|jdd\}}ttdt|}|j||}|rd|}|jj|jd|krwdS|j|j||j j |d|j }|SdS)NrrNz Digest %srG) r=parse_keqv_listfilterparse_http_listget_authorizationrarrGrrrDrG)r|rrJtokenZ challengechalZauth_valZresprIrIrJrY s z0AbstractDigestAuthHandler.retry_http_digest_authcCsXd|j|tjf}|jdtd}tj|j}|ddS)Nz %s:%s:%s:rrM)rUtimeZctimer _randombyteshashlibsha1 hexdigest)r|noncesbdigrIrIrJ get_cnoncesz$AbstractDigestAuthHandler.get_cnoncecCsVyK|d}|d}|jd}|jdd}|jdd}Wntk rcdSYnX|j|\}} |dkrdS|jj||j\} } | dkrdS|jdk r|j|j|} nd} d| || f} d|j|j f}|d kr||j kr=|j d 7_ nd |_ ||_ d |j }|j |}d ||||||f}| || |}nD|dkr| || d|||f}nt d |d| |||j |f}|r|d|7}| r*|d| 7}|d|7}|rR|d||f7}|S)Nr%rfqop algorithmMD5opaquez%s:%s:%sz%s:%srJrNz%08xz%s:%s:%s:%s:%szqop '%s' is not supported.z>username="%s", realm="%s", nonce="%s", uri="%s", response="%s"z , opaque="%s"z , digest="%s"z, algorithm="%s"z, qop=auth, nc=%s, cnonce="%s")rKeyErrorget_algorithm_implsrr+rlrFget_entity_digestrrrVrUrjr)r|rr_r%rfrkrlrnHKDrrHZentdigZA1ZA2ZncvalueZcnonceZnoncebitZrespdigr1rIrIrJr]!sV             ( z+AbstractDigestAuthHandler.get_authorizationcsb|dkrddn+|dkr6ddntd|fdd}|fS)NrmcSstj|jdjS)Nr)rcZmd5rre)xrIrIrJr`sz?AbstractDigestAuthHandler.get_algorithm_impls..ZSHAcSstj|jdjS)Nr)rcrdrre)rtrIrIrJrbsz.Unsupported digest authentication algorithm %rcsd||fS)Nz%s:%srI)rgd)rrrIrJrgs)r?)r|rlrsrI)rrrJrp]s   z-AbstractDigestAuthHandler.get_algorithm_implscCsdS)NrI)r|rFr_rIrIrJrqjsz+AbstractDigestAuthHandler.get_entity_digest) rrrrrWrFrYrjr]rprqrIrIrIrJr&s    < c@s.eZdZdZdZdZddZdS)r'zAn authentication protocol defined by RFC 2069 Digest authentication improves on basic authentication because it does not transmit passwords in the clear. rKicCs9t|jd}|jd|||}|j|S)NrNzwww-authenticate)rrlrFrW)r|rr`rrrarrretryrIrIrJrRys   z$HTTPDigestAuthHandler.http_error_401N)rrrrrGrrRrIrIrIrJr'os c@s(eZdZdZdZddZdS)r(zProxy-AuthorizationicCs/|j}|jd|||}|j|S)Nzproxy-authenticate)rrrFrW)r|rr`rrrarrrvrIrIrJrSs    z%ProxyDigestAuthHandler.http_error_407N)rrrrGrrSrIrIrIrJr(s c@sCeZdZdddZddZddZdd Zd S) AbstractHTTPHandlerrcCs ||_dS)N) _debuglevel)r| debuglevelrIrIrJrszAbstractHTTPHandler.__init__cCs ||_dS)N)rx)r|levelrIrIrJset_http_debuglevelsz'AbstractHTTPHandler.set_http_debuglevelc Cs|j}|std|jdk r|j}t|trTd}t||jdss|jdd|jdsyt|}Wn@tk rt|t j rt dt ||fYn"X|jddt ||j|}|jr/t|j\}}t|\}} |jdsN|jd|xE|jjD]7\} } | j} |j| s[|j| | q[W|S) Nz no host givenzLPOST data should be bytes or an iterable of bytes. It cannot be of type str.z Content-typez!application/x-www-form-urlencodedzContent-lengthzBContent-Length should be specified for iterable data of type %r %rz%drk)rrrrFrrrrr memoryview collectionsIterabler?rr\itemsizerr rr rrr) r|rqrrrFrZmvZsel_hostrZselZsel_pathrWr~rIrIrJ do_request_sB        zAbstractHTTPHandler.do_request_c  s|j}|std||d|j|}|j|jt|jjtfdd|jj Ddd.r Connectioncss'|]\}}|j|fVqdS)N)title)rrWrrIrIrJrszProxy-AuthorizationraN)rrrrGZset_debuglevelrxrrtrrarwrvZ set_tunnelrqrrrFri getresponserZsockrrEreasonr) r|Z http_classrZhttp_conn_argsrrrZtunnel_headersZproxy_auth_hdrerrrrI)rarJrs>  /   &     zAbstractHTTPHandler.do_openN)rrrrr{rrrIrIrIrJrws   (rwc@s%eZdZddZejZdS)r)cCs|jtjj|S)N)rrrHTTPConnection)r|rrIrIrJ http_openszHTTPHandler.http_openN)rrrrrwrrMrIrIrIrJr)s  rc@s:eZdZdddddZddZejZdS)rBrNcCs&tj||||_||_dS)N)rwr_context_check_hostname)r|ryr>check_hostnamerIrIrJr s zHTTPSHandler.__init__cCs(|jtjj|d|jd|jS)Nr>r)rrrrrr)r|rrIrIrJ https_openszHTTPSHandler.https_open)rrrrrrwrrQrIrIrIrJrBs  rBc@sCeZdZdddZddZddZeZeZdS)rNcCs4ddl}|dkr'|jj}||_dS)Nr)Zhttp.cookiejar cookiejarZ CookieJar)r|rrrIrIrJrs  zHTTPCookieProcessor.__init__cCs|jj||S)N)rZadd_cookie_header)r|rqrIrIrJrMsz HTTPCookieProcessor.http_requestcCs|jj|||S)N)rZextract_cookies)r|rqrrIrIrJr"sz!HTTPCookieProcessor.http_response)rrrrrMrrQrrIrIrIrJrs   c@seZdZddZdS)r.cCs|j}td|dS)Nzunknown url type: %s)rr)r|rrrIrIrJr*s zUnknownHandler.unknown_openN)rrrrrIrIrIrJr.)s cCsmi}x`|D]X}|jdd\}}|ddkr[|ddkr[|dd}|||Parse list of key=value strings where keys are not duplicated.=rNrr;rOrO)r=)lZparsedZeltrrrIrIrJrZ.s  rZcCsg}d}d}}x|D]}|r<||7}d}q|ry|dkrZd}qn|dkrld}||7}q|dkr|j|d}q|dkrd}||7}qW|r|j|dd|DS) apParse lists as described by RFC 2068 Section 2. In particular, parse comma-separated lists where the elements of the list may include quoted-strings. A quoted-string could contain a comma. A non-quoted string could have quotes in the middle. Neither commas nor quotes count if they are escaped. Only double-quotes count, not single-quotes. rjF\Tr;,cSsg|]}|jqSrI)rL)rpartrIrIrJr#as z#parse_http_list..)rY)rgresrescaper ZcurrIrIrJr\8s4           r\c@s:eZdZddZdZddZddZdS)r*cCs|j}|dddkrq|dddkrq|jrq|jdkrq|j|jkr~tdn |j|SdS)Nrz//r:r localhostz-file:// scheme is supported only on localhost)rrr get_namesropen_local_file)r|rrErIrIrJ file_openes  5zFileHandler.file_openNc Csztjdkrsy7ttjddtjtjdt_Wn*tjk rrtjdft_YnXtjS)Nrr)r*namesr$rgethostbyname_ex gethostnamegaierror gethostbyname)r|rIrIrJrps$zFileHandler.get_namescCs^ddl}ddl}|j}|j}t|}ytj|}|j}|jj |j dd} |j |d} |j d| pd|| f} |rt |\}} | s| rt||jkr|rd||} n d|} tt|d| | SWn1tk rM}zt|WYdd}~XnXtddS) NrusegmtTz6Content-type: %s Content-length: %d Last-modified: %s z text/plainzfile://rbzfile not on local host) email.utils mimetypesrrrr4rSstatst_sizeutils formatdatest_mtime guess_typemessage_from_stringr _safe_gethostbynamerrrDrir)r|remailrrrr^Z localfilestatsremodifiedmtyperar.ZorigurlexprIrIrJr{s0       zFileHandler.open_local_file)rrrrrrrrIrIrIrJr*cs   c Cs1ytj|SWntjk r,dSYnXdS)N)rrr)rrrIrIrJrsrc@s(eZdZddZddZdS)r+cCsddl}ddl}|j}|s3tdt|\}}|dkr]|j}n t|}t|\}}|rt|\}}nd}t |}|pd}|pd}yt j |}Wn1t k r}zt|WYdd}~XnXt |j\} } | jd} ttt | } | dd| d} } | r|| d r|| dd} y|j||||| |j} | rdpd}xJ| D]B}t|\}}|jdkr|dkr|j}qW| j| |\}}d}|j|jd}|rG|d |7}|dk rm|dkrm|d|7}tj|}t|||jSWnQ|jk r}z.td|}|jtj dWYdd}~XnXdS)Nrzftp error: no host givenrjrrNrPDraArruzContent-type: %s zContent-length: %d z ftp error: %rrrOrO)rrrrPrur)!ftplibrrrrr FTP_PORTrZrrr rrrirrr=rmap connect_ftprGrrpupperretrfilerrlrrr all_errorswith_tracebacksysexc_info)r|rrrrrr.rrrrTattrsdirsrKZfwrattrr~r`retrlenrarrexcrIrIrJftp_opens\          !  zFTPHandler.ftp_openc Cst||||||ddS)N persistentF) ftpwrapper)r|rrrrr.rrGrIrIrJrszFTPHandler.connect_ftpN)rrrrrrIrIrIrJr+s  5c@sXeZdZddZddZddZddZd d Zd d Zd S)r,cCs1i|_i|_d|_d|_d|_dS)Nr<r`)cacherGsoonestdelay max_conns)r|rIrIrJrs     zCacheFTPHandler.__init__cCs ||_dS)N)r)r|trIrIrJ setTimeoutszCacheFTPHandler.setTimeoutcCs ||_dS)N)r)r|rrIrIrJ setMaxConnsszCacheFTPHandler.setMaxConnscCs|||dj||f}||jkrJtj|j|j|YqXqW|jdd=|jri|jjdS)N)rrrirr)r|rKrIrIrJrms    zURLopener.cleanupcGs|jj|dS)zdAdd a header to be used by the HTTP interface only e.g. u.addheader('Accept', 'sound/basic')N)rrY)r|rrIrIrJ addheader{szURLopener.addheadercCstt|}t|dd}|jrn||jkrn|j|\}}t|d}t|||St|\}}|sd}||jkr|j|}t|\}} t| \} } | |f}nd}d|} ||_ | j dd} t ||  s"| d krK|r;|j |||S|j ||Sy9|dkrmt|| |St|| ||SWnattfk rYnGtk r} z'td | jtjd WYdd} ~ XnXdS) z6Use URLopener().open(file) instead of open(file, 'r').rz%/:=&?~#+!$,;'@()*[]|rrKNZopen_-rrz socket errorr)rrr rrDrr rr rrropen_unknown_proxy open_unknownrrrrirrr)r|rrFr^rar`urltyperEr  proxyhostrrrrWrrIrIrJrDs<    zURLopener.opencCs(t|\}}tdd|dS)z/Overridable interface to open unknown URL type.z url errorzunknown url typeN)r ri)r|rrFrrErIrIrJrszURLopener.open_unknowncCs,t|\}}tdd||dS)z/Overridable interface to open unknown URL type.z url errorzinvalid proxy for %sN)r ri)r|r rrFrrErIrIrJrszURLopener.open_unknown_proxyc Cstt|}|jr5||jkr5|j|St|\}}|dkr| sf|dkryC|j|}|j}|jtt|d|fSWn%t k r} zWYdd} ~ XnX|j ||}z|j} |r t |d} nddl } t|\} }t|p4d\} }t |pLd\}} t |pdd\}} tjj|d}| j|\}}|jj|tj|d} z|| f}|jdk r||j|Default error handler: close the connection and raise OSError.N)rr)r|rEr`rrrarIrIrJrbs zURLopener.http_error_defaultcCs"tjj|d|jd|jS)Nrr)rrrrr)r|rrrIrIrJ_https_connectionhs zURLopener._https_connectioncCs|j|j||S)zUse HTTPS protocol.)rr)r|rErFrIrIrJ open_httpsmszURLopener.open_httpscCst|tstd|dddkrr|dddkrr|ddjdkrrtd n |j|SdS) z/Use local file or FTP depending on form of URL.zEfile error: proxy support for file protocol currently not implementedNrz//r:r z localhost/z-file:// scheme is supported only on localhost)rrrrpr?r)r|rErIrIrJ open_fileqs  HzURLopener.open_filecCsddl}ddl}t|\}}t|}ytj|}Wn:tk r}zt|j|j WYdd}~XnX|j } |j j |j dd} |j|d} |jd| pd| | f} |s$|} |dddkr d |} tt|d | | St|\}}| rtj|tftkr|} |dddkrd |} n&|dd d krtd |tt|d | | StddS)zUse local file.rNrTz6Content-Type: %s Content-Length: %d Last-modified: %s z text/plainrNrzfile://rrz./zAlocal file url may start with / or file:. Unknown url of type: %sz#local file error: not on local host)rrr r4rSrrirstrerrorr^rrrrrrrrDr rrrthishostr?)r|rErrrrrKZ localnamerererrraZurlfiler.rIrIrJrzs:   (  " zURLopener.open_local_filecCsst|tstdddl}t|\}}|sKtdt|\}}t|\}}|rt|\}}nd}t|}t|pd}t|pd}t j |}|sddl }|j }n t |}t|\}} t|}|jd} | dd| d} } | rb| d rb| dd} | r}| d r}d| d<|||dj| f} t|jtkrxDt|jD]3} | | kr|j| }|j| =|jqWy-| |jkr%t||||| |j| <| s4d}nd }xJ| D]B}t|\}}|jd krA|dkrA|j}qAW|j| j| |\}}|jd|d}d}|r|d|7}|dk r|dkr|d|7}tj|}t||d|SWnKtk rn}z(td|j t!j"dWYdd}~XnXdS)zUse FTP protocol.zCftp error: proxy support for ftp protocol currently not implementedrNzftp error: no host givenrjrrNrrPrrrrruzftp:zContent-Type: %s zContent-Length: %d z ftp error %rrrOrO)rrrrPrur)#rrrrr r rrr rrrrrZrr=rr\r MAXFTPCACHErrrrrprrrrrr ftperrorsrrr)r|rErrrrTr.rrrrrrKr}rrrrr~r`rrrarrIrIrJopen_ftpsp               zURLopener.open_ftpc Cst|tstdy|jdd\}}Wn!tk rZtddYnX|sgd}|jd}|dkrd ||d kr||dd }|d |}nd }g}|jd tj d tj tj|jd||dkr8t j |j djd}n t|}|jdt||jd |j|dj|}tj|}tj|}t|||S)zUse "data" URL.zEdata error: proxy support for data protocol currently not implementedrrNz data errorz bad data URLztext/plain;charset=US-ASCII;rrNrjzDate: %sz%a, %d %b %Y %H:%M:%S GMTzContent-type: %srrzlatin-1zContent-Length: %d )rrrr=r?rirfindrYraZstrftimeZgmtimerrrrr r\rrrrStringIOr) r|rErFrZsemirrrafrIrIrJ open_datas6  " $   zURLopener.open_data)rrrrrrrrrrrrrDrrrrrrrr@rrrrrr rIrIrIrJr8@s.      $B \   :c@seZdZdZddZddZdddZd d Zdd d Zdd dZ dddZ ddddZ ddddZ dddZ dddZdddZdddZddd Zd!d"ZdS)#r9z?Derived class with handlers for errors we can handle (perhaps).cOs2tj|||i|_d|_d|_dS)Nrr)r8r auth_cachetriesmaxtries)r|rr6rIrIrJrs  zFancyURLopener.__init__cCst||d||S)z3Default error handling -- don't raise an exception.zhttp:)r)r|rEr`rrrarIrIrJrsz!FancyURLopener.http_error_defaultNc Cs|jd7_z{|jrg|j|jkrgt|drH|j}n |j}|||dd|S|j||||||}|SWdd|_XdS)z%Error 302 -- relocated (temporarily).rNhttp_error_500iz)Internal Server Error: Redirect RecursionNr)r r rr rredirect_internal) r|rEr`rrrarFrrcrIrIrJr s    zFancyURLopener.http_error_302c Csd|kr|d}nd|kr2|d}ndS|jt|jd||}t|}|jd krt|||d||||j|S) Nrrrrrrrjz( Redirection to url '%s' is not allowed.)rrrrj)rrrrrrrD) r|rEr`rrrarFrrrIrIrJrs        z FancyURLopener.redirect_internalcCs|j||||||S)z*Error 301 -- also relocated (permanently).)r)r|rEr`rrrarFrIrIrJr9szFancyURLopener.http_error_301cCs|j||||||S)z;Error 303 -- also relocated (essentially identical to 302).)r)r|rEr`rrrarFrIrIrJr =szFancyURLopener.http_error_303cCsE|dkr(|j||||||S|j|||||SdS)z1Error 307 -- relocated, but turn POST into error.N)rr)r|rEr`rrrarFrIrIrJr As zFancyURLopener.http_error_307Fc Csd|kr(tj|||||||d}tjd|} | sftj||||||| j\} } | jdkrtj|||||||stj||||||d|jd} |dkrt|| || St|| || |SdS)z_Error 401 -- authentication required. This function supports Basic authentication only.zwww-authenticatez![ ]*([^ ]+)[ ]+realm="([^"]*)"r:Zretry_ _basic_authN)r8rrNmatchr@rprr) r|rEr`rrrarFrvstuffrrr%rWrIrIrJrRHs&      zFancyURLopener.http_error_401c Csd|kr(tj|||||||d}tjd|} | sftj||||||| j\} } | jdkrtj|||||||stj||||||d|jd} |dkrt|| || St|| || |SdS)zeError 407 -- proxy authentication required. This function supports Basic authentication only.zproxy-authenticatez![ ]*([^ ]+)[ ]+realm="([^"]*)"r:Z retry_proxy_rN)r8rrNrr@rprr) r|rEr`rrrarFrvrrrr%rWrIrIrJrSas&      zFancyURLopener.http_error_407cCs t|\}}d||}|jd}t|\}} t| \} } | jdd} | | d} |j| || \} } | p| sdSdt| ddt| dd| f} d| | |jd<|dkr|j|S|j||SdS)Nzhttp://r@rNz%s:%s@%srrj)r rr rget_user_passwdr rD)r|rEr%rFrrrrr rr proxyselectorrrrrIrIrJretry_proxy_http_basic_authzs     z*FancyURLopener.retry_proxy_http_basic_authcCs t|\}}d||}|jd}t|\}} t| \} } | jdd} | | d} |j| || \} } | p| sdSdt| ddt| dd| f} d| | |jd<|dkr|j|S|j||SdS)Nzhttps://rrrNz%s:%s@%srrj)r rr rrr rD)r|rEr%rFrrrrr rrrrrrrIrIrJretry_proxy_https_basic_auths     z+FancyURLopener.retry_proxy_https_basic_authc Cst|\}}|jdd}||d}|j|||\}}|pY|s`dSdt|ddt|dd|f}d||} |dkr|j| S|j| |SdS)NrrNz%s:%s@%srrjzhttp://)r rrr rD) r|rEr%rFrrrrrrrrIrIrJrDs   z$FancyURLopener.retry_http_basic_authc Cst|\}}|jdd}||d}|j|||\}}|pY|s`dSdt|ddt|dd|f}d||} |dkr|j| S|j| |SdS)NrrNz%s:%s@%srrjzhttps://)r rrr rD) r|rEr%rFrrrrrrrrIrIrJretry_https_basic_auths   z%FancyURLopener.retry_https_basic_authrcCs|d|j}||jkrA|r6|j|=n |j|S|j||\}}|se|rx||f|j|<||fS)Nr)rpr prompt_user_passwd)r|rrr%rr}rrrIrIrJrs   zFancyURLopener.get_user_passwdc Cspddl}y@td||f}|jd|||f}||fSWntk rktdSYnXdS)z#Override this in a GUI environment!rNzEnter username for %s at %s: z#Enter password for %s in %s at %s: )NN)getpassinputKeyboardInterruptprint)r|rrr%rrrrIrIrJrs   z!FancyURLopener.prompt_user_passwd)rrrrrrrrrr r rRrSrrrDrrrrIrIrIrJr9s$     cCstdkrtjdatS)z8Return the IP address of the magic hostname 'localhost'.Nr) _localhostrrrIrIrIrJrs rc Csdtdkr`y#ttjtjdaWn.tjk r_ttjddaYnXtS)z,Return the IP addresses of the current host.Nrr) _thishostr$rrrrrIrIrIrJrs  #rcCs%tdkr!ddl}|jatS)z1Return the set of errors raised by the FTP class.Nr) _ftperrorsrr)rrIrIrJrs   rcCstdkrtjdatS)z%Return an empty email Message object.Nrj) _noheadersrrrIrIrIrJ noheaderss r!c@speZdZdZddddZddZdd Zd d Zd d ZddZ ddZ dS)rz;Class used by open_ftp() for cache of open FTP connections.NTc Csr||_||_||_||_||_||_d|_||_y|jWn|j YnXdS)Nr) rrrrr.rrGrefcount keepaliveinitr)r|rrrrr.rrGrrIrIrJrs         zftpwrapper.__init__cCsddl}d|_|j|_|jj|j|j|j|jj|j |j dj |j }|jj |dS)Nrr)rbusyZFTPrZconnectrrr.rGZloginrrrrcwd)r|rZ_targetrIrIrJr$ s  zftpwrapper.initc -Cs1ddl}|j|dkr1d}d}nd|}d}y|jj|Wn/|jk r|j|jj|YnXd}|r*| r*y&d|}|jj|\}}Wng|jk r)}zDt|ddd krt d |j t j d WYdd}~XnX|s|jjd|r|jj } zVy|jj|Wn;|jk r}zt d ||WYdd}~XnXWd|jj| Xd |}nd }|jj|\}}d|_t|jd|j} |jd7_|j| |fS)NrrurzTYPE ArNzTYPE zRETR r:Z550z ftp error: %rrzLIST ZLISTr)rur)r endtransferrZvoidcmdrr$Z ntransfercmdZ error_permrrrrrpwdr&r%rmakefile file_closer"r) r|rKrrcmdisdirrrrr(ZftpobjrIrIrJr sN        &*   zftpwrapper.retrfilecCs d|_dS)Nr)r%)r|rIrIrJr'D szftpwrapper.endtransfercCs&d|_|jdkr"|jdS)NFr)r#r" real_close)r|rIrIrJrG s zftpwrapper.closecCs@|j|jd8_|jdkr<|j r<|jdS)NrNr)r'r"r#r-)r|rIrIrJr*L s zftpwrapper.file_closec Cs7|jy|jjWntk r2YnXdS)N)r'rrr)r|rIrIrJr-R s  zftpwrapper.real_close) rrrrrr$rr'rr*r-rIrIrIrJrs  -   rcCsi}xYtjjD]H\}}|j}|r|dddkr|||dd proxy server URL mappings. Scan the environment for variables named _proxy; this seems to be the standard convention. If you need a different way, you can pass a proxies dictionary to the [Fancy]URLopener constructor. N_proxyZREQUEST_METHODriiiii)rSenvironrwrpr)rrWr~rIrIrJgetproxies_environmentZ s   r1c Cs|dkrt}y|d}Wntk r;dSYnX|dkrLdSt|\}}dd|jdD}xp|D]h}|r|jd }tj|}d |}tj||tjstj||tjrdSqWdS) zTest if proxies should not be used for a particular host. Checks the proxy dict for the value of no_proxy, which should be a list of comma separated DNS suffixes, or '*' for all hosts. Nnor*rNcSsg|]}|jqSrI)rL)rr rIrIrJr# s z,proxy_bypass_environment..r.z (.+\.)?%s$) r1ror r=lstriprNrrrP)rrrZno_proxyhostonlyr.Z no_proxy_listrWpatternrIrIrJproxy_bypass_environmenty s&       r8c Cszddlm}t|\}}dd}d|krH|drHdSd}x%|jd fD]}|spqatjd |}|dk r_|dkrytj|}||}Wntk rwaYnX||jd } |jd } | dkr$d |jd j dd } nt | d d} d| } || ?| | ?krrdSqa|||radSqaWdS)aj Return True iff this host shouldn't be accessed using a proxy This function uses the MacOSX framework SystemConfiguration to fetch the proxy information. proxy_settings come from _scproxy._get_proxy_settings or get mocked ie: { 'exclude_simple': bool, 'exceptions': ['foo.bar', '*.bar.com', '127.0.0.1', '10.1', '10.0/16'] } r)fnmatchcSs|jd}ttt|}t|dkrV|ddddgdd}|dd>|dd>B|dd>B|d BS) Nr4rrrNr`rrMr:)r=rrrZr\)ZipAddrr-rIrIrJip2num s  z,_proxy_bypass_macosx_sysconf..ip2numr4Zexclude_simpleTN exceptionsz(\d+(?:\.\d+)*)(/\d+)?rNrrM F) r9r rrNrrrrigroupcountrZ) rrproxy_settingsr9r6r.r;ZhostIPr~rr1maskrIrIrJ_proxy_bypass_macosx_sysconf s:        # rBdarwin)_get_proxy_settings _get_proxiescCst}t||S)N)rDrB)rrr@rIrIrJproxy_bypass_macosx_sysconf s rFcCstS)zReturn a dictionary of scheme -> proxy server URL mappings. This function uses the MacOSX framework SystemConfiguration to fetch the proxy information. )rErIrIrIrJgetproxies_macosx_sysconf srGcCs*t}|rt||St|SdS)zReturn True, if host should be bypassed. Checks proxy settings gathered from the environment, if specified, or from the MacOSX framework SystemConfiguration. N)r1r8rF)rrrrIrIrJr s  rcCstptS)N)r1rGrIrIrIrJr5 scCsfi}yddl}Wntk r.|SYnXy|j|jd}|j|dd}|r9t|j|dd}d|krx|jdD]J}|jdd\}}tjd |sd ||f}||| proxy server URL mappings. Win32 uses the registry to store proxies. rNz;Software\Microsoft\Windows\CurrentVersion\Internet Settings ProxyEnableZ ProxyServerrrrNz ^([^/:]+)://z%s://%srXzhttp:rz http://%sz https://%srzftp://%sr) winreg ImportErrorOpenKeyHKEY_CURRENT_USER QueryValueExrr=rNrZCloserir?r)rrIinternetSettings proxyEnableZ proxyServerprZaddressrIrIrJgetproxies_registry s8         rQcCstptS)zReturn a dictionary of scheme -> proxy server URL mappings. Returns settings gathered from the environment, if specified, or the registry. )r1rQrIrIrIrJr5( sc &Csyddl}Wntk r(dSYnXyK|j|jd}|j|dd}t|j|dd}Wntk rdSYnX| s| rdSt|\}}|g}y,tj |}||kr|j |Wntk rYnXy,tj |}||kr&|j |Wntk r;YnX|j d}x|D]} | dkrtd|krtdS| j dd } | j d d } | j d d} x*|D]"} tj| | tjrdSqWqRWdS) Nrz;Software\Microsoft\Windows\CurrentVersion\Internet SettingsrHZ ProxyOverriderzr4rNz\.r3z.*?)rIrJrKrLrMrrir rrrYZgetfqdnr=rrNrrP) rrrIrNrOZ proxyOverrideZrawHostr.ZaddrZfqdnr2rrIrIrJproxy_bypass_registry1 sR                   rScCs*t}|rt||St|SdS)zReturn True, if host should be bypassed. Checks proxy settings gathered from the environment, if specified, or the registry. N)r1r8rS)rrrrIrIrJrc s  )rrrrrcZ http.clientrrrSr/rNrrrrar}rVrPrAZ urllib.errorrrrZ urllib.parserrrrr r r r r rrrrrrrrrZurllib.responserrrArJr@__all__rrrCrr0r1rXr6r7rOASCIIrnrsrrr2rr/rrrrr r!r"r#r$r%urandomrbr&r'r(rwr)rrrBrYrr.rZr\r*rr+r,r-rrWZ nturl2pathr4r3rr8r9rrrrrrr r!rr1r8rBplatformZ_scproxyrDrErFrGrr5rQrSrIrIrIrJDs                 v          ! ?  n $q *@ P   r  +3 :5!      _ # <   - 2