Re#
@ s d Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l m
Z
y d d l m Z Wn" e
k
r d d l m Z Yn Xd d l m Z d d l m Z m Z m Z m Z d d l Z d d l m Z m Z m Z m Z m Z m Z m Z m Z m Z m! Z! m" Z" m# Z# d d l m$ Z$ d d l% m& Z& d d
l' m( Z( d d l) m* Z* d d l+ m, Z, d d
l- m. Z. e j/ d Z0 e j/ d e j1 Z2 e j/ d Z3 e j/ d e j1 j4 Z5 d j6 Z7 d d d d g Z8 d Z9 d Z: e: j; d e j< d d d e Z= d d Z> d d Z? d d Z@ d d! d ZA d d" d# ZB d d$ d% ZC d e d d& d ZD d d' d( ZE d) d* ZF e j/ d+ e j1 ZG eF d, d- ZH Gd. d/ d/ eI ZJ Gd0 d1 d1 eJ ZK Gd2 d d e ZL e j/ d3 jM ZN d4 d5 ZO d6 d7 ZP d8 d9 ZQ d d: d; ZR d< d= ZS Gd> d? d? eI ZT Gd@ dA dA e jU ZV e jW jX dB dC ZY eR e9 eY ZY dD dE ZZ dF dG Z[ d S)Hz#PyPI and direct package downloading N)wraps) splituser)six)urllib http_client configparsermap)
CHECKOUT_DIST Distribution BINARY_DISTnormalize_path SOURCE_DIST Environmentfind_distributions safe_name safe_version to_filename Requirement DEVELOP_DIST) ssl_support)log)DistutilsError) translate)strip_fragment)get_all_headersz^egg=([-A-Za-z0-9_.+!]+)$zhref\s*=\s*['"]?([^'"> ]+)z([^<]+)
\s+\(md5\)z([-+.a-z0-9]{2,}):z .tar.gz .tar.bz2 .tar .zip .tgz PackageIndexdistros_for_urlparse_bdist_wininstinterpret_distro_name z s z(interpret_distro_name..r: Nr8
py_versionrJ rW )r@ anyrangerT r
join)rU rV rO r` rJ rW rB r^ r% r% r& r s * 5 c c s t } | j } | d k rS xj t j j | j | D] } | | | Vq7 Wn8 x5 | D]- } | | } | | k rZ | | | VqZ Wd S)zHList unique elements, preserving order. Remember all elements ever seen.N)setaddr moves filterfalse __contains__)iterablekeyseenZseen_addelementkr% r% r& unique_everseen s
rn c s" t f d d } | S)zs
Wrap a function returning an iterable such that the resulting iterable
only ever yields unique items.
c s t | | S)N)rn )argskwargs)funcr% r& wrapper s z unique_values..wrapper)r )rq rr r% )rq r&
unique_values s rs z(<([^>]*\srel\s*=\s*['"]?([^'">]+)[^>]*)>c c s x t j | D] } | j \ } } t t t j | j j d } d | k sg d | k r x: t j | D]) } t
j j | t
| j d Vqw Wq Wxd d D]\ } | j | } | d k r t j | | } | r t
j j | t
| j d Vq Wd S)
zEFind rel="homepage" and rel="download" links in `page`, yielding URLs,Zhomepager9 r8
| Home Page | Download URLN)ru rv r< )RELfinditergroupsrd r strstripr0 r@ HREFr r" urljoin
htmldecoderN findsearch)rA pagerM tagrelZrelsposr% r% r& find_external_links s '+
r c @ s: e Z d Z d Z d d Z d d Z d d Z d S) ContentCheckerzP
A null content checker that defines the interface for checking content
c C s d S)z3
Feed a block of data to the hash.
Nr% )selfblockr% r% r& feed s zContentChecker.feedc C s d S)zC
Check the hash. Return False if validation fails.
Tr% )r r% r% r& is_valid s zContentChecker.is_validc C s d S)zu
Call reporter with information about the checker (hash name)
substituted into the template.
Nr% )r reportertemplater% r% r& report s zContentChecker.reportN)__name__
__module__ __qualname____doc__r r r r% r% r% r& r s r c @ sa e Z d Z e j d Z d d Z e d d Z d d Z d d Z
d
d Z d S)
HashCheckerzK(?Psha1|sha224|sha384|sha256|sha512|md5)=(?P[a-f0-9]+)c C s( | | _ t j | | _ | | _ d S)N) hash_namehashlibnewhashexpected)r r r r% r% r& __init__ s zHashChecker.__init__c C sR t j j | d } | s# t S| j j | } | sB t S| | j S)z5Construct a (possibly null) ContentChecker from a URLr8 r< )r r" r> r patternr groupdict)clsrA rH rM r% r% r& from_url s zHashChecker.from_urlc C s | j j | d S)N)r update)r r r% r% r& r s zHashChecker.feedc C s | j j | j k S)N)r hexdigestr )r r% r% r& r s zHashChecker.is_validc C s | | j } | | S)N)r )r r r msgr% r% r& r s
zHashChecker.reportN) r r r r\ compiler r classmethodr r r r r% r% r% r& r s r c s e Z d Z d Z d dJ d d d d Z d d d
Z d d d Z d d
d Z d d Z d d Z d d Z
d d Z d d d Z d d Z
d f d d Z d d Z d d Z d! d" Z d# d$ Z d% d& Z d d d d d' d( Z d d d) d* Z d+ d, Z d- Z d. d/ Z d0 d1 Z d d2 d3 Z d4 d5 Z d6 d7 Z d8 d9 Z d: d; Z d< d= Z e d d> d? Z! d@ dA Z" dB dC Z# dD dE Z$ dF dG Z% dH dI Z& S)Kr z;A distribution index that scans web pages for download URLsz https://pypi.python.org/simple*NTc O s t j | | | | d d | j d | _ i | _ i | _ i | _ t j d j t
t | j | _
g | _ | o t j o | p t j } | r t j | | _ n t j j | _ d S)Nr7 |)r r r1 index_url scanned_urls fetched_urls
package_pagesr\ r rc r r rM allowsto_scanr Z is_availableZfind_ca_bundleZ
opener_foropenerr requesturlopen)r r hostsZ ca_bundleZ
verify_sslro kwZuse_sslr% r% r& r $ s ! ' zPackageIndex.__init__Fc C s^ | | j k r | r d Sd | j | r r )r rA fatalsis_filer r% r% r& r { s !%zPackageIndex.url_okc C sE t t j j | } d d | D } t t j | j | d S)Nc s s@ | ]6 } t j | D] } | j d r | | f Vq q d S)z .egg-linkN)rY r r1 )r] rE entryr% r% r& r_ s z.PackageIndex.scan_egg_links..)filterrY rE r r itertoolsstarmap
scan_egg_link)r search_pathdirsZ egg_linksr% r% r& scan_egg_links s
zPackageIndex.scan_egg_linksc
C s t t j j | | ( } t t d t t j | } Wd QRXt | d k r\ d S| \ } } xQ t
t j j | | D]4 } t j j | | | _ t | _
| j | q Wd S)Nr: )openrY rE rc r r r rz r{ rT r rU r
rJ re )r rE r Z raw_lineslinesZegg_pathZ
setup_pathrP r% r% r& r s ( " zPackageIndex.scan_egg_linkc
s f d d } xX t j | D]G } y, | t j j | t | j d Wq" t k
rh Yq" Xq" W| | \ } } | rxv t | | D]e } t | \ } } | j
d r | r | r | d | | f 7} n
j | j | q Wt
j d d | Sd Sd S)
z#Process the contents of a PyPI pagec s | j j r t t t j j | t j d j d } t | d k r d | d k r t | d } t
| d } d j j | j
i | .scanr8 z.pyz
#egg=%s-%sc S s d | j d d d S)Nz%sr8 r r: )rN )mr% r% r& s z,PackageIndex.process_index..r N)r| rx r r" r} r~ rN r# r rI r1 need_version_infoscan_urlPYPI_MD5sub)
r rA r r rM r r new_urlr4 fragr% )r r& r s$ ,
zPackageIndex.process_indexc C s | j d | d S)NzPPage at %s links to .py file(s) without version info; an index scan is required.)scan_all)r rA r% r% r& r s z PackageIndex.need_version_infoc G sI | j | j k r5 | r( | j | | | j d | j | j d S)Nz6Scanning index of all packages (this may take a while))r r r r r )r r ro r% r% r& r s zPackageIndex.scan_allc C s | j | j | j d | j j | j sK | j | j | j d | j j | j sm | j | x3 t | j j | j f D] } | j | q Wd S)Nr7 ) r r unsafe_namer r rj project_namenot_found_in_indexr )r requirementrA r% r% r&
find_packages s
%zPackageIndex.find_packagesc sk | j | j | x8 | | j D]) } | | k r; | S| j d | | q% Wt t | j | | S)Nz%s does not match %s)prescanr rj r superr obtain)r r installerrP ) __class__r% r& r s
zPackageIndex.obtainc C sf | j | j d | | j sb | j t j | t d | j j t j j
| f d S)z-
checker is a ContentChecker
z Validating %%s checksum for %sz7%s validation failed for %s; possible download problem?N) r r r r rY unlinkr r r3 rE rV )r checkerrZ tfpr% r% r&
check_hash s
zPackageIndex.check_hashc C sr xk | D]c } | j d k sJ t | sJ | j d sJ t t | rZ | j | q | j j | q Wd S)z;Add `urls` to the list that will be prescanned for searchesNzfile:)r r r2 r r r append)r urlsrA r% r% r& add_find_links s
zPackageIndex.add_find_linksc C s/ | j r" t t | j | j d | _ d S)z7Scan urls scheduled for prescanning (e.g. --find-links)N)r r r r )r r% r% r& r s zPackageIndex.prescanc C sN | | j r | j d } } n | j d } } | | | j | j d S)Nz#Couldn't retrieve index page for %rz3Couldn't find index page for %r (maybe misspelled?))rj r r r r )r r methr r% r% r& r s
z PackageIndex.not_found_in_indexc C s t | t s t | } | ry | j | j d | | } t | \ } } | j d ru | j | | | } | St j j
| r | St | } t | j
| | d d S)a Locate and/or download `spec` to `tmpdir`, returning a local path
`spec` may be a ``Requirement`` object, or a string containing a URL,
an existing local filename, or a project/version requirement spec
(i.e. the string form of a ``Requirement`` object). If it is the URL
of a .py file with an unambiguous ``#egg=name-version`` tag (i.e., one
that escapes ``-`` as ``_`` throughout), a trivial ``setup.py`` is
automatically created alongside the downloaded file.
If `spec` is a ``Requirement`` object or a string containing a
project/version requirement spec, this method returns the location of
a matching distribution (possibly after downloading it to `tmpdir`).
If `spec` is a locally existing file or directory name, it is simply
returned unchanged. If `spec` is a URL, it is downloaded to a subpath
of `tmpdir`, and the local filename is returned. Various errors may be
raised if a problem occurs during downloading.
r8 z.pyrU N)r r r
_download_urlrN rI r1 gen_setuprY rE r r' r fetch_distribution)r r$ tmpdirrC foundr4 rH r% r% r& r9 s zPackageIndex.downloadc s: j d | i d } d f d d } | rf j j | | | } | r | d k r | | | } | d k r j d k r j | | } | d k r | r j | | | } | d k r j d rd p d | n# j d | | j d | j Sd S)
a| Obtain a distribution suitable for fulfilling `requirement`
`requirement` must be a ``pkg_resources.Requirement`` instance.
If necessary, or if the `force_scan` flag is set, the requirement is
searched for in the (online) package index as well as the locally
installed packages. If a distribution matching `requirement` is found,
the returned distribution's ``location`` is the value you would have
gotten from calling the ``download()`` method with the matching
distribution's URL or filename. If no matching distribution is found,
``None`` is returned.
If the `source` flag is set, only source distributions and source
checkout links will be considered. Unless the `develop_ok` flag is
set, development and system eggs (i.e., those using the ``.egg-info``
format) will be ignored.
zSearching for %sNc s | d k r } x | | j D] } | j t k re re | k r j d | d | .findz:No local packages or working download links found for %s%sza source distribution of r zBest match: %srU )r r r r r cloner ) r r r
force_scanr r Z local_indexrP r r% )r r r
r r r& r B s0 !
z PackageIndex.fetch_distributionc C s/ | j | | | | } | d k r+ | j Sd S)a3 Obtain a file suitable for fulfilling `requirement`
DEPRECATED; use the ``fetch_distribution()`` method now instead. For
backward compatibility, this routine is identical but returns the
``location`` of the downloaded distribution instead of a distribution
object.
N)r rU )r r r r
r rP r% r% r& fetch s zPackageIndex.fetchc
C s\ t j | } | r= d d t | | j d d D p@ g } t | d k r-t j j | } t j j | | k r t j j | | } d d l
m } | | | s t j
| | | } t t j j | d d ? } | j d | d j | d j t j j | d f Wd QRX| S| rLt d | | f n t d
d S) Nc S s g | ] } | j r | q Sr% )version)r] dr% r% r&
s z*PackageIndex.gen_setup..r8 r )samefilezsetup.pywzIfrom setuptools import setup
setup(name=%r, version=%r, py_modules=[%r])
zCan't unambiguously interpret project/version identifier %r; any dashes in the name or version should be escaped using underscores. %rzpCan't process plain .py files without an '#egg=name-version' suffix to enable automatic setup script generation.)rL rM r rN rT rY rE rV dirnamerc Z setuptools.command.easy_installr shutilcopy2r writer r splitextr )
r rZ rH r rM r rV dstr r r% r% r& r s2 !"zPackageIndex.gen_setupi c
C s | j d | d \ } } zSt j | } | j t | } t | t j j rw t d | | j
| j f | j } d } | j } d } d | k r t
| d }
t t t |
} | j | | | | | t | d w } xZ | j | } | rK| j | | j | | d 7} | j | | | | | q Pq W| j | | | Wd QRX| SWd | r| j Xd S)
NzDownloading %szCan't download %s: %s %sr r8 zcontent-lengthzContent-Lengthwb)NNr< )r r r r r r r r r r r r dl_blocksizer maxr int
reporthookr r r r r r )
r rA rZ fpr r r blocknumbssizesizesr r r% r% r& _download_to s:
zPackageIndex._download_toc C s d S)Nr% )r rA rZ r Zblksizer" r% r% r& r s zPackageIndex.reporthookc C s | j d r t | Sy t | | j SWnt t j f k
r } zS d j d d | j D } | r | j | | n t
d | | f WYd d } ~ XnIt j j
k
r } z | SWYd d } ~ Xnt j j k
r8} z: | r
| j | | j n t
d | | j f WYd d } ~ Xn t j k
r} z: | ri| j | | j n t
d | | j f WYd d } ~ Xn` t j t j f k
r} z4 | r| j | | n t
d | | f WYd d } ~ Xn Xd S)Nzfile: c S s g | ] } t | q Sr% )rz )r] argr% r% r& r s z)PackageIndex.open_url..z%s %szDownload error for %s: %sz;%s returned a bad status line. The server might be down, %s)r2
local_openopen_with_authr r# r
InvalidURLrc ro r r r r r URLErrorreason
BadStatusLineline
HTTPExceptionsocket)r rA warningvr r% r% r& r s6
(%% zPackageIndex.open_urlc C sK t | \ } } | rL x7 d | k rH | j d d j d d } q Wn d } | j d rq | d d } t j j | | } | d k s | j d r | j | | S| d
k s | j d r | j | | S| j d r | j | | S| d
k r't
j j t
j
j | d S| j | d | j | | Sd S)Nz...\_Z__downloaded__z.egg.zipr, svnzsvn+gitzgit+zhg+r r: Tr/ )rI replacer1 rY rE rc r2
_download_svn
_download_git _download_hgr r url2pathnamer" r> r _attempt_download)r rC rA r r3 rH rZ r% r% r& r
s$ % zPackageIndex._download_urlc C s | j | d d S)NT)r )r rA r% r% r& r ' s zPackageIndex.scan_urlc C sK | j | | } d | j d d j k rC | j | | | S| Sd S)Nr z content-typer )r$ r r0 _download_html)r rA rZ r r% r% r& r< * s z PackageIndex._attempt_downloadc C s t | } xT | D]L } | j r t j d | r^ | j t j | | j | | SPq W| j t j | t d | d S)Nz ([^- ]+ - )?Revision \d+:z Unexpected HTML page found at ) r r{ r\ r r rY r r8 r )r rA r rZ r r- r% r% r& r= 1 s
zPackageIndex._download_htmlc C si | j d d d } d } | j j d r8d | k r8t j j | \ } } } } } } | r8| j d r8d | d d k r8| d d j d d \ } } t | \ }
} |
r8d
|
k r |
j d
d \ } }
d | |
f } n
d |
} | } | | | | | | f } t j j | } | j d
| | t j
d | | | f | S)Nr; r8 r r zsvn:@z//r7 r: :z --username=%s --password=%sz --username=z'Doing subversion checkout from %s to %szsvn checkout%s -q %s %s) r@ r0 r2 r r" r> r
urlunparser rY system)r rA rZ credsrC netlocrE r^ qr authhostuserpwrB r% r% r& r8 @ s$ !$,"
zPackageIndex._download_svnc C s t j j | \ } } } } } | j d d d } | j d d d } d } d | k rw | j d d \ } } t j j | | | | d f } | | f S)N+r8 r; r r> r r< )r r" urlsplitr@ rsplit
urlunsplit)rA
pop_prefixrC rC rE rG r revr% r% r& _vcs_split_rev_from_urlU s ! !z$PackageIndex._vcs_split_rev_from_urlc C s | j d d d } | j | d d \ } } | j d | | t j d | | f | d k r | j d | t j d | | f | S)
Nr; r8 r rM Tz Doing git clone from %s to %szgit clone --quiet %s %szChecking out %sz"(cd %s && git checkout --quiet %s))r@ rO r rY rA )r rA rZ rN r% r% r& r9 g s zPackageIndex._download_gitc C s | j d d d } | j | d d \ } } | j d | | t j d | | f | d k r | j d | t j d | | f | S)
Nr; r8 r rM Tz Doing hg clone from %s to %szhg clone --quiet %s %szUpdating to %sz (cd %s && hg up -C -r %s >&-))r@ rO r rY rA )r rA rZ rN r% r% r& r: w s zPackageIndex._download_hgc G s t j | | d S)N)r r )r r ro r% r% r& r s zPackageIndex.debugc G s t j | | d S)N)r r )r r ro r% r% r& r s zPackageIndex.infoc G s t j | | d S)N)r r )r r ro r% r% r& r s zPackageIndex.warn)r )'r r r r r r r r r r r r r r r r r r r r9 r r r r r$ r r r r r< r= r8 staticmethodrO r9 r: r r r r% r% )r r& r ! sL 2
+ #D
) $ # z!&(#(\d+|x[\da-fA-F]+)|[\w.:-]+);?c C s6 t | t s | S| d k r, t j | St | S)N )r r r unichrchr)cr% r% r& uchr s
rU c C s | j d } | j d r: t | d d d } nL | j d rb t | d d } n$ t j j j j | | j d } t | S)Nr8 z#xr: r* r; r ) rN r2 r r rf
html_entitiesname2codepointr rU )rM whatr% r% r&
decode_entity s $rY c C s
t t | S)z'Decode HTML entities in the given text.)
entity_subrY )textr% r% r& r~ s r~ c s f d d } | S)Nc s f d d } | S)Nc s? t j } t j z | | SWd t j | Xd S)N)r/ getdefaulttimeoutsetdefaulttimeout)ro rp Z old_timeout)rq timeoutr% r& _socket_timeout s
z@socket_timeout.<locals>._socket_timeout.<locals>._socket_timeoutr% )rq r_ )r^ )rq r& r_ s z'socket_timeout.<locals>._socket_timeoutr% )r^ r_ r% )r^ r& socket_timeout s r` c C sI t j j | } | j } t j | } | j } | j d d S)aq
A function compatible with Python 2.3-3.3 that will encode
auth from a URL suitable for an HTTP header.
>>> str(_encode_auth('username%3Apassword'))
'dXNlcm5hbWU6cGFzc3dvcmQ='
Long auth strings should not cause a newline to be inserted.
>>> long_auth = 'username:' + 'password'*10
>>> chr(10) in str(_encode_auth(long_auth))
False
r )r r" r? encodebase64 encodestringr r7 )rE Zauth_sZ
auth_bytesZ
encoded_bytesencodedr% r% r& _encode_auth s
rf c @ s: e Z d Z d Z d d Z d d Z d d Z d S)
Credentialz:
A username/password pair. Use like a namedtuple.
c C s | | _ | | _ d S)N)usernamepassword)r rh ri r% r% r& r s zCredential.__init__c c s | j V| j Vd S)N)rh ri )r r% r% r& __iter__ s zCredential.__iter__c C s d t | S)Nz%(username)s:%(password)s)vars)r r% r% r& __str__ s zCredential.__str__N)r r r r r rj rl r% r% r% r& rg s rg c @ sF e Z d Z d d Z e d d Z d d Z d d Z d S)
PyPIConfigc C sr t j d d d g d } t j j | | t j j t j j d d } t j j | rn | j
| d S)z%
Load from ~/.pypirc
rh ri
repositoryr ~z.pypircN) dictfromkeysr RawConfigParserr rY rE rc
expanduserr r )r defaultsrcr% r% r& r s
!zPyPIConfig.__init__c s5 f d d j D } t t j | S)Nc s. g | ]$ } j | d j r | q S)rn )r r{ )r] section)r r% r& r s z2PyPIConfig.creds_by_repository.<locals>.<listcomp>)sectionsrp r _get_repo_cred)r Zsections_with_repositoriesr% )r r& creds_by_repository s z PyPIConfig.creds_by_repositoryc C sO | j | d j } | t | j | d j | j | d j f S)Nrn rh ri )r r{ rg )r rv repor% r% r& rx s zPyPIConfig._get_repo_credc C s7 x0 | j j D] \ } } | j | r | Sq Wd S)z
If the URL indicated appears to be a repository defined in this
config, return the credential for that repository.
N)ry itemsr2 )r rA rn credr% r% r& find_credential s zPyPIConfig.find_credentialN)r r r r propertyry rx r} r% r% r% r& rm s rm c C s t j j | \ } } } } } } | j d rB t j d | d
k rc t | \ } } n d } | s t j | }
|
r t |
} |
j
| f } t j d | | rd t
| } | | | | | | f } t j j | }
t j j |
} | j d | n t j j | } | j d t | | } | rt j j | j \ } } } } } } | | k r| | k r| | | | | | f } t j j | | _ | S) z4Open a urllib2 request, handling HTTP authenticationr? znonnumeric port: ''httphttpsNz*Authenticating as %s for %s (from .pypirc)zBasic
Authorizationz
User-Agent)r r )r r" r> r1 r r) r rm r} rz rh r r rf r@ r Request
add_header
user_agentrA )rA r rC rC rE paramsrG r rE rF r| r rB r r r s2h2path2Zparam2Zquery2Zfrag2r% r% r& r( s6 $ 'r( c C s | S)Nr% )rA r% r% r&
fix_sf_url< s r c C s t j j | \ } } } } } } t j j | } t j j | rX t j j | S| j d rNt j j
| rNg } x t j | D] } t j j | | }
| d k r t
|
d } | j } Wd QRXPn t j j
|
r | d 7} | j d j d | q Wd }
|
j d | d d
j | } d \ } } n d \ } } } d d i } t j | } t j j | | | | | S)z7Read a local path, with special support for directoriesr7 z
index.htmlrNz<a href="{name}">{name}</a>r3 zB<html><head><title>{url}{files} |