U cf7@s.ddlZddlmZddlZddlZddlZddlZddlZz ddlZWne k rdddl ZYnXddl Z ddl m Z ddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-dd l.m/Z/m0Z0dd l1m2Z2m3Z3e4e5Z6e7d Z8e7d ej9Z:e7d Z;dZGddde?Z@Gddde@ZAGddde@ZBGddde?ZCGddde@ZDGddde@ZEGdd d e@ZFGd!d"d"e@ZGGd#d$d$e@ZHeHeDd%d&d'd(d)ZIeIjJZJGd*d+d+e?ZKdS)-N)BytesIO)DistlibException)urljoinurlparse urlunparse url2pathname pathname2urlqueuequoteunescape build_openerHTTPRedirectHandler text_typeRequest HTTPErrorURLError) DistributionDistributionPath make_dist)MetadataMetadataInvalidError)cached_property ensure_slashsplit_filenameget_project_dataparse_requirementparse_name_and_version ServerProxynormalize_name) get_schemeUnsupportedVersionError)Wheel is_compatiblez^(\w+)=([a-f0-9]+)z;\s*charset\s*=\s*(.*)\s*$ztext/html|application/x(ht)?mlzhttps://pypi.org/pypicCs6|dkr t}t|dd}z |WS|dXdS)z Return all distribution names known by an index. :param url: The URL of the index. :return: A list of all known distribution names. N@timeoutclose) DEFAULT_INDEXr list_packages)urlclientr,J/opt/hc_python/lib/python3.8/site-packages/pip/_vendor/distlib/locators.pyget_all_distribution_names*s   r.c@s$eZdZdZddZeZZZdS)RedirectHandlerzE A class to work around a bug in some Python 3.2.x releases. c Csd}dD]}||kr||}q"q|dkr.dSt|}|jdkrnt||}t|drf|||n|||<t||||||S)N)locationurireplace_header)rschemer get_full_urlhasattrr3BaseRedirectHandlerhttp_error_302) selfreqfpcodemsgheadersnewurlkeyurlpartsr,r,r-r8Bs   zRedirectHandler.http_error_302N)__name__ __module__ __qualname____doc__r8http_error_301http_error_303http_error_307r,r,r,r-r/9sr/c@seZdZdZdZdZdZdZedZd)dd Z d d Z d d Z ddZ ddZ ddZee eZddZddZddZddZddZddZd d!Zd"d#Zd$d%Zd*d'd(ZdS)+LocatorzG A base class for locators - things that locate distributions. )z.tar.gzz.tar.bz2z.tarz.zipz.tgzz.tbz)z.eggz.exe.whl)z.pdfN)rJdefaultcCs,i|_||_tt|_d|_t|_dS)a^ Initialise an instance. :param scheme: Because locators look for most recent versions, they need to know the version scheme to use. This specifies the current PEP-recommended scheme - use ``'legacy'`` if you need to support existing distributions on PyPI. N) _cacher4r r/openermatcherr Queueerrors)r9r4r,r,r-__init__is  zLocator.__init__cCsVg}|jsRz|jd}||Wn|jjk rDYqYnX|jq|S)z8 Return any errors which have occurred. F)rPemptygetappendEmpty task_done)r9resulter,r,r- get_errors|s    zLocator.get_errorscCs |dS)z> Clear any errors which may have been logged. N)rYr9r,r,r- clear_errorsszLocator.clear_errorscCs|jdSN)rLclearrZr,r,r- clear_cacheszLocator.clear_cachecCs|jSr\_schemerZr,r,r- _get_schemeszLocator._get_schemecCs ||_dSr\r_)r9valuer,r,r- _set_schemeszLocator._set_schemecCs tddS)a= For a given project, get a dictionary mapping available versions to Distribution instances. This should be implemented in subclasses. If called from a locate() request, self.matcher will be set to a matcher for the requirement to satisfy, otherwise it will be None. Please implement in the subclassNNotImplementedError)r9namer,r,r- _get_projects zLocator._get_projectcCs tddS)J Return all the distribution names known to this locator. rdNrerZr,r,r-get_distribution_namesszLocator.get_distribution_namescCsL|jdkr||}n2||jkr,|j|}n|||}||j|<|S)z For a given project, get a dictionary mapping available versions to Distribution instances. This calls _get_project to do all the work, and just implements a caching layer on top. N)rLrhr[)r9rgrWr,r,r- get_projects      zLocator.get_projectcCs^t|}t|j}d}|d}||j}|rBtt||j}|j dkd|j k||||fS)zu Give an url a score which can be used to choose preferred URLs for a given project release. TrJhttpszpypi.org) r posixpathbasenamepathendswithdownloadable_extensionsr#r" wheel_tagsr4netloc)r9r*trn compatibleis_wheelZis_downloadabler,r,r- score_urls   zLocator.score_urlcCsR|}|rN||}||}||kr(|}||kr@td||ntd|||S)a{ Choose one of two URLs where both are candidates for distribution archives for the same version of a distribution (for example, .tar.gz vs. zip). The current implementation favours https:// URLs over http://, archives from PyPI over those from other locations, wheel compatibility (if a wheel) and then the archive name. zNot replacing %r with %rzReplacing %r with %r)rwloggerdebug)r9url1url2rWs1s2r,r,r- prefer_urls   zLocator.prefer_urlcCs t||S)zZ Attempt to split a filename in project name, version and Python version. )r)r9filename project_namer,r,r-rszLocator.split_filenamec Csdd}d}t|\}}}}} } | dr.same_projectNzegg=z %s: version hint in fragment: %r)NN/rJzWheel not compatible: %sTr2z, cSs"g|]}dt|ddqS).N)joinlist).0vr,r,r- sz8Locator.convert_url_to_download_info..)rgversionrr*python-versionzinvalid path for wheel: %szNot downloadable: %sz No match for project/version: %s)rgrrr*r %s_digest)rlower startswithrxry HASHER_HASHmatchgroupsrpr"r#rrrgrrrrpyver Exceptionwarningrqrmrnlenr)r9r*rrrWr4rsroparamsqueryfragmalgodigestZorigpathwheelincluderextrtrgrrr,r,r-convert_url_to_download_infos             z$Locator.convert_url_to_download_infocCshd}d|kr6|d}dD]}||kr|||f}q6q|sddD]$}d|}||kr>|||f}qdq>|S)z Get a digest from a dictionary by looking at a "digests" dictionary or keys of the form 'algo_digest'. Returns a 2-tuple (algo, digest) if found, else None. Currently looks only for SHA256, then MD5. Ndigests)sha256md5rr,)r9inforWrrr@r,r,r- _get_digest3s  zLocator._get_digestc Cs|d}|d}||kr,||}|j}nt|||jd}|j}|||_}|d}||d|<|j|dkr||j||_|d|t  |||_ |||<dS)z Update a result dictionary (the final result from _get_project) with a dictionary for a specific version, which typically holds information gleaned from a filename or URL for an archive for the distribution. rgrr4r*rurlsN) popmetadatarr4rr source_urlr~ setdefaultsetaddlocator) r9rWrrgrdistmdrr*r,r,r-_update_version_dataJs   zLocator._update_version_dataFc Csd}t|}|dkr td|t|j}||j|_}td|t|j | |j }t |dkrg}|j } |D]Z} | dkrqxz(|| sn|s| | js|| Wqxtk rtd|| YqxXqxt |dkrt||jd}|rtd ||d } || }|rx|jr&|j|_|d i| t|_i} |d i} |jD]}|| krT| || |<qT| |_d|_|S) a Find the most recent distribution which matches the given requirement. :param requirement: A requirement of the form 'foo (1.0)' or perhaps 'foo (>= 1.0, < 2.0, != 1.3)' :param prereleases: If ``True``, allow pre-release versions to be located. Otherwise, pre-release versions are not returned. :return: A :class:`Distribution` instance, or ``None`` if no such distribution could be located. NzNot a valid requirement: %rzmatcher: %s (%s)rrrzerror matching %s with %rr)r@zsorted list: %srrr)rrr r4rN requirementrxrytyperBrkrgrZ version_classr is_prereleaserTrrsortedr@extrasrSr download_urlsr)r9r prereleasesrWrr4rNversionsslistZvclskrdsdr*r,r,r-locateasP          zLocator.locate)rK)F)rBrCrDrEsource_extensionsbinary_extensionsexcluded_extensionsrrrqrQrYr[r^rarcpropertyr4rhrjrkrwr~rrrrrr,r,r,r-rIYs.   IrIcs0eZdZdZfddZddZddZZS)PyPIRPCLocatorz This locator uses XML-RPC to locate distributions. It therefore cannot be used with simple mirrors (that only mirror file content). c s*tt|jf|||_t|dd|_dS)z Initialise an instance. :param url: The URL to use for XML-RPC. :param kwargs: Passed to the superclass constructor. r$r%N)superrrQbase_urlrr+r9r*kwargs __class__r,r-rQszPyPIRPCLocator.__init__cCst|jSri)rr+r)rZr,r,r-rjsz%PyPIRPCLocator.get_distribution_namesc Csiid}|j|d}|D]}|j||}|j||}t|jd}|d|_|d|_|d|_ |dg|_ |d|_ t |}|r|d } | d |_ || |_||_|||<|D]:} | d } || } |d |t| | |d | <qq|S) NrTrrgrlicensekeywordssummaryrr*rr)r+Zpackage_releasesZ release_urlsZ release_datarr4rgrrSrrrrrrrrrrr) r9rgrWrrrdatarrrr*rr,r,r-rhs0         zPyPIRPCLocator._get_projectrBrCrDrErQrjrh __classcell__r,r,rr-rs rcs0eZdZdZfddZddZddZZS)PyPIJSONLocatorzw This locator uses PyPI's JSON interface. It's very limited in functionality and probably not worth using. c s tt|jf|t||_dSr\)rrrQrrrrr,r-rQszPyPIJSONLocator.__init__cCs tddSrizNot available from this locatorNrerZr,r,r-rjsz&PyPIJSONLocator.get_distribution_namesc Csiid}t|jdt|}z|j|}|}t|}t |j d}|d}|d|_ |d|_ | d|_| dg|_| d |_t|}||_|||j <|d D]T} | d }|j||| |j|<|d |j t||| |d |<q|d D]\} } | |j kr,qt |j d} |j | _ | | _ t| } || _| || <| D]T} | d }| j||| | j|<|d | t||| |d |<q`qWn@tk r}z |jt|td|W5d}~XYnX|S)Nrz%s/jsonrrrgrrrrrr*rZreleaseszJSON fetch failed: %s) rrr rMopenreaddecodejsonloadsrr4rgrrSrrrrrrrrrrritemsrrPputrrx exception)r9rgrWr*resprrrrrrinfosZomdodistrXr,r,r-rhsR                zPyPIJSONLocator._get_projectrr,r,rr-rs rc@s`eZdZdZedejejBejBZ edejejBZ ddZ edejZ e ddZd S) Pagez4 This class represents a scraped HTML page. z (rel\s*=\s*(?:"(?P[^"]*)"|'(?P[^']*)'|(?P[^>\s ]*))\s+)? href\s*=\s*(?:"(?P[^"]*)"|'(?P[^']*)'|(?P[^>\s ]*)) (\s+rel\s*=\s*(?:"(?P[^"]*)"|'(?P[^']*)'|(?P[^>\s ]*)))? z!]+)cCs4||_||_|_|j|j}|r0|d|_dS)zk Initialise an instance with the Unicode page contents and the URL they came from. rN)rrr*_basesearchgroup)r9rr*rr,r,r-rQ s  z Page.__init__z[^a-z0-9$&+,/:;=?@.#%_\\|-]cCsdd}t}|j|jD]}|d}|dpX|dpX|dpX|dpX|dpX|d }|d pp|d pp|d }t|j|}t|}|j d d|}| ||fqt |dddd}|S)z Return the URLs of all the links on a page together with information about their "rel" attribute, for determining which ones to treat as downloads and which ones to queue for further scraping. cSs,t|\}}}}}}t||t||||fS)zTidy up an URL.)rrr )r*r4rsrorrrr,r,r-clean4s  zPage.links..cleanr2Zrel1Zrel2Zrel3Zrel4Zrel5Zrel6rzr{Zurl3cSsdt|dS)Nz%%%2xr)ordr)rr,r,r-BzPage.links..cSs|dS)Nrr,)rtr,r,r-rFrT)r@reverse) r_hreffinditerr groupdictrrr _clean_resubrr)r9rrWrrrelr*r,r,r-links-s$  z Page.linksN)rBrCrDrErecompileISXrrrQrrrr,r,r,r-rs rcseZdZdZejdddddZdfdd Zd d Zd d Z ddZ e de j ZddZddZddZddZddZe dZddZZS)SimpleScrapingLocatorz A locator which scrapes HTML pages to locate downloads for a distribution. This runs multiple threads to do the I/O; performance is at least as good as pip's PackageFinder, which works in an analogous fashion. cCstjt|dS)N)fileobj)gzipGzipFilerrbr,r,r-rTrzSimpleScrapingLocator.cCs|Sr\r,rr,r,r-rUr)deflaternoneN c sltt|jf|t||_||_i|_t|_t |_ t|_ d|_ ||_t|_t|_d|_dS)a Initialise an instance. :param url: The root URL to use for scraping. :param timeout: The timeout, in seconds, to be applied to requests. This defaults to ``None`` (no timeout specified). :param num_workers: The number of worker threads you want to do I/O, This defaults to 10. :param kwargs: Passed to the superclass. FN)rrrQrrr& _page_cacher_seenr rO _to_fetch _bad_hostsskip_externals num_workers threadingRLock_lock_gplockplatform_check)r9r*r&rrrr,r-rQXs     zSimpleScrapingLocator.__init__cCsBg|_t|jD],}tj|jd}d|_||j|qdS)z Threads are created only when get_project is called, and terminate before it returns. They are there primarily to parallelise I/O (i.e. fetching web pages). )targetTN) _threadsrangerrThread_fetchdaemonstartrT)r9irtr,r,r-_prepare_threadsss z&SimpleScrapingLocator._prepare_threadscCs6|jD]}|jdq|jD] }|qg|_dS)zu Tell all the threads to terminate (by sending a sentinel value) and wait for them to do so. N)r rrr)r9rtr,r,r- _wait_threadss    z#SimpleScrapingLocator._wait_threadsc Csiid}|jx||_||_t|jdt|}|j|j| z&t d||j ||j W5| X|`W5QRX|S)Nrz%s/z Queueing %s)rrWrrrr rr]rrrrxryrrr)r9rgrWr*r,r,r-rhs      z"SimpleScrapingLocator._get_projectz<\b(linux_(i\d86|x86_64|arm\w+)|win(32|_amd64)|macosx_?\d+)\bcCs |j|S)zD Does an URL refer to a platform-specific download? )platform_dependentr)r9r*r,r,r-_is_platform_dependentsz,SimpleScrapingLocator._is_platform_dependentc CsZ|jr||rd}n|||j}td|||rV|j||j|W5QRX|S)a% See if an URL is a suitable download for a project. If it is, register information in the result dictionary (for _get_project) about the specific version it's for. Note that the return value isn't actually used other than as a boolean value. Nzprocess_download: %s -> %s) rrrrrxryrrrW)r9r*rr,r,r-_process_downloads z'SimpleScrapingLocator._process_downloadc Cst|\}}}}}}||j|j|jr2d}n||jrJ||jsJd}nd||js\d}nR|dkrjd}nD|dkrxd}n6||rd}n&| ddd} | dkrd}nd}t d |||||S) z Determine whether a link URL from a referring page and with a particular "rel" attribute should be queued for scraping. F)Zhomepagedownload)httprlftp:rr localhostTz#should_queue: %s (%s) from %s -> %s) rrprrrrrrrsplitrrxry) r9linkZreferrerrr4rsro_rWhostr,r,r- _should_queues0    z#SimpleScrapingLocator._should_queuec Cs|j}zz|r||}|dkr,WWq|jD]j\}}||jkr2zB|j|||s||||rt d|||j |Wq2t k rYq2Xq2Wn2t k r}z|j t|W5d}~XYnXW5|jX|sqqdS)z Get a URL to fetch from the work queue, get the HTML page, examine its links for download candidates and candidates for further scraping. This is a handy method to run in a thread. NzQueueing %s from %s)rrSrVget_pagerrrrrrxryrrrrPr)r9r*pagerrrXr,r,r-r s,       & zSimpleScrapingLocator._fetchc CsXt|\}}}}}}|dkr:tjt|r:tt|d}||jkr`|j|}t d||n| ddd}d}||j krt d||nt |d d id }zzt d ||j j||jd } t d|| } | dd} t| r| } | } | d}|r"|j|}|| } d}t| }|r@|d}z| |} Wn tk rn| d} YnXt| | }||j| <Wntk r}z|jdkrtd||W5d}~XYnt k r}z0td|||j!|j "|W5QRXW5d}~XYn2t#k rB}ztd||W5d}~XYnXW5||j|<X|S)a Get the HTML for an URL, possibly from an in-memory cache. XXX TODO Note: this cache is never actually cleared. It's assumed that the data won't get stale over the lifetime of a locator instance (not necessarily true for the default_locator). filez index.htmlzReturning %s from cache: %srrrNzSkipping %s due to bad host %szAccept-encodingidentity)r>z Fetching %sr%z Fetched %sz Content-Typer2zContent-Encodingzutf-8zlatin-1izFetch failed: %s: %s)$rosroisdirrrrrrxryrrrrMrr&rrSHTML_CONTENT_TYPErgeturlrdecodersCHARSETrrr UnicodeErrorrrr<rrrrr)r9r*r4rsrorrWrr:rr> content_typeZ final_urlrencodingdecoderrrXr,r,r-r sZ              &$ zSimpleScrapingLocator.get_pagez]*>([^<]+)= 1.0) while C requires (B >= 1.1). For successful replacement, ``provider`` must meet all the requirements which ``other`` fulfills. :param provider: The provider we are trying to replace with. :param other: The provider we're trying to replace. :param problems: If False is returned, this will contain what problems prevented replacement. This is currently a tuple of the literal string 'cantreplace', ``provider``, ``other`` and the set of requirements that ``provider`` couldn't fulfill. :return: True if we can replace ``other`` with ``provider``, else False. Z cantreplaceFT) reqtsrr]rrr frozensetr[rrX) r9r^otherproblemsZrlist unmatchedrZrNrWr,r,r-try_to_replaces$       zDependencyFinder.try_to_replaceFcCsi|_i|_i|_i|_t|p g}d|krH|d|tdddgO}t|trh|}}t d|n4|j j ||d}}|dkrt d|t d |d |_ t}t|g}t|g}|r|}|j} | |jkr||n"|j| } | |kr||| ||j|jB} |j} t} |r`||kr`d D]*}d |}||kr4| t|d |O} q4| | B| B}|D].}||}|sFt d||j j ||d}|dkr|s|j j |d d}|dkrt d||d|fn^|j|j}}||f|jkr|||||| krF||krF||t d|j|D]R}|j} | |jkrx|j|t|n"|j| } | |krJ||| |qJqpqt|j}|D]&}||k|_|jrt d|jqt d|||fS)a Find a distribution and all distributions it depends on. :param requirement: The requirement specifying the distribution to find, or a Distribution instance. :param meta_extras: A list of meta extras such as :test:, :build: and so on. :param prereleases: If ``True``, allow pre-release versions to be returned - otherwise, don't return prereleases unless they're all that's available. Return a set of :class:`Distribution` instances and a set of problems. The distributions returned should be such that they have the :attr:`required` attribute set to ``True`` if they were from the ``requirement`` passed to ``find()``, and they have the :attr:`build_time_dependency` attribute set to ``True`` unless they are post-installation dependencies of the ``requirement``. The problems should be a tuple consisting of the string ``'unsatisfied'`` and the requirement which couldn't be satisfied by any distribution known to the locator. z:*:z:test:z:build:z:dev:zpassed %s as requirement)rNzUnable to locate %rz located %sT)testbuilddevz:%s:z %s_requireszNo providers found for %rzCannot satisfy %rZ unsatisfiedzAdding %s to install_distsz#%s is a build-time dependency only.zfind done for %s)rVrTrSr`rrYrDrrxryrrr requestedrr@rXreZ run_requiresZ meta_requiresZbuild_requiresgetattrr_rrZname_and_versionrvaluesZbuild_time_dependency)r9rZ meta_extrasrrrrctodoZ install_distsrgrbZireqtsZsreqtsZereqtsr@rXZ all_reqtsr providersr^nrrWrTr,r,r-finds                            zDependencyFinder.find)N)NF) rBrCrDrErQrXr[r]r_reror,r,r,r-rQ-s (rQ)N)Lriorrloggingr$rmrr ImportErrorZdummy_threadingr/r2rcompatrrrrr r r r r rr7rrrrZdatabaserrrrrrutilrrrrrrrrrr r!rr"r# getLoggerrBrxrrrr)r&r(r.r/objectrIrrrrr1r<rCrHrRrrQr,r,r,r-sZ   @(     C1E:{B'[