a \wiĔ@sxddlmZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z z ddlZWneyzdZYn0ddlmZddlmZmZdd lmZmZmZmZmZmZmZmZmZmZm Z dd l!m"Z"dd l#m$Z$m%Z%e&e'Z(Gd d d e)Z*Gddde)Z+Gddde)Z,edur   zDeflateDecoder.decompresscCst|jpt|jjo|j Sr%)boolrrr.rr r"r"r#has_unconsumed_tailZs z"DeflateDecoder.has_unconsumed_tailN)r+__name__ __module__ __qualname__r$r*r/propertyr3r"r"r"r#r&s  *rc@seZdZdZdZdZdS)GzipDecoderStaterrN)r5r6r7 FIRST_MEMBER OTHER_MEMBERS SWALLOW_DATAr"r"r"r#r9asr9c@s2eZdZddZddZd ddZedd Zd S) GzipDecodercCs$tdtj|_tj|_d|_dS)Nr)rrr0rr9r;_state_unconsumed_tailr r"r"r#r$iszGzipDecoder.__init__cCs t|j|Sr%r&r(r"r"r#r*nszGzipDecoder.__getattr__r+cCs t}|jtjkrt|S|dkr4|j|7_dS|j|}|sR|jjrRt|Sz&||jj|t |t |dd7}Wn@t j y|j}tj|_d|_|tj krt|YSYn0|jjp|jj|_}|dkrt ||krq|st|S|jjrRtj |_t dt j|_qRt|S)Nrrr,r?) bytearrayr@r9r=bytesrAreofr/maxlenrrr<r. unused_datarr0)r!r1r-retZprevious_stater"r"r#r/qs<      zGzipDecoder.decompresscCs t|jSr%)r2rAr r"r"r#r3szGzipDecoder.has_unconsumed_tailN)r+r4r"r"r"r#r>hs  2r>c@s<eZdZddZd ddZdddZedd Zd d Zd S) BrotliDecodercCs0t|_t|jdr"|jj|_n |jj|_dS)Nr/)brotliZ Decompressorrhasattrr/ _decompressZprocessr r"r"r#r$s   zBrotliDecoder.__init__r+cCs tdSr%)NotImplementedError)r!r1output_buffer_limitr"r"r#rLszBrotliDecoder._decompresscCsVz(|dkr|j||dWS||WSWn(tyPtdt||YS0dS)Nr)rNz;Brotli >= 1.2.0 is required to prevent decompression bombs.)rL TypeErrorwarningswarnr )r!r1r-r"r"r#r/s zBrotliDecoder.decompresscCs(z|j WSty"YdS0dS)NF)rZcan_accept_more_dataAttributeErrorr r"r"r#r3s z!BrotliDecoder.has_unconsumed_tailcCst|jdr|jSdS)Nflushr)rKrrSr r"r"r#rSs  zBrotliDecoder.flushN)r+)r+) r5r6r7r$rLr/r8r3rSr"r"r"r#rIs    rIc@s:eZdZdZdZddZddZd dd Zed d Z d S) MultiDecodera From RFC7231: If one or more encodings have been applied to a representation, the sender that applied the encodings MUST generate a Content-Encoding header field that lists the content codings in the order in which they were applied. cCsPdd|dD}t||jkrrz)MultiDecoder.__init__..,z)Too many content encodings in the chain: z > cSsg|] }t|qSr") _get_decoderrXer"r"r#rZr)splitrFmax_decode_linksr _decoders)r!Zmodes encodingsr"r"r#r$szMultiDecoder.__init__cCs|jdSNr)rarSr r"r"r#rSszMultiDecoder.flushr+cCs|dkr&t|jD]}||}q|St}d}t|jD]"}|j||t|d}|r:d}q:||7}|rvt||kr~t|Sd}q,dS)NrFr,Tr)reversedrar/rBrFrC)r!r1r-drHZany_datar"r"r#r/s zMultiDecoder.decompresscCstdd|jDS)Ncss|] }|jVqdSr%)r3)rXrer"r"r# rz3MultiDecoder.has_unconsumed_tail..)anyrar r"r"r#r3sz MultiDecoder.has_unconsumed_tailN)r+) r5r6r7__doc__r`r$rSr/r8r3r"r"r"r#rTs   rTcCs:d|vrt|S|dkrtStdur4|dkr4tStS)Nr[gzipbr)rTr>rJrIr)moder"r"r#r\ sr\c@s0eZdZdZddZddZddZdd Zd S) BytesQueueBufferaMemory-efficient bytes buffer To return decoded data in read() and still follow the BufferedIOBase API, we need a buffer to always return the correct amount of bytes. This buffer should be filled using calls to put() Our maximum memory usage is determined by the sum of the size of: * self.buffer, which contains the full data * the largest chunk that we will copy in get() cCst|_d|_dSrc) collectionsdequebuffer_sizer r"r"r#r$(s zBytesQueueBuffer.__init__cCs|jSr%)rpr r"r"r#__len__,szBytesQueueBuffer.__len__cCs"|j||jt|7_dSr%)roappendrprF)r!r1r"r"r#put/s zBytesQueueBuffer.putc Cs|dkr dS|jstdn|dkr,tdt|jd|krft|jdtrf|j|8_|jSd}t }||kr||}|j}t|}||kr|d|||d}}| ||j ||j|8_qn| ||j|8_||7}|jsrqqr| S)Nrrzbuffer is emptyzn should be > 0) ro RuntimeError ValueErrorrF isinstancerCrppopleftioBytesIOwrite appendleftgetvalue) r!nZfetchedrHZ remainingchunkZ chunk_lengthZ left_chunkZ right_chunkr"r"r#get3s6 "      zBytesQueueBuffer.getN)r5r6r7rhr$rqrsrr"r"r"r#rls  rlc@sreZdZdZddgZedur&edg7ZgdZdLd d Zd dZddZ ddZ e ddZ e ddZ ddZddZddZddZeejfZedureejf7ZdMdd Zd!d"Zed#d$ZdNd%d&ZdOd'd(ZdPd*d+Zed,d-Zd.d/ZdQd0d1Z d2d3Z!d4d5Z"e d6d7Z#d8d9Z$d:d;Z%dd?Z'd@dAZ(dBdCZ)dDdEZ*dRdFdGZ+dHdIZ,dJdKZ-dS)S HTTPResponsea HTTP Response container. Backwards-compatible with :class:`http.client.HTTPResponse` but the response ``body`` is loaded and decoded on-demand when the ``data`` property is accessed. This class is also compatible with the Python standard library's :mod:`io` module, and can hence be treated as a readable object in the context of that framework. Extra parameters for behaviour not present in :class:`http.client.HTTPResponse`: :param preload_content: If True, the response's body will be preloaded during construction. :param decode_content: If True, will attempt to decode the body based on the 'content-encoding' header. :param original_response: When this HTTPResponse wrapper is generated from an :class:`http.client.HTTPResponse` object, it's convenient to include the original for debug purposes. It's otherwise unused. :param retries: The retries contains the last :class:`~urllib3.util.retry.Retry` that was used during the request. :param enforce_content_length: Enforce content length checking. Body returned by server must match value of Content-Length header, if present. Otherwise, raise error. riZdeflateNrj)i-i.i/i3i4rTFcCs&t|tr||_n t||_||_||_||_||_||_d|_| |_ ||_ ||_ d|_ d|_ d|_| |_d|_| |_||_|rt|tjtfr||_ | |_| |_t|dr||_d|_d|_|jdd}dd|dD}d |vrd |_|||_t |_!|r"|j s"|j"|d |_ dS) NFrreadztransfer-encodingrcss|]}|VqdSr%rV)rXencr"r"r#rfrz(HTTPResponse.__init__..r[chunkedTdecode_content)#rvrheadersstatusversionreasonstrictr_has_decoded_contentretriesenforce_content_length auto_close_decoder_body_fp_original_response_fp_bytes_readmsg _request_urlrZ string_typesrC_pool _connectionrKr chunk_leftrlowerr_ _init_lengthlength_remainingrl_decoded_bufferr)r!bodyrrrrrZpreload_contentroriginal_responseZpool connectionrrrrequest_methodZ request_urlrZtr_encrbr"r"r#r$|sF    zHTTPResponse.__init__cCs|j|jvr|jdSdS)a Should we redirect and where to? :returns: Truthy redirect location string if we got a redirect status code and valid location. ``None`` if redirect status and no location. ``False`` if not a redirect status code. locationF)rREDIRECT_STATUSESrrr r"r"r#get_redirect_locations  z"HTTPResponse.get_redirect_locationcCs(|jr |jsdS|j|jd|_dSr%)rrZ _put_connr r"r"r# release_conns zHTTPResponse.release_connc Cs2z|j|jdWnttttfy,Yn0dS)z Read and discard any remaining HTTP response data in the response connection. Unread data in the HTTPResponse connection blocks the connection from being released back to the pool. rN)rrr SocketErrorrr r r"r"r# drain_conns  zHTTPResponse.drain_conncCs"|jr |jS|jr|jddSdS)NT) cache_content)rrrr r"r"r#r1szHTTPResponse.datacCs|jSr%)rr r"r"r#rszHTTPResponse.connectioncCs t|jSr%)rrr r"r"r#isclosedszHTTPResponse.isclosedcCs|jS)z Obtain the number of bytes pulled over the wire so far. May differ from the amount of content returned by :meth:``urllib3.response.HTTPResponse.read`` if bytes are encoded on the wire (e.g, compressed). )rr r"r"r#tellszHTTPResponse.tellcCs|jd}|dur|jr(tddSz.r[rz8Content-Length contained multiple unmatching values (%s)r)i0dZHEAD) rrrlogZwarningsetr_rFrpoprurr)r!rlengthZlengthsrr"r"r#rs6       &zHTTPResponse._init_lengthcshjdd}jdurd|jvr2t|_n2d|vrdfdd|dD}t|rdt|_dS)z= Set-up the _decoder attribute if necessary. content-encodingrNr[cs"g|]}|jvr|qSr")rWCONTENT_DECODERSr]r r"r#rZ=sz.HTTPResponse._init_decoder..)rrrrrr\r_rF)r!content_encodingrbr"r r# _init_decoder2s    zHTTPResponse._init_decoderc Cs|s|jrtd|S|dus"|r&d}z |jrD|jj||d}d|_WnF|jy}z,|jdd}td||WYd}~n d}~00|r|| 7}|S) zN Decode the data passed in and potentially flush the decoder. _Calling read(decode_content=False) is not supported after read(decode_content=True) was called.Nr+r,TrrzEReceived response with content-encoding: %s, but failed to decode it.) rrtrr/DECODER_ERROR_CLASSESrrrr _flush_decoder)r!r1r flush_decoderr-r^rr"r"r#_decodeIs.   zHTTPResponse._decodecCs$|jr |jd}||jSdS)zk Flushes the decoder. Should only be called if the decoder is actually being used. r)rr/rS)r!bufr"r"r#rhs zHTTPResponse._flush_decoderc cs:d}zz dVWnty0t|jddYnztyt}z.dt|vrRt|t|jddWYd}~n>d}~0ttfy}ztd||WYd}~n d}~00d}W|s|j r|j |j r|j |j r|j r| nH|s|j r|j |j r|j |j r4|j r4| 0dS)z Catch low-level python exceptions, instead re-raising urllib3 variants, so that low-level exceptions are not leaked in the high-level api. On exit, release the connection back to the pool. FNzRead timed out.zread operation timed outzConnection broken: %rT) SocketTimeoutrrrstrrr rrrcloserrr)r!Z clean_exitr^r"r"r#_error_catcherss6    "$     zHTTPResponse._error_catchercCs|jdurdSt|jdd}||durB|s<|jnd}n|sR|j|nd}|dur|dkr|s|j|jr|jdur|jdkrt|j|jWdn1s0Y|r|jt |7_|jdur|jt |8_|S)z7 Reads `amt` of bytes from the socket. NclosedFrr) rr'rrrrrrrrF)r!amtZ fp_closedr1r"r"r# _raw_reads*    , zHTTPResponse._raw_readcCs~||dur|j}|durrd}|jrX|jjrX|jd|d|t|jd}|j|t|j|krr|j|S| |}d}|durd}n|dkr|sd}|st|jdkr|jr|jjs|S|dur||||}|r||_ n|s|j rt d|S|j||||t|jd}|j|t|j|krn|rn| |}|j||||t|jd}|j|q"|j|}|S) aT Similar to :meth:`http.client.HTTPResponse.read`, but with two additional parameters: ``decode_content`` and ``cache_content``. :param amt: How much of the content to read. If specified, caching is skipped because it doesn't make sense to cache partial content as the full response. :param decode_content: If True, will attempt to decode the body based on the 'content-encoding' header. :param cache_content: If True, will save the returned data such that the same result is returned despite of the state of the underlying file object. This is useful if you want the ``.data`` property to continue working after having ``.read()`` the file object. (Overridden if ``amt`` is set.) NFr)rr-Trrr,) rrrr3rrFrrsrrrrrt)r!rrrZ decoded_datar1rr"r"r#rst           zHTTPResponse.readccsn|jr*|r*|j||dD] }|Vqn@t|jrPt|jdksP|jrj|jjrj|j ||d}|r*|Vq*dS)a_ A generator wrapper for the read() method. A call will block until ``amt`` bytes have been read from the connection or until the connection is closed. :param amt: How much of the content to read. The generator will return up to much data per iteration, but may return less. This is particularly likely when using compressed data. However, the empty string will never be returned. :param decode_content: If True, will attempt to decode the body based on the 'content-encoding' header. rr)rrN) rsupports_chunked_reads read_chunkedrrrFrrr3r)r!rrliner1r"r"r#stream2s  zHTTPResponse.streamc Ksb|j}t|ts.tjr"t|}n t|}t|dd}|f|||j|j |j ||d|}|S)a Given an :class:`http.client.HTTPResponse` instance ``r``, return a corresponding :class:`urllib3.response.HTTPResponse` object. Remaining parameters are passed to the HTTPResponse constructor, along with ``original_response=r``. rr)rrrrrrr) rrvrrZPY2 from_httplibitemsr'rrr)Z ResponseClsrZ response_kwrrZrespr"r"r#rPs$      zHTTPResponse.from_httplibcCs|jSr%rr r"r"r# getheadersqszHTTPResponse.getheaderscCs|j||Sr%)rr)r!r)defaultr"r"r# getheadertszHTTPResponse.getheadercCs|jSr%rr r"r"r#infoxszHTTPResponse.infocCs6|js|j|jr |j|js2tj|dSr%)rrrrrrxIOBaser r"r"r#r|s   zHTTPResponse.closecCsT|jstjj|S|jdur"dSt|jdr8|jSt|jdrL|jjSdSdS)NTrr)rrxrr__get__rrKrr r"r"r#rs    zHTTPResponse.closedcCs6|jdurtdnt|jdr*|jStddS)Nz-HTTPResponse has no file to get a fileno fromfilenozOThe file-like object this HTTPResponse is wrapped around has no file descriptor)rIOErrorrKrr r"r"r#rs    zHTTPResponse.filenocCs2|jdur.t|jdr.t|jdds.|jSdS)NrSrF)rrKr'rSr r"r"r#rSs  zHTTPResponse.flushcCsdS)NTr"r r"r"r#readableszHTTPResponse.readablecCs:|t|}t|dkrdS||dt|<t|SdSrc)rrF)r!bZtempr"r"r#readintos  zHTTPResponse.readintocCs t|jdS)a Checks if the underlying file-like object looks like a :class:`http.client.HTTPResponse` object. We do this by testing for the fp attribute. If it is present we assume it returns raw chunks as processed by read_chunked(). fp)rKrr r"r"r#rsz#HTTPResponse.supports_chunked_readscCsd|jdurdS|jj}|ddd}zt|d|_Wn$ty^|t||Yn0dS)N;rrr?) rrrreadliner_rrurr)r!rr"r"r#_update_chunk_lengths   z!HTTPResponse._update_chunk_lengthcCsd}|dur2|j|j}|}|jdd|_nv||jkrZ|j|}|j||_|}nN||jkr|j|}|jdd|_|}n |j|j}|jdd|_|S)Nr:)rZ _safe_readr)r!rZreturned_chunkr~valuer"r"r# _handle_chunks&        zHTTPResponse._handle_chunkccs||jstd|s&td||jrZt|jrZ|jWddS|j j durvWddS| |j dkrq| |}|j||d|d}|rv|Vqv|r|}|r|V|j j }|sq|dkrqq|jr|jWdn1s 0YdS)a Similar to :meth:`HTTPResponse.read`, but with an additional parameter: ``decode_content``. :param amt: How much of the content to read. If specified, caching is skipped because it doesn't make sense to cache partial content as the full response. :param decode_content: If True, will attempt to decode the body based on the 'content-encoding' header. zHResponse is not chunked. Header 'transfer-encoding: chunked' is missing.zkBody should be http.client.HTTPResponse like. It should have have an fp attribute which returns raw chunks.NrF)rrr-s )rrrrr rrrrrrrrrrrr)r!rrr~Zdecodedrr"r"r#rsL      zHTTPResponse.read_chunkedcCs.|jdur$t|jjr$|jjdjS|jSdS)z Returns the URL that was the source of this response. If the request that generated this response redirected, this method will return the final redirect location. Nr+)rrFhistoryZredirect_locationrr r"r"r#geturl)szHTTPResponse.geturlccsg}|jddD]n}d|vrt|d}d||ddV|ddD]}|dVqJ|drn|dg}q~g}q||q|rd|VdS)NTr rrrr+)rr_joinrr)r!ror~xr"r"r#__iter__4s    zHTTPResponse.__iter__)rNrrNrTTNNNNNFNNT)N)N)NNF)rN)N)NN).r5r6r7rhrrJrr$rrrr8r1rrrrrrrrrrrrrrrr classmethodrrrrrrrrSrrrrrrrrr"r"r"r#rVs|   G   4     7 + \         I r)3Z __future__rrmrxZloggingrrP contextlibrZsocketrrrrrJ ImportError _collectionsrrrr exceptionsr r r r rrrrrrrZpackagesrZ util.responserrZ getLoggerr5robjectrr9r>rIrTr\rlrrr"r"r"r#s6        4  ;@ +8 <