a `{v@sdZddlZddlmZddlmZddlmZddlm Z ddlm Z dd lm Z dd lm Z dd l mZdd lmZed ZedejZedZedejZedZedejejBZzedddWn eyedZdZYn0ddlm ZdZe dZ!e dZ"e dZ#e dZ$e dZ%e dZ&e d Z'e d!Z(e d"Z)e d#Z*e d$Z+e d%Z,e d&Z-e d'Z.e d(Z/e d)Z0e d*Z1e d+Z2e d,Z3e d-Z4e d.Z5e d/Z6e d0Z7e d1Z8e d2Z9e d3Z:e d4Z;e d5Ze d8Z?e d9Z@e d:ZAe d;ZBe d<ZCe d=ZDe d>ZEe d?ZFe d@ZGe dAZHe dBZIe dCZJe dDZKe dEZLe dFZMe dGZNe dHZOe dIZPe dJZQe!e9e%e(e1e0e4e:e,e6e-e7e+e5e'e2e)e*e.e/e"e&e#e3e$e8dKZReSdLdMe eRDZTedNdOUdPdQeVeRdRdSdTDZWeXeGeIeHe;eLeMeNgZYeXe;eOeIeNgZZdUdVZ[dWdXZ\dYdZZ]d[d\Z^d]d^Z_Gd_d`d`e`ZaGdadbdbebZce Gdcdddde`Zde Gdedfdfe`ZedgdhZfGdidjdjebZgGdkdldle`ZhdS)mzImplements a Jinja / Python combination lexer. The ``Lexer`` class is used to do some preprocessing. It filters out invalid operators like the bitshift operators we don't allow in templates. It separates template code and python code in expressions. N) literal_eval)deque) itemgetter)implements_iterator)intern) iteritems) text_type)TemplateSyntaxError)LRUCache2z\s+z (\r\n|\r|\n)z7('([^'\\]*(?:\\.[^'\\]*)*)'|"([^"\\]*(?:\\.[^"\\]*)*)")z (\d+_)*\d+z (?evalz[a-zA-Z_][a-zA-Z0-9_]*F)patternTaddZassigncolonZcommaZdivdoteqfloordivgtZgteqZlbraceZlbracketZlparenltZlteqmodmulnepipepowrbraceZrbracketZrparenZ semicolonsubtildeZ whitespacefloatZintegernamestringoperatorZ block_beginZ block_endZvariable_beginZ variable_endZ raw_beginZraw_endZ comment_beginZ comment_endcommentZlinestatement_beginZlinestatement_endZlinecomment_beginZlinecomment_endZ linecommentdatainitialeof)+-/z//*%z**~[](){}z==z!=>z>=r?z(%s)r7ccs|]}t|VqdSN)reescaper;xr:r:r> r@rFcCs t| SrA)lenrEr:r:r>r@rI)keycCsL|tvrt|Stdtdtdtdtdtdtdtdt dt d t d t d i ||S) Nzbegin of commentzend of commentr"zbegin of statement blockzend of statement blockzbegin of print statementzend of print statementzbegin of line statementzend of line statementztemplate data / textzend of template)reverse_operatorsTOKEN_COMMENT_BEGINTOKEN_COMMENT_END TOKEN_COMMENTTOKEN_LINECOMMENTTOKEN_BLOCK_BEGINTOKEN_BLOCK_ENDTOKEN_VARIABLE_BEGINTOKEN_VARIABLE_ENDTOKEN_LINESTATEMENT_BEGINTOKEN_LINESTATEMENT_END TOKEN_DATA TOKEN_EOFget) token_typer:r:r>_describe_token_types" rZcCs|jtkr|jSt|jS)z#Returns a description of the token.)type TOKEN_NAMEvaluerZ)tokenr:r:r>describe_tokens r_cCs2d|vr&|dd\}}|tkr*|Sn|}t|S)z0Like `describe_token` but for token expressions.r6r)splitr\rZ)exprr[r]r:r:r>describe_token_exprs rbcCstt|S)zsCount the number of newline characters in the string. This is useful for extensions that filter a stream. )rG newline_refindall)r]r:r:r>count_newlinessrecCstj}t|jt||jft|jt||jft|jt||jfg}|j durp| t|j t d||j f|j dur| t|j t d||j fddt|ddDS)zACompiles all the rules from the environment into a list of rules.Nz ^[ \t\v]*z(?:^|(?<=\S))[^\S\r\n]*cSsg|]}|ddqS)rNr:rDr:r:r>r?r@z!compile_rules..T)reverse)rBrCrGcomment_start_stringrLblock_start_stringrPvariable_start_stringrRline_statement_prefixappendrTline_comment_prefixTOKEN_LINECOMMENT_BEGINsorted) environmenterulesr:r:r> compile_ruless:    rrc@s$eZdZdZefddZddZdS)FailurezjClass that raises a `TemplateSyntaxError` if called. Used by the `Lexer` to specify known errors. cCs||_||_dSrA)message error_class)selfrtclsr:r:r>__init__szFailure.__init__cCs||j||dSrA)rurt)rvlinenofilenamer:r:r>__call__szFailure.__call__N)__name__ __module__ __qualname____doc__r rxr{r:r:r:r>rss rsc@sTeZdZdZdZddedD\ZZZddZ dd Z d d Z d d Z ddZ dS)Tokenz Token class.r:ccs|]}tt|VqdSrA)propertyrrDr:r:r>rFr@zToken.cCst||tt||fSrA)tuple__new__rstr)rwryr[r]r:r:r>r sz Token.__new__cCs*|jtvrt|jS|jdkr$|jS|jS)Nr)r[rKr]rvr:r:r>__str__ s    z Token.__str__cCs2|j|krdSd|vr.|dd|j|jgkSdS)zTest a token against a token expression. This can either be a token type or ``'token_type:token_value'``. This can only test against string values and types. Tr6rF)r[r`r]rvrar:r:r>tests  z Token.testcGs|D]}||rdSqdS)z(Test against multiple token expressions.TF)r)rviterablerar:r:r>test_any s zToken.test_anycCsd|j|j|jfS)NzToken(%r, %r, %r))ryr[r]rr:r:r>__repr__'szToken.__repr__N)r|r}r~r __slots__rangeryr[r]rrrrrr:r:r:r>rs rc@s(eZdZdZddZddZddZdS) TokenStreamIteratorz`The iterator for tokenstreams. Iterate over the stream until the eof token is reached. cCs ||_dSrA)stream)rvrr:r:r>rx1szTokenStreamIterator.__init__cCs|SrAr:rr:r:r>__iter__4szTokenStreamIterator.__iter__cCs0|jj}|jtur"|jtt|j|SrA)rcurrentr[rWclose StopIterationnextrvr^r:r:r>__next__7s    zTokenStreamIterator.__next__N)r|r}r~rrxrrr:r:r:r>r+src@szeZdZdZddZddZddZeZedd Z d d Z d d Z dddZ ddZ ddZddZddZddZdS) TokenStreamzA token stream is an iterable that yields :class:`Token`\s. The parser however does not iterate over it but calls :meth:`next` to go one token ahead. The current active token is stored as :attr:`current`. cCs>t||_t|_||_||_d|_tdtd|_ t |dS)NFr) iter_iterr_pushedrrzclosedr TOKEN_INITIALrr)rv generatorrrzr:r:r>rxGs zTokenStream.__init__cCst|SrA)rrr:r:r>rPszTokenStream.__iter__cCst|jp|jjtuSrA)boolrrr[rWrr:r:r>__bool__SszTokenStream.__bool__cCs| S)z Are we at the end of the stream?r:rr:r:r>eosXszTokenStream.eoscCs|j|dS)z Push a token back to the stream.N)rrkrr:r:r>push]szTokenStream.pushcCs"t|}|j}||||_|S)zLook at the next token.)rrr)rvZ old_tokenresultr:r:r>lookas  zTokenStream.lookrcCst|D] }t|qdS)zGot n tokens ahead.N)rr)rvn_r:r:r>skipis zTokenStream.skipcCs|j|rt|SdS)zqPerform the token test and return the token if it matched. Otherwise the return value is `None`. N)rrrrr:r:r>next_ifns zTokenStream.next_ifcCs||duS)z8Like :meth:`next_if` but only returns `True` or `False`.N)rrr:r:r>skip_ifuszTokenStream.skip_ifcCsV|j}|jr|j|_n8|jjturRzt|j|_WntyP|Yn0|S)z|Go one token ahead and return the old one. Use the built-in :func:`next` instead of calling this directly. ) rrpopleftr[rWrrrr)rvrvr:r:r>rys  zTokenStream.__next__cCs"t|jjtd|_d|_d|_dS)zClose the stream.rNT)rrryrWrrrr:r:r>rszTokenStream.closecCs~|j|s^t|}|jjtur:td||jj|j|jtd|t |jf|jj|j|jz|jWt |St |0dS)z}Expect a given token type and return it. This accepts the same argument as :meth:`jinja2.lexer.Token.test`. z(unexpected end of template, expected %r.zexpected token %r, got %rN) rrrbr[rWr ryrrzr_rrr:r:r>expects(  zTokenStream.expectN)r)r|r}r~rrxrrZ __nonzero__rrrrrrrrrrr:r:r:r>r@s   rc CsZ|j|j|j|j|j|j|j|j|j|j |j |j f }t |}|durVt|}|t |<|S)z(Return a lexer which is probably cached.N)rhblock_end_stringrivariable_end_stringrgcomment_end_stringrjrl trim_blocks lstrip_blocksnewline_sequencekeep_trailing_newline _lexer_cacherXLexer)rorJZlexerr:r:r> get_lexers$ rcs$eZdZdZdZfddZZS)OptionalLStripzWA special tuple for marking a point in the state that can have lstrip applied. r:cstt|||SrA)superrr)rwmemberskwargs __class__r:r>rszOptionalLStrip.__new__)r|r}r~rrr __classcell__r:r:rr>rsrc@s>eZdZdZddZddZd ddZdd d Zdd d ZdS)ra Class that implements a lexer for a given environment. Automatically created by the environment class, usually you don't have to do that. Note that the lexer is not automatically bound to an environment. Multiple environments can share the same lexer. cCstj}dd}ttdfttdfttdftt dft t dft t dfg}t|}|jrTdpVd}|jrf|dnd|_|j|_|j|_d|ddd ||j||j||jfgd d |Dttd d f|d tdfgt|d||j||j|fttfdf|dtdfdfgt|d||j||j|ftdfg|t |d||j!||j!ft"dfg|t#|d||j||j||j|fttt$df|dtdfdfgt%|dt&dfg|t'|dt(t)fdfgi|_*dS)NcSst|tjtjBSrA)rBcompileMSrHr:r:r>cszLexer.__init__..cz\n?rz[^ \t]rootz (.*?)(?:%s)r7z0(?P%s(\-|\+|)\s*raw\s*(?:\-%s\s*|%s))cSsg|]\}}d||fqS)z(?P<%s>%s(\-|\+|))r:)r;rrr:r:r>r?sz"Lexer.__init__..#bygroupz.+z(.*?)((?:\-%s\s*|%s)%s)#popz(.)zMissing end of comment tagz(?:\-%s\s*|%s)%sz \-%s\s*|%sz1(.*?)((?:%s(\-|\+|))\s*endraw\s*(?:\-%s\s*|%s%s))zMissing end of raw directivez \s*(\n|$)z(.*?)()(?=\n|$))+rBrC whitespace_reTOKEN_WHITESPACEfloat_re TOKEN_FLOAT integer_re TOKEN_INTEGERname_rer\ string_re TOKEN_STRING operator_reTOKEN_OPERATORrrrrlstrip_unless_rerrjoinrhrrrVrLrrNrMrsrPrQrRrrSTOKEN_RAW_BEGIN TOKEN_RAW_ENDrTrUrmrOTOKEN_LINECOMMENT_ENDrq)rvrorprZ tag_rulesZroot_tag_rulesZblock_suffix_rer:r:r>rxs      zLexer.__init__cCst|j|S)z@Called for strings and template data to normalize it to unicode.)rcrr)rvr]r:r:r>_normalize_newlinesaszLexer._normalize_newlinesNcCs&|||||}t||||||S)z:Calls tokeniter + tokenize and wraps it in a token stream.) tokeniterrwrap)rvsourcerrzstaterr:r:r>tokenizeeszLexer.tokenizec csz|D]n\}}}|tvrqnH|tkr.t}n8|tkr>t}n(|ttfvrPqn|tkrf||}n|dkrt|}n|t krt |}t r| st d|||n|tkrz$||ddddd}WnJty}z0t |dd}t ||||WYd }~n d }~00nJ|tkr8t|d d }n.|tkrTt|d d }n|tkrft|}t|||Vqd S) zThis is called with the stream as returned by `tokenize` and wraps every token in a :class:`Token` and converts the value. keywordzInvalid character in identifierrasciibackslashreplacezunicode-escaper6Nrr)ignored_tokensrTrPrUrQrrrVrr\r check_ident isidentifierr rencodedecode Exceptionr`striprintreplacerrr operatorsr) rvrrrzryr^r]rpmsgr:r:r>rjsJ   &   z Lexer.wrapc cs4t|}|}|jr<|rrFr@z"Lexer.tokeniter..r'r&rz?%r wanted to resolve the token dynamically but no group matchedr0r1r.r/r,r-)r1r/r-zunexpected '%s'zunexpected '%s', expected '%s'rzC%r wanted to resolve the new state dynamically but no group matchedz,%r yielded empty string without stack changezunexpected char %r at %d)$r splitlinesrendswithrkrrqrGrmatchrSrQrU isinstancergroupsrrrstripcount groupdictrXrRrfindsearch enumeraterrsr RuntimeErrorignore_if_emptygrouprr popend) rvrrrzrlinesnewlineposrystackZ statetokensZ source_lengthZbalancing_stackrZnewlines_strippedZ line_startingZregextokensZ new_statemrtextZ strip_signstrippedZl_posidxr^rJr]r#Z expected_opZpos2r:r:r>rs                                       zLexer.tokeniter)NNN)NN)NN) r|r}r~rrxrrrrr:r:r:r>rs  ,r)irrBastr collectionsrr!rZ_compatrrrr exceptionsr Zutilsr rrUrrcrrr IGNORECASEVERBOSEr SyntaxErrorrrZ _identifierrZ TOKEN_ADDZ TOKEN_ASSIGNZ TOKEN_COLONZ TOKEN_COMMAZ TOKEN_DIVZ TOKEN_DOTZTOKEN_EQZTOKEN_FLOORDIVZTOKEN_GTZ TOKEN_GTEQZ TOKEN_LBRACEZTOKEN_LBRACKETZ TOKEN_LPARENZTOKEN_LTZ TOKEN_LTEQZ TOKEN_MODZ TOKEN_MULZTOKEN_NEZ TOKEN_PIPEZ TOKEN_POWZ TOKEN_RBRACEZTOKEN_RBRACKETZ TOKEN_RPARENZTOKEN_SEMICOLONZ TOKEN_SUBZ TOKEN_TILDErrrr\rrrPrQrRrSrrrLrMrNrTrUrmrrOrVrrWrdictrKrrnr frozensetrrrZr_rbrerrobjectrsrrrrrrrr:r:r:r>s                "   ) (f