tag, if the del looks like (DEL_START,
'Text
', DEL_END)r
rF rn z']
>>> doc = ['
', 'Text', '
', '', 'More Text', '
']
>>> pre, post = doc[:3], doc[3:]
>>> pre, post
(['', 'Text', '
'], ['', 'More Text', '
'])
>>> locate_unbalanced_start(unbalanced_start, pre, post)
>>> pre, post
(['', 'Text', '
', ''], ['More Text', '
'])
As you can see, we moved the point so that the dangling that
we found will be effectively replaced by the div in the original
document. If this doesn't work out, we just throw away
unbalanced_start without doing anything.
r
r z<>r r insdelzUnexpected delete tag: %rN)r r rv r r rJ )rr r r finding finding_namenextr s r r r m s , E"1% }}q)//55 E1~ 9 DOOC$8$8 E 7c>> Ezz||A$$T** 5== Eu}}} '$ . }} < # # # kooa 0 0 1 1 1 1
E5r c | sdS | d }| d d }|sdS |d }|t u s| d sdS | d d }|dk s|dk rdS ||k r=| | d| ndS ) zt like locate_unbalanced_start, except handling end tags and
possibly moving the point earlier in the document. r
rF r r tag, which takes up visible space just like a word but
is only represented in a document by a tag. Nr c t | t d|||| }||_ ||_ ||_ |S )Nz: rL )rO r rN r data html_repr)r r r r rH rG rM r s r r ztag_token.__new__ sP mmCTTT44!8%-&/0C E E !
r c
h d| j d| j d| j d| j d| j d| j d
S )Nz
tag_token(r z , html_repr=z , post_tags=z , pre_tags=z, trailing_whitespace=r )r r r rH rG rM r s r r ztag_token.__repr__ sG HHH III NNN MMM NNN $ $ $
& &r c | j S r )r r s r rS ztag_token.html s
~ r r )rx ry rz r r r rS r r r r r sX 5 5 5946 & & & r r c e Zd ZdZdZd ZdS )
href_tokenzh Represents the href in an anchor tag. Unlike other words, we only
show the href when it changes. Tc d| z S )Nz Link: %sr r s r rS zhref_token.html s
T!!r N)rx ry rz r rl rS r r r r r s4 ( ( O" " " " "r r Tc t j | r| }nt | d }t |d| }t | S )ak
Parse the given HTML and returns token objects (words with attached tags).
This parses only the content of a page; anything in the head is
ignored, and the and elements are themselves
optional. The content is then parsed by lxml, which ensures the
validity of the resulting parsed document (though lxml may make
incorrect guesses when the markup is particular bad).
and tags are also eliminated from the document, as
that gets confusing.
If include_hrefs is true, then the href attribute of tags is
included as a special kind of diffable token.Tcleanup)skip_tagr' )r iselement
parse_html
flatten_el fixup_chunks)rS r' body_elr s r r( r( sQ
t 1 T4000
$m
L
L
LF r c F |rt | } t | d S )a
Parses an HTML fragment, returning an lxml element. Note that the HTML will be
wrapped in a
tag that was not in the original document.
If cleanup is true, make sure there's no or , and get
rid of any and tags.
T)
create_parent) cleanup_htmlr )rS r s r r r s, "D!! t4 8 8 88r z z
zc t | }|r | | d } t | }|r | d| } t
d| } | S )z This 'cleans' the HTML, meaning that any page structure is removed
(only the contents of are used, if there is any and tags are removed. Nr )_body_researchr _end_body_rer _ins_del_resub)rS matchs r r r 1 s|
OOD ! !E "EIIKKLL! % %E $NU[[]]N# ??2t $ $D Kr z
[ \t\n\r]$c l t | }| d| | |d fS )zP
This function takes a word, such as 'test
' and returns ('test','
')
r N)r? rstrip)wordstripped_lengths r split_trailing_whitespacer A s9 $++--((O /! "D )9)9$: ::r c
x g }d}g }| D ]{}t |t r|d dk rL|d }t |d \ }}t d|||| }g }| | n=|d dk r1|d }t ||d }g }| | t
| r> )B5)I)I &E& UYL_```HI MM( # # # #
%
U # # # #
1 '''' 9 9 9 xx89 9x "))%0000 5 /b9---..r
##I... Mr )
paramr areabrbasefontinputbasemetalinkcol)address
blockquotecenterdirdivdlfieldsetformh1h2h3h4h5h6hrisindexmenunoframesnoscriptolprU tableul)
dddtframesetlitbodytdtfootththeadtrc # r K |sD| j dk r(d| d t | fV nt | V | j t v r | j st | s | j sdS t | j }|D ]}t | V | D ]}t || D ]}|V | j dk r0| d r|rd| d fV |s;t | V t | j }|D ]}t | V dS dS )a Takes an lxml element el, and generates all the text chunks for
that tag. Each start tag is a chunk, each word is a chunk, and each
end tag is a chunk.
If skip_tag is true, then the outermost container tag is
not returned (just its contents).r rA Nr& r- r ) r get start_tagr r
r? tail split_wordsr r end_tag)elr' r start_wordsr childitem end_wordss r r r s
6U??"&&--2 7 7 7 7 7B-- vBGCGGBG bg&&K $ uMBBB DJJJJ v}} }M}rvvf~~&&&& $bkk (( $ $D d## # # # # $ $ $ $r z \S+(?:\s+|$)c j | r| sg S t | }|S )z_ Splits some text into words. Includes trailing whitespace
on each word when appropriate. )r split_words_refindall)r
wordss r r r s8 tzz|| " "4 ( (E Lr z
^[ \t\n\r]c d| j d d | j D dS )z=
The text representation of the start tag for a tag.
r r c B g | ] \ }}d |dt |d d S )rn z="T")r )r r values r r z start_tag.. sG ? ? ? (T55 (,tt[-E-E-E-EF ? ? ?r >)r r attribitems)r s r r r s_
? ?,.IOO,=,=? ? ? @ @ @ @ A Ar c r | j r"t | j rd}nd}d| j d|S )zg The text representation of an end tag for a tag. Includes
trailing whitespace when appropriate. rn r r r! )r start_whitespace_rer r )r extras r r r sF
w &--bg66 &&r c . | d S )Nr r r* s r r r s ~~c"" ""r c , | d S )Nr r( r) s r r r s >>$ r c X | d o| d S )Nr r r( r) s r r r s( >># ;s~~d';';#;;r c h t | d }t | t |d } | S )z Given an html string, move any or tags inside of any
block-level elements, e.g. transform , etc.
TF)r block_level_tagsblock_level_container_tagsr9 )r r s r r9 r9 sT
v!!!RV/I%I%It $U + + 44 5r c | D ]}t | r nTt j | }| j |_ d| _ | t | |g| dd< dS t | D ]}t | rkt
|| |j rSt j | }|j |_ d|_ | | | dz | |t j | }|
|| | | | j r?t j | }| j |_ d| _ | d| dS dS )zt helper for _fixup_ins_del_tags; actually takes the etc tags
and moves them inside any block-level tags. Nr
r ) r9 r Elementr
rb listr: r r_ r r` rJ )r r r children_tagtail_tag child_tagtext_tags r r: r: s
$U + + E }S)) G DHH%%% 111b $ $ $U + +
$ !% - - -z
7 =-- %
!
"((5//!+X666
c**I JJui ( ( ( U # # # # w =%%
!X r c | }| j pd}| j rUt | s || j z
}n;| d j r | d xj | j z
c_ n| j | d _ | | }|rU|dk rd}n ||dz
}| |j r|xj |z
c_ n'||_ n |j r|xj |z
c_ n||_ | |||dz < dS )z
Removes an element, but merges its contents into its place, e.g.,
given .m s= 7Y&&Aw ' &&&r )minr? r. rP difflibSequenceMatcherget_matching_blocks)r sizeactualrP s @r rT z.InsensitiveSequenceMatcher.get_matching_blocksi sw 3tv;;DF ,,q11 (< r Ur r r% r r r r r rY r/ r. r9 r: rK rS r0 rx rX mainr r r rj s^ ' & & & & & ) ) ) ) ) ) J
'******** * * *))))))))*HH HHHJJ JJJ1 1 1 #1 &# &# &# &#P - - -1 1 1
$ *& & &.$ $ $L
. % % %N& & & P
=
=
=0 0 0d 4' ' ' ' ' H ' ' ' R 8" " " " " " " " 0 9 9 9 9 2:lBDI . .rz-bd33 bj,bd24i88 BJ}-- ; ; ;2 2 2l#
6 $ $ $ $8 ORT22 !bj // A A A' ' '# # # < < < $ @ - - -D !8 z&&&&&&L s- ' 55< AA
A
AA
word
toword
Fr T) skip_outer)r _fixup_ins_del_tagsserialize_html_fragment)rS r s r rY rY s; T5 ) ) )C "34 8 8 8D Kr c ( t | t r J d| z t j | dt }|rQ|| d dz d }|d| d }| S |S )z Serialize a single lxml element as HTML. The serialized form includes the elements tail. If skip_outer is true, then don't serialize the outermost tag z3You should pass in an element, not a string like %rrS )methodencodingr! r Nr )r basestringr tostringr findrfindr )r r- rS s r r/ r/ s "j)) D D=BD D ) >"Vh ? ? ?D DIIcNN1$%%&$TZZ__$%zz|| r c dD ]R}| d|z D ]7}t | st || | 8SdS )z?fixup_ins_del_tags that works on an lxml document in-place )r r zdescendant-or-self::%s)r N)xpath_contains_block_level_tag_move_el_inside_blockdrop_tag)r r r s r r. r. sx )) 4s :;; B,R00 !"# . . . . KKMMMM r c p | j t v s| j t v rdS | D ]}t | r dS dS )zPTrue if the element contains any block-level elements, like,
Hi there!
, if you remove the element you getHi there!
r rF r Nr ) getparentr r r? r getchildren)r parentr r previouss r _merge_element_contentsrK ? s \\^^F 7=bD w &2ww & BGODD"v{ &2 rw& g2 LL E % A::HH eAgH { # t# " } % % $ NN , ,F5q=r c e Zd ZdZdZd ZdS )r0 zt Acts like SequenceMatcher, but tries not to find very small equal blocks amidst large spans of changes r c t t | j t | j }t | j |dz t j | }fd|D S )N c < g | ]}|d k s|d |S )r r )r r thresholds r r zBInsensitiveSequenceMatcher.get_matching_blocks.