,hN 8 d Z dZddlZddlmZ ddlZddlZddlZdZ ddl Z d Z
ddl
Z
dZd Z e Z ej$ ej' d
ej( ej$ ej' d
ej( d ee< ej$ eej( ej$ eej( d ee< G d d
e Z G d d Z G d d Zy# e $ r ddl Z d Z
n# e $ r d Z
Y nw xY wY w xY w# e $ r Y w xY w)aB Beautiful Soup bonus library: Unicode, Dammit
This library converts a bytestream to Unicode through any means
necessary. It is heavily based on code from Mark Pilgrim's Universal
Feed Parser. It works best on XML and HTML, but it does not rewrite the
XML or HTML to reflect a new encoding; that's the tree builder's job.
MIT N)codepoint2namec T t | t ry t j | d S Nencoding)
isinstancestrcchardetdetectss L/var/www/html/Resume-Scraper/venv/lib/python3.12/site-packages/bs4/dammit.pychardet_dammitr s# a q!*-- c T t | t ry t j | d S r )r r chardetr r s r r r " s# !S!>>!$Z0 0r c y N r s r r r * s r z$^\s*<\?.*encoding=['"](.*?)['"].*\?>z0<\s*meta[^>]+charset\s*=\s*["']?([^>]*?)[ /;'">]ascii)htmlxmlc e Zd ZdZd Z e \ ZZZddddddZ e j d Z e j d
Z e
d Ze
d Ze
d
Ze
dd Ze
dd Ze
d Zy)EntitySubstitutionzFThe ability to substitute XML or HTML entities for certain characters.c i } i }g }dg}t t j |z D ]/ \ }}t | }|dvr|j | || |< |||< 1 ddj | z }| |t
j | fS )N)' apos)" r z[%s] )listr itemschrappendjoinrecompile)lookupreverse_lookupcharacters_for_reextra codepointname character
re_definitions r _populate_class_variablesz,EntitySubstitution._populate_class_variablesG s #N$8$8$: ;e C -OItII(
"((3$(y! $-N4 - ):!;;
~rzz-'@@@r r quotampltgt)'"&<>z&([<>]|&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;))z([<>&])c ` | j j |j d }d|z S )ziUsed with a regular expression to substitute the
appropriate HTML entity for a special character.r &%s;)CHARACTER_TO_HTML_ENTITYgetgroupclsmatchobjentitys r _substitute_html_entityz*EntitySubstitution._substitute_html_entityq s- --11(..2CD r c H | j |j d }d|z S )zhUsed with a regular expression to substitute the
appropriate XML entity for a special character.r r: )CHARACTER_TO_XML_ENTITYr= r> s r _substitute_xml_entityz)EntitySubstitution._substitute_xml_entityx s' ,,X^^A->? r c T d}d|v rd|v rd}|j d| }nd}||z |z S )a* Make a value into a quoted XML attribute, possibly escaping it.
Most strings will be quoted using double quotes.
Bob's Bar -> "Bob's Bar"
If a string contains double quotes, it will be quoted using
single quotes.
Welcome to "my bar" -> 'Welcome to "my bar"'
If a string contains both single and double quotes, the
double quotes will be escaped, and the string will be quoted
using double quotes.
Welcome to "Bob's Bar" -> "Welcome to "Bob's bar"
r5 r4 z")replace)selfvalue
quote_with replace_withs r quoted_attribute_valuez)EntitySubstitution.quoted_attribute_value sE &
%<e| (
c<8 !
E!J..r c x | j j | j | }|r| j | }|S )a Substitute XML entities for special XML characters.
:param value: A string to be substituted. The less-than sign
will become <, the greater-than sign will become >,
and any ampersands will become &. If you want ampersands
that appear to be part of an entity definition to be left
alone, use substitute_xml_containing_entities() instead.
:param make_quoted_attribute: If True, then the string will be
quoted, as befits an attribute value.
)AMPERSAND_OR_BRACKETsubrE rL r? rI make_quoted_attributes r substitute_xmlz!EntitySubstitution.substitute_xml s? ((,, & &/ !..u5E r c x | j j | j | }|r| j | }|S )a Substitute XML entities for special XML characters.
:param value: A string to be substituted. The less-than sign will
become <, the greater-than sign will become >, and any
ampersands that are not part of an entity defition will
become &.
:param make_quoted_attribute: If True, then the string will be
quoted, as befits an attribute value.
)BARE_AMPERSAND_OR_BRACKETrO rE rL rP s r "substitute_xml_containing_entitiesz5EntitySubstitution.substitute_xml_containing_entities s? --11 & &/ !..u5E r c N | j j | j | S )a Replace certain Unicode characters with named HTML entities.
This differs from data.encode(encoding, 'xmlcharrefreplace')
in that the goal is to make the result more readable (to those
with ASCII displays) rather than to recover from
errors. There's absolutely nothing wrong with a UTF-8 string
containg a LATIN SMALL LETTER E WITH ACUTE, but replacing that
character with "é" will make it more readable to some
people.
:param s: A Unicode string.
)CHARACTER_TO_HTML_ENTITY_RErO rB )r? r
s r substitute_htmlz"EntitySubstitution.substitute_html s( ..22 ' ', ,r N)F)__name__
__module__ __qualname____doc__r/ r; HTML_ENTITY_TO_CHARACTERrW rD r% r& rT rN classmethodrB rE rL rR rU rX r r r r r D s PA4 %>$?" 7
!+
,/ !0 &2::i0 "/ "/H ( */ * , ,r r c T e Zd ZdZ ddZd Zed Zed Z ed d Z
y)
EncodingDetectora^ Suggests a number of possible encodings for a bytestring.
Order of precedence:
1. Encodings you specifically tell EncodingDetector to try first
(the override_encodings argument to the constructor).
2. An encoding declared within the bytestring itself, either in an
XML declaration (if the bytestring is to be interpreted as an XML
document), or in a tag (if the bytestring is to be
interpreted as an HTML document.)
3. An encoding detected through textual analysis by chardet,
cchardet, or a similar external library.
4. UTF-8.
5. Windows-1252.
Nc |xs g | _ |xs g }t |D cg c] }|j c} | _ d| _ || _ d| _ | j | \ | _ | _ yc c}w )a Constructor.
:param markup: Some markup in an unknown encoding.
:param override_encodings: These encodings will be tried first.
:param is_html: If True, this markup is considered to be HTML. Otherwise
it's assumed to be XML.
:param exclude_encodings: These encodings will not be tried, even
if they otherwise would be.
N)
override_encodingssetlowerexclude_encodingschardet_encodingis_htmldeclared_encodingstrip_byte_order_markmarkupsniffed_encoding)rH rj rb rg re xs r __init__zEncodingDetector.__init__ st #5": - 3!$9J%KAaggi%K!L $ !% .2-G-G-O* T*
&Ls A2c r |5|j }|| j v ry||vr|j | yy)zShould we even bother to try this encoding?
:param encoding: Name of an encoding.
:param tried: Encodings that have already been tried. This will be modified
as a side effect.
FT)rd re add)rH r trieds r _usablezEncodingDetector._usable
sA ~~'H4111 u$ (#r c # b K t }| j D ] }| j || s| | j | j | r| j | j +| j | j | j | _ | j | j | r| j | j t | j | _ | j | j | r| j dD ] }| j || s| yw)zmYield a number of encodings that might work for this markup.
:yield: A sequence of strings.
N)utf-8 windows-1252)
rc rb rq rk rh find_declared_encodingrj rg rf r )rH rp es r encodingszEncodingDetector.encodings s
(( A||Au% <<--u 5'' ' ! ! )%)%@%@ T\\&+D " <<.. 6(( ( ($24;;$?D ! <<--u 5'' ' + A||Au% s ,D/C8D/(D/c D d}t |t r||fS t | dk\ r|dd dk( r|dd dk7 r d}|dd }||fS t | dk\ r|dd dk( r|dd dk7 r d}|dd }||fS |dd d
k( r d }|d d }||fS |dd d k( r d
}|dd }||fS |dd dk( rd}|dd }||fS )zIf a byte-order mark is present, strip it and return the encoding it implies.
:param data: Some markup.
:return: A 2-tuple (modified data, implied encoding)
N s z zutf-16bes zutf-16le s rs s zutf-32bes zutf-32le)r r len)r? datar s r ri z&EncodingDetector.strip_byte_order_mark> s' dC > ! INbq[!81I+!H8D X~ $i1n48{#:1Qi:-!H8D X~ "1X
( H8D X~
"1X,
,!H8D X~ "1X,
,!H8DX~ r c |rt | x}}n#d}t dt t | dz }t |t rt
t }n
t
t }|d }|d }d} |j || }
|
s|r|j || }
|
|
j d } | r2t | t r| j d d
} | j S y) a Given a document, tries to find its declared encoding.
An XML encoding is declared at the beginning of the document.
An HTML encoding is declared in a tag, hopefully near the
beginning of the document.
:param markup: Some markup.
:param is_html: If True, this markup is considered to be HTML. Otherwise
it's assumed to be XML.
:param search_entire_document: Since an encoding is supposed to declared near the beginning
of the document, most of the time it's only necessary to search a few kilobytes of data.
Set this to True to force this method to search the entire document.
i i g?r r N)endposr r rG ) r| maxintr bytes encoding_resr searchgroupsdecoderd ) r? rj rg search_entire_document
xml_endpos html_endposresxml_rehtml_rerh declared_encoding_matchs r ru z'EncodingDetector.find_declared_encoding\ s "'*6{ 2J J dCF d(:$; > @ C +U3$5$<$dddd?dd@ddAddBddCd dDd dEd!dFd#dGd%dHd'dId)d*d+dJd-d*d.d*i dKdLdMdLdNdOdPdOdQdRdSdTdUdVdWdXdYdZd[d\d]d^d_d`dad*dbdcdddedfddgdhi didjdkdldmdndodpdqdrdsdGdtdudvdwdxdydzd{d|dhd}dd~dddTddddddi ddddddddddRdd?ddddydddddddddd*ddddddddi ddddddddddddddddddddddddddddddddddi ddddddRddddddddddddeddœddǓddɓddɓddɓddɓddɓddɓi ddГddjddӓddӓddӓddӓddؓddؓddؓddؓddddޓdddddddddddddddddddd Zi dddddddddddddddddddddddddd ddddddddi d d
d d d
ddddddddddddddd d d d d d!d"d#d$d%d&d'd(d)d*i d+d,d-d.d/d0d1d2d3d4d5d6d7d8d9d:d;d<d=d>d?d@dAdBdCdDdEdFdGdHdIdJdKdLi dMdNdOdPdQdRdSdTdUdVdWdXdYdZd[d\d]d^d_d`dadbdcdddedfdgdhdidjdkdldmdni dodpdqdrdsdtdudvdwdxdydzd{d|d}d~ddddddddddddddddddi ddddddddddddddddgddddddddddddddddddi ddddddddddddddddddÓdĐdœdƐdǓdȐdɓdʐd˓d̐d͓dΐdϓdАdѓdҐdӓdԐdՐddלZg dآZed d Zed d Ze dސd܄ Zy(
UnicodeDammitzA class for detecting the encoding of a *ML document and
converting it to a Unicode string. If the source encoding is
windows-1252, can replace MS smart quotes with their HTML or XML
equivalents.z mac-romanz shift-jis) macintoshzx-sjis)rt z
iso-8859-1z
iso-8859-2NFc || _ g | _ d| _ || _ t j
t | _ t |||| | _ t |t s|dk( r || _ t | | _
d| _ y| j j | _ d}| j j D ]- }| j j }| j! | }|- n |sX| j j D ]? }|dk7 r| j! |d }| | j j# d d| _ n || _
|sd| _ yy)aP Constructor.
:param markup: A bytestring representing markup in an unknown encoding.
:param override_encodings: These encodings will be tried first,
before any sniffing code is run.
:param smart_quotes_to: By default, Microsoft smart quotes will, like all other characters, be converted
to Unicode characters. Setting this to 'ascii' will convert them to ASCII quotes instead.
Setting it to 'xml' will convert them to XML entity references, and setting it to 'html'
will convert them to HTML entity references.
:param is_html: If True, this markup is considered to be HTML. Otherwise
it's assumed to be XML.
:param exclude_encodings: These encodings will not be considered, even
if the sniffing code thinks they might make sense.
Fr Nr rG zSSome characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.T)smart_quotes_totried_encodings contains_replacement_charactersrg logging getLoggerrY logr` detectorr r rj unicode_markuporiginal_encodingrw
_convert_fromwarning)rH rj rb r rg re ur s r rm zUnicodeDammit.__init__ sT " / ! /4, $$X.( &1BD
fc "fl DK"%f+D %)D " mm**
// H]]))F""8,A} !MM 3 3
w& **8Y?A=HH$$ C <@D8
%)D " r c |j d }| j dk( r+| j j | j }|S | j
j | }t
| t k( r}| j dk( r7dj |d j z dj z }|S dj |d j z dj z }|S |j }|S )z[Changes a MS smart quote character to an XML or HTML
entity, or an ASCII character. r r z&#x;r6 r )r= r MS_CHARS_TO_ASCIIr< encodeMS_CHARStypetuple)rH matchorigrO s r _sub_ms_charzUnicodeDammit._sub_ms_char s {{1~ 7 *((,,T299;C
--##D)CCyE!''50 ,,.3q6==?:SZZ\IC
**,Q 83::<GC
jjl
r c | j | }|r||f| j v ry| j j ||f | j }| j A|| j
v r3d}t
j | }|j | j | } | j ||| }|| _ || _ | j S # t $ r
}Y d}~yd}~ww xY w)z|Attempt to convert the markup to the proposed encoding.
:param proposed: The name of a character encoding.
Ns ([-]))
find_codecr r# rj r ENCODINGS_WITH_SMART_QUOTESr% r& rO r _to_unicoder Exception)rH proposederrorsrj smart_quotes_resmart_quotes_compiledr rv s r r zUnicodeDammit._convert_from s
??8,Hf-1E1EE ##Xv$67
,D < << .O$&JJ$? !*..t/@/@&IF 6:ADK%-D " {{ s !C C C c t ||| S )z}Given a string and its encoding, decodes the string into Unicode.
:param encoding: The name of an encoding.
)r )rH r} r r s r r zUnicodeDammit._to_unicode s
46**r c H | j sy| j j S )zhIf the markup is an HTML document, returns the encoding declared _within_
the document.
N)rg r rh )rH s r declared_html_encodingz$UnicodeDammit.declared_html_encoding s
||}}...r c N | j | j j || xsf |xr! | j |j dd xs? |xr! | j |j dd xs |xr |j xs |}|r|j S y)zConvert the name of a character set to a codec name.
:param charset: The name of a character set.
:return: The name of a codec.
-r _N)_codecCHARSET_ALIASESr< rG rd )rH charsetrI s r r zUnicodeDammit.find_codec s T 1 1 5 5gw GH Dt{{7??3+C DEt{{7??3+D E .w}} ;;= r c n |s|S d } t j | |}|S # t t f$ r Y |S w xY wr )codecsr' LookupError
ValueError)rH r codecs r r zUnicodeDammit._codec) sJ N MM' "E Z( s ! 44 )euro20AC )sbquo201A )fnof192 )bdquo201E )hellip2026 )dagger2020 )Dagger2021 )circ2C6 )permil2030 )Scaron160 )lsaquo2039 )OElig152 ? )z#x17D17D )lsquo2018)rsquo2019)ldquo201C)rdquo201D)bull2022)ndash2013)mdash2014)tilde2DC)trade2122)scaron161)rsaquo203A)oelig153)z#x17E17E)Yumlr ) EUR,fz,,z...+z++^%Sr7 OEZr r4 r r r5 r
r *r r r
z--r ~r z(TM)r r
r r8 r oer r zr Y ! c GBP $ YEN | z.. r z(th) z<< z(R) o z+- 2 3 )r4 acute r P 1 z>> z1/4 z1/2 z3/4 A AE C E I D N O U b B a ae rv i n /y) s € s ‚ s ƒ s „ s … s † s ‡ s ˆ s ‰ s Š s ‹ s Œ s Ž s ‘ s ’ s “ s ” s • s – s — s ˜ s ™ s š s › s œ s ž s Ÿ s s ¡ s ¢ s £ s ¤ s ¥ s ¦ s § s ¨ s © s ª s « s ¬ s s ® s ¯ s ° s ± s ² s ³ s ´ s µ s ¶ s · s ¸ s ¹ s º s » s ¼ s ½ s ¾ s ¿ s À s Á s  s à s Ä s Å s Æ s Ç s È s É s Ê s Ë s Ì s Í s Î s Ï s Ð s Ñ s Ò s Ó s Ô s Õ s Ö s × s Ø s Ù s Ú s Û s Ü s Ý s Þ s ß s à s â s ã s ä s å s æ s ç s è s é s ê s ë s ì s í s î s ï s ð s ñ s ò s ó s ô s õ s ö s ÷ s ø s ù s ú s ûs üs ýs þ) ))r r rz )r r r{ )r r ry r r c |j dd j dvr t d |j dvr t d g }d}d}|t | k r|| }t |t
s t
| }|| j k\ r7|| j k r(| j D ] \ }} }
||k\ s
|| k s||
z
} nT nR|dk\ rH|| j v r:|j ||| |j | j | |d z
}|}n|d z
}|t | k r|dk( r|S |j ||d
d j | S ) a Fix characters from one encoding embedded in some other encoding.
Currently the only situation supported is Windows-1252 (or its
subset ISO-8859-1), embedded in UTF-8.
:param in_bytes: A bytestring that you suspect contains
characters from multiple encodings. Note that this _must_
be a bytestring. If you've already converted the document
to Unicode, you're too late.
:param main_encoding: The primary encoding of `in_bytes`.
:param embedded_encoding: The encoding that was used to embed characters
in the main document.
:return: A bytestring in which `embedded_encoding`
characters have been converted to their `main_encoding`
equivalents.
r r )rt windows_1252zPWindows-1252 and ISO-8859-1 are the only currently supported embedded encodings.)utf8rs z4UTF-8 is the only currently supported main encoding.r r r Nr )
rG rd NotImplementedErrorr| r r ordFIRST_MULTIBYTE_MARKERLAST_MULTIBYTE_MARKERMULTIBYTE_MARKERS_AND_SIZESWINDOWS_1252_TO_UTF8r# r$ ) r? in_bytes
main_encodingembedded_encoding byte_chunks chunk_startposbytestartendsizes r detwinglezUnicodeDammit.detwinglei s & $ $S# . 4 4 6 ?, ,%&'
' (9 9%FH
H CM!C=D dC(4y222C 5 55 ),(G(G $E3u}t $#*B*B"B ""8K#<= ""3#;#;D#ABq ! q3 CM!4 ! O
x 5 6xx $$r )strict)r rt )rY rZ r[ r\ r r rm r r r r r r r r r r$ r# r! r" r^ r/ r r r r r sg %0!,.O#
35!%u?*B" >+ / / () ( (* ( ( * (
+ ( +
( + ( ( + ( * ( + ( ) ( ( ) ( ( ! (" +***)**)**+) )%? (HHB
%B
#B
#B
# B
$ B
%
B
#B
$B
#B
#B
#B
#B
$B
# B
# B
#!B
" ##B
$ #%B
& #'B
( #)B
* #+B
, #-B
. #/B
0 $1B
2 #3B
4 &5B
6 #7B
8 #9B
: $;B
< #=B
> #?B
@ #AB
B #CB
D #EB
F #GB
H %IB
J #KB
N %OB
P #QB
R #SB
T $UB
V "WB
X &YB
Z $[B
\ #]B
^ #_B
` %aB
b #cB
d #eB
f $gB
h #iB
j #kB
l .mB
n #oB
p #qB
r #sB
t #uB
v #wB
x &yB
z ${B
| %}B
~ %B
@ %AB
B #CB
D #EB
F #GB
H #IB
J #KB
L #MB
N #OB
P $QB
R #SB
T #UB
V #WB
X #YB
Z #[B
\ #]B
^ #_B
` #aB
b #cB
d #eB
f #gB
h #iB
j #kB
l #mB
n #oB
p #qB
r #sB
t #uB
v #wB
x #yB
z #{B
| #}B
~ #B
@ #AB
B #CB
D #EB
F #GB
H #IB
J #KB
L #MB
N #OB
P $QB
R #SB
T #UB
V #WB
X #YB
Z #[B
\ #]B
^ #_B
` #aB
b #cB
d #eB
f #gB
h #iB
j #kB
l #mB
n #oB
p #qB
r CB
R{
{
{
{{
{
{
{
{
{{
{
{{
{
{{
{{
{
{
!{
"
#{
$
%{
&
'{
(
){
*
{+{
,
-{
.
{/{
0
1{
2
{3{
4
{5{
6
{7{
8
{9{
:
{;{
<
{={
>
{?{
@
{A{
B
{C{
D
{E{
F
{G{
H
{I{
J
{K{
L
{M{
N
{O{
P
{Q{
R
{S{
T
{U{
V
{W{
X
{Y{
Z
{[{
\
{]{
^
{_{
`
{a{
b
{c{
d
{e{
f
{g{
h
{i{
j
{k{
l
{m{
n
{o{
p
{q{
r
{s{
t
{u{
v
{w{
x
{y{
z
{{{
|
{}{
~
{{
@
{A{
B
{C{
D
{E{
F
{G{
H
{I{
J
{K{
L
{M{
N
{O{
P
{Q{
R
{S{
T
{U{
V
{W{
X
{Y{
Z
{[{
\
{]{
^
{_{
`
{a{
b
{c{
d
{e{
f
{g{
h
{i{
j
{k{
l
{m{
n
{o{
p
{q{
r
{s{
t
{u{
v
{w{
x
{y{
z
w{{
|
{}{
~
{{
@
{A{
B
{C{
D
{E{
F
{G{
H
{I{
J
{K{
L
{M{
N
{O{
P
{Q{
R
{S{
T
{U{
V
{W{
X
{Y{
Z
{[{
\
{]{
^
{_{
`
{a{
b
{c{
d
{e{
f
{g{
h
{i{
j
{k{
l
{m{
n
{o{
p u{
z#
9 ;A > 7 ;A >/5$2@% @%r r )r\ __license__r
html.entitiesr r% r string chardet_typer
r ImportErrorr iconv_codec xml_encoding html_metadictr r& r ra r r objectr r` r r r r r; sC
(
.2 ; C v
RZZ ((1244
8 BJJ|**73RTT : U
RZZ 244
( BJJ|RTT * S
Z, Z,zb bHe% e%S 1 . sF C/ D /D5C=<D=DDDD
DDD