`i' d dl Z d dlmZ ddlmZ ddlmZ ddlmZ ddl m
Z
m Z ddl m
Z
mZ dd lmZmZ d
d lmZ e r
d dlZd dlmZ d Zd
ZdZdZdZ ddZ G d d Z dS ) N)Image )create_causal_mask)_get_model_class)
AutoConfig) MODEL_FOR_PRETRAINING_MAPPING
MODEL_MAPPING)PROCESSOR_MAPPING_NAMES
AutoProcessor)TOKENIZER_MAPPING_NAMES
AutoTokenizer )is_torch_availablez[92mz[93mz[0mu ■u ⬚![]()
c
^ j dk rdddddf j dk rddddddf t t d D }d}g }t D ]F\ } | k r |s }df< |dk r*| k s dz
k rdz
k rdz
d||f< d}Gd fdt D d fd
t D }
||dk } t j | d d z | z
| d d t j d|dz | } t j |
t t t dt t t d}
| d |
z d |d z z d t |
dz z }|dz
}| | g }t D ]\ }||f dk rg| d t' t) | t t) D z| t' t) | t t) t' t- t& t/ | }|D ]P}| 7|d z d z d | z dz d | z nd Qt D ]\ t1 | }v rt | t n|}d fdt D }d}3d fdt D }| | dt) d d | d| d | S )z
Generates an attention matrix from a given attention mask.
Optionally applies a sliding window mask (e.g., for Gemma2/3) and
marks regions where image tokens occur based on the specified `img_token`.
r N c 3 N K | ] }t t | V !d S N)lenrepr).0words {/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/utils/attention_visualizer.py z6generate_attention_matrix_from_mask..8 s. < g | ]}d |z
cxk rk rn ndnd S )r r )r jisliding_windows r
zBgenerate_attention_matrix_from_mask...I sB \ \ \aa1q5&A&A&A&A>&A&A&A&A&A \ \ \r )range)r r! nr" s @r r# z7generate_attention_matrix_from_mask..I s= o o oab \ \ \ \ \SXYZS[S[ \ \ \ o o or c 3 K | ]Z}d |f rt t t n4d |k rt t t nd |f rt nt V [dS r NYELLOW BLACK_SQUARERESETGREEN WHITE_SQUARE)r r masks r r z6generate_attention_matrix_from_mask..K s
1: 6(<((((
66
,|
,U
,
,
, 1:\\
r )
boundariesz: i == j (diagonal) z: token_type_idszAttention MatrixzSliding Window Maskc 2 g | ]}t | t S r )r* r, )r ks r r# z7generate_attention_matrix_from_mask..i s( #e#e#eav$9q$9%$9$9#e#e#er z | c 3 K | ]h}| v r%|f rv rt t t n4|k rt t t n|f rt nt V id S r r) )r r r! img_tokenr/ r wordss r r z6generate_attention_matrix_from_mask..x s
E!H$$ad$ T8I8I ,| ,U , , , Avv 0<0000 AqDz
r c 3 K | ]x}| v r1v r-d f d |f k rt t t n8|k rt t t n | rt nt V ydS r( r) )r r r! r7 sliding_window_masktoken_type_bucketsr r8 s r r z6generate_attention_matrix_from_mask.. s * *
a((Y$->->CUVWYZVZC[_qrsuvrv_wCwCw 0<0000 66 4|4U444 'q)!,"\\! * * * * * *r z:
)intndimr max enumerater$ jointorchwherecumsumboolarange bucketizer- r+ r, r* appendljustliststrrjustmapzipr )r8 r/ r7 r" token_type_idsimage_seq_lengthmax_word_length
first_img_idxoutputr4 row_dummy
is_specialr2 legendf_stringvertical_headeridxrow word_repr colored_word row_displaysliding_window_rowr! r% r: r; r s ```` @@@@@r #generate_attention_matrix_from_maskr_ ( sv 88::D yA~~Aqqq!!!G} yA~~Aq!!!QQQJ E
A<>-> MDAJ 1 !y..AQJJAEzzQ56D q -/1 2 M ! o o o o ofklmfnfn o o o q I !#q(
"[
"
"2
&
&
*Z
7 = = ? ? AVAVWYAZAZ\]
\!%5%9;KLL
"_-?J W W W
o|
oU
o
o&
o,
oX]
o
o
oF
MM#, o)*-?-E-Ec)nnXYFY-Z-ZZH!))
MM(O u%% F F T S>Q " "#e#e4CWZ[^_`[a[aWbWbHcHcCdCd#e#e#e f f f f " "4Cs3q66{{(C(C#D#D E E E E3tS/%:;;<._ModelWrapperc t j | t j dd | _ || _ d S )Nr )nnModule__init__Linear dummy_moduleconfig)selfrl rb s r ri z@AttentionMaskVisualizer.__init__.._ModelWrapper.__init__ s5 ""4((($&IaOO! $ r N)__name__
__module__ __qualname__ri r r r
_ModelWrapperre s#
%
%
%
%
%r rq )r from_pretrained image_tokenhasattrget_text_configgetattrr" r r Exceptionr
ValueError
mapped_clsrg rh modeltodtyperepo_idrl )rm rb rl ry rq s r ri z AttentionMaskVisualizer.__init__ s: +J77" 6))++-= > > \")&*@*@*B*BDTVZ"["[D Q)&-@@JJ Q Q Q)&2OPPJJJ Q d: d d dee e$ % % % % %J % % % #]6:66
fl###! s 'A= = B B r5 input_sentencec 4 | || d S )N)suffix)visualize_attention_mask)rm r~ r s r __call__z AttentionMaskVisualizer.__call__ s! %%nV%DDDDDr c
| j }i }d }| j j t v rd}t j t
j |d j }d}t j
| j | }t |d r|j
}n&|j |j g d }|r| d| } ||||d
} |j |j g d | _
| d }
d | v r | d |d < |j | d
d } nr| j j t$ v rBt' j
| j } | | } | |d d }
n t+ d|j j d d|j _ | |
j \ }
}t3 j |
||j j f| j j }t3 j | }t= |j ||
|d }|| }
n>|
d d ! |
d|| }
dtE d| j j d| j# dz z }d}tI d| tI dd| j j d| j d | j# j% & tE | z d z |z tI | tO | |
| j
tQ | j d d | d | }tI | tI | d S ) Nzchttps://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg?download=trueT)streamr1 )rP rs r r pt)imagestextr return_tensorsattention_maskrO input_ids)r z Model type z) does not support attention visualizationeager)r| )rl input_embedsr cache_positionpast_key_valuesr z##z Attention visualization for z | r r< z" Attention visualization for [1m:z[0m z r" )r7 r" rO rP ))rz rl
model_typer
r openrequestsgetrawr rr r} rt rs tokenizerconvert_ids_to_tokensimage_token_idreplacer r
tokenizerx _attn_implementationtrainshaperB zeros hidden_sizer| rF r rE unsqueezeexpandr ry printrn centerr_ rv )rm r~ r rz kwargsrP img processorrs inputsr tokensr
batch_size
seq_lengthr r causal_masktop_bottom_border side_borderrW s r r z0AttentionMaskVisualizer.visualize_attention_mask s
; !%< < <wC*X\#d ; ; ; ?@@C %5dlUefffIy-00
g ' 3 ' 1 G GIaHb c cde f
N!/!7!7!M!M Ycv^bcccF ( 2 H H)JbIc d def gD #$4 5N 6))+12B+C'( (>>vk?RST?UVVFF
[
#'>
>
>%5dlCCI ''77F &Y~d K K KL\ ]NN m5<+B m m mnn n,3 )
!/!5
J {J
ELr sW . . . . . . 7 7 7 7 7 7 7 7 7 7 7 7 T T T T T T T T P P P P P P P P R R R R R R R R , , , , , , LLL `di i i i Xg& g& g& g& g& g& g& g& g& g&r