`i30TddlmZmZddlZddlmZddlmZmZmZm Z m Z ddl m Z ddl mZddlmZmZd d lmZejeZGd d e ZGd deZGddejZGdde ZGddeZGddeZgdZdS))OptionalUnionN)nn)LlavaCausalLMOutputWithPastLlavaForConditionalGeneration LlavaModelLlavaModelOutputWithPastLlavaPreTrainedModel)ACT2FN)Cache)auto_docstringlogging)VipLlavaConfigceZdZdS)VipLlavaModelOutputWithPastN__name__ __module__ __qualname__/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/vipllava/modular_vipllava.pyrr&DrrceZdZdS)VipLlavaCausalLMOutputWithPastNrrrrrr*rrrc*eZdZdeffd ZdZxZS)VipLlavaMultiModalProjectorconfigctt|jtrdnt |j}t j||jj z|j |_ t j ||jj z|j j d|_t|j|_t j |j j |j j d|_dS)Nr)epsT)bias)super__init__ isinstancevision_feature_layersintlenr LayerNorm vision_config hidden_sizeprojector_layernorm_epsprojector_layernormLinear text_configlinear_1r projector_hidden_actactlinear_2)selfr num_feature_layers __class__s rr%z$VipLlavaMultiModalProjector.__init__/s ",V-I3"O"OvQQUXY_YuUvUv#%< !5!A AvGe$ $ $   !5!A A   *    &56 &"4"@&BTB`gklll rc||}||}||}||}|SN)r.r1r3r4)r5 hidden_statess rforwardz#VipLlavaMultiModalProjector.forward>sN00??  m44 //  m44 r)rrrrr%r; __classcell__)r7s@rrr.sZ m~ m m m m m mrrceZdZdS)VipLlavaPreTrainedModelNrrrrr>r>Frrr>ceZdZ ddejdeeeeeffdZ e ddeej deejdeej deej dee d eejdeeeeefd eed eed eed eedeej deeeffdZdS) VipLlavaModelN pixel_valuesr'c&||n |jj}||dt|trj|ddddf}n$fd|D}t j|d}||}|S)aW Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layers (`Union[int, list[int]]`): The vision feature layer, or the list of indexes of the layers to select the vision feature. Returns: image_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`). NT)output_hidden_statesrcBg|]}j|ddddfS)Nr)r:).0index image_outputss r z4VipLlavaModel.get_image_features..es2kkkEm9%@ABBGkkkr)dim) r r' vision_towerr&r(r:torchcatmulti_modal_projector)r5rAr'image_featuresrGs @rget_image_featuresz VipLlavaModel.get_image_featuresKs &;%F ! !DKLm )),T)RR  +S 1 1 ?*89NOPQPQPQSTSUSUPUVNNlkkkUjkkkN"Y~2>>>N33NCCr input_idsattention_mask position_idspast_key_values inputs_embeds use_cacheoutput_attentionsrC return_dictcache_positionreturnc t| | n |jj} | | n |jj} | | n |jj} ||n |jj}|du|duz rt d|||}|e|||}||j |j }| |||}| ||}|j d|||||| | d| d | }t|j|j|j|j||nd}| r|n|S) z vision_feature_layers (`Union[int, list[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. Nz:You must specify exactly one of input_ids or inputs_embedsrAr')rUrOT) rRrSrTrUrVrWrCrXrY)last_hidden_staterTr: attentionsimage_hidden_statesr)r rWrCuse_return_dictr' ValueErrorget_input_embeddingsrPtodevicedtypeget_placeholder_maskmasked_scatterlanguage_modelrr]rTr:r^to_tuple)r5rQrArRrSrTrUr'rVrWrCrXrY lm_kwargsrOspecial_image_maskoutputsoutputs rr;zVipLlavaModel.forwardjs,2C1N--TXT_Tq$8$D $+Jj &1%2JPT    %;vv&//*;*;;rr9) NNNNNNNNNNNN)rrrrL FloatTensorrrr(listrPr LongTensorTensorr booltuplerr;rrrr@r@Jshl!-FNuUXZ^_bZcUcOdFe>15481537+/59AE$(,0/3&*59B<B<E,-B<u01B<!. B< u/0 B< "% B<  12B< (c49n(=>B<D>B<$D>B<'tnB<d^B<!!12B< u11 2B<B<B<^B<B<B>> import torch >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, VipLlavaForConditionalGeneration >>> model = VipLlavaForConditionalGeneration.from_pretrained("llava-hf/vip-llava-7b-hf", device_map="auto", dtype=torch.float16) >>> processor = AutoProcessor.from_pretrained("llava-hf/vip-llava-7b-hf") >>> prompt = "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: \n{}###Assistant:" >>> question = "Can you please describe this image?" >>> prompt = prompt.format(question) >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/compel-neg.png" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=text, images=image, return_tensors="pt").to(0, torch.float16) >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=20) >>> processor.decode(generate_ids[0][len(inputs["input_ids"][0]):], skip_special_tokens=True) The image features a brown and white cat sitting on a green surface, with a red ball in its ```NT) rQrArRrSrTrUrVr'rWrCrXrYr)logitsrx vocab_size)lossr{rTr:r^r_r)r rWrCr`r'rwr&r(slicelm_head loss_functionr0r|rrTr:r^r_)r5rQrArRrSrTrUr'rxrVrWrCrXrYryrjrlr: slice_indicesr{r}s rr;z(VipLlavaForConditionalGeneration.forwardsih2C1N--TXT_Tq$8$D $+Jj &1%SV8W8Wk~ot444]k mAAA}aaa,?@AA  %%VFt{OfOq%rrD-#3!/) ' ;     rr9)NNNNNNNNNNNNNr)rrrrLrnrrr(rorPrprqr rrrsrr;rrrrurushluu!-uFNuUXZ^_bZcUcOdFeuuuu15481537+/59AE-1$(,0/3&*5934] ] E,-] u01] !. ] u/0 ] "% ]   12]  (c49n(=>] )*] D>] $D>] 'tn] d^] !!12] c5</0] " u44 5#] ] ] ] ] ] rru)r@rur>)typingrrrLr(transformers.models.llava.modeling_llavarrrr r activationsr cache_utilsr utilsrrconfiguration_vipllavar get_loggerrloggerrrModulerr>r@ru__all__rrrrs  #""""""" "!!!!! ,,,,,,,,222222  H % %     ":        %@   ")0     2   c<c<c<c<c