`i/ddlZddlmZmZddlZddlmZddlmZddlm Z ddl m Z ddl m Z mZmZdd lmZdd lmZmZmZdd lmZd d lmZmZd dlmZmZmZmZddl m!Z!GddeZ"GddeZ#GddeZ$edGddeZ%Gddej&Z'edGdde$e Z(gd Z)dS)!N)OptionalUnion)nn)ACT2FN)Cache)GenerationMixin)BaseModelOutputBaseModelOutputWithPastCausalLMOutputWithPast)Unpack)TransformersKwargsauto_docstringcan_return_tuple)check_model_inputs) AutoModelAutoModelForCausalLM)Qwen2AudioAttentionQwen2AudioEncoderQwen2AudioEncoderLayerQwen2AudioPreTrainedModel) VoxtralConfigceZdZdS)VoxtralAttentionN__name__ __module__ __qualname__/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/voxtral/modular_voxtral.pyrr'Dr"rceZdZdS)VoxtralEncoderLayerNrr!r"r#r&r&+r$r"r&c&eZdZdZdZdZdZdZdZdS)VoxtralPreTrainedModelTN)rrr _supports_flex_attn_supports_cache_class_supports_attention_backend_can_compile_fullgraph_no_split_modulesr!r"r#r(r(/s4 "&!"&r"r(z: The Voxtral encoder, which is a Whisper encoder. ) custom_introcDeZdZeedZe ddeefdZ dS)VoxtralEncoder) attentions hidden_statesNkwargsc |jj|jjdz|jjdz}|jd|kr$t d|d|jdd|d||jjj |jjj }tj ||}tj ||}|ddd }|jj}||z|j }tj ||j|j }t%|jD]\}} | ||d } | d}||}t+| S)a Args: input_features (`torch.LongTensor` of shape `(batch_size, feature_size, sequence_length)`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] attention_mask (`torch.Tensor`)`, *optional*): Voxtral does not support masking of the `input_features`, this argument is preserved for compatibility, but it is not used. By default the silence in the input log mel spectrogram are ignored. rz:Qwen2Audio expects the mel input features to be of length z , but found z-. Make sure to pad the input mel features to .)dtypedevicerr)ptrainingN)attention_masklayer_head_mask)last_hidden_state)configmax_source_positionsconv1strideconv2shape ValueErrortoweightr7r8r functionalgelupermuteembed_positionsdropoutr: enumeratelayers layer_normr ) selfinput_featuresr;r3expected_seq_length inputs_embeds embed_posr2idx encoder_layer layer_outputss r#forwardzVoxtralEncoder.forwardDs&#k>ARSTAUUX\XbXijkXll   #': : :LM`LLn|oCDFoGLLvILLL (**1B1HQUQ[QbQi*jj **4::n+E+EFF  **4::m+D+DEE %--aA66 (/ &266}7JKK  --mt|VZVc-dd "+DK"8"8 - - C)M- $M *!,MM 66 +    r"N) rrr rr&_can_record_outputsrr rrWr!r"r#r0r09sd', - - +, - - - - - - r"r0c*eZdZdeffd ZdZxZS)VoxtralMultiModalProjectorr>c6ttj|jj|jjd|_t|j |_ tj|jj|jjd|_ dS)NF)bias) super__init__rLinear audio_configintermediate_size text_config hidden_sizelinear_1rprojector_hidden_actactlinear_2rOr> __class__s r#r_z#VoxtralMultiModalProjector.__init__vsv  &"5"GI[Ignsttt &56 &"4"@&BTB`glmmm r"c||}||}||}|SrX)rergrh)rOaudio_featuresr2s r#rWz"VoxtralMultiModalProjector.forward|s; n55 //  m44 r")rrr rr_rW __classcell__rjs@r#r[r[usZn}nnnnnn r"r[zs The Voxtral model, which consists of Whisper encoder, a multi-modal projector and a LLama language model. ceZdZdgZddiZddgdgfiZdgZfdZdZd Z d Z d Z d Z d Z dejfdZdejfdZee d deejdeejdeejdeejdeedeejdeejdeedeejdeeejfdeedefdZfdZxZ S)!VoxtralForConditionalGenerationzlm_head.weightlm_head colwise_repr2logitsrJc4t||jj|_t j|j|_tj|j|_ t||_ | dSrX) r^r_rc vocab_sizer from_configra audio_towerrlanguage_modelr[multi_modal_projector post_initris r#r_z(VoxtralForConditionalGeneration.__init__s|     ,7$01DEE2>v?QRR%?%G%G" r"c4|jSrX)rxget_input_embeddingsrOs r#r|z4VoxtralForConditionalGeneration.get_input_embeddingss"77999r"c:|j|dSrX)rxset_input_embeddings)rOvalues r#rz4VoxtralForConditionalGeneration.set_input_embeddingss 0077777r"c4|jSrX)rxget_output_embeddingsr}s r#rz5VoxtralForConditionalGeneration.get_output_embeddingss"88:::r"c:|j|dSrX)rxset_output_embeddings)rOnew_embeddingss r#rz5VoxtralForConditionalGeneration.set_output_embeddingss 11.AAAAAr"c:|j|dSrX)rx set_decoder)rOdecoders r#rz+VoxtralForConditionalGeneration.set_decoders ''00000r"c4|jSrX)rx get_decoderr}s r#rz+VoxtralForConditionalGeneration.get_decoders"..000r"rPc||}|j}|d|jjj}||}|S)a This method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector. Args: input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] Returns: `torch.FloatTensor`: The audio embeddings. r5)rwr=reshaper>rarbry)rOrP audio_outputsaudio_hidden_states audio_embedss r#get_audio_featuresz2VoxtralForConditionalGeneration.get_audio_featuressX((88 +=199"dk>V>hii112EFF r"c`tjdt||S)NzUThe method `get_audio_embeds` is deprecated. Please use `get_audio_features` instead.)warningswarn FutureWarningr)rOrPs r#get_audio_embedsz0VoxtralForConditionalGeneration.get_audio_embedss2 cer   &&~666r"Nr input_idsr; position_idspast_key_valuesrRlabels use_cachecache_positionlogits_to_keepr3returnc t|||}||~||} ||jjkd} || |j| |j}|jd||||||| | d| }|S)aj Example: ```python >>> from transformers import VoxtralForConditionalGeneration, AutoProcessor >>> import torch >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> repo_id = "mistralai/Voxtral-Mini-3B-2507" >>> processor = AutoProcessor.from_pretrained(repo_id) >>> model = VoxtralForConditionalGeneration.from_pretrained(repo_id, dtype=torch.bfloat16, device_map=device) >>> conversation = [ { "role": "user", "content": [ { "type": "audio", "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/dude_where_is_my_car.wav", }, {"type": "text", "text": "What can you tell me about this audio?"}, ], } ] >>> inputs = processor.apply_chat_template(conversation) >>> inputs = inputs.to(device, dtype=torch.bfloat16) >>> outputs = model.generate(**inputs, max_new_tokens=30) >>> processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True) ["This audio is a humorous conversation between two friends, likely in English, where one of them is trying to figure out what the other's tattoo says."] ```Nr5)r;rrrRrrrrr!) r|rr>audio_token_id unsqueezemasked_scatterrEr8rx)rOrrPr;rrrRrrrrr3raudio_token_maskoutputss r#rWz'VoxtralForConditionalGeneration.forwardsb  7D5577 BBM  %)*?22>BBL!*T[-G GRRSUVV )88 ##M$899 , )%+')) , ,  , , r"c|dd}|d}tj|i|}||ddkr||d<|S)NrPrr)popgetr^prepare_inputs_for_generation)rOargsr3rPr model_inputsrjs r#rz=VoxtralForConditionalGeneration.prepare_inputs_for_generationsl $4d;;$455F!!12Fc5</0F+,F FFF^FP         r"rp)r(r0rp)*rtypingrrrr activationsr cache_utilsr generationr modeling_outputsr r r processing_utilsr utilsrrr utils.genericrautorr qwen2_audio.modeling_qwen2_audiorrrrconfiguration_voxtralrrr&r(r0Moduler[rp__all__r!r"r#rs """""""" !!!!!! ))))))``````````&&&&&&IIIIIIIIII//////22222222 100000     *        0   6 4 4 4 4 4 &4 4  4 n         SSSSS&