`iddZddlmZddlmZmZddlZddlmZddlm Z ddl m Z dd l m Z mZd d lmZejeZee d Gdde ZGddejZGddejZGddejZGddejZe Gdde ZdgZdS)zPyTorch UnivNetModel model.) dataclass)OptionalUnionN)nn) ModelOutput)PreTrainedModel)auto_docstringlogging) UnivNetConfigz Output class for the [`UnivNetModel`], which includes the generated audio waveforms and the original unpadded lengths of those waveforms (so that the padding can be removed by [`UnivNetModel.batch_decode`]). ) custom_introc\eZdZUdZdZeejed<dZ eejed<dS)UnivNetModelOutputa" waveforms (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Batched 1D (mono-channel) output audio waveforms. waveform_lengths (`torch.FloatTensor` of shape `(batch_size,)`): The batched length in samples of each unpadded waveform in `waveforms`. N waveformswaveform_lengths) __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__r/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/univnet/modeling_univnet.pyrrsO.2Ix)*11148hu0188888rrcJeZdZdZdeffd ZdejfdZdZ dZ xZ S)#UnivNetKernelPredictorResidualBlockz Implementation of the residual block for the kernel predictor network inside each location variable convolution block (LVCBlock). Parameters: config: (`UnivNetConfig`): Config for the `UnivNetModel` model. configct|j|_|j|_|j|_|j|_|jdz dz}tj |j|_ tj |j|j|j|d|_ tj |j|j|j|d|_dS)Nr Tpaddingbias)super__init__model_in_channelschannelskernel_predictor_conv_size kernel_sizekernel_predictor_dropout dropout_probleaky_relu_sloperDropoutdropoutConv1dconv1conv2)selfrr# __class__s rr&z,UnivNetKernelPredictorResidualBlock.__init__<s 0 !<"; & 7#a'A-z$"344 Yt}dmT=MW^eijjj Yt}dmT=MW^eijjj r hidden_statesc"|}||}||}tj||j}||}tj||j}||zSN)r/r1r functional leaky_relur-r2)r3r5residuals rforwardz+UnivNetKernelPredictorResidualBlock.forwardLsy  ]33  =11  00@UVV  =11  00@UVV x''rctjj}ttjjdrtjjj}||j||jdSN weight_norm)rutilsr>hasattrparametrizationsr1r2r3r>s rapply_weight_normz5UnivNetKernelPredictorResidualBlock.apply_weight_normVs[h* 28,m < < @(3?K DJ DJrctj|jtj|jdSr7)rr?remove_weight_normr1r2r3s rrEz6UnivNetKernelPredictorResidualBlock.remove_weight_norm^s8 ##DJ/// ##DJ/////r) rrrrr r&rrr;rCrE __classcell__r4s@rrr2skkkkkkk (U%6((((   0000000rrcXeZdZdZ d dededeffd Zdejfd Z d Z d Z xZ S) UnivNetKernelPredictora  Implementation of the kernel predictor network which supplies the kernel and bias for the location variable convolutional layers (LVCs) in each UnivNet LVCBlock. Based on the KernelPredictor implementation in [maum-ai/univnet](https://github.com/maum-ai/univnet/blob/9bb2b54838bb6d7ce767131cc7b8b61198bc7558/model/lvcnet.py#L7). Parameters: config: (`UnivNetConfig`): Config for the `UnivNetModel` model. conv_kernel_size (`int`, *optional*, defaults to 3): The kernel size for the location variable convolutional layer kernels (convolutional weight tensor). conv_layers (`int`, *optional*, defaults to 4): The number of location variable convolutional layers to output kernels and biases for. rrconv_kernel_size conv_layersctj|_djz|_||_||_|j|jz|jz|jz|_|j|jz|_j |_ j |_ j |_j|_j|_|jdz dz}t%j|j |j ddd|_t%jfdt-|jD|_t%j|j |j|j|d|_t%j|j |j|j|d|_dS)Nr!r Tr"c.g|]}tSr)r).0_rs r z3UnivNetKernelPredictor.__init__..s#'t't'tXY(KF(S(S't't'tr)r%r&model_hidden_channelsconv_in_channelsconv_out_channelsrLrMkernel_channels bias_channels num_mel_binsresnet_in_channels kernel_predictor_hidden_channelsresnet_hidden_channelsr)resnet_kernel_sizekernel_predictor_num_blocks num_blocksr-rr0 input_conv ModuleListrange resblocks kernel_conv bias_conv)r3rrLrMr#r4s ` rr&zUnivNetKernelPredictor.__init__ts  & <!"V%A!A 0&  !D$: :T=R RUYUe e "3d6FF"("5&,&M#"("C < & 7*Q.14)D$;T=XZ[efmqrrr't't't't]bcgcr]s]s't't'tuu9  ')=t?V`gnr     ');T=T^elp   r spectrogramc|j\}}}||}tj||j}|jD] }||}||}||}| ||j |j |j |j |} | ||j |j |} | | fS)a Maps a conditioning log-mel spectrogram to a tensor of convolutional kernels and biases, for use in location variable convolutional layers. Note that the input spectrogram should have shape (batch_size, input_channels, seq_length). Args: spectrogram (`torch.FloatTensor` of shape `(batch_size, input_channels, seq_length)`): Tensor containing the log-mel spectrograms. Returns: tuple[`torch.FloatTensor, `torch.FloatTensor`]: tuple of tensors where the first element is the tensor of location variable convolution kernels of shape `(batch_size, self.conv_layers, self.conv_in_channels, self.conv_out_channels, self.conv_kernel_size, seq_length)` and the second element is the tensor of location variable convolution biases of shape `(batch_size, self.conv_layers. self.conv_out_channels, seq_length)`. )shaper`rr8r9r-rcrdreviewrMrUrVrL contiguous) r3rf batch_sizerR seq_lengthr5resblockkernel_hidden_statesbias_hidden_stateskernelsbiasess rr;zUnivNetKernelPredictor.forwards "%0$5! Az 44  00@UVV  4 4H$H]33MM#// >>!^^M::'++     !  "  !     *,, $((     "    *,, rc.tjj}ttjjdrtjjj}||j|jD]}|||j||j dSr=) rr?r>r@rAr`rcrCrdrer3r>layers rrCz(UnivNetKernelPredictor.apply_weight_normsh* 28,m < < @(3?K DO$$$^ & &E  # # % % % % D$%%% DN#####rctj|j|jD]}|tj|jtj|jdSr7)rr?rEr`rcrdrer3rts rrEz)UnivNetKernelPredictor.remove_weight_normsw ##DO444^ ' 'E  $ $ & & & & ##D$4555 ##DN33333r)rrK rrrrr intr&rrr;rCrErGrHs@rrJrJcs&!" $ $ $ $  $ $ $ $ $ $ L,5#4,,,,\ $ $ $4444444rrJc eZdZdZdededeffd ZddZ dd ej d ej d ej ded ef d Z dZ dZ xZ S)UnivNetLvcResidualBlocka Implementation of the location variable convolution (LVC) residual block for the UnivNet residual network. Parameters: config: (`UnivNetConfig`): Config for the `UnivNetModel` model. kernel_size (`int`): The kernel size for the dilated 1D convolutional layer. dilation (`int`): The dilation for the dilated 1D convolutional layer. rr*dilationc"t|j|_||_||_|j|_|j|jdz zdz}tj|j|j|j||j|_ dS)Nr r!)r#r{) r%r&rThidden_channelsr*r{r-rr0conv)r3rr*r{r#r4s rr&z UnivNetLvcResidualBlock.__init__s %;&  & 7-4#3a#78A=I    ]     rc|}tj||j}||}tj||j}|||||}t j|ddd|jddft j |dd|jdddfz}||z}|SN)hop_size) rr8r9r-r~location_variable_convolutionrsigmoidr}tanh)r3r5kernelr$rr:s rr;zUnivNetLvcResidualBlock.forwards  00@UVV  -00  00@UVV ::=&RVai:jj  mAAA7M9M7Mqqq4P&QRRUZU_ !!!T133QQQ6 7V V  !=0 rr r5rr$rc|j\}}}|j\}}} } } || |zkrtd| |zd|d|t| dz dz z} tj|| | fdd}|d|d| zz|}||kr$tj|d|fdd}|d||}|d d d d d d d d d |f}|dd }|d | d}tj d ||} | tj } | d  d  tj }| |z} | || d } | S)u Performs location-variable convolution operation on the input sequence (hidden_states) using the local convolution kernel. This was introduced in [LVCNet: Efficient Condition-Dependent Modeling Network for Waveform Generation](https://huggingface.co/papers/2102.10815) by Zhen Zheng, Jianzong Wang, Ning Cheng, and Jing Xiao. Time: 414 μs ± 309 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each), test on NVIDIA V100. Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, in_channels, in_length)`): The input sequence of shape (batch, in_channels, in_length). kernel (`torch.FloatTensor` of shape `(batch_size, in_channels, out_channels, kernel_size, kernel_length)`): The local convolution kernel of shape (batch, in_channels, out_channels, kernel_size, kernel_length). bias (`torch.FloatTensor` of shape `(batch_size, out_channels, kernel_length)`): The bias for the local convolution of shape (batch, out_channels, kernel_length). dilation (`int`, *optional*, defaults to 1): The dilation of convolution. hop_size (`int`, *optional*, defaults to 256): The hop_size of the conditioning sequence. Returns: `torch.FloatTensor`: the output sequence after performing local convolution with shape (batch_size, out_channels, in_length). z#Dim 2 of `hidden_states` should be z ) but got zX. Please check `hidden_states` or `kernel` and `hop_size` to make sure they are correct.r r!constantrrNrKzbildsk,biokl->bolsd) memory_format)rh ValueErrorrxrr8padunfold transposereinsumtochannels_last_3d unsqueezerjri)r3r5rr$r{rbatchrR in_length out_channelsr* kernel_lengthr#output_hidden_statess rrz5UnivNetLvcResidualBlock.location_variable_convolutions<,1q)=C\:q, ] 1 2 2]mh6N]]Zc]]]  S+/Q!6777 ))-'79KZYZ[[ %,,Q1w;0FQQ h  M--ma]JXYZZM%,,Q(CC %aaaAAAqqq)8)&;< %//155 %,,Q Q??  %|,A=RXYY366UE[6\\~~b!!++B//22AW2XX3d:3>>@@EEe\[]^^##rctjj}ttjjdrtjjj}||jdSr=)rr?r>r@rAr~rBs rrCz)UnivNetLvcResidualBlock.apply_weight_normOsIh* 28,m < < @(3?K DIrcNtj|jdSr7)rr?rEr~rFs rrEz*UnivNetLvcResidualBlock.remove_weight_normVs  ##DI.....rr)r r)rrrrr rxr&r;rrrrCrErGrHs@rrzrzs            ,    * ?$?$(?$!?$ ?$  ?$  ?$?$?$?$B///////rrzcdeZdZdZ d dededeffd Zdejdejfd Z d Z d Z xZ S) UnivNetLvcBlocka# Implementation of the location variable convolution (LVC) residual block of the UnivNet residual block. Includes a `UnivNetKernelPredictor` inside to predict the kernels and biases of the LVC layers. Based on LVCBlock in [maum-ai/univnet](https://github.com/maum-ai/univnet/blob/9bb2b54838bb6d7ce767131cc7b8b61198bc7558/model/lvcnet.py#L98) Parameters: config (`UnivNetConfig`): Config for the `UnivNetModel` model. layer_id (`int`): An integer corresponding to the index of the current LVC resnet block layer. This should be between 0 and `len(config.resblock_stride_sizes) - 1)` inclusive. lvc_hop_size (`int`, *optional*, defaults to 256): The hop size for the location variable convolutional layers. rrlayer_id lvc_hop_sizec ztj_j|_j|_j|_ |_ j _ tj _ tjjjdjzjjdzjdzzjdz_t#jj _tjfdt)j D_dS)Nr!)strider#output_paddingcRg|]#}tjj|$Sr)rzr* dilations)rQirr3s rrSz,UnivNetLvcBlock.__init__..s1 r r rVW $VT-=t~a?P Q Q r r rr)r%r&rTr}resblock_kernel_sizesr*resblock_stride_sizesrresblock_dilation_sizesrcond_hop_lengthr-lenr_rConvTranspose1d convt_prerJkernel_predictorrarbrc)r3rrrr4s`` rr&zUnivNetLvcBlock.__init__ls& %;!7A28< 7A+ & 7dn--+    O;K1$t{Q6;?    !7vt?OQUQ` a a r r r r r[`aeap[q[q r r r  rr5rfc Ztj||j}||}||\}}t |jD]C\}}|dd|ddddddddf}|dd|ddddf}|||||j}D|Sr) rr8r9r-rr enumeratercr) r3r5rfrprqrrmrr$s rr;zUnivNetLvcBlock.forwards 00@UVV }55 // <<$T^44 a aKAxQQQ111aaaAAA-.F!!!Q111*%D$H]FD4K_```MMrc tjj}ttjjdrtjjj}||j|j|jD]}|dSr=) rr?r>r@rArrrCrcrss rrCz!UnivNetLvcBlock.apply_weight_normsh* 28,m < < @(3?K DN### //111^ & &E  # # % % % % & &rctj|j|j|jD]}|dSr7)rr?rErrrcrvs rrEz"UnivNetLvcBlock.remove_weight_norms_ ##DN333 00222^ ' 'E  $ $ & & & & ' 'rrrwrHs@rrrZs*            < U%6 UEV    &&&'''''''rrceZdZUeed<dZdeffd Ze ddej de ej de ej de ej de e d e eej eff d Zd Zd Zd ZxZS) UnivNetModelrinput_featuresc6ttj|_j|_t jjj dddd|_ tj }d}gj D]}||z} |t j fdt|D|_t jj dddd|_|dS)Nr rreflect)r*rr# padding_modec@g|]}t||S))rr)r)rQrr hop_lengthss rrSz)UnivNetModel.__init__..sG      !,Q   r)r#r)r%r&rr num_kernelsr-rr0r'rTconv_prerappendrarbrc conv_post post_init)r3r num_layers hop_lengthrrr4s ` @rr&zUnivNetModel.__init__s<    v;<< & 7  $  ("     566   2 + +F#f,J   z * * * *      z**       6#?Aq_hiii rNnoise_sequence padding_mask generator return_dictreturnc||n |jj}|dk}|s|d}|j\}}} |0|dk} | s|d}n1|||jjf} t j| ||j|j }|jd} |dkr| dkr| |dd}n#| dkr|dkr| | dd}| |krtd| d|d|V|dkr|d}|jd} | |krtd | d|d| d d}| d d}| |}|jD]}|||}tj||j}||}t j|}|d}d}|t j|d }|s||f}|St/|| S) a noise_sequence (`torch.FloatTensor`, *optional*): Tensor containing a noise sequence of standard Gaussian noise. Can be batched and of shape `(batch_size, sequence_length, config.model_in_channels)`, or un-batched and of shape (sequence_length, config.model_in_channels)`. If not supplied, will be randomly generated. padding_mask (`torch.BoolTensor`, *optional*): Mask indicating which parts of each sequence are padded. Mask values are selected in `[0, 1]`: - 1 for tokens that are **not masked** - 0 for tokens that are **masked** The mask can be batched and of shape `(batch_size, sequence_length)` or un-batched and of shape `(sequence_length,)`. generator (`torch.Generator`, *optional*): A [torch generator](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. return_dict: Whether to return a [`~utils.ModelOutput`] subclass instead of a plain tuple. Example: ```python >>> from transformers import UnivNetFeatureExtractor, UnivNetModel >>> from datasets import load_dataset, Audio >>> model = UnivNetModel.from_pretrained("dg845/univnet-dev") >>> feature_extractor = UnivNetFeatureExtractor.from_pretrained("dg845/univnet-dev") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> # Resample the audio to the feature extractor's sampling rate. >>> ds = ds.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate)) >>> inputs = feature_extractor( ... ds[0]["audio"]["array"], sampling_rate=ds[0]["audio"]["sampling_rate"], return_tensors="pt" ... ) >>> audio = model(**inputs).waveforms >>> list(audio.shape) [1, 140288] ``` Nrr)rdtypedevicer z&The batch size of `noise_sequence` is z+ and the batch size of `input_features` is z', but the two are expected to be equal.z$The batch size of `padding_mask` is r!)dim)rr)ruse_return_dictrrrhr'rrandnrrrepeatrrrrcrr8r9r-rrsqueezesumr)r3rrrrrspectrogram_batchedspectrogram_batch_sizespectrogram_lengthrRnoise_sequence_batchednoise_sequence_shapenoise_sequence_batch_sizepadding_mask_batch_sizer5rmwaveformroutputss rr;zUnivNetModel.forwards`&1% ") =!/!9!9!! > >h9Rhh(>hhh   #!!Q&&+55a88 &2&8&; #&*@@@ l;Rll,Blll '00A66 '11!Q77 m44  D DH$H]NCCMM 00@UVV }55  =11 !((++   #$y1===  !12GN!-    rct|tjtjtjfrR|jjd|jj |j "|j j dSdSdS)zInitialize the weights.g)meanstdN) isinstancerLinearr0rweightdatanormal_rinitializer_ranger$zero_)r3modules r _init_weightszUnivNetModel._init_weightsQsz fry")R5GH I I ) M  & &CT[5R & S S S{&  &&((((( ) )&&rctjj}ttjjdrtjjj}||j|jD]}|||jdSr=) rr?r>r@rArrcrCrrss rrCzUnivNetModel.apply_weight_normXsh* 28,m < < @(3?K DM"""^ & &E  # # % % % % DN#####rctj|j|jD]}|tj|jdSr7)rr?rErrcrrvs rrEzUnivNetModel.remove_weight_normbs^ ##DM222^ ' 'E  $ $ & & & & ##DN33333r)NNNN)rrrr rmain_input_namer&r rrr Generatorboolrtuplerr;rrCrErGrHs@rrrs/ &O%}%%%%%%N7;48/3&* x x )x !!23x u01 x EO, x d^ x  uU&');; <x x x ^x t)))$$$4444444rr)r dataclassesrtypingrrrrmodeling_outputsrmodeling_utilsr r?r r configuration_univnetr get_loggerrloggerrModulerrJrzrr__all__rrrrs&"!!!!!!!"""""""" ++++++------,,,,,,,,000000  H % %  9 9 9 9 9 9 9   9.0.0.0.0.0").0.0.0bu4u4u4u4u4RYu4u4u4p|/|/|/|/|/bi|/|/|/~M'M'M'M'M'biM'M'M'`{4{4{4{4{4?{4{4{4|  r