PiF-vddlZddlmZmZmZmZddlZddlZddl m Z ddl m Z e j dZ ddejded ed ejfd Z ddejded ed ejfd Z ddejded ed ejfd Z ddejded ed ejfdZ ddejded ed ejfdZ ddejded ed ejfdZdefde deeejfdejdejjd edeejeegejfd ejfdZGddZGddeZGddeZdS)N)CallableDictListOptional)utils)TransformerDecoderDEBUG? layer_idsn_layerse_scalereturnctjt||j}|tj||dz k|dz}|tj|z S)N)devicer )torchoneslenrwheresumr r r loss_scaless u/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/modules/early_exit_loss.pyuniform_loss_scalersW*S^^I4DEEEK I1 ,Dgs S SSK ;// //ctj|dz}|tj||dz k|dz}|tj|z SNrr )rTensorrrrs rlinear_l_loss_scalersM,y1}--K I1 ,Dgs S SSK ;// //rctj|dzd}|tj||dz k|dz}|tj|z S)Nrr)dimr )rcumsumrrrs rsum_l_loss_scaler##sR,y1}!444K I1 ,Dgs S SSK ;// //rctj|dz}|tj||dz k|dz}|tj|z Sr)rsqrtrrrs rsqrt_l_loss_scaler&+sM*Y]++K I1 ,Dgs S SSK ;// //rcd|dzz }|tj||dz k|dz}|tj|z S)Nr r)rrrrs rinv_l_loss_scaler(3sHQ'K I1 ,Dgs S SSK ;// //rctjtj|dz}|tj||dz k|dz}|tj|z Sr)r reciprocalr%rrrs rinv_sqrt_l_loss_scaler+;sX"5:i!m#<#<==K I1 ,Dgs S SSK ;// //rmodelhidden_states_dictlabelsloss_fn loss_scale_fnctj|}d|_t|}t |}t |} tj|} | | } | d| d} | } | |d d} || | } | |d} ||jk}| d|z } |tj| | t|j|}tj|| zS)a Compute the early exit loss for a given model and outputs of intermediate layers. This function takes in a model, a dictionary of hidden states, labels, a loss function, and optional parameters for scaling the loss. It computes the early exit loss by iterating over the hidden states, computing the logits and losses at each layer, and then scaling and summing these losses. Args: model (TransformerDecoder): The model to compute the early exit loss for. hidden_states_dict (Dict[int, torch.Tensor]): A dictionary of hidden states, where each key is a layer index and each value is a tensor of shape [b, s, d]. labels (torch.Tensor): The labels for the input data. loss_fn (torch.nn.Module): The loss function to use (should be the same as the standard loss function for last layer). e_scale (float): A scaling factor for the early exit losses. Defaults to 1.0. loss_scale_fn (Callable[[torch.Tensor, int, float], torch.Tensor]): A function to determine scale of each layer's loss. Defaults to uniform_loss_scale. Returns: torch.Tensor: The computed early exit loss. noner)copydeepcopy reductionrtuplevalueskeysrstackunembedreshapesize contiguousrepeatview ignore_indexrfloatrtolayers)r,r-r.r/r r0 batch_loss_fne hidden_stateshidden_layer_idshidden_states_stacked logits_earlylabels_repeated losses_early s_unpadded losses_scaless rearly_exit_lossrOCs:M'**M$M A,335566M/446677!K 66==!677L''L,=,=b,A,ABBL**,,LmmAq))11"55O =??L$$Q++LG005577J%%''++B//*