PiGLddlmZddlZddlmcmZGddejjZGddejjZ GddejjZ Gd d ejjZ Gd d ejjZ Gd dejjZ dS))ListNc reZdZdZd deffd Z d dejdejdejd ed ejf d Z xZ S) ForwardKLLoss The Kullback-Leibler divergence loss for valid indexes. Implementation of https://github.com/jongwooko/distillm/blob/17c0f98bc263b1861a02d5df578c84aea652ee65/distillm/losses.py Args: ignore_index (int): Specifies a target value that is ignored and does not contribute to the input gradient. The loss is divided over non-ignored targets. Default: -100. ignore_indexcVt||_dSNsuper__init__rselfr __class__s t/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/modules/loss/kd_losses.pyr zForwardKLLoss.__init__' (Tstudent_logitsteacher_logitslabels normalizereturnc tj|dtj}tj|}tj|dtj}tj||z|d}tj|dd} ||j k } |s-tj| | dzd Stj| dd} | dkrtj d| j Stj| | dzd tj| ddz S5 Args: student_logits (torch.Tensor): logits from student model of shape (batch_size*num_tokens, vocab_size). teacher_logits (torch.Tensor): logits from teacher model of shape (batch_size*num_tokens, vocab_size). labels (torch.Tensor): Ground truth labels of shape (batch_size, vocab_size). normalize (bool): Whether to normalize the loss by the number of unmasked elements. Returns: torch.Tensor: KL divergence loss of shape (1,). )dimdtyperrdevice) Fsoftmaxtorchfloat32isinf log_softmax masked_fillsumviewrinttensorr") rrrrr teacher_probinf_maskstudent_logprob prod_probsxmask sum_maskss rforwardzForwardKLLoss.forwardsK*yRu}MMM ;~..-BemTTT&|o'ExQRSS Ijb ) ) ) . .r 2 2$++0022 8Ia$))B--/Q7777 7Idiimm333 >><AH555 5 !diimm+3333ei " ST6U6U6UUUrrT __name__ __module__ __qualname____doc__r,r r%Tensorboolr5 __classcell__rs@rrr s))S)))))) !V!V !V !V !V  !V  !V!V!V!V!V!V!V!Vrrc reZdZdZd deffd Z d dejdejdejd ed ejf d Z xZ S) ReverseKLLossrrrcVt||_dSr r rs rr zReverseKLLoss.__init__KrrTrrrrrctj|dtj}tj|dtj}tj|dtj}tj|tj|z}tj||z|d} | tj||z|dz} tj| dd} ||j k } |s-tj| | dzd Stj| dddkrtj d| j Stj| | dzd tj| ddz Sr) r#r$r%r&r(r'r)r*r+rr,r-r") rrrrr student_probr0teacher_logprobr/r1r2r3s rr5zReverseKLLoss.forwardOs*yRu}MMM -BemTTT-BemTTT;~..^1L1LL&|o'ExQRSS e' (FRSTTT Ijb ) ) ) . .r 2 2$++0022 8Ia$))B--/Q7777 7 9TYYr]] * * *a / /<AH555 5 !diimm+3333ei " ST6U6U6UUUrr6r7r8r@s@rrBrB@s))S)))))) %V%V %V %V %V  %V  %V%V%V%V%V%V%V%VrrBc veZdZdZddedeffd Z ddejd ejd ejd e d ejf d Z xZ S)SymmetricKLLossa The Symmetric Kullback-Leibler divergence loss for valid indexes. Implementation of https://github.com/jongwooko/distillm/blob/17c0f98bc263b1861a02d5df578c84aea652ee65/distillm/losses.py Args: sym_kd_ratio (float): Ratio of symmetric KL divergence loss. When set to 1 this loss reduces to forward KL divergence, when set to 0 this loss reduces to reverse kl divergence. ignore_index (int): Specifies a target value that is ignored and does not contribute to the input gradient. The loss is divided over non-ignored targets. Default: -100. Raises: ValueError: If sym_kd_ratio is not in the range [0, 1]. ?r sym_kd_ratiorct|dks|dkrtd||_||_t ||_t||_dS)Nr g?z(sym_kd_ratio must be in the range [0, 1]) r r ValueErrorrrJrfklrBrkl)rrJrrs rr zSymmetricKLLoss.__init__so  #  !3!3GHH H(( .. ..rTrrrrrc|j|||||zd|jz |||||zzS)r)rJrMrN)rrrrrs rr5zSymmetricKLLoss.forwards[* 488 NFI$ $  " "dhh NFI' '    r)rIrr7) r9r:r;r<floatr,r r%r=r>r5r?r@s@rrHrHws  //U///////                    rrHceZdZdZd dedeffd Zdeejdeejd ejd ejfd Z xZ S) ForwardKLWithChunkedOutputLossa Forward KL with chunked outputs that saves memory by only upcasting one chunk at a time. Since the model is trained with bf16, before computing KL divergence, we have to upcast it to fp32 for better accuracy and stability. When upcasting happens, the memory usage doubles. Models like llama3 have large vocabulary size and, therefore, have a large output result (bsz, num_tokens, vocab_size). If we chunk on the token level, you can still compute the cross entropy normally, but upcasting only one chunk at a time saves considerable memory. Args: num_output_chunks (int): Number of chunks to chunk the output into. Each chunk has shape (batch_size, num_tokens / num_output_chunks, vocab_size). Default: 8 ignore_index (int): Specifies a target value that is ignored and does not contribute to the input gradient. The loss is divided over non-ignored targets. Default: -100 rnum_output_chunksrct||_||_t ||_dSr )r r rUrrfkl_lossrrUrrs rr z'ForwardKLWithChunkedOutputLoss.__init__< !2(%l33 rrrrrcd|D}d|D}||jk}d||jdD}d}t |||D]"\}}}|||||dz }#t j|d d } | d kr!t j d|d j S|t j|d d z S) a Args: student_logits (List[torch.Tensor]): List of chunked logits from student model of length ``self.num_output_chunks``, where each chunk has shape (batch_size, num_tokens / num_output_chunks, vocab_size). teacher_logits (List[torch.Tensor]): List of chunked logits from teacher model of length ``self.num_output_chunks``, where each chunk has shape (batch_size, num_tokens / num_output_chunks, vocab_size). labels (torch.Tensor): Ground truth labels of shape (batch_size, num_tokens). Returns: torch.Tensor: KL divergence loss of shape (1,). Example: >>> loss_fn = ForwardKLWithChunkedOutputLoss() >>> >>> h = torch.tensor([bsz, num_tokens, dim]) >>> output_chunks = [model.output(chunk) for chunk in h.chunk(num_chunks, dim=1)] >>> teacher_chunks = [teacher_model.output(chunk) for chunk in h.chunk(num_chunks, dim=1)] >>> labels = torch.tensor([bsz, num_tokens]) >>> loss = loss_fn(output_chunks, teacher_chunks, labels) c`g|]+}|d|d,Srreshapesize.0teacher_logits_chunks r z:ForwardKLWithChunkedOutputLoss.forward..G   $ ! ( (-A-F-Fr-J-J K K   rc`g|]+}|d|d,Sr\r]rastudent_logits_chunks rrcz:ForwardKLWithChunkedOutputLoss.forward..rdrc8g|]}|dSr\r^ra target_chunks rrcz:ForwardKLWithChunkedOutputLoss.forward..6      $ $   rrPrr Frrrr!) rr,chunkrUziprWr%r*r+r-r") rrrrr3total_fkl_loss student_chunk teacher_chunk label_chunkr4s rr5z&ForwardKLWithChunkedOutputLoss.forwardsN<  (6     (6   $++0022  & T-C K K    9< NF: :    5M=+ dmm}kU, NNIdiimm333 >><N1,=,DEEE E $))B--Q ? ? ???rrTr r9r:r;r<r,r rr%r=r5r?r@s@rrSrSs$44#4444444 9@U\*9@U\*9@ 9@  9@9@9@9@9@9@9@9@rrSceZdZdZd dedeffd Zdeejdeejd ejd ejfd Z xZ S) ReverseKLWithChunkedOutputLossa Reverse KL with chunked outputs that saves memory by only upcasting one chunk at a time. Since the model is trained with bf16, before computing KL divergence, we have to upcast it to fp32 for better accuracy and stability. When upcasting happens, the memory usage doubles. Models like llama3 have large vocabulary size and, therefore, have a large output result (bsz, num_tokens, vocab_size). If we chunk on the token level, you can still compute the cross entropy normally, but upcasting only one chunk at a time saves considerable memory. Args: num_output_chunks (int): Number of chunks to chunk the output into. Each chunk has shape (batch_size, num_tokens / num_output_chunks, vocab_size). Default: 8 ignore_index (int): Specifies a target value that is ignored and does not contribute to the input gradient. The loss is divided over non-ignored targets. Default: -100 rTrrUrct||_||_t ||_dSr )r r rUrrBrkl_lossrXs rr z'ReverseKLWithChunkedOutputLoss.__init__rYrrrrrc|d|D}d|D}||jk}d||jdD}d}t |||D]"\}}}|||||dz }#|t j|d d z S) a Args: student_logits (List[torch.Tensor]): List of chunked logits from student model of length ``self.num_output_chunks``, where each chunk has shape (batch_size, num_tokens / num_output_chunks, vocab_size). teacher_logits (List[torch.Tensor]): List of chunked logits from teacher model of length ``self.num_output_chunks``, where each chunk has shape (batch_size, num_tokens / num_output_chunks, vocab_size). labels (torch.Tensor): Ground truth labels of shape (batch_size, num_tokens). Returns: torch.Tensor: KL divergence loss of shape (1,). Example: >>> loss_fn = ReverseKLWithChunkedOutputLoss() >>> >>> h = torch.tensor([bsz, num_tokens, dim]) >>> output_chunks = [model.output(chunk) for chunk in h.chunk(num_chunks, dim=1)] >>> teacher_chunks = [teacher_model.output(chunk) for chunk in h.chunk(num_chunks, dim=1)] >>> labels = torch.tensor([bsz, num_tokens]) >>> loss = loss_fn(output_chunks, teacher_chunks, labels) c`g|]+}|d|d,Sr\r]r`s rrcz:ReverseKLWithChunkedOutputLoss.forward..8rdrc`g|]+}|d|d,Sr\r]rfs rrcz:ReverseKLWithChunkedOutputLoss.forward..<rdrc8g|]}|dSr\rirjs rrcz:ReverseKLWithChunkedOutputLoss.forward..BrlrrPrr Frmrr) rr,rnrUroryr%r*r+) rrrrr3total_rkl_lossrqrrrss rr5z&ReverseKLWithChunkedOutputLoss.forwards <  (6     (6   $++0022  & T-C K K    9< NF: :    5M=+ dmm}kU, NN $))B--Q ? ? ???rrtrur@s@rrwrws$44#4444444 5@U\*5@U\*5@ 5@  5@5@5@5@5@5@5@5@rrwceZdZdZ ddededeffd Zd eej d eej d ej d ej fd Z xZ S) SymmetricKLWithChunkedOutputLossa Symmetric KL with chunked outputs that saves memory by only upcasting one chunk at a time. Since the model is trained with bf16, before computing KL divergence, we have to upcast it to fp32 for better accuracy and stability. When upcasting happens, the memory usage doubles. Models like llama3 have large vocabulary size and, therefore, have a large output result (bsz, num_tokens, vocab_size). If we chunk on the token level, you can still compute the cross entropy normally, but upcasting only one chunk at a time saves considerable memory. Args: num_output_chunks (int): Number of chunks to chunk the output into. Each chunk has shape (batch_size, num_tokens / num_output_chunks, vocab_size). Default: 8 sym_kd_ratio (float): Ratio of symmetric KL divergence loss. When set to 1 this loss reduces to forward KL divergence, when set to 0 this loss reduces to reverse kl divergence. Default: 0.5 ignore_index (int): Specifies a target value that is ignored and does not contribute to the input gradient. The loss is divided over non-ignored targets. Default: -100 rTrIrrUrJrct||_||_||_t |j|j|_dS)N)rJr)r r rUrJrrH sym_kl_loss)rrUrJrrs rr z)SymmetricKLWithChunkedOutputLoss.__init__hs[ !2((**9J   rrrrrc|d|D}d|D}||jk}d||jdD}d}t |||D]"\}}}|||||dz }#|t j|d d z S) a Args: student_logits (List[torch.Tensor]): List of chunked logits from student model of length ``self.num_output_chunks``, where each chunk has shape (batch_size, num_tokens / num_output_chunks, vocab_size). teacher_logits (List[torch.Tensor]): List of chunked logits from teacher model of length ``self.num_output_chunks``, where each chunk has shape (batch_size, num_tokens / num_output_chunks, vocab_size). labels (torch.Tensor): Ground truth labels of shape (batch_size, num_tokens). Returns: torch.Tensor: KL divergence loss of shape (1,). Example: >>> loss_fn = SymmetricKLWithChunkedOutputLoss() >>> >>> h = torch.tensor([bsz, num_tokens, dim]) >>> output_chunks = [model.output(chunk) for chunk in h.chunk(num_chunks, dim=1)] >>> teacher_chunks = [teacher_model.output(chunk) for chunk in h.chunk(num_chunks, dim=1)] >>> labels = torch.tensor([bsz, num_tokens]) >>> loss = loss_fn(output_chunks, teacher_chunks, labels) c`g|]+}|d|d,Sr\r]r`s rrcz.rdrc`g|]+}|d|d,Sr\r]rfs rrcz.rdrc8g|]}|dSr\rirjs rrcz.rlrrPrr Frmrr) rr,rnrUrorr%r*r+) rrrrr3total_sym_kl_lossrqrrrss rr5z(SymmetricKLWithChunkedOutputLoss.forwardvs<  (6     (6   $++0022  & T-C K K     9< NF: :    5M=+ !1!1}kU"2""   !59TYYr]]#B#B#BBBr)rTrIr) r9r:r;r<r,rQr rr%r=r5r?r@s@rrrRs."#!              5CU\*5CU\*5C 5C  5C5C5C5C5C5C5C5Crr)typingrr%torch.nn.functionalnn functionalr#ModulerrBrHrSrwrrrrs 0V0V0V0V0VEHO0V0V0Vf4V4V4V4V4VEHO4V4V4Vn2 2 2 2 2 eho2 2 2 jR@R@R@R@R@UX_R@R@R@jN@N@N@N@N@UX_N@N@N@bYCYCYCYCYCuxYCYCYCYCYCr