Pis$ddlmZddlmZmZmZmZmZmZm Z ddl m Z m Z ddl mZedZGddZGdd eZGd d eZGd d eZGddeZdeeddfdZGddeZdS))Path)AnyDictListLiteralMappingOptionalUnion)format_content_with_images load_image) Transform)systemuser assistantipythonc eZdZdZ ddedeeeeee ffde de de f d Z d eeee ffd Z e d ed dfd Zd edfdZed e fdZed efdZddZd efdZdS)Messageai This class represents individual messages in a fine-tuning dataset. It supports text-only content, text with interleaved images, and tool calls. The :class:`~torchtune.modules.transforms.tokenizers.ModelTokenizer` will tokenize the content of the message using ``tokenize_messages`` and attach the appropriate special tokens based on the flags set in this class. Args: role (Role): role of the message writer. Can be "system" for system prompts, "user" for human prompts, "assistant" for model responses, or "ipython" for tool call returns. content (Union[str, List[Dict[str, Any]]]): content of the message. If it is text only content, you can pass in a string. If it is multimodal content, pass in a list of dictionaries formatted as follows:: [ {"type": "image", "content": }, {"type": "text", "content": "What is in this image?"}, ] masked (bool): whether the message is masked in the sample. If True, do not use in loss calculation. Default: False ipython (bool): whether the message is a tool call. Default: False eot (bool): whether the message corresponds to the end of a turn, where control is handed over to the assistant from the user or the user from the assistant. Default: True. Should be true in most cases except for: - For multiple consecutive assistant messages (i.e., tool calls by assistant), only the last assistant message will have ``eot=True`` - All ipython messages (tool call returns) should set ``eot=False``. Note: Message class expects any image content to be in `PIL Image format `_. FTrolecontentmaskedreotc||_|||_||_||_||_|dS)N)r_convert_to_list_of_dictrrrr_validate_message)selfrrrrrs l/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/data/_messages.py__init__zMessage.__init__;sM 44W==         returnct|trd|dgSt|ts Jd||S)zUser is currently allowed to pass in a string for text-only content. This ensures that the content is formatted as a list of dictionaries.texttyperz2content must be of type List[Dict[str, Any]], got ) isinstancestrlist)rrs rrz Message._convert_to_list_of_dictKsh gs # # :#889 9 T   J J I I I J J rdc ||d|d|dd|dd|ddS) z Construct a Message from a dictionary. Args: d (dict): dictionary containing the fields of the Message. Returns: Message: constructed Message. rrrFrrT)rrrrr)get)clsr's r from_dictzMessage.from_dictWs^s6iL555))EE)U++eT""     rzPIL.Image.Imagec$d|jDS)z7 Returns media content of the message. c6g|]}|ddk|dS)r#imager.0rs r z%Message.get_media..ns3   #*gfoQX>X>XGI >X>X>Xrrrs r get_mediazMessage.get_mediajs%  .2l    rc>td|jDS)z= Returns whether the message contains media. c3.K|]}|ddkVdS)r#r.Nr/r0s r z)Message.contains_media..ws+JJ'76?g-JJJJJJr)anyrr4s rcontains_mediazMessage.contains_mediars# JJT\JJJJJJrcJdd|jDS)z; Returns text-only content of the message. c3>K|]}|ddk|dVdS)r#r!rNr/r0s rr8z'Message.text_content..~s?  #*gfoQW>W>WGI >W>W>W>W  r)joinrr4s r text_contentzMessage.text_contentys8 ww  .2l      rNc|jr|jrtd|j|jr*|jdkr!td|jd|jdSdS)NzGMedia tokens in tool calls are not supported. Both are set in message: rz6Only assistant messages can be tool calls. Found role z in message: )rr: ValueErrorr?rr4s rrzMessage._validate_messages < D/ mZ^Zkmm  < DI44tttaeartt   44rc@d|jD}d|jd|dS)Ncg|] }|d Sr3r/r0s rr2z$Message.__repr__..sGGGw *GGGrzMessage(role='z ', content=))rr)r content_onlys r__repr__zMessage.__repr__s2GG$,GGG G GGlGGGGr)FFT)rN)__name__ __module__ __qualname____doc__Roler r%rrrboolrr classmethoddictr+r5propertyr:r?rrFr/rrrrs""P !!!sDc3h001! !  !  !!!! 4S#X3G     $ 9   [ $ 4 12    KKKKXK  c   X H#HHHHHHrrc eZdZdZ d dedeeeefdeedeefdZ d e ee fd e ee ffd Z dS) InputOutputToMessagesae Message transform class that converts a single sample with "input" and "output" fields, (or equivalent fields specified in column_map) to user and assistant messages, respectively. This is useful for datasets that have two columns, one containing the user prompt string and the other containing the model response string:: | input | output | |-----------------|------------------| | "user prompt" | "model response" | Args: train_on_input (bool): Whether the model is trained on the user prompt or not. Default is False. column_map (Optional[Dict[str, str]]): a mapping to change the expected "input" and "output" column names to the actual column names in the dataset. Keys should be "input" and "output" and values should be the actual column names. Default is None, keeping the default "input" and "output" column names. new_system_prompt (Optional[str]): if specified, prepend a system message. This can serve as instructions to guide the model response. Default is None. image_dir (Optional[Path]): path to the directory containing the images that is prepended to all image paths in the dataset. For example, if ``image_dir="/home/user/dataset/"` and the sample image path was ``"images/1.jpg"``, the final image path that will be loaded is ``"/home/user/dataset/images/1.jpg"``. If None, assume images are available in current working directory or are located on a remote url. For text-only, leave as None. Default is None. Raises: ValueError: If ``column_map`` is provided and ``input`` not in ``column_map``, or ``output`` not in ``column_map``, **or** if ``image_dir`` is provided but ``image`` not in ``column_map``. FNtrain_on_input column_mapnew_system_prompt image_dirc||_||_||_|jgd|jvr*td|jdd|jvr*td|jdn dddd|_d|jvr|td|d ||_dS) Ninputz2Expected a key of 'input' in column_map but found .outputz3Expected a key of 'output' in column_map but found r.)rWrYr.zimage_dir is specified as zO but 'image' is not in column_map. Please specify an 'image' key in column_map.)rRrTrSrAkeysrU)rrRrSrTrUs rrzInputOutputToMessages.__init__s-!2$ ? &do-- bI]I]I_I_bbbt.. c$/J^J^J`J`ccc/ )08gVVDO $/..00 0 0Y5J?Y???  #rsamplerc,d|vpd|jvo|jd|v}|rv||jd}t|tr0t|}|j |j|z }t |}n|}d|dd||jddg}nd||jddg}d||jddg}t d||j dt d |d dg}|jt d |jddg|z}d |iS) Nr.r"r!rWrYrTrrrrrFrmessages) rSr$r%rrUr rrRrT)rr[ is_multimodal image_path pil_imageroutput_contentr^s r__call__zInputOutputToMessages.__call__s6) t & M4?7+Cv+M   V 89J*c** '!*-- >-!%*! # # # #D2&wsCx02&WS#X5F2&2&2&2&2&2&rrQceZdZdZ d dedeeeefdeefdZde ee fd e ee ffd Z dS) ChosenRejectedToMessagesa^ Transform for converting a single sample from datasets with "chosen" and "rejected" columns containing conversations to a list of chosen and rejected messages. For example:: | chosen | rejected | |----------------------------------------|----------------------------------------| | [{"role": "user", "content": Q1}, | [{"role": "user", "content": Q1}, | | {"role": "assistant", "content": A1}] | {"role": "assistant", "content": A2}] | will be converted to: .. code-block:: python chosen = [ Message(role="user", content="Q1"), Message(role="assistant", content="A1"), ] rejected = [ Message(role="user", content="Q1"), Message(role="assistant", content="A2"), ] A single sample typically consists of a single optional system prompt and one or multiple turns of user and assistant messages. Args: train_on_input (bool): Whether the model is trained on the user prompt or not. Default is False. column_map (Optional[Dict[str, str]]): a mapping to change the expected "chosen" and "rejected" column names to the actual column names in the dataset. Keys should be "chosen" and "rejected" and values should be the actual column names. Default is None, keeping the default column names. new_system_prompt (Optional[str]): if specified, prepend a system message. This can serve as instructions to guide the model response. Setting this will OVERRIDE any system messages already present in the dataset. Default is None. Raises: ValueError: If ``column_map`` is provided and ``chosen`` not in ``column_map``, or ``rejected`` not in ``column_map``. FNrRrSrTc||_||_|r[d|vr%td|dd|vr%td|d||_dSddd|_dS)Nchosenz3Expected a key of 'chosen' in column_map but found rXrejectedz5Expected a key of 'rejected' in column_map but found rhrirRrTrArZ _column_maprrRrSrTs rrz!ChosenRejectedToMessages.__init__3s -!2  Lz)) ^*//J[J[^^^++ `JOOL]L]``` *D   *2 KKD   rr[rcJg}||jdD]Z}|ddkr|j|ddko|j |d<|t|[g}||jdD]Z}|ddkr|j|ddko|j |d<|t|[|j8t d|jddg|z}t d|jddg|z}||d S) NrhrrrrriTr]rj)rlrTrRappendrr+)rr[chosen_messagesmessagerejected_messagess rrcz!ChosenRejectedToMessages.__call__Hsd.x89 ? ?Gv(**t/E/Q!(K!?!'' H   " "7#4#4W#=#= > > > >d.z:; A AGv(**t/E/Q!(K!?!'' H   $ $W%6%6w%?%? @ @ @ @  ! -!4+A$TX  O !4+A$TX!" !"  *7HIIIrFNN rGrHrIrJrLr rr%rrrrcr/rrrfrf s''V %/3+/ LLLT#s(^,L$C= LLLL*JwsCx0JWS#X5FJJJJJJrrfc eZdZdZ ddedeeeefdeedeed eef d Z d e ee fd e ee ffd Z dS)ShareGPTToMessagesa Convert a single chat sample adhering to the ShareGPT JSON structure to torchtune's :class:`~torchtune.data.Message` structure. A single sample typically consists of a single optional system prompt and one or multiple turns of user and assistant messages. ShareGPT follows:: { "conversations": [ { "from": , "value": , }, ... ] } :class:`~torchtune.data.Message` follows:: [ { "role": , "content": , }, ... ] Args: train_on_input (bool): whether the prompt should remain unmasked. For multimodal datasets, ``train_on_input`` is always False and this value is ignored. Default: False column_map (Optional[Dict[str, str]]): a mapping from the expected columns ("conversations") to the new column names in the dataset. Key should be "conversations" and value should be the new column name. If None, keep the default "conversations". Default is None. new_system_prompt (Optional[str]): if specified, prepend a system message. This can serve as instructions to guide the model response. Setting this will OVERRIDE any system messages already present in the dataset. Default is None. image_dir (Optional[Path]): path to the directory containing the images that is prepended to all image paths in the dataset. For example, if ``image_dir="/home/user/dataset/"` and the sample image path was ``"images/1.jpg"``, the final image path that will be loaded is ``"/home/user/dataset/images/1.jpg"``. If None, assume images are available in current working directory or are located on a remote url. For text-only, leave as None. Default is None. image_tag (Optional[str]): placeholder tags in the text content of each message to be replaced by image special tokens. If images are present and this is None, then will prepend image tokens to the first user message in the sample by default. If text-only, this field is ignored. Default is ``""``. Raises: ValueError: If ``column_map`` is provided and ``conversations`` not in ``column_map``. FNrRrSrTrU image_tagc||_||_|r1d|vr%td|d||_n ddd|_||_||_dS)N conversationsz:Expected a key of 'conversations' in column_map but found rXr.)rzr.)rRrTrArZrlrUrx)rrRrSrTrUrxs rrzShareGPTToMessages.__init__s-!2  Tj00 eQ[Q`Q`QbQbeee *D  1@7SSD ""rr[rcjdddd}g}|j+|td|jddd|vpd|jvo|jd|v}d }||jd D]}||d }|d }|dkr|j&|dkrc|ra|s_||jd} |j |j| z } t | } |j d| d d|d g}nt||j| g}d}|dko |j p|} |t||| d|iS)J Return a list of Message objects from the provided sample dict. Args: sample (Mapping[str, Any]): a single data sample with "messages" field pointing to a list of dict messages. Returns: List[Message]: A list of messages with "role" and "content" fields. rrr)rhumangptNTr]r.Frzfromvaluer"r!)rximagesrrrr^) rTrorrlrUr rxr rR) rr[role_mapr^r_ image_loadedrqrrr`rars rrczShareGPTToMessages.__call__s' LL  ! - OO!4+A$TX     6) t' ' OD, # C= ####*:&wsCx0:&WS#X5F:&:&:&:&:&:&rrvceZdZdZ d dedeeeefdeefdZde eee fd e eee ffd Z d e ee fd e ee ffd Z dS)OpenAIToMessagesa Convert a single chat sample adhering to the `OpenAI chat completion `_ JSON structure to torchtune's :class:`~torchtune.data.Message` structure. This supports both text and image messages. A single sample typically consists of a single optional system prompt and one or multiple turns of user and assistant messages. For example:: { "messages": [ { "role": , "content": [ { "type": "text", "text": "What'''s in this image?", }, { "type": "image_url", "image_url": { "url": , }, }, }, ... ] } :class:`~torchtune.data.Message` follows:: [ { "role": , "content": [ { "type": "text", "content": "What'''s in this image?", }, { "type": "image", "content": , }, ], }, ... ] Args: train_on_input (bool): whether the prompt should remain unmasked. Default: False column_map (Optional[Dict[str, str]]): a mapping from the expected columns ("messages") to the new column names in the dataset. Key should be "messages" and value should be the new column name. If None, keep the default "messages". Default is None. new_system_prompt (Optional[str]): if specified, prepend a system message. This can serve as instructions to guide the model response. Setting this will OVERRIDE any system messages already present in the dataset. Default is None. Raises: ValueError: If ``column_map`` is provided and ``messages`` not in ``column_map``. FNrRrSrTc||_||_|r2d|vr%td|d||_dSddi|_dS)Nr^z5Expected a key of 'messages' in column_map but found rXrkrms rrzOpenAIToMessages.__init__1st -!2  8++ `JOOL]L]``` *D    *J7D   rrrcg}|D]j}|ddkr|d|dd-|ddkr1|dt|dddk|S)zPConverts a list of content dicts from the OpenAI format to the torchtune format.r#r!r" image_urlr.url)ror )rrconverted_content content_dicts r_convert_from_openai_contentz-OpenAIToMessages._convert_from_openai_contentBs#  LF#v--!((# V0DEEf%44!(( '#-l;.G.N#O#O ! rr[cg}|j+|td|jdd||jdD]}|ddkr|j|ddko|j }t |dt r||d}n#t |dtr|d}|t|d|| d|iS) r|NrTr]r^rrrr) rTrorrlrRr$r&rr%)rr[updated_messagesrqrrs rrczOpenAIToMessages.__call__Us7  ! -  # #!4+A$TX    d.z:;  Gv(**t/E/Qfo4St?R;RF'),d33 -;;GI+>Z1ZZZ  <6 ! !i6&9&9U!UU!a%UUU  <8 # #A_1___ L  ! !rcreZdZdZ d dedeeeeffdZde ee fde ee ffd Z dS) AlpacaToMessagesa| Message transform class for Alpaca-style datasets with "instruction", "input", and "output" (or equivalent fields specified in column_map) columns. User messages are formed from the instruction + input columns and assistant messages are formed from the output column. Prompt templating is conditional on the presence of the "input" column, and thus is handled directly in this transform class instead of a dedicated :class:`~torchtune.data.PromptTemplate` class due to this custom logic. Args: train_on_input (bool): Whether the model is trained on the user prompt or not. Default is True. column_map (Optional[Dict[str, str]]): a mapping to change the expected "instruction", "input", and "output" column names to the actual column names in the dataset. Default is None, keeping the default column names. TNrRrSc6||_||_ddd|_dS)NzBelow is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. ### Instruction: {instruction} ### Input: {input} ### Response: zBelow is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: {instruction} ### Response: ) prompt_inputprompt_no_input)rRrStemplate)rrRrSs rrzAlpacaToMessages.__init__s1-$\ E    rr[rc|jpi}|dd}|dd}|dd}||vr7||r/|jd||||}n'|jd||}t d||j d t d ||d d g}d |iS)NrW instructionrYr)rrWr)rrTr]rFr^)rSr)rformatrrR)rr[rS key_inputkey_instruction key_outputpromptr^s rrczAlpacaToMessages.__call__s_* NN7G44 $.. FF^^Hh77   6)#4 ]>299"?36);L:FF]#45<<"?3=F ..      z*      H%%r)TNrtr/rrrrs"SW  " 7?S#X7O    $&wsCx0&WS#X5F&&&&&&rr)pathlibrtypingrrrrrr r torchtune.data._utilsr r torchtune.modules.transformsr rKrrQrfrvrrrr/rrrsEEEEEEEEEEEEEEEEEEHHHHHHHH222222xHxHxHxHxHxHxHxHvu&u&u&u&u&Iu&u&u&p^J^J^J^J^Jy^J^J^JBD&D&D&D&D&D&D&D&NF.F.F.F.F.yF.F.F.R'!7m'! '!'!'!'!T@&@&@&@&@&y@&@&@&@&@&r