`i,pdZddlZddlmZmZddlmZddlmZddl m Z gd Z Gd d eZ d gZ dS) z Processor class for EVOLLA. N)OptionalUnion) BatchFeature)ProcessorMixin) AutoTokenizer)aa_seqfoldseekmsac 4eZdZdZddgZdgZdZdZdZdfd Z dd Z dd e fd Z dd e eeeefde eeeeeefde e d e e fdZdZdZdZdZfdZefdZxZS)EvollaProcessoran Constructs a EVOLLA processor which wraps a LLama tokenizer and SaProt tokenizer (EsmTokenizer) into a single processor. [`EvollaProcessor`] offers all the functionalities of [`EsmTokenizer`] and [`LlamaTokenizerFast`]. See the docstring of [`~EvollaProcessor.__call__`] and [`~EvollaProcessor.decode`] for more information. Args: protein_tokenizer (`EsmTokenizer`): An instance of [`EsmTokenizer`]. The protein tokenizer is a required input. tokenizer (`LlamaTokenizerFast`, *optional*): An instance of [`LlamaTokenizerFast`]. The tokenizer is a required input. protein_max_length (`int`, *optional*, defaults to 1024): The maximum length of the sequence to be generated. text_max_length (`int`, *optional*, defaults to 512): The maximum length of the text to be generated. protein_tokenizer tokenizersequence_max_lengthr Nc |td|tdt||d|j_||_||_dS)Nz+You need to specify an `protein_tokenizer`.z"You need to specify a `tokenizer`.z<|reserved_special_token_0|>) ValueErrorsuper__init__r pad_tokenprotein_max_lengthtext_max_length)selfrrrrkwargs __class__s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/evolla/processing_evolla.pyrzEvollaProcessor.__init__;sh  $JKK K  ABB B *I666#A "4.c*g}|D]n}|d}|d}ddt||D}||o|j|dd|d}|S)Nr r cfg|].\}}||z/S)upperlower).0sfs r z4EvollaProcessor.process_proteins..Ls1"["["[TQ17799qwwyy#8"["["[rptT)return_tensors truncation max_lengthpadding)getjoinzipappendrbatch_encode_plus) rproteinsr sa_sequencesproteinr r sa_sequence sa_tokenss rprocess_proteinsz EvollaProcessor.process_proteinsGs  - -G[[**F{{:..H''"["[SQYEZEZ"["["[\\K    , , , ,*<< $K]gk=  rrcg}|D]4}|j|dd}||5||dddd|}|S)NFT)tokenizeadd_generation_promptr*longest)add_special_tokensr+r.r,r-)rapply_chat_templater2)rtextsrpromptsmessagesprompt prompt_inputss r process_textzEvollaProcessor.process_textTs  # #H^77&*8F NN6 " " " " $& '  rr4 messages_listrc ||td||n|j}||n|j}t|tr|g}t|t t fr%t|dt t fs|g}t|t t fr(td|Dstdt|t t frFtd|Ds-tddtd |t|t t fr|D]}t|t t fs td t|d td |Dstd td|Dstd|Drtd|n tdt|d | ||}| ||}t|d|d|d|ddS)avThis method takes batched or non-batched proteins and messages_list and converts them into format that can be used by the model. Args: proteins (`Union[List[dict], dict]`): A list of dictionaries or a single dictionary containing the following keys: - `"aa_seq"` (`str`) -- The amino acid sequence of the protein. - `"foldseek"` (`str`) -- The foldseek string of the protein. messages_list (`Union[List[List[dict]], List[dict]]`): A list of lists of dictionaries or a list of dictionaries containing the following keys: - `"role"` (`str`) -- The role of the message. - `"content"` (`str`) -- The content of the message. protein_max_length (`int`, *optional*, defaults to 1024): The maximum length of the sequence to be generated. text_max_length (`int`, *optional*, defaults to 512): The maximum length of the text. Return: a dict with following keys: - `protein_input_ids` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The input IDs for the protein sequence. - `protein_attention_mask` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The attention mask for the protein sequence. - `text_input_ids` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The input IDs for the text sequence. - `text_attention_mask` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The attention mask for the text sequence. Nz3You need to specify `messages_list` and `proteins`.rc3@K|]}t|tVdSN isinstancedictr&ps r z+EvollaProcessor.__call__..s-:a:aST:a;N;N:a:a:a:a:a:arzUThe proteins should be a list of dictionaries, but not all elements are dictionaries.c3lK|]/}td|DV0dS)c3(K|] }|tvVdSrI)PROTEIN_VALID_KEYS)r&ks rrOz5EvollaProcessor.__call__...s(::A''::::::rN)allkeysrMs rrOz+EvollaProcessor.__call__..sR; ; ?@C::::: : :; ; ; ; ; ; rz2There should be a list of dictionaries with keys: z, z for each protein.But got: z;Each messages in messages_list should be a list instead of .c3@K|]}t|tVdSrIrJr&ms rrOz+EvollaProcessor.__call__..s,AA1:a..AAAAAArzfEach message in messages_list should be a list of dictionaries, but not all elements are dictionaries.c3`K|])}t|dkV*dS)rN)lenrUrXs rrOz+EvollaProcessor.__call__..s5<.sSDD=>CMMfi%88DDDDDDrzlEach message in messages_list should be a list of dictionaries with two keys: 'role' and 'content'.But got: zFThe messages_list should be a list of lists of dictionaries, but it's input_idsattention_mask)protein_input_idsprotein_attention_maskr`ra)data)rrrrKrLlisttuplerTr0rRtypeanyr9rEr) rr4rFrrrrBr8 text_tokenss r__call__zEvollaProcessor.__call__lsB  }4RSS S3E3Q//W[Wn-<-H//dNb h % % " zH mdE] 3 3 ,J}UVGWZ^`eYf"?     rc&|jj|i|SrI)r batch_decoderargsrs rrlzEvollaProcessor.batch_decodes*t~*D;F;;;rc&|jj|i|SrI)rdecoderms rrpzEvollaProcessor.decodes$t~$d5f555rc&|jj|i|SrI)rrlrms rprotein_batch_decodez$EvollaProcessor.protein_batch_decodes2t%2DCFCCCrc&|jj|i|SrI)rrprms rprotein_decodezEvollaProcessor.protein_decodes,t%,d=f===rc ~|jtj||jd|jv}|r|jdnd}|r||jdtj|fi|}|r||j |d|S)Nr) rsave_pretrainedospathr0protein_tokenizer_dir_name attributesindexremoverinsert)rsave_directoryrprotein_tokenizer_presentprotein_tokenizer_indexoutputsrs rrvzEvollaProcessor.save_pretraineds ..rw||NDLk/l/lmmm%84?$J!Pi"s$/"7"78K"L"L"Los $ 8)@)L O " "#6 7 7 7)%'').CCFCC $ Q)@)L O " "#:>> (    [     rr)rrwtypingrrfeature_extraction_utilsrprocessing_utilsrautor rRr__all__r#rrrs """"""""444444! 322TTTTTnTTTn  r