.`iK LddlmZddlZddlZddlmZmZddlm Z ddl m Z m Z ddl mZddlmZmZmZmZmZddlmZdd lmZdd lmZdd lmZdd lmZd dlm Z e e!Z"GddeZ#GddeZ$e dGddeZ%dS))SetN) ModelConfig VllmConfig) init_logger)DispatchPoolerPoolingParamsUpdate)PoolerNormalize)EmbeddingPoolerHeadSequencePoolerSequencePoolingMethodSequencePoolingMethodOutputget_seq_pooling_method)pooler_for_token_embed)LlamaForCausalLM) PoolingTask)cached_tokenizer_from_config)PoolingMetadata)default_pooling_typec eZdZdZdeffd Z ddejdejded edzd ef d Z d ejd efd Z d e e fdZ de d efdZdejded efdZxZS)GritLMMeanPoolz8As `MeanPool`, but only includes non-instruction tokens. model_configc\t|_tjfddD_dt t dtjffd }|gd_ |gd_ |gd_ dS) NcJi|]}||gd S)r)convert_tokens_to_ids).0tok tokenizers u/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/model_executor/models/gritlm.py z+GritLMMeanPool.__init__../s@    00#77:   )▁<<|embed><0x0A>usertokensreturncDtjfd|DS)Nc*g|]}j|S) token_ids)rtokenselfs r zBGritLMMeanPool.__init__..tokens_to_ids..5s GGGuT^E2GGGr!)nparray)r*r1s r tokens_to_idsz.GritLMMeanPool.__init__..tokens_to_ids4s(8GGGGGGGHH Hr!)r#r%r)r%r'r()r(r$r%r&r%r'r()r#r%r&r%r'r() super__init__rrr/liststrr3ndarrayuser_pattern_idsembed_newline_pattern_idsembed_pattern_ids)r1rr5r __class__s` @rr7zGritLMMeanPool.__init__$s (01BCC     P     I$s) I  I I I I I I!. .W.W.W X X)6 = = =* * &"//Y/Y/Y!Z!Zr!rNarrtarget start_idxend_idxr+cz|dkrtdt|dkst|dkrtdt|}t|}||}t|t|||z dzD])}||||z|kr|cS*dS)a Find the first occurrence of `target` in `arr` starting from `start_idx`. Args: arr: The array to search within. target: The consecutive subsequence to find. start_idx: The starting index to search from (inclusive). end_idx: The ending index to search from (exclusive). Returns: The index of the first occurrence of `target` in `arr`. rz `start_idx` must be non-negativez#Empty `arr` or `target` not allowedNr) ValueErrorlenrangeminall)r1r?r@rArBarr_len target_lenis r _find_arrayzGritLMMeanPool._find_array=s( q==?@@ @ s88q==CKK1,,BCC Cc(([[ ?Gy#gw/Ca/G"H"HII  AAJ&'616688  rr!prompt_token_idscjd}|d|jdkrtd|S|j}|||jdddkr|j}|||d}|dkr|t|z}ntd d}|S) aF Get the length of the instruction in the prompt. We do a pattern matching to find the instruction in the prompt, and then return the length of the instruction. The pattern matching is done using integers instead of strings because the prompt is given as a list of token IDs. rr"zlBOS token not found in prompt, thus using empty string for instruction. GritLM requires BOS token in prompt.r)rArB)rArDzQuery instruction not found in prompt, thus using BOS token as instruction instead. GritLM requires query instruction in prompt.)r/loggerwarningr=rMr;r<rF)r1rNinstruction_lenr=found_embed_pattern_idxs r_get_instruction_lenz#GritLMMeanPool._get_instruction_lenbs A $."7 7 7 NN7    # "!2    $"71a      !% > #'"2"2 /1#3# #  #b ( (5.sK   )))--//*?*?*A*ABB   r!r`)devicer)dimdtype) prompt_lenstorchtensorget_prompt_token_idsr8Tensorzipappendmeanfloat32) r1r\r]re instr_lensoffset pooled_data prompt_len instr_lens ` rforwardzGritLMMeanPool.forwards '2 \    !1!F!F!H!H       5<(** %(j%A%A ! ! !J   fy06J3FFGLLM    j FFr!)rN)__name__ __module__ __qualname____doc__rr7r3r:intrMrUrrrWrr[rfrirr rs __classcell__r>s@rrr!sEBB[[[[[[[[:" ## Z# # # t # ####J/RZ/C////bS%5< <8K<<<<|* % r!rc$eZdZdeffd ZxZS) GritLMPoolerrc|j}|Jt|jdkrt |nt |jt |jtdS)NMEAN) head_dtype activation)poolinghead) pooler_configr6r7seq_pooling_typerrr rr )r1rrr>s rr7zGritLMPooler.__init__s$2 ((( !1V;;|,,,+M,JKK$'2*,,  r!)rtrurvrr7ryrzs@rr|r|sD [          r!r|r~)rc8eZdZdZdZ d dededdffd ZxZS) GritLMaqThis class implements the embedding model for parasail-ai/GritLM-7B-vllm. The class inherits from LlamaForCausalLM and provides a custom pooling layer. The main difference between the pooling layer in GritLM and the one in LlamaForCausalLM is that GritLM ignores the query instruction in the prompt when pooling the hidden states. Embedding prompts should be in the following format: - With instruction: "<|user|> INSTRUCTION <|embed|> PROMPT". - Without instruction: "<|embed|> PROMPT". Generation prompts should be in the following format: - "<|user|> PROMPT <|assistant|> " T vllm_configprefixr+Nc 8|jjdkr&|jj}d|_d|j_d|_t jd||d||jj}|8tt|t|jd|_ dSdS)NrF)rr) token_embedr&r.) r runner_type hf_config is_causal cache_configsliding_windowr6r7rrrr|pooler)r1rrkwargsrrr>s rr7zGritLM.__init__s  # /9 < <#0:I"'I 6:K $ 3'+I $J[JJ6JJJ#0>  $(#9-#H#H)+*BCCDKKK % $r!)r) rtrurvrwis_pooling_modelrr9r7ryrzs@rrrsu"   r!r)&collections.abcrrar3rf vllm.configrr vllm.loggerr!vllm.model_executor.layers.poolerrr-vllm.model_executor.layers.pooler.activationsr )vllm.model_executor.layers.pooler.seqwiser r r r r)vllm.model_executor.layers.pooler.tokwiser vllm.model_executor.models.llamar vllm.tasksrvllm.tokenizersrvllm.v1.pool.metadatarinterfaces_baserrtrQrr|rr.r!rrs  ////////######JIIIIIMLLLLL======""""""888888111111111111 X  PPPPP*PPPf     >   $v...+++++ ++/.+++r!