.`i{9 ddlZddlmZddlmZmZddlZddlmZddl m Z ddl m Z m Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZmZdd lmZmZddlmZddl m!Z!m"Z"ee#Z$GddZ%eGddZ&dede'ddfdZ(eddej)dej)dzdej)fdZ*eddej)dej)dzdej)fdZ+ded e de'fd!Z, d/d"e-e.e.fd#e/e!d$e0e.dzddfd%Z1 d0d'e-e.ej)fd(e-e.e fd)e/ej)d*e'ddf d+Z2d,e d-e'de3fd.Z4dS)1N) defaultdict) dataclassfield) deprecated) Attention) CacheConfig VllmConfig) init_logger)MultiModalEmbeddings)extract_layer_index)MultiModalRegistry)current_platform)MemorySnapshot format_gib)AttentionBackendAttentionMetadataBuilder)compute_mm_encoder_budget)KVCacheGroupSpec KVCacheSpeccteZdZdZdededdffd ZdefdZde fdZ d ed e de e e ffd Z d d Z xZS)MultiModalBudgetzDHelper class to calculate budget information for multi-modal models. vllm_config mm_registryreturnNct|jx|_}|jx|_}||_||x|_}|j|_|j|_ | |||_ | |||j }t||\}}||_||_t!t"t$f} t!t"t$f} |D](\} } || | \} }| | | <|| | <)||_| |_| |_dS)N)cache)rprofiler_limits)super__init__ model_configscheduler_configr processor_only_cache_from_configr max_model_len max_num_seqs max_num_reqsget_mm_limits_per_prompt mm_limits#get_max_tokens_per_item_by_modalityrencoder_compute_budgetencoder_cache_sizedictstrintitems get_max_itemsmax_tokens_by_modality max_items_per_prompt_by_modalitymax_items_per_batch_by_modality)selfrrr r!rr0r)r*r1r2modality max_tokensmax_items_per_promptmax_items_per_batch __class__s h/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/v1/worker/utils.pyrzMultiModalBudget.__init__s +6+CCL3>3OO 0&(II+VVV U)7,9$==lRW=XX!,!P!P  N"Q" "  6O  "6 6 2 2 '=#"4+/S>+;+;(*.sCx.*:*:'$:$@$@$B$B L L Hj""8Z88 $#:N ,X 68K +H 5 5&<#0P-/N,,,cb|j}t|d\}}|S)Nc|dS)N)xs r9z?MultiModalBudget.get_modality_with_max_tokens..Ms !r:)key)r0maxr.)r3r0r4_s r9get_modality_with_max_tokensz-MultiModalBudget.get_modality_with_max_tokensKs5!%!<06688nnMMM !r:c6t|j|jSN)minr)r*r3s r9get_encoder_budgetz#MultiModalBudget.get_encoder_budgetQs4.0GHHHr:r4max_tokens_per_itemcb|dkrdS|}|dkrdS||z}|j|}tdt||j|z}|j}|j}|jst||j|z}||z} tdt|| } || fS)Nr)rrr=) rIr'rBrGr#r!r%enable_chunked_prefillmax_num_batched_tokens) r3r4rJencoder_budgetmax_encoder_items_per_batchmm_limitr6r!r%max_decoder_items_per_batchr7s r9r/zMultiModalBudget.get_max_itemsTs ! # #40022 Q  4&48K&K#>(+" $,0CC D D    0( 6  7;NNL '35I&I#! +-H I I   $%888r:cJ|j|jdSdSrF)r clear_cacherHs r9 reset_cachezMultiModalBudget.reset_caches, : ! J " " $ $ $ $ $ " !r:)rN)__name__ __module__ __qualname____doc__r r rr,rDr-rItupler/rT __classcell__)r8s@r9rrsNN-O-O(-O  -O-O-O-O-O-O^c ICIIII+9+9!+9 sCx +9+9+9+9Z%%%%%%%%r:rceZdZUeeed<eeed<eed<e ed<e dZ ee ed< dd e d zd e fd Z dde de fdZd S)AttentionGroupbackend layer_names kv_cache_speckv_cache_group_idcgSrFr>r>r:r9r@zAttentionGroup.sr:)default_factorymetadata_buildersr=kernel_block_sizeNnum_metadata_buildersc|j|njfdt|D_dS)Ncdg|],}jj-Sr>)r]get_builder_clsr^).0rCdevicekv_cache_spec_builderr3rs r9 z;AttentionGroup.create_metadata_builders..sT" " "  +DL ( ( * *%    " " " r:)r_copy_with_new_block_sizerangerc)r3rrjrdrerks``` @r9create_metadata_buildersz'AttentionGroup.create_metadata_builderss~!,   7 78I J J J#  " " " " " " " 011" " " r:r ubatch_idrcPt|j|ksJ|j|SrF)lenrc)r3rps r9get_metadata_builderz#AttentionGroup.get_metadata_builders,4)**Y6666%i00r:r=)r)rUrVrWtyper__annotations__listr,rr-rrcrrorsr>r:r9r\r\s " ####c9>" 999t45&'   :  #     ,11c1:R111111r:r\ mm_embeddingsexpected_num_itemsrcNt|tttjfsJdt |dt ||ksJd|dt |dtd|DsJdd|DddS) z{ Perform sanity checks for the result of [`vllm.model_executor.models.SupportsMultiModal.embed_multimodal`][]. z`Expected multimodal embeddings to be a list/tuple of 2D tensors, or a single 3D tensor, but got zg instead. This is most likely due to incorrect implementation of the model's `embed_multimodal` method.zIExpected number of multimodal embeddings to match number of input items: z, but got len(mm_embeddings)=c3,K|]}|jdkVdS)N)ndimries r9 z2sanity_check_mm_encoder_outputs..s(22qqv{222222r:z[Expected multimodal embeddings to be a sequence of 2D tensors, but got tensors with shapes cg|] }|j Sr>)shaper~s r9rlz3sanity_check_mm_encoder_outputs..s'G'G'GA'G'G'Gr:N) isinstancerwrYtorchTensorrurrall)rxrys r9sanity_check_mm_encoder_outputsrs mdE5<%@ A A 4*.}*=*= 4 4 4 A }  !3 3 3 3 4* 4 469-6H6H 4 4 4 4 3 3 22M222 2 2 4'G'G'G'G'G 4 4 4 2r:zG`scatter_mm_placeholders` is deprecated and will be removed in v0.15.0.embedsis_embedc||S||jd|jdftj}|||<|S)a Scatter the multimodal embeddings into a contiguous tensor that represents the placeholder tokens. [`vllm.multimodal.processing.PromptUpdateDetails.is_embed`][]. Args: embeds: The multimodal embeddings. Shape: `(num_embeds, embed_dim)` is_embed: A boolean mask indicating which positions in the placeholder tokens need to be filled with multimodal embeddings. Shape: `(num_placeholders, num_embeds)` Nr) fill_value)new_fullrrnan)rr placeholderss r9scatter_mm_placeholdersrsT$ ??  FL,-9#L$L r:zF`gather_mm_placeholders` is deprecated and will be removed in v0.15.0.rc||S||S)z Reconstructs the embeddings from the placeholder tokens. This is the operation of [`scatter_mm_placeholders`] [vllm.v1.worker.utils.scatter_mm_placeholders]. r>)rrs r9gather_mm_placeholdersrs  !!r: init_snapshot cache_configctj|j|jz}|j|krZt d|jdt|jdt|jd|jdt|d |S)z Calculate the amount of memory required by vLLM, then validate that the current amount of free memory is sufficient for that. zFree memory on device z (/z> GiB) on startup is less than desired GPU memory utilization (z, zT GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes.)mathceil total_memorygpu_memory_utilization free_memory ValueErrordevice_r)rrrequested_memorys r9request_memoryrs y"\%HH #333 I]%: I I=455 I I-455 I I3  I I *++  I I I    r:shared_kv_cache_layerskv_cache_groupsrunner_only_attn_layersci}|D]}|jD]}|||<|D]>\}}||}|j||||?dS)a Sets up KV cache sharing by reusing the allocated KV caches in `kv_caches` for layers that do not allocate its own KV cache, based on the mapping in `shared_kv_cache_layers`. Adds these layers to the corresponding KV cache group, which is needed to ensure that attention metadata is assigned later. Args: shared_kv_cache_layers: Layer pairings for cross-layer KV sharing. If an Attention layer `layer_name` is in the keys of this dict, it means this layer will perform attention using the keys and values from the KV cache of `shared_kv_cache_layers[layer_name]`. kv_cache_groups: The KV cache groups of the model. N)r^r.appendadd)rrrlayer_to_kv_cache_groupkv_cache_group layer_nametarget_layer_nametgt_kv_cache_groups r9(add_kv_sharing_layers_to_kv_cache_groupsrs$<>)AA(4 A AJ2@ #J / / A*@)E)E)G)G44% %45FG&--j999 " . # ' ' 3 3 3 44r:r= kv_cachesforward_contextrunner_kv_cachesnum_attn_modulecDt|dksJtt}|D]+}|t|||,t |D]~}||}t|dkrAtjs&tj stj rnt|D]}|||| D]\}}|g||_ dS)a{ Bind the allocated KV cache to both ModelRunner and forward context so that the KV cache can be used in the forward pass. This function: 1) Fills the ModelRunner's kv cache list (`runner_kv_caches`) with kv_caches. 2) Associates each attention layer in the `forward_context` with its corresponding KV cache in kv_caches. Args: kv_caches: The allocated kv_caches with layer names as keys. forward_context: The global forward context containing all Attention layers with layer names as keys. runner_kv_caches: The kv_cache declared by ModelRunner. rr=N)rrrrwr rsortedkeysr is_cuda_alikeis_xpuis_cpuNotImplementedErrorr.kv_cache) rrrr index2namer layer_indexr^rs r9 bind_kv_cacher0sQ.  A % % % %T""JXX &z?CCDKKJWWWWjoo//00;;  - {  a  !.00 *#*,, *$*,, *))% ; ;J  # #Ij$9 : : : : ;!* 1 1:: H08z #,,::r:rnum_input_tokensc|jjjsdS|jj}|dkrdS||zdksJ|jjr |jjrdS|jj}|dS||vS)aCheck if the residual tensor is scattered for sequence parallelism. The residual tensor is scattered across tensor parallel ranks when sequence parallelism and tensor parallelism is enabled. This follows the same logic as SequenceParallelismPass.is_applicable_for_range(): - In full-graph compilation mode (no splitting ops or using inductor graph partition), SP is always applied - Otherwise, SP is only applied for specific shapes in compile_sizes Fr=rT)compilation_config pass_config enable_spparallel_configtensor_parallel_size splitting_opsuse_inductor_graph_partition compile_sizes)rrtprs r9is_residual_scattered_for_sprls  ) 5 ?u  $ 9B Qwwu b A % % % %  * 8  ) Ft2@Mu } ,,r:rFrt)5r collectionsr dataclassesrrrtyping_extensionsrvllm.attention.layerr vllm.configrr vllm.loggerr %vllm.model_executor.models.interfacesr vllm.model_executor.models.utilsr vllm.multimodal.registryr vllm.platformsrvllm.utils.mem_utilsrrvllm.v1.attention.backendrr"vllm.v1.core.encoder_cache_managerrvllm.v1.kv_cache_interfacerrrUloggerrr\r-rrrrrr+r,rwsetrrboolrr>r:r9rs ######(((((((( ((((((******////////######FFFFFF@@@@@@777777++++++;;;;;;;;PPPPPPPPHHHHHHDDDDDDDD X  j%j%j%j%j%j%j%j%Z $1$1$1$1$1$1$1 $1N' <  UVV LlT! \WV8  TUU ", "lT! " \ " " "VU " . PS40444 cN4*+4!X_4 4444F 9:9:C%&9:#y.)9:5<(9: 9:  9:9:9:9:x!-!-/2!- !-!-!-!-!-!-r: