.`i&XUddlZddlZddlmZmZddlmZmZddlm Z ddl m Z m Z mZmZmZddlZddlmZddlmZddlmZmZmZddlmZdd lmZmZdd lm Z m!Z!m"Z"dd l#m$Z$m%Z%dd l&m'Z'd dl(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.e rddl/m0Z0m1Z1d dl2m3Z3ee4Z5GddZ6GddZ7e6e7ze-ze,zee8e.fzZ9ee:d<ede9Z;GddZ<eddZ=eddZ>Gdd ee e=e>fZ?e@e,ed!fdzZAee:d"<e@e,dzed!fZBee:d#<Gd$d%e?eAeBfZCGd&d'eCZDGd(d)eCZEGd*d+eCZFGd,d-e?e,dze,fZGGd.d/eGZHGd0d1eGZIdS)2N)ABCabstractmethod)MappingSequence)Lock) TYPE_CHECKINGGeneric TypeAliasTypeVarcast)override) MsgpackSerdeSingleWriterShmObjectStorageSingleWriterShmRingBuffer) init_logger) CacheInfoLRUCache)json_count_leavesjson_map_leavesjson_reduce_leaves) GiB_bytes MiB_bytes) format_gib)MultiModalBatchedFieldMultiModalFeatureSpecMultiModalFieldElemMultiModalKwargsItemMultiModalKwargsItems NestedTensors) ModelConfig VllmConfig)ResolvedPromptUpdatec<eZdZdZdededddffd ZxZS)MultiModalProcessorCacheItemz The data to store inside `MultiModalProcessorOnlyCache`. Args: item: The processed tensor data corresponding to a multi-modal item. prompt_updates: The prompt updates corresponding to `item`. itemprompt_updatesr#returnNcdt||_||_dSN)super__init__r&r'selfr&r' __class__s i/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/multimodal/cache.pyr,z%MultiModalProcessorCacheItem.__init__3s0  ,__name__ __module__ __qualname____doc__rrr, __classcell__r/s@r0r%r%*si-"-!!78-  ----------r1r%c<eZdZdZdededddffd ZxZS)$MultiModalProcessorCacheItemMetadataaI The metadata to store inside `MultiModalProcessorSenderCache`. Args: item: The processed tensor data corresponding to a multi-modal item. Since P1 already stores the tensor data, we only store its size metadata in P0 to reduce memory usage. The size metadata is still needed to keep the same cache eviction policy as P0. prompt_updates: The prompt updates corresponding to `item`. This needs to stay on P0 because for some models, they are dependent on the processed tensor data (cached on P1). r&r'r#r(Nctt||_||_dSr*)r+r,MultiModalCache get_item_size item_sizer'r-s r0r,z-MultiModalProcessorCacheItemMetadata.__init__Ls> (66t<<,r1r2r8s@r0r:r:>si  -"-!!78-  ----------r1r:MultiModalCacheValue_V)boundc eZdZededefdZedddededefdZ ededefd Z eddd e d e e dedeee ffd Zd S)r<leafr(czt|tr||jSt|tr|jSt|t ttfr| |j St|tj r|j Stj|Sr*) isinstancer% get_leaf_sizer&r:r>rrrr=datatorchTensornbytessys getsizeof)clsrCs r0rFzMultiModalCache.get_leaf_sizecs d8 9 9 0$$TY// / d@ A A "> !   "$8:M N   0$$TY// / dEL ) ) ; }T"""r1FdebugvaluerOcttjt|j|}|rFt |}t dt|t|||S)Nz.Calculated size of %s to be %s GiB (%d leaves)) roperatoraddrrFrloggerrOtyper)rMrPrOsize leaf_counts r0r=zMultiModalCache.get_item_sizewsu" L/#*;UCC    *511J LL@U 4       r1c t|S)a Get the number of leaf elements in a multi-modal cache value. This provides a measure of structural complexity that can be useful for debugging cache performance and understanding data patterns. Args: value: The multi-modal cache value to analyze. Returns: The number of leaf elements in the nested structure. )r)rMrPs r0get_item_complexityz#MultiModalCache.get_item_complexitys!'''r1 capacity_gb value_typec@tt|zfdS)Nc2|S)NrN)r=)xrMrOs r0z/MultiModalCache.get_lru_cache..s 1 1!5 1 A Ar1)rL)rr)rMrZr[rOs` `r0 get_lru_cachezMultiModalCache.get_lru_caches4  #AAAAA    r1N)r3r4r5 classmethodobjectintrFr?boolr=rYfloatrUr@rstrr`r1r0r<r<bs##C###[#&  #  [* ((< ( ( ( ([ (       H    #r'     [    r1r<_IT) contravariant_O) covariantceZdZdZedededefdZde ede ede efdZ ed d Z d S) BaseMultiModalCacheav Abstract base class to read/write multi-modal items from cache. The idea of multi-modal caching is based on having a client and server where the client executes in the frontend process (=P0) and the server in the core process (=P1). The data flow is as follows: ``` is_cached() x N get_and_update() P0: From API -----------------> -----------------> To P1 get_and_update() P1: From P0 -----------------> To model ``` `is_cached()` can be called any number of times in P0. However, `get_and_update()` must be called in P0 and P1 one after another so that their cache eviction order remains the same. This ensures that the keys in P0 and P1 caches are mirrored, allowing us to determine whether a key is cached in P1 by looking up the P0 cache, without having to communicate with P1. mm_itemmm_hashr(ct)ag Possibly update a multi-modal item based on whether it is in the underlying cache. This update is done out-of-place and updates the cache eviction order. Args: mm_item: The multi-modal item to update. mm_hash: The hash of `mm_item`. Returns: The update multi-modal item. NotImplementedError)r.rnros r0get_and_update_itemz'BaseMultiModalCache.get_and_update_items &"!r1mm_items mm_hashesct|t|ksJfdt||DS)a Possibly update a sequence of multi-modal items based on whether they are in the underlying cache. This update is done out-of-place and updates the cache eviction order. Args: mm_items: The multi-modal items to update. mm_hashes: The hash of each item in `mm_items`. Returns: A new list of updated multi-modal items. cBg|]\}}||Srg)rs).0rnror.s r0 z6BaseMultiModalCache.get_and_update..s=      $ $Wg 6 6   r1)lenzip)r.rtrus` r0get_and_updatez"BaseMultiModalCache.get_and_updatesW$8}}I....    $')$<$<    r1Nct)zClear the underlying cache.rqr.s r0 clear_cachezBaseMultiModalCache.clear_caches "!r1r(N) r3r4r5r6rrhrfrjrsrlistr|rrgr1r0rmrms0""" """^"( 2, 9  b     2"""^"""r1rmr#MultiModalProcessorCacheInItemMultiModalProcessorCacheOutItemceZdZdZededefdZdeedeefdZ ddZ ededdfd Z ed d d ede fd Z dS)BaseMultiModalProcessorCachez(The required interface for caches on P0.ror(ct)a$ Check whether a multi-modal item is in the underlying cache. This **DOES NOT** update the cache eviction order. Args: mm_hash: The hash of the item to check. Returns: `True` if the item is cached, otherwise `False`. rqr.ros r0is_cached_itemz+BaseMultiModalProcessorCache.is_cached_item s "!r1ruc fd|DS)aD Check whether a sequence of multi-modal items are in the underlying cache. This **DOES NOT** update the cache eviction order. Args: mm_hashes: The hash of each item to check. Returns: For each item, `True` if the item is cached, otherwise `False`. c:g|]}|Srg)r)rxror.s r0ryz:BaseMultiModalProcessorCache.is_cached..(s'FFF##G,,FFFr1rg)r.rus` r0 is_cachedz&BaseMultiModalProcessorCache.is_cacheds GFFFIFFFFr1NcdS)z&Close the underlying cache, if needed.Nrgr~s r0closez"BaseMultiModalProcessorCache.close*s r1ct)z Update the cache eviction order for a multi-modal item. This is used to touch the item in the cache without changing its value. Args: mm_hash: The hash of the multi-modal item. rqrs r0touch_sender_cache_itemz4BaseMultiModalProcessorCache.touch_sender_cache_item.s "!r1Fdeltarct)z Get (and reset) the multi-modal cache stats. Returns: The current multi-modal caching stats. rqr.rs r0 make_statsz'BaseMultiModalProcessorCache.make_stats;s "!r1r)r3r4r5r6rrfrdrrrrrrrrgr1r0rrs32 "c "d " " "^ " G49 Gd G G G G     "s "t " " "^ "*/"""4"I"""^"""r1rceZdZdZdfd ZededefdZed e dede fd Z ededdfd Z edd Z ed ddedefdZxZS)MultiModalProcessorOnlyCachea< The cache which is used on P0 when IPC caching is disabled. How to update each item: - If the item is in the cache, replace the input with the cached item. - If the item is not in the cache, store that item (which includes tensor data and metadata) into the cache, and return the input. model_configr!r(Nct|}t|jt |_dSr*)r+r,get_multimodal_configr<r`mm_processor_cache_gbr%_cacher.r mm_configr/s r0r,z%MultiModalProcessorOnlyCache.__init__QsN  6688 %33  + (   r1roc||jvSr*rrs r0rz+MultiModalProcessorOnlyCache.is_cached_item[$+%%r1rnc|j|x}|j|jfS| Jd|t ||j|<|SN#Expected a cached item for mm_hash=)rgetr&r'r%r.rnro cached_items r0rsz0MultiModalProcessorOnlyCache.get_and_update_item_sa  ;??733 3K @#[%?? ?""$L'$L$L""";WE Gr1c:|j|dSr*rtouchrs r0rz4MultiModalProcessorOnlyCache.touch_sender_cache_itemn '"""""r1c8|jdSr*rclearr~s r0rz(MultiModalProcessorOnlyCache.clear_cacher r1Frrc8|j|SNrrstatrs r0rz'MultiModalProcessorOnlyCache.make_statsv{e,,,r1rr!r(Nrr3r4r5r6r,r rfrdrrrrsrrrrr7r8s@r0rrFs:      &c&d&&&X& /   )   X #s#t###X#X*/---4-I---X-----r1rceZdZdZdfd ZededefdZed e dede fd Z ededdfd Z edd Z ed ddedefdZxZS)MultiModalProcessorSenderCachea The cache which is used on P0 when IPC caching is enabled. How to update each item: - If the item is already in the cache, clear the input to avoid unnecessary IPC. - If the item is not in the cache, store the metadata of that item so that the eviction policy remains the same as the cache on P1, and return the input. By only storing the metadata, we avoid keeping the data itself in memory inside P0. rr!r(Nct|}t|jt |_dSr*)r+r,rr<r`rr:rrs r0r,z'MultiModalProcessorSenderCache.__init__sN  6688 %33  + 0   r1roc||jvSr*rrs r0rz-MultiModalProcessorSenderCache.is_cached_itemrr1rnc|j|x} d|jfS| Jd|t||j|<|Sr)rrr'r:rs r0rsz2MultiModalProcessorSenderCache.get_and_update_items^  ;??733 3K @33 3""$L'$L$L"""CWM Gr1c:|j|dSr*rrs r0rz6MultiModalProcessorSenderCache.touch_sender_cache_itemrr1c8|jdSr*rr~s r0rz*MultiModalProcessorSenderCache.clear_cacherr1Frrc8|j|Srrrs r0rz)MultiModalProcessorSenderCache.make_statsrr1rrrr8s@r0rr{s:        &c&d&&&X& /   )   X #s#t###X#X*/---4-I---X-----r1rceZdZdZdfd Zddd edefd Zed e defd Z ed e d e de fdZ ed e ddfdZeddZeddd edefdZeddZddZdedede defdZxZS)ShmObjectStoreSenderCachea The cache which is used on P0 when IPC caching is enabled. How to update each item: - If the item is already in the cache, clear the input to avoid unnecessary IPC. - If the item is not in the cache, store the data in shared memory. vllm_configr"r(Nct|jj|_|j}t t|jtztj d}t|j tz|j|t|_i|_d|_d|_t)dd|_dS)NTdata_buffer_sizenamecreate)max_object_size n_readers ring_buffer serde_classrhitstotal)r+r,parallel_config world_sizerrrrcrrenvs#VLLM_OBJECT_STORAGE_SHM_BUFFER_NAMErmm_shm_cache_max_object_size_mbrr _shm_cache _p0_cache_hits_totalr _last_info)r.rrrr/s r0r,z"ShmObjectStoreSenderCache.__init__s %5@,BBDD / !@9!LMM9   7%E Qo#$    QS  #!444r1Frrcft|j|j}|r||jz }||_|}|S)Nr)rrrr)r.rinfo info_deltas r0_statzShmObjectStoreSenderCache._stats=dj <<<  /J"DOD r1roc6|j|Sr*)rrrs r0rz(ShmObjectStoreSenderCache.is_cached_items((111r1rnc |j|rf|xjdz c_|xjdz c_|j|\}}|j|\}}|||||fS| Jd||xjdz c_ |j||d\}}t|jdt|jj zkr| |d|dj f|j|<||||dj }||dfS#ttf$r(}td|||cYd}~Sd}~wwxYw)Nrrrz)Failed to cache mm_input with hash %s: %s)rrrr get_cachedraddress_as_itemputrz key_indexremove_dangling_itemsmodality ValueError MemoryErrorrTrO) r.rnroaddress monotonic_idr'r address_itemes r0rsz-ShmObjectStoreSenderCache.get_and_update_items ? $ $W - - Y JJ!OJJ KK1 KK$(O$>$>w$G$G !G\'+~g'> $NH''xHH.X X""$L'$L$L""" q  $(O$7$7$L$L !G\4>""a#do.G*H*H&HHH**,,,&-aj'!*2E&EDN7 #//wqz':L + +K(    LLDgq Q Q QNNNNNN  s!B2EF %FF F c:|j|dS)ziTouch the item in shared memory cache to prevent eviction. Increments writer_flag on sender side.N)rrrs r0rz1ShmObjectStoreSenderCache.touch_sender_cache_item s  g&&&&&r1c|j|jd|_d|_t dd|_dS)Nrr)rrrrrrrr~s r0rz%ShmObjectStoreSenderCache.clear_cachesP    #!444r1c.||Sr)rrs r0rz$ShmObjectStoreSenderCache.make_statsszzz&&&r1c8|jdSr*)rrr~s r0rzShmObjectStoreSenderCache.close r1c|jj}t|j|z }|D] }|j|= dS)z;Remove items that are no longer in the shared memory cache.N)rrkeyssetr)r. cached_hashesdangling_hashesros r0rz/ShmObjectStoreSenderCache.remove_dangling_items!s^16688 dn113344}D& ( (Gw'' ( (r1rrrct|d|t}t|d|t}tj||g}|S)Nr)rkeyrGfieldr)rrr from_elems)r.rrr addr_elemid_elemrns r0rz)ShmObjectStoreSenderCache.address_as_item(sn((**    &(**     '19g2FGGr1)rr"r(Nr)r3r4r5r6r,rdrrr rfrrrrsrrrrrrcrrr7r8s@r0rrs  5555550&+dy2c2d222X2 /   )   X D's't'''X' 555X5*/'''4'I'''X'   X ((((*-9< r1rcheZdZdZdeddedfdZe d dededzddfd Z dS) BaseMultiModalReceiverCachez(The required interface for caches on P1. mm_featuresrr(c|D]+}|jp|j}|||j,|D]0}|jp|j}||j||_1|S)a1 Update multimodal features with cached encoder outputs. Touch all identifier at first before update to avoid item in updated list evict during update. Uses mm_hash for cache key to share across LoRAs (falls back to identifier for backward compatibility). )ro identifiertouch_receiver_cache_itemrGrs)r.rfeature cache_keys r0get_and_update_featuresz3BaseMultiModalReceiverCache.get_and_update_features@s# D DG=7+=I  * *9gl C C C C" M MG=7+=I33GL)LLGLLr1Nrornct)ap Update the cache eviction order for a multi-modal item. This is used to touch the item in the cache without changing its value. Args: mm_hash: The hash of the multi-modal item. mm_item: The multi-modal item itself. This is optional and may not be needed by some cache implementations. rqr.rorns r0rz5BaseMultiModalReceiverCache.touch_receiver_cache_itemUs ""!r1r*) r3r4r5r6rrrrfrrrgr1r0rr;s3212 % &*04"""&,"  """^"""r1rceZdZdZd fd Zededzdedefd Ze d dededzddfd Z edd Z xZ S)MultiModalReceiverCachea. The cache which is used on P1 when IPC caching is enabled. How to update each item: - If the item is in the cache, replace the input with the cached item. - If the item is not in the cache, store that item (which includes tensor data) into the cache, and return the input. rr!r(Nct|}t|jt |_dSr*)r+r,rr<r`rrrrs r0r,z MultiModalReceiverCache.__init__tsN  6688 %33  +    r1rnroct|j|x}|S| Jd|||j|<|Sr)rrrs r0rsz+MultiModalReceiverCache.get_and_update_item~sQ  ;??733 3K @ ""$L'$L$L"""& Gr1c:|j|dSr*rrs r0rz1MultiModalReceiverCache.touch_receiver_cache_items '"""""r1c8|jdSr*rr~s r0rz#MultiModalReceiverCache.clear_cacherr1rr*r) r3r4r5r6r,r rrfrsrrr7r8s@r0rris       %,      X 04###&,#  ###X#Xr1rceZdZdZdddeddffd Zededzd edefd Z e d d ededzddfd Z edd Z xZ S)ShmObjectStoreReceiverCachez The cache which is used on P1 Worker Process when IPC caching is enabled. How to update each item: - If the item has an address, replace the input with the cached item. - If not, return the input. rr"shared_worker_lockr(Ncht|jj|_|j}t t|jtztj d}t|j tz|j|t||_dS)NFr)rrrr reader_lock)r+r,rrrrrrcrrrrrrrrr)r.rr rrr/s r0r,z$ShmObjectStoreReceiverCache.__init__s %5@,BBDD / !@9!LMM9   7%E Qo#$*    r1rnroc| Jd|d|vr[tt|dj}tt|dj}|j||S|S)Nz%Expected an address item for mm_hash=rr)r rcrGrr)r.rnrorrs r0rsz/ShmObjectStoreReceiverCache.get_and_update_itemsw ""$NG$N$N"""   3 2 788GW^%<%ABBL?&&w == =r1c|Jd|vr_tt|dj}tt|dj}|j|||dSdS)zlTouch the item in shared memory cache to prevent eviction. Increments reader_count on receiver side.Nrr)rr)r rcrGrr)r.rornrrs r0rz5ShmObjectStoreReceiverCache.touch_receiver_cache_itemst"""   3 2 788GW^%<%ABBL O ! !'7 ! V V V V V r1c8|jdSr*)rrr~s r0rz'ShmObjectStoreReceiverCache.clear_cacherr1r*r) r3r4r5r6LockTyper,r rrfrsrrr7r8s@r0r r s ! %         . %,      X 04 W W W&, W  W W WX W   X     r1r )JrRrKabcrrcollections.abcrrmultiprocessing.synchronizerrtypingrr r r r rHtyping_extensionsr vllm.envsr8vllm.distributed.device_communicators.shm_object_storagerrr vllm.loggerrvllm.utils.cacherrvllm.utils.jsontreerrrvllm.utils.mem_constantsrrvllm.utils.mem_utilsrinputsrrrrrr vllm.configr!r"processing.processorr#r3rTr%r:rfr?__annotations__r@r<rhrjrmtuplerrrrrrrrr rgr1r0r%s ########--------888888CCCCCCCCCCCCCC &&&&&& $#####00000000VVVVVVVVVV99999999++++++;33333333:::::: X  --------(--------4!*+c= ! "i WT-...F F F F F F F F R WT&&& WTT"""J"J"J"J"J"#wr2vJ"J"J"\ )? @ @ADH  .34*@!AA. ="="="="="68WWX="="="@2-2-2-2-2-#?2-2-2-j7-7-7-7-7-%A7-7-7-tCCCCC <CCCL+"+"+"+"+",t35IIJ+"+"+"\-----9---`? ? ? ? ? "=? ? ? ? ? r1