.`i"dZddlmZmZddlmZddlmZddlm Z GddeZ eGdd Z eGd d Z Gd d eZ dS)a: OffloadingManager class for managing KV data offloading in vLLM v1 This class runs in the scheduler, tracks which blocks are offloaded and their address. The class provides the following primitives: lookup() - find the length of the maximal series of blocks, starting from the first one, that are all offloaded. prepare_load() - prepare given blocks to be read. The given blocks will be protected from eviction. This function returns a LoadSpec which encapsulates information required for performing the load. touch() - marks the give blocks as recently used. Can be used to track block's LRU. This function is separated from the prepare_load function to allow setting block recency even for blocks which do not need reading from the cache, such as blocks that are cached by the GPU prefix cache. complete_load() - mark blocks which were previously prepared to be loaded as done loading. This is to re-allow their eviction. prepare_store() - prepare the given blocks to be written. Returns a StoreSpec encapsulating offloading information, as well as a list of blocks that were evicted as a result. complete_store() - marks a previous store as completed. Following this call, the given blocks will become loadable. )ABCabstractmethod)Iterable) dataclass) BlockHashc>eZdZdZeedefdZdS) LoadStoreSpecz Abstract metadata that encapsulates information allowing a worker to load, and optionally also to store, blocks of KV data. returncdS)ze Returns a string representation of the medium type this store/load targets. Nr o/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/v1/kv_offload/abstract.pymediumzLoadStoreSpec.medium+s  r N)__name__ __module__ __qualname____doc__ staticmethodrstrrr r rr r %sO  C   ^\   r r cFeZdZUeeed<eed<eeed<dS)PrepareStoreOutputblock_hashes_to_store store_specblock_hashes_evictedN)rrrlistr__annotations__r r r rrr5s> ?***y/)))))r rcDeZdZUeeed<eed<eed<eed<dS)OffloadingEvent block_hashes block_sizerremovedN) rrrrrrintrboolr r rrr<s;y/!!!OOO KKK MMMMMr rceZdZedeededzfdZedeedefdZ deefdZ deefdZ edeede dzfdZ d deed efd Zdeefd ZdS)OffloadingManagerrr NcdS)a Finds the length of the maximal series of blocks, starting from the first one, that are all offloaded. Args: block_hashes: the hashes identifying the blocks to lookup. Returns: An integer representing the maximal number of blocks that are currently offloaded, or None if the lookup should be retried later. Returning None will delay the request handling by the vLLM scheduler. Nr selfrs rlookupzOffloadingManager.lookupF  r cdS)a Prepare the given blocks to be read. The given blocks will be protected from eviction until complete_load is called. It assumes all given blocks are offloaded. Args: block_hashes: the hashes identifying the blocks. Returns: A LoadStoreSpec that can be used by a worker to locate and load the actual offloaded KV data. Nr r's r prepare_loadzOffloadingManager.prepare_loadWr*r cdS)z Mark the given blocks as recently used. This could in practice mean moving them to the end of an LRU list. Args: block_hashes: the hashes identifying the blocks. Nr r's rtouchzOffloadingManager.touchhs  r cdS)z Marks previous blocks that were prepared to load as done loading. Args: block_hashes: the hashes identifying the blocks. Nr r's r complete_loadzOffloadingManager.complete_loadrs  r cdS)a Prepare the given blocks to be offloaded. The given blocks will be protected from eviction until complete_store is called. Args: block_hashes: the hashes identifying the blocks. Returns: A PrepareStoreOutput indicating which blocks need storing, where to store them (LoadStoreSpec), and list of blocks that were evicted as a result. None is returned if the blocks cannot be stored. Nr r's r prepare_storezOffloadingManager.prepare_store{s $ r TsuccesscdS)a| Marks blocks which were previously prepared to be stored, as stored. Following this call, the blocks become loadable. If if_success is False, blocks that were not marked as stored will be removed. Args: block_hashes: the hashes identifying the blocks. success: whether the blocks were stored successfully. Nr )r(rr3s rcomplete_storez OffloadingManager.complete_stores  r cdS)z Take the offloading events from the manager. Yields: New OffloadingEvents collected since the last call. r r )r(s r take_eventszOffloadingManager.take_eventss rr )T)rrrrrrr"r)r r,r.r0rr2r#r5rr7r r rr%r%EsB 8I#6 3:   ^  ))<    ^ (9"5(9*= $Y/ d "   ^ &  8I+>     Xo6r r%N)rabcrrcollections.abcr dataclassesrvllm.v1.core.kv_cache_utilsrr rrr%r r rr<s/6$#######$$$$$$!!!!!!111111      C     ******* *   ^^^^^^^^^^r