`i.ddlmZmZddlmZddlmZddlmZddl m Z GddeZ Gd d e Z Gd d e Z d S))ABCabstractmethod)deque)ceil)Optional)loggerc DeZdZUdZeed<eeeefed<e dedede ede efdZ dede edd fd Z e ded ed edeefd Ze ded ed edeefdZe ded ed edeeeffdZd S)CacheAllocatorzAbstract base class for cache managers. Cache managers keep track of per-request cache allocations, determine when a new physical block needs to be allocated and compute physical indices for reading or writing to the cache._index _block_tablen_blocks request_id free_blocksreturncdS)zxAllocates n_blocks for a given request_id. Returns the num of blocks allocated if successful and None otherwise.Nselfrrrs /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/generation/continuous_batching/cache_manager.pyallocate_blockszCacheAllocator.allocate_blockss  Nc||jvr1|j|}||dStjd|jd|dS)z.Frees all blocks associated with a request_id.zCacheAllocator z7 attempted to free blocks for non-existent request_id: N)r popextendr warningr )rrrblocks_to_frees rrzCacheAllocator.free_blocks$st * * *!.22:>>N   ~ . . . . . Nr$+rrfprr     r past_length query_lengthcdS)zUReturns the physical indices of where to read request_id's cache in the cache tensor.Nrrrrrs rget_read_indiceszCacheAllocator.get_read_indices.  rcdS)zVReturns the physical indices of where to write request_id's cache in the cache tensor.Nrr!s rget_write_indicesz CacheAllocator.get_write_indices3r#rcdS)gReturns the attention type of the cache allocator and the key sequence length for the given request_id.Nrr!s r get_seqlens_kzCacheAllocator.get_seqlens_k8r#r)__name__ __module__ __qualname____doc__int__annotations__dictstrlistrrrrrr"r%tupler(rrrr r syy KKKsDI~&&&&   5QT: ZbcfZg   ^ cc t 3 S PS X\]`Xa   ^  C c QT Y]^aYb   ^   # S UZ[^`c[cUd   ^   rr c eZdZdZdededdfdZdeded eedeefd Z ded ed ede efd Z ded ed ede efdZ ded ed ede eeffdZdS)FullAttentionCacheAllocatorz3Cache manager for a group of full attention layers.index block_sizerNc0||_||_i|_dS)zInitializes the cache manager for a group of full attention layers. Args: - index: the index of the associated layer group - block_size: the size of the blocks in the cache N)r r6r )rr5r6s r__init__z$FullAttentionCacheAllocator.__init__As  $rrrrct|krdS||jvr g|j|<|j|fdt|D|S)zAllocate blocks for a given request_id. Returns the number of blocks allocated if successful and None otherwise. For group of full attention layers, we always allocate the number of requested blocks.Nc3@K|]}VdSNpopleft.0_rs r z>FullAttentionCacheAllocator.allocate_blocks..Rs/,\,\q[-@-@-B-B,\,\,\,\,\,\r)lenr rrangers `rrz+FullAttentionCacheAllocator.allocate_blocksKsu {  h & &4 T. . .,.D j ) *%,,,\,\,\,\ERZOO,\,\,\\\\rrrc|j|}|td|g}t||zD]>}||jz}||jz}|||jz|z} || ?|S)zReturns the physical indices of where to read request_id's cache. For a group of full attention layers, we first write the new cache to the cache tensor and then read the entire cache from the beginning to the end.N!No block table found for request r get ValueErrorrCr6append rrrr block_tablephysical_indicesi block_idx block_offsetphysical_indexs rr"z,FullAttentionCacheAllocator.get_read_indicesUs'++J77  MMMNN N{\122 4 4AT_,It.L(3doE TN  # #N 3 3 3 3rc |j|}|td|g}t|||zD]>}||jz}||jz}|||jz|z} || ?|S)zReturns the physical indices for writing to the cache. For a group of full attention layers, we write the new cache as a continuation of the existing cache for the same request.NrErFrJs rr%z-FullAttentionCacheAllocator.get_write_indiceses'++J77  MMMNN N{K,$>?? 4 4AT_,It.L(3doE TN  # #N 3 3 3 3rc||z}d|fS)r'full_attentionrrrrr seqlens_ks rr(z)FullAttentionCacheAllocator.get_seqlens_kts,. **rr)r*r+r,r-r8r0rrrr1r"r%r2r(rrrr4r4>s,==cst5QT:ZbcfZg 3 S PS X\]`Xa     C  c  QT  Y]^aYb     ++#+S+UZ[^`c[cUd++++++rr4c eZdZdZdedededdfdZded ed eedeefd Z d ed ed ede efdZ d ed ed ede efdZ d ed ed ede eeffdZdS)SlidingAttentionCacheAllocatorz2Cache manager for sliding window attention layers.r5r6sliding_windowrNc||_||_||_t|j|jz |_i|_dS)aInitializes the cache manager for a group of sliding window attention layers. Args: - index: the index of the associated layer group - block_size: the size of the blocks in the cache - sliding_window: the size of the sliding window N)r r6rYr_max_blocks_per_requestr )rr5r6rYs rr8z'SlidingAttentionCacheAllocator.__init__}sB $,'+D,?$/,Q'R'R$rrrrcR||jvr g|j|<t|j|}||jkrdSt||z|j}||z }t|krdS|j|fdt |D|S)aAllocate blocks for a given request_id. Returns the number of blocks allocated if successful and None otherwise. For group of sliding window attention layers, we only allocate up to the point where we can fit an entire sliding window in the cache tensor.rNc3@K|]}VdSr;r<r>s rrAzASlidingAttentionCacheAllocator.allocate_blocks..s/,c,cq[-@-@-B-B,c,c,c,c,c,cr)r rBr[minrrC)rrrralready_allocatedafter_allocationactual_n_blockss ` rrz.SlidingAttentionCacheAllocator.allocate_blockss T. . .,.D j ) 1* =>>  < < <108;T=YZZ*->> {  o - -4 *%,,,c,c,c,cERaLbLb,c,c,ccccrrrc|j|}|td|||jkrdn ||jz}t ||jdz }g}t |||zD]H}||jz}||jz} ||jz} || |jz| z} || I|dg|zzS)aReturns the physical indices of where to read request_id's cache in the cache tensor. For a group of sliding window attention layers, we read from the cache tensor before writing on it, because the new cache can overwrite the old one. To form the cache + new key / values states, we read the at most sliding_window - 1 cache page and then manually add the new key / values states after. Hence the -1 indices which indicate where to store the new key or values indices.NrErrr rGrHrYr^rCr6rI) rrrrrK start_index cache_lengthrLrMrNrOrPs rr"z/SlidingAttentionCacheAllocator.get_read_indicess'++J77  MMMNN N&)<<?? 4 4A $ $AT_,It.L(3doE TN  # #N 3 3 3 32$"555rc|j|}|td|||jz}t ||j}||z }g}t |||zD]H} | |jz} | |jz} | |jz} || |jz| z} || I|dkr dg|z|z}|S)aBReturns the physical indices of where to write request_id's cache in the cache tensor. For a group of sliding window attention layers, we write the new cache in rolling-buffer kind of way: if we reach the end of the allocated physical cache, we start writing from the beginning of the physical cache again.NrErrcrd) rrrrrKrerfpadding_lengthrLrMrNrOrPs rr%z0SlidingAttentionCacheAllocator.get_write_indicess '++J77  MMMNN N!D$77 <)<== % 4{K,$>?? 4 4A $ $AT_,It.L(3doE TN  # #N 3 3 3 3 A   "tn47GG rc@|t||jdz z}d|fS)r'rsliding_attention)r^rYrTs rr(z,SlidingAttentionCacheAllocator.get_seqlens_ks) 3{D4G!4K#L#LL "I--rrVrrrrXrXzs3<< c s C D    5QT:ZbcfZg&636S6PS6X\]`Xa6666. C c QT Y]^aYb    0..#.S.UZ[^`c[cUd......rrXN)abcrr collectionsrmathrtypingrrequestsr r r4rXrrrrps$#######$ $ $ $ $ S$ $ $ N9+9+9+9+9+.9+9+9+xU.U.U.U.U.^U.U.U.U.U.r