-`i-0ddlZddlZddlZddlmZddlmZddlmZddl Z ddl m Z ddl m Z ddlmZe eZdZ dd lmZmZmZdd lmZed ZeZd Zn#e$rdZdZdZdZdZdZYnwxYweeeeefZejGd dZ deddfdZ!deddfdZ"deegefdeeegdfde j#j$j%fdZ&edeegefdeeegdfddfdZ'GddZ(dS)N)Callable)contextmanager)Any) init_logger)is_pin_memory_available)find_loaded_libraryF) init_modulepython_create_and_mappython_unmap_and_release) CudaRTLibrarycumem_allocatorTcBeZdZUeed<eed<dZejdzed<dS)AllocationDatahandletagNcpu_backup_tensor) __name__ __module__ __qualname__ HandleType__annotations__strrtorchTensoro/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/device_allocator/cumem.pyrr4s?  HHH-1u|d*11111rrallocation_handlereturnct|dSN)r rs rcreate_and_mapr#;s,----rct|dSr!)r r"s runmap_and_releaser%?s/0000rpython_malloc_fnpython_free_funcc|t||tjjt dd}|S)N my_mallocmy_free)r rcudamemoryCUDAPluggableAllocatorlib_name)r&r' new_allocs rget_pluggable_allocatorr0Cs? "2333 !88+yI rc#Kt||}tjj|j}tjj|5||fVddddS#1swxYwYdSr!)r0rr+r,MemPool _allocator use_mem_pool)r&r'r/mem_pools ruse_memory_pool_with_allocatorr6Ms((8:JKKIz (()=>>H   ' ' 1 1"" !!!!""""""""""""""""""s A44A8;A8ceZdZUdZdZded<dZeed<eddZ dZ d e ddfd Z d e de fd Zdd eedfezdzddfdZddeedzddfdZeddedzfdZde fdZdS)CuMemAllocatora' A singleton class that manages a memory pool for CUDA tensors. The memory in this pool can be offloaded or discarded when the allocator sleeps. Inside the `use_memory_pool(tag)` context, all tensors created will be allocated in the memory pool, and has the same tag as the tag passed to the context. When we call `sleep`, all tensors with the specified tag will be offloaded to CPU memory, and the rest of the tensors will be discarded. When we call `wake_up`, all tensors that are previously offloaded will be loaded back to GPU memory, and the rest of the tensors will have empty memory. Why it needs to be a singleton? When allocated tensors are garbage collected, PyTorch will call the free callback, which will call the `python_free_callback` method. The C-extension uses a global variable to store the function of an instance of this class. If we create multiple instances of this class, the global variable will be overwritten and the free callback will not work as expected. Ninstancedefault default_tagrcts Jdtjtt_tjS)z CuMemAllocator is a singleton class. We cannot call the constructor directly. Call this method to get the instance. z cumem allocator is not available)cumem_availabler8r9rrr get_instancezCuMemAllocator.get_instancess8BB BBB  " *&4&6&6N #&&rctjdd}d|vs Jdi|_tj|_i|_|j|_ |j |_ dS)NPYTORCH_CUDA_ALLOC_CONFzexpandable_segments:TruezExpandable segments are not compatible with memory pool. Please track https://github.com/pytorch/pytorch/issues/147851 for the latest updates.) osenvirongetpointer_to_datar8r; current_tagallocator_and_pools_python_malloc_callbackpython_malloc_callback_python_free_callbackpython_free_callback)selfconfs r__init__zCuMemAllocator.__init__sqz~~7<<)555 &655 ;= . :35 '+&B#$($>!!!rrc|d}t||j|j|<td|d|j|dS)zj Internal method to store the allocation data when memory is allocated in the memory pool.z>Allocated %s bytes for %s with address %s from cumem allocatorN)rrFrEloggerdebug)rLrpy_d_mems rrHz&CuMemAllocator._python_malloc_callbacks`%Q')7 t/* * X&  L a       rptrc|j|}|jd|_td|jd|j||jS)zh Internal method to look up the allocation data when memory is freed in the memory pool.Nz:Freed %s bytes for %s with address %s from cumem allocatorrQ)rEpoprrRrSrr)rLrUdatas rrJz$CuMemAllocator._python_free_callbacks_#'',,  ! -%)D " H KN H     {r offload_tags.c|tjf}nt|tr|f}t|tsJd}d}|jD]\}}|j}||dz }|j|vrx||dz }|d}tj |tj dt}| } t| ||||_t#|t$d|dz |dz ||z dz t)jtjdS)aS Put the allocator in sleep mode. All data in the memory allocation with the specified tag will be offloaded to CPU memory, and others will be discarded. :param offload_tags: The tags of the memory allocation that will be offloaded. The rest of the memory allocation will be discarded. NrrQcpu)dtypedevice pin_memoryzCuMemAllocator: sleep freed %.2f GiB memory in total, of which %.2f GiB is backed up in CPU and the rest %.2f GiB is discarded directly.i@)r8r; isinstancertuplerEitemsrrremptyuint8rdata_ptr libcudart cudaMemcpyrr%rRinfogccollectr+ empty_cache) rLrY total_bytes backup_bytesrUrXr size_in_bytesrcpu_ptrs rsleepzCuMemAllocator.sleeps}  +68LL  c * * +(?L,.....  -3355 & &IC[F 6!9 $Kx<''q ) &q $)K!+ 688 %%%! ,4466$$Wc=AAA):& f % % % %   ' ! 7 " < '7 2           rtagscf|jD]\}}| |j|vr|j}t ||ji|j}|`||z}|}t |||d|_dS)a| Wake up the allocator from sleep mode. All data that is previously offloaded will be loaded back to GPU memory, and the rest of the data will have empty memory. :param tags: The tags of the memory allocation that will be loaded back to GPU memory. If None, all memory allocation will be loaded back to GPU memory. N) rErarrr#rnumel element_sizerdrerf)rLrprUrXrrrmrns rwake_upzCuMemAllocator.wake_ups-3355 6 6IC|tx4//v&&&)5(,(>%(4-33558I8V8V8X8XX&#4"<"<">">!,,S'=III15. 6 6rrc#K| tj}t|tsJ|j}||_t |j|j5}||j|<dV|d }|D]8}|ddkr*| |d}t|9||_ddddS#1swxYwYdS)a' A context manager to use the memory pool. All memory allocation created inside the context will be allocated in the memory pool, and has the specified tag. :param tag: The tag of the memory allocation. If None, the default tag will be used. Nrallocated_sizeaddress) r8r;r_rrFr6rIrKrGsnapshotrJr%)rLrold_tagrX allocations allocationrs ruse_memory_poolzCuMemAllocator.use_memory_pools6 ; ,C#s#####" +  ')B   '  -1D $S ) EEEq'**,,K) . . ./144!77 98MNNF%f---&D 3 ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' 'sA+CC C cld}|jD]\}}|j}||dz }|S)zM Get the total number of bytes allocated in the memory pool. rrQ)rErar)rL sum_bytesrUrXrs rget_current_usagez CuMemAllocator.get_current_usage%sH -3355 # #IC[F  "IIr)rr8r!)rrr__doc__r9rr;r staticmethodr>rNrrHintrJr`rolistrtrr|rrrrr8r8Wsj0"&H%%% K    ' ' '\ '???"      0!0!%S/C"7$">0!$0!0!0!0!d66DI,666660)')'3:)')')'^)'V3rr8)) dataclassesrhrBcollections.abcr contextlibrtypingrr vllm.loggerrvllm.utils.platform_utilsrvllm.utils.system_utilsrrrRr=vllm.cumem_allocatorr r r 2vllm.distributed.device_communicators.cuda_wrapperr r.reModuleNotFoundErrorr`rr dataclassrr#r%r+r,r-r0r6r8rrrrs $$$$$$%%%%%% ######======777777 X   QPPPPP""#455H IOOK #MHIII3S#% &  22222222 .j.T....111111ucz*>FSzSWGW>X Z-"ucz*">FSzSWGW>X" """"VVVVVVVVVVs'A++A?>A?