.`i-( zddlZddlZddlZddlmZddlmZmZddlm Z ddl Z ddl Z ddl Z ddl mZddlmZmZded efd Zded efd Ze dd ed efd Zd efdZGddZeGddZeGddZej ddeded eeddffdZdS)N) Generator) dataclassfield)cache)current_platform) GiB_bytes MiB_bytesbreturnc4t|tz dSN)roundr r s h/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/utils/mem_utils.py format_mibrA M1%% ''c4t|tz dSr)rr rs r format_gibrrrgpucvddlm}||}|dks Jdt|S)z    ~  rc2tjjS)z2Returns the total CPU memory of the node in bytes.)psutilvirtual_memorytotalrrget_cpu_memoryr&'s  " " ((rcNeZdZddejjdzfdZdefdZdZ dZ dS) DeviceMemoryProfilerNdevicec||_dSN)r)selfr)s r__init__zDeviceMemoryProfiler.__init__-s  rr cZtjtj|jSr+)gccollectrget_current_memory_usager)r-s rcurrent_memory_usagez)DeviceMemoryProfiler.current_memory_usage0s  8EEErc8||_|Sr+)r4initial_memoryr3s r __enter__zDeviceMemoryProfiler.__enter__5s"7799 rc||_|j|jz |_t jdSr+)r4 final_memoryr6consumed_memoryr0r1)r-exc_typeexc_valexc_tbs r__exit__zDeviceMemoryProfiler.__exit__:s: 5577#043FF rr+) __name__ __module__ __qualname__torchtypesDevicer.floatr4r7r>r%rrr(r(,s|u{1D8FeFFFF  rr(ceZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed<dZ eed<d Z e ed <d Zejjed <d Zeed<ddZddZddZdefdZd S)MemorySnapshotzMemory snapshot.r torch_peak free_memory total_memory cuda_memory torch_memorynon_torch_memory timestampNr)T auto_measurer c|j2tj}|Jtj||_ntj|j|_|jr|dSdSr+)r)rcurrent_devicerBdevice_rPmeasure)r- device_fns r __post_init__zMemorySnapshot.__post_init__Qsp ; (7I((( < 44DLL < 44DL    LLNNNNN  rc|j}tj|dd|_tj|\|_|_d}tjr8tj |j |vrtj j |_|j|jz |_tj||_|j|jz |_t%j|_dS)Nzallocated_bytes.all.peakr))) r) r)rSr memory_statsgetrH mem_get_inforIrJis_cudaget_device_capabilityindexr"r# availablerKmemory_reservedrLrMtimerO)r-r)shared_sysmem_device_mem_smss rrTzMemorySnapshot.measure\s+7??CC &  /?.KF.S.S+$+'A$  $ & & A 6v|DD+,, &466@D ,t/?? -> E E E rrm)r?r@rArnrtrrorurvrwrrGryrzrErVrqrlr%rrrsrss<<        NC$)E.$I$I$IM>IIIL%OOOO  #       rrsbaseline_snapshotrwc#&Ktjtjtj|jt ||}|j|Vtjtj|j |j |jz }|j |j z }|j |_ |j |_|j|_|j}|j }||z|jz|_dS)a  Memory profiling context manager. baseline_snapshot: the memory snapshot before the current vLLM instance. weights_memory: memory used by PyTorch when loading the model weights. Note that, before loading the model weights, we also initialize the device and distributed environment, which may consume some memory. This part is not included in the weights_memory because PyTorch does not control it. The memory in one GPU can be classified into 3 categories: 1. memory used by anything other than the current vLLM instance. 2. memory used by torch in the current vLLM instance. 3. memory used in the current vLLM instance, but not by torch. A quantitive example: Before creating the current vLLM instance: category 1: 1 GiB category 2: 0 GiB category 3: 0 GiB After creating the current vLLM instance and loading the model, (i.e. before profiling): category 1: 1 GiB category 2: 2 GiB (model weights take 2 GiB) category 3: 0.5 GiB (memory used by NCCL) During profiling (peak): category 1: 1 GiB category 2: 4 GiB (peak activation tensors take 2 GiB) category 3: 1 GiB (memory used by NCCL + buffers for some attention backends) After profiling: category 1: 1 GiB category 2: 3 GiB (after garbage-collecting activation tensors) category 3: 1 GiB (memory used by NCCL + buffers for some attention backends) In this case, non-kv cache takes 5 GiB in total, including: a. 2 GiB used by the model weights (category 2) b. 2 GiB reserved for the peak activation tensors (category 2) c. 1 GiB used by non-torch components (category 3) The memory used for loading weights (a.) is directly given from the argument `weights_memory`. The increase of `torch.cuda.memory_stats()["allocated_bytes.all.peak"]` during profiling gives (b.). The increase of `non_torch_memory` from creating the current vLLM instance until after profiling to get (c.). )ryrwN)r0r1r empty_cachereset_peak_memory_statsrSrsr|rTr}ryrHrurMrvrOrzrwrt)rrwresult diff_profilediff_from_createrMpeak_activation_memorys rmemory_profilingrspJLLL """,->-FGGG "'%F  !!### LLLJLLL """   """'&*??L+f.BB!-!8F 0 AF&0F0#711F4II r)r) contextlibr0rdcollections.abcr dataclassesrr functoolsrr"rB torch.typesvllm.platformsr mem_constantsr r rrqrrr r&r(rGrscontextmanagerrr%rrrs= %%%%%%(((((((( ++++++////////(#(#(((((#(#((((S))))) , ` ` ` ` ` ` `  ` F          : TT%TT$dD01TTTTTTr