)`iq ddlZddlmZmZmZmZmZmZmZmZm Z ddl Z Gddej Z Gddej Z Gddej ZedeejZGd d ej Zd efd efd edeefge_edeedZGddZdZdedededede jf dZdS)N) CFUNCTYPEPOINTERc_intc_int64c_size_tc_uint8c_uint16c_void_ppointerc$eZdZdefdefdefgZdS) DLDataTypecodebitslanesN)__name__ __module__ __qualname__rr _fields_p/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/comm/dlpack_utils.pyr r s-   (HHHrr ceZdZdefdefgZdS)DLDevice device_type device_idN)rrrrrrrrrr(s%  eHHHrrc `eZdZdefdefdefdefdeefdeefde fgZ dS) DLTensordatadevicendimdtypeshapestrides byte_offsetN) rrrr rrr rrrrrrrrr/se  8  * '''""#  GG    ! HHHrrceZdZdS)DLManagedTensorN)rrrrrrr&r&EsDrr& dl_tensor manager_ctxdeletercdS)Nr)dmt_ptrs r no_op_deleterr,Qs  DrceZdZdZdZdS)CapsuleWrapperaE A wrapper class that holds references to the PyCapsule and its associated data. This class prevents Python's garbage collector from collecting the shape_array and managed_tensor objects while the capsule is still in use. It serves as a container to maintain the lifecycle of all DLPack-related objects. c0||_||_||_dS)aF Initialize the CapsuleWrapper with the necessary objects. Parameters: capsule: The PyCapsule object that follows the DLPack protocol shape_array: The array containing tensor shape information managed_tensor: The DLManagedTensor instance that the capsule points to N)capsule _shape_array_managed_tensor)selfr0 shape_arraymanaged_tensors r__init__zCapsuleWrapper.__init__as)  (  rN)rrr__doc__r6rrrr.r.Xs-     rr.cTd}d}|tjtjtjtjtjtjfvrtj|j}d}n|tj tj tj tj fvrtj |j}d}n[|tjtjtjtjfvrtj |j}d}nt%||dz}t&dz} | |||z} | ||zd} t)d|} t+||d} t-}t/||_| |_d|_| |_t9j| t=t&|_t9j| t=t&|_ d|_!tE}||_#d|_$tJ|_&t8j'j(}t.|_)t.t8j*t.g|_+tY|}||dd}t9j|t8j-j.}t_|| |}|S) a Parameters: ptr: GPU memory address obtained from cudaMalloc (Python int) segment_size: Memory size of each segments in bytes segment_stride: Memory stride size between segments in bytes num_segments: Number of segments torch_dtype: torch dtype dev_id: device id. Returns: A PyCapsule object compliant with DLPack specification, which can be directly converted to a tensor using torch.utils.dlpack.from_dlpack r)rr)rrrNsdltensor)0torch float8_e5m2 float8_e4m3fnbfloat16float16float32float64finforint8int16int32int64iinfouint8uint16uint32uint64NotImplementedErrorrrr rr rrr r!ctypescastrr"r#r$r&r'r(r,r) pythonapi PyCapsule_Newrestypec_char_pargtypesr py_objectvaluer.)ptr segment_sizesegment_stride num_segments torch_dtypedev_idbits_per_elementsdldata_type_codebytes_per_elementShapeArrayTyper4 stride_arrayrr!dltensorr5rQmanaged_tensor_ptr capsule_ptrr0capsule_wrappers rcreate_dlpack_capsulerfssK      "K 449 U[%+u{K K K!K 449 elEL%,O O O!K 449!+...)Q.q[N .|?P/PQQK!>.4E"EqIIL !v 6 6 6F ,3DA N N NEzzHSMMHMHOHMHN[gg.>.>??HN{<1A1ABBHH$&&N'N!%N*N$2M$M&BM 00- 2KFFKk+v'788>G$Wk>JJO rrWrXrYrZr!ct||||||}tjj|j}||_|S)ax Pack GPU memory into a PyTorch tensor with specified stride. Parameters: ptr: GPU memory address obtained from cudaMalloc segment_size: Memory size of each segment in bytes segment_stride: Memory stride size between segments in bytes num_segments: Number of segments dtype: PyTorch data type for the resulting tensor dev_id: CUDA device ID Returns: PyTorch tensor that references the provided memory Note: This function creates a new DLPack capsule each time it's called, even with the same pointer. Each capsule is consumed only once. )rfr<utilsdlpack from_dlpackr0_capsule_wrapper)rWrXrYrZr!r\re torch_tensors rpack_strided_memoryrmsK6, \><O;%11/2IJJL$3L! r)rNrrrrrrr r r r< Structurer rrDLManagedTensorDeleterr&rr,r.rfintr!rmrrrrqsW                        !v     v    #''&/ " "      f&   (H $ 8 899: 4))**  +*         6IIIX       ;      r