`i,j dZddlmZddlZddlZddlZddlZddlmZm Z m Z m Z m Z m Z ddlZddlZddlmZmZ ddlZn #e$rdZYnwxYwgdZeZdaejeZdejdd d ejfd Zde eejd fd e eejfd e eejfd efdZdedejde d e d fdZ!ddejded d fdZ"dde dd efdZ#dZ$ej%e$GddZ&dS)aWrapper for cuDNN frontend to improve user experience. This wrapper provides a more user-friendly interface for cuDNN frontend. It allows users to create a graph, add operations to the graph, and then compile the graph to a cuDNN plan. This wrapper is designed to avoid boilerplate code. Key Features: - Automatic graph validation and compilation - Simplified tensor management - Support for both named and positional tensor inputs - Automatic workspace management - PyTorch integration with DLPack support Example: >>> x = torch.randn(8, 56, 56, 64, device=device, dtype=torch.float16).permute(0, 3, 1, 2) >>> w = torch.randn(32, 3, 3, 64, device=device, dtype=torch.float16).permute(0, 3, 1, 2) >>> with Graph() as graph: ... y = graph.conv_fprop( ... image=x, weight=w, ... padding=[1,1], stride=[1,1], dilation=[1,1], ... compute_data_type=data_type.FLOAT, ... name="conv2d", ... ) ... y.set_output(True).set_data_type(data_type.HALF) ... # Graph is automatically validated and compiled on exit >>> graph.set_io_tuples(["conv2d::image", "conv2d::weight"], ["conv2d::Y"]) >>> # Execute the graph >>> output = graph(x, w) ) OrderedDictN)AnyDictListOptionalTupleUnion) data_type heur_mode)Graphr r cudnngraphtensor torch.Tensorreturnct|dr.|jr'||S||S)aCreate a tensor in the graph object. Args: graph: The cuDNN graph object to create the tensor in tensor: The dlpack tensor to create a graph tensor from Returns: A cuDNN tensor object representing the input tensor in the graph Note: If the input tensor has requires_grad=True, it will be detached before creating the graph tensor to avoid gradient tracking issues. requires_grad)hasattrr tensor_likedetach)rrs a/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/cudnn/wrapper.py _graph_tensorr:sPv'')F,@)  111  ((( tensor_map dlpack_mapct|tr=||vr|S|D]!\}}||kr|cS"nt|tr7|D]!\}}||kr|cS"nt|t jr#|D] \}}||ur|cSnot|dr_t|trJt||vr9|t|}|D]\}}||kr|cStd)a7Find the mapping name for a tensor used in a graph. This function searches for a tensor in the tensor map and returns its corresponding name. The tensor can be specified in multiple ways: - As a string (either the assigned tensor name or the node::input_name) - As a cuDNN tensor object - As a DLPack-compatible tensor (e.g., PyTorch tensor) that was used in creating the graph Args: tensor: The tensor to find, can be a string name, cuDNN tensor, or DLPack tensor tensor_map: Dictionary mapping tensor names to cuDNN tensor objects dlpack_map: Dictionary mapping DLPack tensor IDs to cuDNN tensor objects Returns: The key in tensor_map that the provided tensor is mapped to Raises: ValueError: If the tensor cannot be found in the tensor map __dlpack__zInput not found in tensor map) isinstancestritemsget_nameintget_uidr rrdictid ValueError)rrr tensor_name tensor_values r _find_tensorr)Os0&## Z  M)3)9)9);); # # %K$$&&&00""""1 # FC #)3)9)9);); # # %K##%%//""""0 # FEL ) )#)3)9)9);); # # %K%%""""& #  & &#:j$+G+G#BvJJZdLdLdBvJJ')3)9)9);); # # %Kv%%""""& 4 5 55rnamearg_dictc||vr||S||vr||S ||S#t$rYnwxYw ||S#t$rYdSwxYw)a3Extract a dlpack tensor from the arg_dict that matches the provided name or cudnn tensor Args: name: The name of the tensor to extract tensor: The cudnn tensor to extract arg_dict: The dictionary of arguments to extract the tensor from Returns: A dlpack tensor N)r!KeyErrorr#)r*rr+s r_extract_tensorr.s x~  ))**      (()) tts!4 AAA A-,A-pyt cudnn_tensor tensor_typec|dkrtdtjst dtj|}|$td|tj | d|}tj || | }|S)a0Create a tensor like the provided cudnn tensor Args: cudnn_tensor: The cuDNN tensor to create a dlpack tensor from tensor_type: The type of tensor to create, currently only "pyt" is supported Returns: A dlpack tensor allocated that is like the provided cuDNN tensor r/z(Only PyTorch tensor is supported for nowPyTorch is not availableNz0cuDNN uses an unsupported data type in PyTorch: cudadevicedtype) NotImplementedErrorr datatypesis_torch_available RuntimeError_cudnn_to_torch_data_type get_data_type TypeErrortorchemptyget_dim as_strided get_stride)r0r1r7rs r _tensor_likerDse!"LMMM ? - - / /75666 O 5 5l6P6P6R6R S SE }iU>U>W>W X XF Mrstreamztorch.cuda.Streamcttdttja|#tjj}tjt|tS)z>Get the default cuDNN handle and set to torch's current streamNr3)handlerE) r?r;_default_cudnn_handler create_handler4current_stream cuda_stream set_stream)rEs rget_default_handlerMse }5666$ % 3 5 5 ~**,,8 1&AAAA  rcJttjtdSdSN)rHr destroy_handlerrdestroy_default_handlerRs(( 233333)(rc eZdZUdZdZeeed<dddddddeedeee e de j fd eee e de j fd eee d ed df d ZdZdZdZde d efdZdZd e dedffdZdee dfd ee dffdZdee e de j fd ee e de j fd dfdZdS)r aWrapper object for cuDNN computation graph This class simplifies the process of creating, compiling, and executing cuDNN computation graphs. It handles common boilerplate code and provides a more Pythonic interface to the cuDNN frontend API. Key features: - Automatic graph validation and compilation - Simplified tensor management with PyTorch integration - Support for both named and positional tensor inputs - Automatic workspace management Note: The graph is automatically validated and compiled when exiting the context manager. Any errors in graph construction will be raised at that point. N_Graph__handleT)rGinputsoutputs heuristicsworkspace_allocrGrUrrVrWrXrc Xttd|r+t|ttfst d|r+t|ttfst d|r+t|ttfst dt jdkrtd||_d|_ i|_ t|_ t|_ g|_i|_t!|_d|_d|_|pg|_|pg|_|pt,jt,jg|_|sd|_|r||_dD]6}||vr0t j||p||||<7dS) Nr3inputs must be a list or tupleoutputs must be a list or tuplez"heuristics must be a list or tuplei@dz*cuDNN version 9.12.0 or higher is requiredF) io_data_typeintermediate_data_typecompute_data_type)r?r;rlisttupler&r backend_version_Graph__kwargs _Graph__graph_Graph__tensor_mapr_Graph__tensor_in_Graph__tensor_out_Graph__tensor_unknown_Graph__node_countset_Graph__node_names_Graph__input_tuples_Graph__output_tuples_Graph__inputs_Graph__outputsr AFALLBACK_Graph__heuristics_Graph__workspacerTr9_torch_to_cudnn_data_type)selfrGrUrVrWrXkwargskeys r__init__zGraph.__init__s =9:: :  ?*VdE];; ?=>> >  @:ge}== @>?? ?  CjdE]CC CABB B  " "U * *KLL L  &=='MM "EE"# "  B&K9; 8J*K %$D   #"DMR d dCf}}#oGGs TTcX^_bXcs  d drcdSrOrQrts r__del__z Graph.__del__s rc|jtdtjdi|jdvr d|jini|j|_|S)NzGraph already created)autoNrGrQ)rcr;r pygraphrTrbrys r __enter__zGraph.__enter__se < #677 7}  ,0M,O,O$-((UW m    rc|#td|d|_|j|j|j|j|j|jt|ds=tj |j dtj |_|js|jr ||j|j|`|`td|jtd|jtd|j|jS) a%Exit the context manager, validating and compiling the graph. This method performs the following steps: 1. Validates the graph structure 2. Builds the operation graph 3. Creates execution plans 4. Checks hardware support 5. Builds the final plans 6. Allocates workspace memory Raises: ValidationError: If graph validation fails GraphStateError: If graph operations are performed in invalid order CudnnError: For other cuDNN-related errors Nz'Exception during graph construction: %s __workspacer4r5z Inputs: %sz Outputs: %szNode count: %s)loggererrorrcvalidatebuild_operation_graphcreate_execution_plansrq check_support build_plansrr?r@get_workspace_sizeuint8rrrmrn set_io_tuplesdebugrerfrh)rtexc_type exc_valuetbs r__exit__zGraph.__exit__s\"   LLBI N N NDL   **,,, ++D,=>>> ""$$$   """t]++ ${ //11k   D  = >DN >   t}dn = = = M4> \4#3444 ]D$5666 %t'8999|rr*ctjgd}|vrSgd}|vrtddtjsSfd}dvr|Sfd}|S)N)r get_workspace_size_plan_at_index serialize deserializequery_tensor_attributes_of_uid)buildrbuild_plan_at_indexrrcreate_execution_planrdeselect_behavior_notesdeselect_enginesdeselect_numeric_notesdeselect_workspace_greater_thanexecuteexecute_plan_at_indexget_behavior_notes$get_behavior_notes_for_plan_at_indexget_engine_countget_execution_plan_countget_knobs_for_engineget_plan_name_at_indexrvpopulate_cuda_graphrselect_behavior_notesselect_numeric_notesupdate_cuda_graphrzCalling z via wrapper is not allowedcL|i|}j||SrO)rgappend)argsruoutputattrrts rtensor_capturez)Graph.__getattr__..tensor_capturems3T4*6**F  ! ( ( 0 0 0Mr)rrct|} jvr d j < j xxdz cc<d|vr |d}n d j dz }||d<| jvrtd|d j|t |D]\}}t |drGt|}| jvrt j | j|< j|x}||<t|tj r| j|d|<|D]\}}t |drGt|}| jvrt j | j|< j|x}||<t|tj r| j|d|< |i|}t|tj r|g}nt|tt fr|}t |D]i\}}t|tj rJt |d r)|r|} n|d|} | j| <j|S) Nrr*.z Node name z already usedrz::r!)r_rhrjr&add enumeraterr%rdrrcrr rrer r`r!rf) rru node_nameiobjobj_idrvr output_listr'rr*rts rwrapperz"Graph.__getattr__..wrappervs::D4,,,*+!$'  d # # #q ( # # #"6N #AAd&7&=a&?AA !*vD--- !Fi!F!F!FGGG   ! !) , , ,#D// @ @33 -->WWFT%6664A$,PS4T4T)&1$($5f$==C$q'c5<00@AD$ %:%:S%:%:;T4*6**F&%,// %%h FT5M22 %$ #K00 9 93c5<009sJ//:CLLNN:&)llnn )2&9&9a&9&9 58D%k2Mr)getattrrcr;inspectismethod)rtr* pass_throughblocked_methodsrrrs`` @r __getattr__zGraph.__getattr__;st|T**    <  K   : ? " "K$KKKLL L%% K       , , ,! !- - - - - - - ^rc"|jtd|jstdt|dkr/t |dt r|j|dfi|St|dkr|jstdt|t|jkr5tdt|dt|jd |j |i|S) z"Execute the graph with tensor dictNzGraph not createdz8You should not invoke the graph before the context exitsrrzXYou should not invoke the graph with positional arguments before running set_io_tuples()zNumber of arguments (z#) does not match number of inputs ()) rcr;rlenrr$_Graph__call_with_tensor_dictrkr&!_Graph__call_with_positional_args)rtrrus r__call__zGraph.__call__s( < 233 3|4466 [YZZ Z t99>>ja$77>/4/QBB6BB B4yy1}}T%8} !{|||4yyC 34444 "CT"C"Cgjkok~gg"C"C"CDDD343TDVDD Dr)r.c>i}t|j|D]\}}|||<d|jD}t|j|D]\}}|||<t |}d|vr>|jdkrt |d<n!|j |j|d<ntdd|vr"|jdurtd|j|d<|j j |fi|t|d kr|d S|S) a?Execute the graph with positional arguments. Args: *args: Positional arguments to pass to the graph **kwargs: Additional keyword arguments to pass to the graph execution Returns: A single tensor or a tuple of tensors Note: This method is called by __call__() when the graph is executed with positional arguments. It is not intended to be called directly by the user. The `args` should be a list of dlpack tensors that matches the input order of `self.__input_tuples`. c.g|]}t|dSr/)rD).0r0s r z5Graph.__call_with_positional_args..s"cccl \599cccrrGr|N-Need to specify cudnn handle to execute graph workspaceF*Need to specify workspace to execute graphrr) ziprkr#rlr$rTrMr;rrrcrr)rtrru variant_packr0 user_tensor output_tuples r__call_with_positional_argsz!Graph.__call_with_positional_argssd" ),T-@$)G)G ? ? %L+3>L--// 0 0ccdNbccc ),T-A<)P)P ? ? %L+3>L--// 0 0f 6 ! !}&#7#7x  *#'=x  "#RSSS f $ $5(("#OPPP&*&6{# \44V444 |   ! !? " r tensor_dictc : i}i}tj|j|jD]\}}||vs|r0t|||}||||<It|dstd|d|||<g}g} |D]z\}}||vr||jvr@t|dx||<||<| |e| |{|rtd|| rt d| t|}d|vr>|jd krt!|d<n!|j |j|d<ntd d |vr"|jd urtd |j|d <|jj|fi||S)aExecute the graph with a dictionary of tensors. Args: tensor_dict: Dictionary of tensor names to tensors **kwargs: Additional keyword arguments to pass to the graph execution Returns: Dictionary of tensor names to tensors Raises: RuntimeError: If a non-virtual tensor in the graph is not found in `tensor_dict`, or the tensor in `tensor_dict` is not a dlpack tensor NrzTensor z# is not provided as a dlpack tensorr/z5Non-virtual input tensors not found in variant pack: zAdded output tensors: %srGr|rrFr) itertoolschainrer rfr#get_is_virtualr.rr;rDrrrr$rTrMrrrcr) rtrrurmissing_tensorsr*rrmissing_inputsmissing_outputss r__call_with_tensor_dictzGraph.__call_with_tensor_dicts}&  %OD,<,B,B,D,DdFWF]F]F_F_`` 9 9LD&~~<//63H3H3J3J/)$ DDK"*.'; 55 X"#VT#V#V#VWWW-8L)) * *+1133 , ,LFD~~<//t(((EQRXZ_E`E`` V^^--.T1B&&t,,,,%%d++++  igWegghh h  F LL3_ E E Ef 6 ! !}&#7#7x  *#'=x  "#RSSS f $ $5(("#OPPP&*&6{# \44V444rc t|ttfstdt|ttfstd|js||_||_dSg}t}t|D]\}} ||j vr| |%t||j |j}|j |}t||vrtd|d|d|t|| |#t$rtd|d|ddwxYwg}t|D]\}} ||j vr| |%t||j|j}|j|}t||vrtd|d|d|t|| |#t$rtd|d|ddwxYwt|D],\}}|rtd|d -|jD]0\}}|s||vrtd |d 1|j D]=\}}|s$t||vrtd |d >t||_t||_dS)a4Set order of input and output tensors to allow graph to be executed with positional arguments. Args: inputs: List of input tensors or names outputs: List of output tensors or names Raises: ValueError: If inputs or outputs are not lists or tuples rZr[NzInput at index z (z) is a duplicatez) not found in tensor mapzOutput at index z is a virtual tensorz Node output z4 is a non-virtual tensor but not specified as outputz Node input z= is a non-virtual tensor but not specified as input or output)rr_r`r&rcrrmrnrirrgrr)rerdr%rrfrr rkrl) rtrUrV input_tensors tensors_foundrr*routput_tensorss rrzGraph.set_io_tuples+s&4-00 ?=>> >'D%=11 @>?? ?|4466 "DM$DN F    (( c cGAt c4000!((...#D$*:Do+ + + + ZF#~-.F c>! " FFFFPG5U3 <=>G5eC=>?G5  G5G5G5G5G5G5rr rrO)'r collectionsratexitrrloggingtypingrrrrrr r cudnn.datatypesr r r? ImportError__all__r"rrH getLoggerrrr}rrrr)r$r.rDrMrRregisterr rQrrrs>$##### :::::::::::::::: &&&&&&&&LLLL EEE 7 6 6   8 $ $)))5<))))*-6 #u|^3 4-6S%,&'-6S%,&'-6 -6-6-6-6`#u|tQ_H`2u|#., ! !x(;< ! ! ! ! !444 &'''o5o5o5o5o5o5o5o5o5o5s?A A