.`i*)ddlmZddlZddlmcmZd dZdZ d dZGddZ ed dZ Gd d Z dS) )contextmanagerNc|g}t|ttfr|D]}t|||}nPt|tjr6t|}|||S)zDescends iterators that contains Tensors and prints the Tensor. Recursive function that descends iterator type arguments until it finds a Tensor object. ) isinstancelisttuple print_tensortorchTensorsizeappend) tensor_objprefix tensor_listten tensor_dimss q/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/utils/nvtx_pytorch_hooks.pyrr s  *tUm,,( A AC&sFK@@KK A J - -(:??,,-- ;''' ci}t|tjjtjjtjjfrji}|j|d<|j|d<|j|d<|j |d<|j |d<|j |d<|j |d<|j |d<|j|d <|j|d <|}nt|tjjtjjtjjfrji}|j|d<|j|d<|j|d<|j |d<|j |d<|j |d<|j |d<|j |d<|j|d <|j|d <|}nt|tjjtjjtjjfrUd }i}||j|d<||j |d<||j |d<||j |d<|}nTt|tjjtjjtjjfrMi}|j|jg|d<|j |j g|d<|j |j g|d<|j|d <|j|d <|}nt|tjjtjjtjjfri}|j|jg|d<|}npt|tjjr|j |d<|j!|d<n;t|tjj"tjj#tjj$fr|j%|d<|j&|d<|j'|d<nt|tjj(r |j)|d<nt|tjj*r|j+|d<|j)|d<n~t|tjj,r|j-|d<|j.|d<nJt|tjj/tjj0tjj1fr |j2|d<|S)zCExtract the static parameters from LLM and VLM relevant layer typesin_chanout_chan filter_dimstridepaddingdilation transposedoutput_paddinggroups padding_modect|trt|St|tr||gSdSN)rrrint) parameters r_handle_int_or_tuplez2process_layer_params.._handle_int_or_tupleDsG)U++ .I&Is++ .!9-- . .r ceil_modecount_include_pad output_size in_features out_features num_featuresepsilonmomentumin_placepnum_embeddings embedding_dim scale_factor)3rr nnConv1dConv2dConv3d in_channels out_channels kernel_sizerrrrrrrConvTranspose1dConvTranspose2dConvTranspose3d MaxPool1d MaxPool2d MaxPool3d AvgPool1d AvgPool2d AvgPool3dr$r%AdaptiveAvgPool1dAdaptiveAvgPool2dAdaptiveAvgPool3dr&Linearr'r( BatchNorm1d BatchNorm2d BatchNorm3dr)epsr+ReLUinplaceDropoutr- Embeddingr.r/UpsampleUpsamplingNearest2dUpsamplingBilinear2dr0) module_obj param_info conv_paramsconvtranspose_paramsr#pooling_paramss rprocess_layer_paramsrUsJ*uxQRRf= !+!7 I","9 J$.$: L! * 1 H!+!3 I","5 J$.$9 L!(2(A $% * 1 H&0&= N#  H $ H $ H $   Y= "*4*@Y'+5+BZ(-7-C\*)3):X&*4*<Y'+5+>Z(-7-B\*1;1J-.)3):X&/9/F^,) UX');UX=OP  E= . . . ';';J""   ux3UX5IJ  =&0%< >" * 9!+!4 : J . .=!+!3 : J 0 1 1=$, 3!+!3 : J 2 3 3 ='1'@ #$&0&> ?##  H  H ( H )   =&0%< >" rcdi}||d<|d}t|D]=\}\}} |dkri|d<t| |d|<>t |d} | r| |d<t |d} | r| |d <|rX|D]C\} } t | tjttfrt | | }|r||| <Dt|}|r||d <tj d |dS) NModuleF)recurserTrainableParamsInputInputsOutputOutputs StaticParamsz{})named_parameters enumeraterr ritemsrr r rrUnvtx range_pushformat) module_namerP in_tensorkwargs out_tensor marker_dict module_paramsidx param_name param_objin_tensor_listout_tensor_listkeyvaluerrQs rconstruct_marker_dict_and_pushrrslK'K///>>M(1-(@(@LL$ $j) !88-/K) *59)..:J:J5K5K %&z22!)W55N/ . H":x88O1!0 I3 ,,.. 3 3JC%%,e!<== 3*5#66 3'2K$%j11J1&0 N#ODKK ,,-----rceZdZdZdZdS) ResultHolderz9Holder for storing results from within a context manager.N)__name__ __module__ __qualname____doc__resultrrrtrtsCC FFFrrtc #Kt}t|||| |Vtj|dd}t||dd|jtjdS#tj|dd}t||dd|jtjwxYw)a;Context manager for NVTX markers that automatically pushes on enter and pops on exit. Example: with nvtx_marker_context("Module:MyModule", module, in_tensor=args, kwargs=kwargs) as ctx: ctx.result = module(*args, **kwargs) return ctx.result )rfrgz(input)z(output)Nrfrgrh)rtrrrb range_popreplacery)rerPrfrgholder output_names rlayerwise_nvtx_marker_contextrs ^^F#    !)))Z@@ &  }       !)))Z@@ &  }      s BACc<eZdZdZfdZdZdZdZddZxZ S) PytHooksaDThis module contains all the code needed to enable forward hooks in a pytorch network. To register the hooks for a given network, the user needs to instantiate a PytHook object. Then call the register_hooks method. Example: my_hook = PytHook() my_hook.register_hooks(my_network_model) cVti|_dS)zInitialize module variables.N)super__init__module_to_name_map)self __class__s rrzPytHooks.__init__s' "$rc t|Sr )rU)rrPs r_process_layer_paramszPytHooks._process_layer_paramss#J///rctj|j|d}t ||dd|tjdS)zCallback function that ends the NVTX marker. Records the module name and tensor information. Called after the module executes the forward method. unknownNr|)rbr}rgetrr)rrPrfrhres rmodule_fwd_hookzPytHooks.module_fwd_hooksa -11*iHH & tDZ     rcd|j|d}t||||ddS)zwCreates an NVTX marker with the module name in it. This function is called before the module executes. rNr|)rrrr)rrPrfrgres rmodule_fwd_pre_hookzPytHooks.module_fwd_pre_hooksF-11*iHH & yTX     rtopctjjtjjtjjtjjtjjf}||D]\}}t||r| |j d| |j ||j vr ||j |<`td|dS)zUser level function that activates all the hooks. The user needs to call this method from the network source code. The code descends all the modules in the network and registers their respective hooks. )rT) with_kwargsz!Module instance {} is not unique N)r r1IdentityrK Dropout1d Dropout2d Dropout3d named_modulesrregister_forward_pre_hookrregister_forward_hookrr ValueErrorrd)r network_model module_prefix skip_typesnamemodules rregister_hookszPytHooks.register_hookss H  H  H  H  H   *77}7MM U ULD&&*--   , ,T-ESW , X X X  ( ()= > > >T44426'// !D!K!KF!S!STTTr)r) rurvrwrxrrrrr __classcell__)rs@rrrs  %%%%% 000   rrr )NN) contextlibrr torch.cuda.nvtxcudarbrrUrrrtrrrzrrrs&%%%%% "lll`AE . . . .F !!!!HFFFFFFFFFFr