`i@ddlmZmZmZmZddlZddlmZddlm Z ddl m Z ddl m Z mZddlmZmZdd lmZmZmZdd lmZmZmZmZmZmZmZmZmZm Z d ed e d ee!fdZ"d ed e dede!de!de!de!de!de!de#de#dee!d efdZ$d e de%ee&e!e!ffde%ee&e!e!ffdede!d e f dZ'dedede d ed!eej(e)fd"eej(e#fd#e!d efd$Z*ded%edeee+efd&e de d ed'e!ded(e%e!e,efd eee+effd)Z-ded&e de d ed ef d*Z.d+ed&e d,e#d e/fd-Z0d.eee+efd/eeee+efd+ed&e de d'e!d efd0Z1d.eee+efd/eeee+efded&e de d'e!d efd1Z2 d7d2e!d&e d3e!de d4e%e!e&eeffded5e/d(ee%e!e,efd e fd6Z3dS)8)AnyCallableOptionalUnionN)get_node_type_to_io_type_map)get_new_attr_name_with_prefix)_is_activation_post_process) GraphModulemap_arg)GraphNode)NSNodeTargetTypeNSSingleResultValuesType NSSubgraph) get_arg_indices_of_inputs_to_log$get_node_first_input_and_output_typeget_node_input_qparamsget_normalized_nth_inputget_number_of_non_param_argsget_target_type_strgetattr_from_fqnNodeInputOrOutputTypeop_type_supports_shadowingreturn_first_non_observer_nodenodegmreturncd}t|drv|}|jdkrQt|jtsJt ||j}t |rt||d}|j|j d}|S)N_node_name_to_scope call_moduler) hasattrop isinstancetargetstrrr rr name)rrfqnnode_to_use_for_fqnmodules o/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torch/ao/ns/fx/graph_passes.py_maybe_get_fqnr,s Cr()) B# 7m # #dk3// / //%b$+66F*622 L&>tR&K&K#$%8%=>qA J logger_clslogger_node_name_suffix ref_node_name model_nameref_nameref_node_target_type results_typeindex_within_arg index_of_argr(c t|j|z|} t||} |||j||| ||| | | }t|| ||jd| |fi}|S)z Given a starting graph of prev_node -> node -> next_node This function creates a new logger_cls obj and adds it after node, resulting in prev_node -> node -> logger_obj -> next_node r!)rr'rsetattrgraph create_node)rrr.r/r0r1r2r3r4r5r6r(logger_node_name target_type logger_obj logger_nodes r+_insert_logger_after_noder?*s24 ++  &dB//K    J B *---*((8H4'SUVVK r-*node_to_instrument_inputs_to_ref_node_name+node_to_instrument_outputs_to_ref_node_namecft}ifd}|jjD]}|jdkr3|t t ||d|A||vs||vrt||}||vr||\} } t|} | D]} t ||| } t| tkrE| j }t|||d|j || | tjjd| | | j <pt| t jjjkrZt)| D]I\}}|j }t|||d|j || | tjj|| | |j <J||||j <||vrM||\} } t|j ||d|j || | tjjdd| |j <||||j <t/||}|S)z Takes the graph of gm, adds loggers to the output of each node in nodes_to_instrument. Returns a GraphModule with the new graph. c*t|fdS)Nc|jSNr')renvs r+z8add_loggers_to_model..load_arg..ks s49~r-r )arGs r+load_argz&add_loggers_to_model..load_argjsq5555666r-outputr _ns_logger_r5r6r()r r9nodesr#rLr rr,rtyper r'r?r NODE_INPUTvaluetorchfximmutable_collectionsimmutable_list enumerate node_copy NODE_OUTPUTr )rr@rAr.r1 new_graphrKrr(r2 ref_node_typearg_indices_to_log node_arg_idxnode_arg prev_nodearg_idxargnew_gmrGs @r+add_loggers_to_modelrcZsIC77777VAVA 7h     W%=dB%J%JHUU V V V  > > > ? ? ? r**CAAA*T+'-&Fd%K%K"$6((L7b,OOHH~~--$' $6 -F%&) I&$)4?E-.)5 # . . .HM**X%(*H*WWW-6h,?,?LGS(+CH I2K ) " * - $ * ( - 8 C I18-9$' 3 3 3C //"'00x@@C NBBB*U+'-"; N!I!,8>%&!" " " "DI '00x@@C NN Y ' 'F Mr- prev_node_cnode_agm_bgraph_cscale zero_pointdtype_cast_namect|jdz|}t||||d|di|}t|jdz|} t|| ||d| di| } |dtj||| tjfi|S)N _input_scale_get_attr_input_zero_point_ call_function)rr'r8r:rSquantize_per_tensorquint8) rdrerfrgrhrirjscale_node_name scale_nodezero_point_node_namezero_point_nodes r+ _insert_quantize_per_tensor_noderwsS3FK/4QRRSWXXO D/5)))$$OR_J8 **    D& 333))("b2FO    ! j/5<@    r-node_cgm_anode_name_prefixnode_type_to_io_type_mapc "d} d} d} d} d} d}t||||\}}t||||\}}|tjkr|tjks@|tjkr|tjks |tjkr|tjkr t j} n||kr"|tjkrt j j } n|tjkr5|tjkr%t|||}|t j } |\} }nr|tjkr|tjkrd} t j } nCtd|d|d|d|dzt!|t"rt%||}| r3| |t'||||| ||S|d| |fi|S| r|d| || fi|S| sJ| }t+||||d ||fi|St!|t,rg}|D]}t%||}| r0|d| |fi|}||L| sJ| }t+||||d ||fi|}|||Std t1|d ) a Given a starting graph C (derived from graph B) of ... -> prev_node_c -> node_c -> ... And a corresponding related node_a, inserts the correct dtype cast node after prev_node_c to cast into the dtype expected by node_a, resulting in: dtype_cast / ... -> prev_node_c -> node_c -> ... For example, if node_c is an int8 op and node_a is an fp32 op, this function will insert a dequant. Ntozdtype cast from  z to z needs to be implementedrp call_methodr!ztype fz is not handled)rrFP32INT8FP16 FP32_OR_INT8rS dequantizeUNKNOWNnnIdentityrrqfloat16AssertionError format_noder$r rrwr:r8listappendrP)rerxrdryrfrgrzr.r{ dtype_cast_opdtype_cast_mod_clsdtype_cast_methoddtype_cast_method_dtypedtype_cast_scaledtype_cast_zero_pointnode_input_type_a_node_output_type_anode_input_type_c_node_output_type_cnode_a_input_qparamsnew_dtype_cast_namedtype_cast_modresultsprev_node_c_innernew_dtype_cast_nodes r+_insert_dtype_cast_after_noders$6M" -Qj":..**.Rj":..** !6!; ; ;!%:%??? !6!; ; ;!%:%??? !6!; ; ;!%:%GGG( ... !6!> > >"X.2777 !6!; ; ; 6 D2   +!5M6J 3 32777 !6!; ; ; "'- M0 M M63E3E3G3G M M M"SSV%7%7%9%9SSS T   +t$$DJM; > >&&##   K & &J!, 4 4 "Q"?@P"Q"QRV"W"W  4&-&9&9#!&(' ''#23333))))!3!3!5!51>BBB&-&9&9!'&(' ''#23333Hd;&7&7HHHIIIr-c |jdkrt|jdz|}t||j}t j|r|}t|||| |j|di|}|S|jdkr|jdvsJd|jd|jdkrgtt||d |||}t|jdz|}| |j|j|fi|}|Stt||d |||}t|jdz|}| |j|j|t||d fi|}|Std | d |jd) z+ Simple copy of node_a to graph_c. rm _shadow_copy_rnrrr}ztarget  is not implementedrrrzhandling of node z with op )r#rr'rr%rS is_tensordetachr8r:_copy_node_from_a_to_crrr)reryrfrgnode_a_copy_name node_a_obj node_a_copyarg_copys r+rrsNyJW89VWW   &dFM:: ?: & & -#**,,J& 333)) I'R1A   m # #}!     8V] 7 7 7    =L ( (-(q994wH < o-     "-- 6=8+r;KK -(q994wH < o-     "--  3FD!DDE K  ] 2 2 4 4 ] ]vy ] ] ]   r- subgraph_anum_non_param_args_node_acg}|j}||jkr1||t||d}||jk1|||d}|D]}||dur|nd}||d}||\} } n|j|j} } d} | t| krA| dkrn"| dkr|dkrn|| | |sdS| dz } | t| kA| D]+} | dkrn| dkr|dkrn|| |sdS| dz } ,dS) z This function returns `False` if the input subgraph cannot be copied by `_insert_copy_of_subgraph_a_after_input_node_c`. This usually means that there is a corner case logic for which copy is not yet implemented. rct|tr3t||}|jdkr |jdvS|jdkrdSdSt|t t fr|D]}t|tsdSdS)NrrrmTF)r$r rr#r%rtuple) node_a_argryarg_aels r+ _can_insertz3_can_insert_copy_of_subgraph_a.._can_inserts j$ ' ' !2:tDDEx=((|';;;Z''tu  T5M 2 2 !  ! !!"d++! 55!tr-rTnormalize_to_only_use_kwargsNF) end_node start_noderrreversenormalized_argumentsargskwargslenvalues) rryrrOcur_noderrelocal_num_non_param_args_node_anorm_args_kwargs norm_args norm_kwargscur_idx kwarg_vals r+_can_insert_copy_of_subgraph_ars E"H j+ + + X+HdA>> j+ + + LL MMOOO   """)/58);); % % ("66 t7    '%5 "I{{%+[&-{II&&!||A"AQ"F"F"{9W#5t<<! 55 qLGI&&%++--  I!||A"AQ"F"F"{9d33! 555 qLGG  4r- input_node_cinput_node_c_2c t|ttfsJ|jg}|j}||jkr2t ||d}|d|||jk2|d}t||||||} tdt|D] } || }| } t| d||||} !| S)z* TODO(before land): real docblock rrN) r$r rrrrinsert)_insert_copy_of_node_a_after_input_node_cranger) rrrryrfrz nodes_of_ar cur_node_a cur_node_c cur_idx_ards r+-_insert_copy_of_subgraph_a_after_input_node_crs lT4L 1 111 1%&J"H j+ + ++HdA>>!X&&& j+ + + AJ:nj$>NJ1c*oo..    *  >         r-ct|tr|jn$t|tsJ|dj|d}||\}}n|j|j}}g} i} fd} d} | t|krR| dkr|} n| dkr||} n| || } | | | dz } | t|kR| D]2\}| dkr|| |<n| dkr||| |<n| | |<| dz } 3t| } t|}|j dkrzt|}t|j tsJt|j }t!|||j || | |}|S|j dvsJ|j |j | | |}|S) a Assume that node_a from graph_a has args (input, (input2)?, arg1, ...), and kwargs {kw0: kwarg0, ...} Note: input2 is optional. If it equals to None, we assume that the op has a single non-param input. If it is specified, we assume that the op has two non-param inputs. Copies the underlying values of arg1..argn and kwarg0..kwargn into gm_b, and creates the corresponding nodes in graph_c. Note: observers are ignored, so if an arg is an observer we navigate up until we find a non-observer parent. If node_a is a call_module, points the module pointed to by node_a to gm_b. Creates the copy of node_a in graph_c, with input as the first arg, and all other args and kwargs pointing to the copies of the objects in gm_b created above. An example in pictures: graph A: ======== input -------------> node_a / / / (input_2)?----------/ / / / / weight -> weight_obs / / bias ---------------- graph C (derived from B): ========================= input_node_c --> node_a_copy / / / (input_node_c_2)? / / / / weight_copy ----/ / / bias_copy ------/ rTrNct|tr$t|}t|}|St|tt t jfr|Stttfr&D]!}t|tr Jd"|Stdtd)Nz/handling of Node inside list is not implementedzhandling for kwarg of type r) r$r rrintfloatrSdtyperrrrP)rarryrfrgrs r+ _copy_argz<_insert_copy_of_node_a_after_input_node_c.._copy_args c4  0d;;C(dD'BBCJ c5%+6 7 7 J  D%= 1 1   %b$//E/J Rd9ooRRR r-rr!)rpr)r$r r9rrrrrritemsrrr#r%r&rr8r:)rrreryrfrzrrrnew_args new_kwargsrrnew_arg kwarg_namenode_a_shadows_c_namenew_mod_copy_namemod_anode_a_shadows_crgrs `` @@r+rr;sf,%%($,-----q/'22 43#!1 ;;!'fm; HJ&G C NN " " a<<"GG \\n8$GGi ' 233G   1  C NN " ""-!2!2!4!4 I a<<%1Jz " " \\n8%3Jz " "%.Yy%9%9Jz "1 XHK9:JKKDQQ yM!!K9:JKKDQQ&------ v}55'///".. I    !    y<<<<<".. I M   !    r-name_aname_bmatched_subgraph_pairsshould_log_inputsc3|t}t}i33fd} i} i} |D]P\} } | \}}t|j|}t|j}|| ||f| |j<|| ||f| |j<QjjD]}|j dkr/| t|j d| =|| v}|| v}|s|r|r| |\}}}}n|sJ| |\}}}}t|jot|}|s[tdt|dt|j|zdz||| 3|j<t#|j|||\}}t#|||\}}|t$jko/|t$jko|t$jko|t$jk}|s\tdt|dt|j|zdz||| 3|j<|t$jkr|t$jkrtt-|j||}|s\tdt|dt|j|zd z||| 3|j<Lt/|j|}t1|||s\tdt|dt|j|zd z||| 3|j<t3|j|}t3|j}|r|rt5|d} t7| t8rE3| j}!t;|!|d |j|||t<jj dd| 3|!j<nt7| tBrc3fd | D}"tE| D]D\}#}$|"|#}!t;|!|d |j|||t<jj |#d| 3|!j<En tGdtI| d|s|r+||| 3|j<3|j}%|rt5|%d}!|rJt7|!t8rt5|%d}!n#t7|!tBrfd|!D}!tK|j|%|!|||jdz|| }&|rd}'t7|&t8r-t;|&|d|'|||t<jj dd| }&|&}(nqt7|&tBsJg})tE|&D]D\}*}+t;|+|d|'|||t<jj |*d| },|)&|,E|)}&|&}(d}-t/|j|}|dkrt5|%d}-tO|&|-|||%jdz}.|.3|.j<|r|.}/t5|/d|(kr&t5|/d}/t5|/d|(k&t7|(t8r"tQ|(j}0|/j|0_)n=t7|(tBsJ|(D]#}1tQ|1j}0|/j|0_)$t;3|.j|d|.j|||t<j*j dd| 3|.j<|rBt;3|j|d|j|||t<j*j dd| 3|j<||| 3|j<tW|}2|2S)a Creates a new GraphModule consisting of the graph of C, with the meaningful nodes of A shadowing the corresponding nodes of B. For example, Graph A: a0 -> op0_fp32 -> a1 -> op1_fp32 -> a2 Graph B: b0 -> op0_int8 -> b1 -> op1_int8 -> b2 matched_node_pairs: {'op0': (op0_fp32, op0_int8), 'op1': (op1_fp32, op1_int8)} Graph C (A shadows B): / dequant0 -> op0_fp32 -> logger_a_0 / dequant_1 -> op1_fp32 -> logger_a_1 / / b0 -------------> op0_int8 -> logger_b_0 --------------> op1_int8 -> logger_b_1 In a nutshell, this function does the following for each node pair: * copies the necessary attributes and modules from gm_a to gm_b, keeping names unique * adds a dtype cast op (dequant, quant, etc) * adds a copy of node_a in gm_b's graph * adds loggers to the outputs of node_a and node_b Nc*t|fdS)Nc|jSrErF)renv_cs r+rHz6create_a_shadows_b..load_arg..suTY'7r-rI)rJrs r+rKz$create_a_shadows_b..load_argsq7777888r-rLrz$skipping shadow loggers for node_b: z, start_node_a: z , unsupportedz, unknown dtype castz, unknown input qparamsz", unhandled logic in subgraph copy_ns_logger_b_inp_rNc*g|]}|jSrnrF).0rars r+ z&create_a_shadows_b..s+S+S+SE#(O+S+S+Sr-ztype z is not handled yetc2g|]}t|dS)r)r)rrarfs r+rz&create_a_shadows_b..s5''' #5S$BB'''r- _dtype_cast__ns_logger_a_inp_rrr _ns_logger_a_ _ns_logger_b_),rr rr base_op_noderrr9rOr#rLr rrprintrXr'rrrrrrrrr,rr$r r?rrQrRrrWrrPrrrgetattrr0rYr )4rryrrfrr.rr{rgrK+start_node_b_to_matched_subgraph_a_and_name)end_node_b_to_matched_subgraph_a_and_name match_namematchr subgraph_bref_node_type_aref_node_type_bnode_bnode_b_is_start_nodenode_b_is_end_noder2all_op_types_support_shadowingrnode_output_type_anode_input_type_bnode_output_type_bnode_io_types_known_a_and_brr fqn_base_a fqn_base_b prev_node_brdprev_node_c_listr`rarxdtype_cast_noder0 input_logger new_loggersdtype_cast_idxdtype_cast_node_innerdtype_cast_loggernode_c_second_non_param_argrrinput_logger_modinput_logger_innergm_crs4 ` @r+create_a_shadows_brs| H '#?#A#A ggGE9999935/02-399;;   E!& J-j.EtLL-j.EtLL     N 3J4IJ     J 1*2EFF*"aEaE 9 NN76;q>8<< = = =  &)TT#'PP X E#5X E# F @G ##O*))) >fE ##.H%...5,V44 +2 ^;NvW[;\;\^^[))FFH'*?*GG ( / ^;NvW[;\;\^^[))CCC'=)41I(($,b?RSY[_?`?`bb_-@AVX\-]-]__`34 *1):):68)L)LE&+&(D%t)) %2D"; ^;NvW[;\;\^^[)vtQ&O&O #K66'''''2''' #@)K.0, # #*%)7$&M!/488&7*C+ &/)"$+4?E-.)* * + + +AP )/4@@@@@&( EN+FFBBAN,A1J 5 $ * 3 - & ( / 8 C I1?-.$. 1 1 1-(../@AAAA*5'6 /3+,H)4--)-112Ja33/$Q#/K/1 $$ 0@&+,%K 0H28T1EEUU#;HdA#N#N38T1EEUU!,55K+249J+K+K(9A(66),=====2>KK./6t=O=T/U/U,=E],::0I*/0#$)#,8>%&!"" 0 0 0&+,(" %>&+&#K#,8>%&!"" & & &fk"0")!2!268!D!DE&+   tW % %D Kr-rE)4typingrrrrrStorch.ao.ns.fx.mappingsrtorch.ao.quantization.fx.utilsrtorch.ao.quantization.observerr torch.fxr r torch.fx.graphr r ns_typesrrrutilsrrrrrrrrrrr&r,rr?dictrrcTensorrrwrsetrrboolrrrrrnr-r+rs111111111111 @@@@@@HHHHHHFFFFFF))))))))&&&&&&&&LLLLLLLLLL                          ; 8C=     - ---! -  -  ------ #- ----`ll04T5c?5J0Kl26dE#s(O6K1Ll l  l  llll^    u$ %  elC'(  B\J \J \JtT$Z'(\J  \J  \J  \J\J\J#3,<(=#=>\J 4d \J\J\J\J@8 8 8  8  8  8 8 8 8 vHH H #H HHHHV(d4j()(U4d#345((  (  (  ( ((((VI d4j()I U4d#345I  I   I  I  I  I I I I hLPff f f f  f !eJ ,B&C!CD f  ff'tC5E1F,F'GHfffffffr-