`i*.UddlZddlmZddlmZddlmZmZddlm Z  ddl m Z ddl m Z n#e$rddlZeejdZejdeeddl m Z ddl m Z eejdkr"ejdeekrejd=YnwxYwgZeeed <d eefd Ze d e deeded efdZe de dedededeeeeeffdZ deede!ed e!efdZ"ded efdZ#ded efdZ$ded ee%eeefe%eefe!ee%eeefffdZ&de!ed ee%eeefe%eefe!ee%eeefffd Z'd=ded"ed eefd#Z(ded eefd$Z) d>deeeefde!ed&e!ed'ed(e%eefd)ed eefd*Z*d+Z+eed,<d-d.hZ,e!eed/<e!Z-e!eed0<d1d1d2Z.e%eefed3<d4Z/eed5<d-d.hZ0e!eed6<e!Z1e!eed7<d8d9iZ2e%eefed:<d?d;Z3e4dz"find_file_paths..Ns*LLL%rss8K8K8K8K8Krch|]}|v| Srr)r;r<r7s rr=z"find_file_paths..Os+   u.Rs)NNNbgll8U33NNNr)setr*listdirupdate)r6r7paths all_files python_files filter_filesrAs ` @rfind_file_pathsrIEs eeEPPJx(( LL9LLL     +     NNNNNNNOOOO Lrrcd|vrd\}}nd|vrd\}}ntd|||t|z||}}|||S)zXExtract method name from decorator in the form of "@functional_datapipe({method_name})".(")rKz")(')rLz')z4Unable to find appropriate method name within line: ) RuntimeErrorfindlenr start_token end_tokenstartends rextract_method_namerUVs t||!+ YY !+ YY JD J J   ;''#k*:*::DIIiT;Q;Q!T))'++K888 A%  #S1EFF|!## DJJsOO3  DJJsOO3 #q((**S//Cdsd+I7H7S7S' 48B(59C6KYY%))&Q-IC! . %.%.%.%.%.%.%.%.%.%.%.%.%.%.%.N   sFGG G  file_pathsci}i}t}i}|D]j}t|\}}}} |||||||| k||||fSr)rBrorD) rpmethods_and_signaturesmethods_and_class_names!methods_with_special_output_typesmethods_and_doc_stringsr+rfrg$methods_needing_special_output_typesris rparse_datapipe_filesrws  (+%  8 8  % %   0 %%&9:::&&';<<<)001UVVV&&7777)  r, delimitercd}d}g}|D]C}|dkr|dz }n0|dkr|dz}n$||kr|dkr||d}>||z }D|||S)zRGiven a line of text, split it on comma unless the comma is within a bracket '[]'.rr[r]])ra)rry bracket_count curr_tokenreschars rsplit_outside_bracketrsMJ C   3;; Q MM S[[ Q MM Y  =A#5#5 JJz " " "J d JJz Jrc|t|}t|D]\}}|d||<|dkrd||<)|dkr'd||dz kr||ddkrd||<Vd|vr6|d d}|dd z||<d |D}|S) z Clean up a given raw function signature. This includes removing the self-referential datapipe argument, default arguments of input functions, newlines, and spaces. rclsselfrr]*rz Callable ==z = ...cg|] }|dk| Srr)r;ts r z%process_signature..s + + +A177a777r)rrre rpartition)rtokensrtokenheads rrdrds.d33Ff%% 3 35KK$$q E>>F1II UU&Q-//fQilc6I6IF1II U " "##C((+D 3(2F1I + + + + +F Mrrdeprecated_filesdefault_output_typemethod_to_special_output_typerootcdkr8tttjt |tr|gn|}fd|D}t |||}t|\}}} } |D]} | | vr| | g} | D]\} }|| }| | vr || }n|}d | | }|dkrd}nd|z}t| ||}| d|d|dz|z| d | S) a #.pyi generation for functional DataPipes Process. # 1. Find files that we want to process (exclude the ones who don't) # 2. Parse method name and signature # 3. Remove first argument after self (unless it is "*datapipes"), default args, and spaces rcPg|]"}tj|#Srr@)r;r+rs rrz*get_method_definitions..s)@@@ddD))@@@r)r7z ...rz# Functional form of 'z' z...c8|ddS)Nrr])split)ss rz(get_method_definitions..saggdmmA&r)key)strr__file__parentresolve isinstancerIunionrwrbitemsr,rra removesuffixrstripsort)rZr7rrrrrprrrsmethods_w_special_output_typesrufn_namemethod_definitionsrk argumentsrl output_type doc_string definitions ` rget_method_definitionsrs rzz4>>(002233))S99H yI@@@@i@@@I $4$:$:;K$L$LJ Z(( &188 8 8 8 * . .w 7 7 7"8">">"@"@   Y,[9 8 8 87 DKK-KWW4[ABB   JJ *J.{I{SS !! 4Z 4 4 4%%e,,3355 6      & & riteriterDP_file_pathz __init__.pyzutils.pyiterDP_files_to_excludeiterDP_deprecated_fileszlist[IterDataPipe])demuxfork$iterDP_method_to_special_output_typemapmapDP_file_pathmapDP_files_to_excludemapDP_deprecated_filesshuffle IterDataPipe#mapDP_method_to_special_output_typecVttttdttt t tdttt j }t||d}|ddfddS) a # Inject file into template datapipe.pyi.in. TODO: The current implementation of this script only generates interfaces for built-in methods. To generate interface for user-defined DataPipes, consider changing `IterDataPipe.register_datapipe_as_function`. r MapDataPipeF) install_dir template_dirdry_runz datapipe.pyizdatapipe.pyi.incdS)N)IterDataPipeMethodsMapDataPipeMethodsr)iter_method_definitionsmap_method_definitionssrrzmain..Hs#:"8  rN)rrrrrrrrrrrabsoluterr write_with_template)r+fmrrs @@rmainr,s5, 4+  >> " " $ $ +D D% H H HB     r__main__)rxr)r N)5r* collectionsrpathlibrtypingrrtyping_extensionsrtorchgen.api.pythonrtorchgen.utilsr ImportErrorsysrrparents REPO_ROOTr+insertrrOr r___annotations__r FutureWarningintr tupler5rBrIrUrYdictrorwrrdrrrrrrrrrr__name__rrrrsY ######(((((( ======9999999   JJJX''))1!4IHOOAss9~~&&&======****** s38}}qSXa[CC NN:: HQK cc m T#YSS  m  uS#s]+,  &tCyCHS" c c    SS11 4T#Y c3hS4T#Y;O OP1111hC 4T#Y c3hS4T#Y;O OP4d3i&CDI8 77S$s)^$7#h7#h7 7 $(S> 7  7 #Y7777v#%2J$?S???$'CEES))) ! 88$d38n $1:#>C>>>#&355C(((7@.6Q#T#s(^QQQ    F zDFFFFFs .BCC