`iV ddlZddlZddlZddlZddlZddlmZmZddlm Z ddl m Z ddl m Z mZmZmZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddl m!Z!ddl"m"Z"erddl#Z$ddl#m%Z%ddl#m&Z'Gdde%Z(e j)j*Z+e e,j-j-a.t\j-j-a/de+d0Z1GddZ2dee3ej4fde3de3fdZ5de2de3de3d e6e7e3e8ffd!Z9d"e3d#e6e3fd$Z:d%e3de3de3fd&Z;d%e3de3d e6e7e3e8ffd'Zd(e3de2d+e3d)e7e3e3e6ffd,Z?de2de3d e6e7e3e8fd)e3fd-Z@de2d e6e7e3e8ffd.ZAde2de3de3d e6e7e3e8fd/e8f d0ZB d=d1e3d2ee3d3eed4ee3d)e f d5ZCd6e3d)e8fd7ZDd8ZEd9efd:ZFGd;dN)ArgumentParser Namespace)date)Path)AnyCallableOptionalUnion)CONFIG_MAPPING_NAMESMODEL_NAMES_MAPPING)FEATURE_EXTRACTOR_MAPPING_NAMES)IMAGE_PROCESSOR_MAPPING_NAMES)PROCESSOR_MAPPING_NAMES)TOKENIZER_MAPPING_NAMES)VIDEO_PROCESSOR_MAPPING_NAMES)is_libcst_available)BaseTransformersCLICommand)add_fast_image_processor) CSTVisitor)matchersc^eZdZdZdZdejddfdZdejfdZdej fdZ dS) ClassFinderzC A visitor to find all classes in a python module. c0g|_g|_d|_dSNF)classespublic_classes is_in_classselfs |/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/commands/add_new_model_like.py__init__zClassFinder.__init__/s!#DL(*D $D   nodereturnNc\|j|jjd|_dS)zwRecord class names. We assume classes always only appear at top-level (i.e. no class definition in function or similar)TN)rappendnamevaluerr!r%s r"visit_ClassDefzClassFinder.visit_ClassDef4s* L    0 0 0#D   r$cd|_dSr)rr+s r"leave_ClassDefzClassFinder.leave_ClassDef9s$D   r$ctjtjtjtjgg}|jsgtj||rT|jdjdj j }|dkr.|jdj j }d|D|_ dSdSdSdS)z:Record all public classes inside the `__all__` assignment.)target)targets)bodyr__all__c&g|]}|jjS)r*).0elements r" z9ClassFinder.visit_SimpleStatementLine..Es*W*W*W77=+>*W*W*Wr$N) mSimpleStatementLineAssign AssignTargetNamermatchesr2r1r0r*elementsr)r!r%!simple_top_level_assign_structureassigned_variabler?s r"visit_SimpleStatementLinez%ClassFinder.visit_SimpleStatementLine<s010Ehafhh(G(G(G'HIIIJ111 -# X $8Y(Z(Z X$(IaL$8$;$B$H!$ 11#y|1:H*W*Wh*W*W*WD'''  X X X X11r$) __name__ __module__ __qualname____doc__r#cstClassDefr,r.r:rBr5r$r"rr*s   % % %  $s| $ $ $ $ $  %s| % % % % X#2I X X X X X Xr$rz # coding=utf-8 # Copyright aN the HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. ceZdZdZdefdZdS) ModelInfoszI Retrieve the basic information about an existing model classes. lowercase_namecj|dddd|_|jtvr|jdd|jtvrt |dt |j|_t|j|_|jdd|_|jtvr9t|j\|_ |_ |j dkrdn|j |_ nd\|_ |_ tj |jd\|_|_t!j |jd|_t%j |jd|_t)j |jd|_dS) N _-z is not a valid model nameConfigPreTrainedTokenizerFast)NN)lowerreplacerKr ValueErrorr paper_name config_classcamelcase_namertokenizer_classfast_tokenizer_classrgetimage_processor_classfast_image_processor_classrvideo_processor_classrfeature_extractor_classrprocessor_class)r!rKs r"r#zModelInfos.__init__cs,2244<UVZVi>j ;D $";15NNNTXTm  % %?I ;D $";FcFg  G G C"D$C&C%FtGZ\`%a%a"'F'J4K^`d'e'e$6:4;NPTUUr$N)rCrDrErFstrr#r5r$r"rJrJ^s@VsVVVVVVr$rJ file_name new_content add_aftercJt|dd5}|}dddn #1swxYwY||d\}}||z|z|z}t|dd5}||ddddS#1swxYwYdS)ac A utility to add some content inside a given file. Args: file_name (`str` or `os.PathLike`): The name of the file in which we want to insert some content. new_content (`str`): The content to add. add_after (`str`): The new content is added just after the first instance matching it. rutf-8encodingNrw)openreadsplitwrite)rbrcrdf old_contentbeforeafters r"add_content_to_filerss/ iw / / /1ffhh  %%i33MFE9${2U:K iw / / /1  s4885BBBold_model_infosnew_lowercase_namenew_model_paper_namefilenames_to_addc dd|dddD}|j|j}fd|ddD}g}|D]F\}}t jd|r|d \} } | | p|f|d </|||fGttd z d z d z d |d|ddttd z d z d z d |d|dd|D]\} }|r| dd} ttd z d z | z 5} | }dddn #1swxYwYt j dd|t j } | D]J}ttd z d z | z |||||KdS)a Add a model to all the relevant mappings in the auto module. Args: old_model_infos (`ModelInfos`): The structure containing the class information of the old model. new_lowercase_name (`str`): The new lowercase model name. new_model_paper_name (`str`): The fully cased name (as in the official paper name) of the new model. filenames_to_add (`list[tuple[str, bool]]`): A list of tuples of all potential filenames to add for a new model, along a boolean flag describing if we should add this file or not. For example, [(`modeling_xxx.px`, True), (`configuration_xxx.py`, True), (`tokenization_xxx.py`, False),...] rQc3>K|]}|VdSNtitler6xs r" z-add_model_to_auto_mappings..*``1QWWYY``````r$rOrNcFg|]\}}|d|fS)autorT)r6filenameto_addold_lowercase_names r"r8z.add_model_to_auto_mappings..s@CS8V  ,f 5 5v>r$rNz2(?:tokenization)|(?:image_processing)_auto_fast.pymodelsrzconfiguration_auto.pyz ("z", "z Config"), zOCONFIG_MAPPING_NAMES = OrderedDict[str, str]( [ # Add configs here rcrdz"), zcMODEL_NAMES_MAPPING = OrderedDict[str, str]( [ # Add full (and cased) model names here _fast.py.pyz( {8,12}\(\s*"z",.*?\),\n)(?: {4,12}\(|\]))joinrTrmrKrXresearchr(rsTRANSFORMERS_PATHrkrlfindallDOTALL)rtrurvrwnew_cased_nameold_cased_namecorrected_filenames_to_addfiler previous_fileprevious_to_addrromatching_linesmatchrs @r"add_model_to_auto_mappingsrs(WW``0B0J0J3PS0T0T0Z0Z[^0_0_`````N(7$3NWghihjhjWk"$(>> f 9JD Q Q >-G-K *M?.;_=VPV-W &r * * & - -tVn = = = =H$v-0GGT!3TTTTTf H$v-0GGT!3TT9MTTTz 7&  '' E::H'(2V;hFGG 1vvxx                ZU$6UUUW[]_]fN(  #%069HD % .@BT U U ] ]&!!$ sE""E& )E& new_paper_namerc4d}tjddtdd|z}t jd|d|d}g}|D](}d |d |}d |vr|d z }||)d |}||z|zS)a4 Create a new doc file to fill for the new model. Args: new_paper_name (`str`): The fully cased name (as in the official paper name) of the new model. public_classes (`list[str]`): A list of all the public classes that the model will have in the library. u ⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be rendered properly in your Markdown viewer. --> z# ?rQz coding=utf-8 z