`iUZdZddlZddlmZddlZddlZddlm Z ddl m Z ddl m Z mZmZer ddlZddlmZejeZdZe rerdd lmZndd lmZGd d eZGd deZdadZdZdZdZ dZ!dZ"ddZ#ddZ$dS)z Integration with Deepspeed N) partialmethod)dep_version_check)is_accelerate_availableis_torch_availablelogging)nnctjddu}|r- tjd}dS#tj$rYdSwxYwdS)N deepspeedTF) importlibutil find_specimportlib_metadatametadataPackageNotFoundError)package_exists_s w/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/integrations/deepspeed.pyis_deepspeed_availabler$so^--k::$FN "+K88A4!6   55  s;A A)HfDeepSpeedConfig)objectc"eZdZdZfdZxZS)raJ This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage. A `weakref` of this object is stored in the module's globals to be able to access the config from areas where things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore it's important that this object remains alive while the program is still running. [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic the DeepSpeed configuration is not modified in any way. Args: config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict. ct|tdtdt|dS)N accelerater )set_hf_deepspeed_configrsuper__init__selfconfig_file_or_dict __class__s rrzHfDeepSpeedConfig.__init__JsL%%%,'''+&&& ,-----)__name__ __module__ __qualname____doc__r __classcell__r!s@rrr9sB .........r"rc^eZdZdZfdZdZdZd dZeed Z d d Z d Z xZ S)HfTrainerDeepSpeedConfigz The `HfTrainerDeepSpeedConfig` object is meant to be created during `TrainingArguments` object creation and has the same lifespan as the latter. cft|d|_g|_dSN)rr_dtype mismatchesrs rrz!HfTrainerDeepSpeedConfig.__init__Xs. ,--- r"c<|jtd|jS)Nz8trainer_config_process() wasn't called yet to tell dtype)r- ValueError)rs rdtypezHfTrainerDeepSpeedConfig.dtype]s! ; WXX X{r"c@||}|dS|dkS)NFauto) get_value)r ds_key_longvals ris_autoz HfTrainerDeepSpeedConfig.is_autobs'nn[)) ;5&= r"NTc ||\}}|dS||dkr|||<dS|sdS||}|.||kr*|jd|d|d|d|dSdSdS)a A utility method that massages the config file and can optionally verify that the values match. 1. Replace "auto" values with `TrainingArguments` value. 2. If it wasn't "auto" and `must_match` is true, then check that DS config matches Trainer config values and if mismatched add the entry to `self.mismatched` - will assert during `trainer_config_finalize` for one or more mismatches. Nr3z- ds =z vs hf )find_config_nodegetr.append)rr5hf_valhf_key must_matchconfigds_keyds_vals r fill_matchz#HfTrainerDeepSpeedConfig.fill_matchis..{;; > F ::f   ' '#F6N F  FF##  &F"2"2 O " "#Y;#Y#Y#Y#Y#Y#YQW#Y#Y Z Z Z Z Z  "2"2r"F)r?c|j|jz|jz}|d|jd| |d|jd|d|d| |d|jd|d|jd |d |j|jgd |d |jd |d|j d| dd|d|jd |j s|j r|j dkrdnd}nd}|jr8|jdi|jd<|j|jdd<|d|j s|j o|dkd|d|dkd|d|jd|d|jp|jd|drt*j|_dS|drt*j|_dSt*j|_dS) z Adjust the config with `TrainingArguments` values. This stage is run during `TrainingArguments` object creation. train_micro_batch_size_per_gpuper_device_train_batch_sizegradient_accumulation_stepstrain_batch_sizeztrain_batch_size (calculated)gradient_clipping max_grad_normzoptimizer.params.lr learning_ratezoptimizer.params.betaszadam_beta1+adam_beta2zoptimizer.params.eps adam_epsilonzoptimizer.params.weight_decay weight_decayzscheduler.params.warmup_min_lrrzscheduler.params.warmup_max_lrapexampN checkpointuse_node_local_storagez fp16.enabledz%fp16|fp16_full_eval+fp16_backend(amp)z amp.enabledzfp16+fp16_backend(apex)z amp.opt_levelfp16_opt_levelz bf16.enabledzbf16|bf16_full_eval) world_sizerFrGrCrJrK adam_beta1 adam_beta2rLrM fill_onlyfp16fp16_full_eval fp16_backendsave_on_each_noder@r;rRbf16bf16_full_evalis_truetorchbfloat16r-is_falsefloat32float16)rargsauto_find_batch_sizerHrYs rtrainer_config_processz/HfTrainerDeepSpeedConfig.trainer_config_processs ?T-MMPTPpp  ,  , )$ $     )  , )      +$ $    +T-?QQQ -t/A?SSS  $ _do . #   .0A>RRR 79JN[[[ 7;;; 8$:Lo^^^ 9 + %)%6&%@%@66eLLL  ! Y(,  b(I(IDK %BFBXDK %&> ?  i.4. ILE4I 3     |v'=?XYYY )<>NOOO )Id6ILabbb << ' ' (.DKKK ]]> * * (-DKKK-DKKKr"cJgd}fd|D}t|dkrud}t|drt|jdr |jj}nt|jdrt |jj}nt|jdr,t|jjdr|jjj}nMt|jdr8t|jjdrt |jjj}|td |d d ||z rBd td |z|zdd|z d|d d| |dtj dkr-dj }td|ddS)z This stage is run after we have the model and know num_training_steps. Now we can complete the configuration process. )$zero_optimization.reduce_bucket_size-zero_optimization.stage3_prefetch_bucket_size4zero_optimization.stage3_param_persistence_thresholdc>g|]}||S)r7).0xrs r zDHfTrainerDeepSpeedConfig.trainer_config_finalize..s) V V VqdllSToo V V V Vr"rNr@ hidden_size hidden_sizes text_configzThe model's config file has neither `hidden_size` nor `hidden_sizes` entry, therefore it's not possible to automatically fill out the following `auto` entries in the DeepSpeed config file: zb. You can fix that by replacing `auto` values for these keys with an integer value of your choice.rgrhg?ri z scheduler.params.total_num_stepsznum_training_steps (calculated)z!scheduler.params.warmup_num_steps warmup_steps z]Please correct the following DeepSpeed config values that mismatch TrainingArguments values: zF The easiest method is to set these DeepSpeed config values to 'auto'.)lenhasattrr@romaxrprqr0rVis_zero3intrCget_warmup_stepsr.join)rrcmodelnum_training_stepshidden_size_based_keyshidden_size_auto_keysror.s` rtrainer_config_finalizez0HfTrainerDeepSpeedConfig.trainer_config_finalizes" " "  !W V V V,B V V V $ % % ) )Kuh'' M5<77 M"',":KKU\>::M"%el&?"@"@KKU\=99MgelF^`m>n>nM"',":"FKKU\=99MgelF^`n>o>oM"%el&>&K"L"LK" Y5JYYY NNA;Q\C\ ] ] ]}} Ck)K788J$  .  -    /  ! !"4 5 5    t  ! # #4?33Jq'qqq  $ #r")NTF) r#r$r%r&rr1r7rCrrVrerr'r(s@rr*r*Rs   !!![[[[4 jU;;;IH(H(H(H(TCCCCCCCr"r*c.tj|adSr,)weakrefref_hf_deepspeed_config_weak_ref)hf_deepspeed_config_objs rrrs %,K0G$H$H!!!r"c dadSr,)rrkr"runset_hf_deepspeed_configr!s%)!!!r"cpt.t tSdS)NF)rrxrkr"ris_deepspeed_zero3_enabledr's1$05R5T5T5`,..77999ur"cVt!ttjSdSr,)rr@rkr"rdeepspeed_configr.s($05R5T5T5`,..55tr"ct|dd|}|_gddtjffd ||dS) z Loads state dict into a model specifically for Zero3, since DeepSpeed does not support the `transformers` tensor parallelism API. Nearly identical code to PyTorch's `_load_from_state_dict` _metadataNFmodulec in ddi}||d<||dgg f}trtfd|Ddkrddl}t |ddd fd|D}t|dkr`|j|d 5tj dkr |j |dddn #1swxYwY|j D]\}} |  | ||zd z|dS) Nassign_to_params_buffersTc>g|]}||Srk) startswith)rlkeyprefixs rrnzC_load_state_dict_into_zero3_model..load..Ms+0e0e0ecnn]cNdNd0e0e0e0er"rF)rrecursec(g|]}|v|Srkrk)rlknamed_parameterss rrnzC_load_state_dict_into_zero3_model..load..Ss*aaa1P`K`K` 0 3K`K`K`r") modifier_rank.)r;rrur dictrzeroGatheredParametersr^ distributedget_rank_load_from_state_dict_modulesitems)r state_dictrrlocal_metadatarcr params_to_gathernamechildr error_msgsloadrs ` @rrz/_load_state_dict_into_zero3_model..loadFs'/X\\&"+r5R5R5M12FND"b*M & ' ' ._lr_scheduler_callablesC#y11 -1 )+<<'9Y =   $#r")lr_scheduler_callabler) accelerate.utilsrrr@optimr0 is_offloadloggerinfocreate_optimizer isinstancer) rhf_deepspeed_configrcr}model_parametersrrr@rrrs ` ` rdeepspeed_optim_schedresQ<;;;;;;; 'FIf : $ $8 J&6777  ) ) + +  KKV   ,,.. 26./Lf%~i00 i , , p $ $ $ $ $ $*>)KabbbLL"33GYen3ooL l ""r"Fcddlm}|j}|j}|jjjj}||||| | |rU| std| d| dd\}}d} nd|_|jdid d } | d kr4ddl} | || ||j }t)t+d |} t/||||| \}}||fS) a Init DeepSpeed, after updating the DeepSpeed configuration with any relevant Trainer's args. If `resume_from_checkpoint` was passed then an attempt to resume from a previously saved checkpoint will be made. Args: trainer: Trainer object num_training_steps: per single gpu resume_from_checkpoint: path to a checkpoint if to resume from after normal DeepSpeedEngine load inference: launch in inference mode (no optimizer and no lr scheduler) auto_find_batch_size: whether to ignore the `train_micro_batch_size_per_gpu` argument as it's being set automatically by the auto batch size finder Returns: optimizer, lr_scheduler We may use `deepspeed_init` more than once during the life of Trainer, when we do - it's a temp hack based on: https://github.com/deepspeedai/DeepSpeed/issues/1394#issuecomment-937405374 until Deepspeed fixes a bug where it can't resume from a checkpoint after it did some stepping https://github.com/deepspeedai/DeepSpeed/issues/1612 r)rzMZeRO inference only makes sense with ZeRO Stage 3 - please adjust your configrr)NNNtensor_parallel autotp_size)r|tp_sizer1r@c|jSr,) requires_grad)ps rz deepspeed_init..sr")deepspeed.utilsrr|rc acceleratorstatedeepspeed_plugin hf_ds_configrsetLevelget_process_log_levelrxr0del_config_sub_treerr@r;r tp_model_initr1listfilter parametersr) rr} inference ds_loggerr|rcrrrrdeepspeed_tp_sizer s rdeepspeed_initrs*433333 ME rs# //////######999999HHHHHHHHHHLLL  H % %   3!7!7!9!93OOOOOOO322222........2@@@@@0@@@H!%III))) ---`:#:#:#z@#@#@#@#FPPPPPPr"