&`i%K6ddlZddlmZddlmZmZmZmZmZm Z ddl Z ddl Z ddlmZddlmZddlmZddlmZddlmZmZdd lmZerdd lmZd ZeZerdd lm Z dd l!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'ej(e)Z*dedee+e e,e-e.ffdefdZ/de j0de j0de-de j0de,de1de j2fdZ3de j4dee+ee5e5ffde-dededee+ee5e5ffdeee j4ffdZ6GddeZ7dS)N)deepcopy) TYPE_CHECKINGCallableDictOptionalTupleUnion) TuneError)Trial)PopulationBasedTraining)_PBTTrialState) flatten_dictunflatten_dict)log_once)TuneControllerc6 ddl}n#t$rd}YnwxYw|S)Nr)sklearn ImportError)rs k/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/tune/schedulers/pb2.pyimport_pb2_dependenciesrs;  Ns  )GaussianProcessRegressor)UCB TV_SquaredExp normalize optimize_acq select_length standardizeconfighyperparam_boundsreturnci}|D]\}}t|tr#||vri||<t|||||<=t|tt frv||vrrt |dzrtd|dt|dksJ|\}}tj ||x||<||<|S)ajFills missing hyperparameters in config by sampling uniformly from the specified `hyperparam_bounds`. Recursively fills the config if `hyperparam_bounds` is a nested dict. This is a helper used to set initial hyperparameter values if the user doesn't specify them in the Tuner `param_space`. Returns the dict of filled hyperparameters. z-missingz Cannot find zU in config. Initializing by sampling uniformly from the provided `hyperparam_bounds`.) items isinstancedict _fill_configlisttuplerloggerdebuglennprandomuniform)rrfilled_hyperparams param_nameboundslowhighs rr&r&,s./5577 F fd # # ''%'z"-9&:Lf-U-U z * * u . . :V3K3K Z/00  P:PPPv;;!####ICBD)BSBSTCC F: !3J!? Xrawyrawcurrentnewpointr1num_fct||||}|| dddf}|| d}tjt|j}|ddd|f}tjtj|dtj|df d|j d} tj| |fd} t|| } t| |j d} t||tddd} t| d d }|| | ng#tjj$rP| tj| j dd zz } t| d d }|| | YnwxYw|t)|}ntjfd t+|j dD}t||}tj||f}tj| |f}tj|j d}| dd}tj| |f}tddd}t|d d }|||t3t4|||}|tj|dtj|dz ztj|dz}|tj}|S)aSelects the next hyperparameter config to try. This function takes the formatted data, fits the GP model and optimizes the UCB acquisition function to select the next point. Args: Xraw: The un-normalized array of hyperparams, Time and Reward yraw: The un-normalized vector of reward changes. current: The hyperparams of trials currently running. This is important so we do not select the same config twice. If there is data here then we fit a second GP including it (with fake y labels). The GP variance doesn't depend on the y labels so it is ok. newpoint: The Reward and Time for the new point. We cannot change these as they are based on the *new weights*. bounds: Bounds for the hyperparameters. Used to normalize. num_f: The number of fixed params. Almost always 2 (reward+time) Return: xt: A vector of new hyperparameters. Nraxisr"g?g?)variance lengthscaleepsilon fmin_l_bfgs_bg|=)kernel optimizeralphagMbP?cg|]}SrF).0_fixeds r z"_select_config..sCCCaECCCr4)rr,arrayr'valuesT concatenatemaxminreshapeshaperrsizerrfitlinalg LinAlgErroreyerrangehstackvstackzerosrrastypefloat32)r5r6r7r8r1r9length base_vals oldpointsold_limslimitsXyrBmm1paddingXnewypadynewkernel1xtrIs @r_select_configrnLsV<4vu 5 5F !!! D >Dfmmoo..//1IQQQYI~   " " "BF91$=$=$=> ga#$$ ^Xy1 : : :F$AD!!$)Q//A h * *E CS# F F FF  $_E    a 9  RVAGAJ  $ && $_E    a  a[[(CCCC5q1A+B+BCCCDDGY//)Wg.//y!W&&x a())||B""y!T###sKKK %oU    tT c1b% / /B rvia(((26)!+D+D+DD EIII B 2:  B Is(E--A!GGdatabaseoldc |dd}|dgt|zd|d<|dgt|zd|d<||dd kd}|j|jz |d <|j|jz |d<||j d}|dd}|j d d d d fd}||dt|k}|j stj|jj}|dd g} ||} t!j| | gd j} ||dt|kj dd d fdd gj} t%| ||| |t'| j} |}g}t-| jD]R\}}t/||}|| |||<||| |S||dt|kj dd d fd}||dt|kj dd d fj}t|g|gz|z|gzg}ddgt|zdgz}t!j||}t!j||gd}n|}||fS)a|Returns next hyperparameter configuration to use. This function primarily processes the data from completed trials and then requests the next config from the select_config function. It then adds the new trial to the dataframe, so that the reward change can be computed using the new weights. It returns the new point and the dataframe with the new entry. Time)byTdropr Rewardret_changerR_beforeiNr=r;rK)r9columns) sort_values reset_indexgroupbyr'keysdiffrwrerxisnailocstremptyr,rLrMpdconcatrnr+r{copy enumeratetypeappend DataFrame)ror1r7rprqrdf dfnewpointret_rhparamsrdr8new new_configrMicoltype_new_T new_Rewardlstcols new_entrys r_explorers"   V  $ $ 0 0d 0 ; ;Bjj'T&++--%8%8899(CHHJJBsGZZ D,?,? ?@@HMMOOBzN BzNQ   + + + 6 6BY%BzNdR[ BsG RTYY[[L  % %4 % 0 0B 6 " " . .D . 9 9B   ' 'T ' 2 2BBwK3t99,-J  !# HRT[ ! !&*%&V[[]]# IsGn1 - - - 4bkSYY./4RU;VZ)`. In {LOG_DIR}/{MY_EXPERIMENT_NAME}/, all mutations are logged in `pb2_global.txt` and individual policy perturbations are recorded in pb2_policy_{i}.txt. Tune logs: [target trial tag, clone trial tag, target trial iteration, clone trial iteration, old config, new config] on each perturbation step. Args: time_attr: The training result attr to use for comparing time. Note that you can pass in something non-temporal such as `training_iteration` as a measure of progress, the only requirement is that the attribute should increase monotonically. metric: The training result objective value attribute. Stopping procedures will use this attribute. mode: One of {min, max}. Determines whether objective is minimizing or maximizing the metric attribute. perturbation_interval: Models will be considered for perturbation at this interval of `time_attr`. Note that perturbation incurs checkpoint overhead, so you shouldn't set this to be too frequent. hyperparam_bounds: Hyperparameters to mutate. The format is as follows: for each key, enter a list of the form [min, max] representing the minimum and maximum possible hyperparam values. A key can also hold a dict for nested hyperparameters. Tune will sample uniformly between the bounds provided by `hyperparam_bounds` for the initial hyperparameter values if the corresponding hyperparameters are not present in a trial's initial `config`. quantile_fraction: Parameters are transferred from the top `quantile_fraction` fraction of trials to the bottom `quantile_fraction` fraction. Needs to be between 0 and 0.5. Setting it to 0 essentially implies doing no exploitation at all. custom_explore_fn: You can also specify a custom exploration function. This function is invoked as `f(config)`, where the input is the new config generated by Bayesian Optimization. This function should return the `config` updated as needed. log_config: Whether to log the ray config of each model to local_dir at each exploit. Allows config schedule to be reconstructed. require_attrs: Whether to require time_attr and metric to appear in result for every iteration. If True, error will be raised if these values are not present in trial result. synch: If False, will use asynchronous implementation of PBT. Trial perturbations occur every perturbation_interval for each trial independently. If True, will use synchronous implementation of PBT. Perturbations will occur only after all trials are synced at the same time_attr every perturbation_interval. Defaults to False. See Appendix A.1 here https://arxiv.org/pdf/1711.09846.pdf. Example: .. code-block:: python from ray import tune from ray.tune.schedulers.pb2 import PB2 from ray.tune.examples.pbt_function import pbt_function pb2 = PB2( metric="mean_accuracy", mode="max", perturbation_interval=20, hyperparam_bounds={"lr": [0.0001, 0.1]}, ) tuner = tune.Tuner( pbt_function, tune_config=tune.TuneConfig( scheduler=pb2, num_samples=8, ), param_space={"lr": 0.0001}, ) tuner.fit() time_total_sNN@?TF time_attrmetricmodeperturbation_intervalrquantile_fraction log_config require_attrssynchcustom_explore_fnc t} | std|pi}|stdtt|||||||d| |||  d|_tj|_ ||_ t|d|_ | |j d|_dS)Nz'Please install scikit-learn to use PB2.z;`hyperparam_bounds` must be specified to use PB2 scheduler.r) rrrrhyperparam_mutationsrresample_probabilityrrrrT)prevent_delimiter)r RuntimeErrorr superr__init__last_exploration_timerrro_hyperparam_boundsr_hyperparam_bounds_flat_validate_hyperparam_boundsr7) selfrrrrrrrrrrsklearn_available __class__s rrz PB2.__init__Rs455  JHII I-3  M  c4!!"7!2/!"/!' " &'"LNN "3'3 ( ( ( $ (()EFFF  r4tune_controllerrtrialct|j|j}|jt |t ||dS)N)r&rrevaluated_paramsupdaterr on_trial_add)rrrr/rs rrzPB2.on_trial_addsX)%,8OPP %%l3E&F&FGGG _e44444r4c|D]k\}}t|ttfrt |dkrt d|d|d|\}}||krt d|d|dldS) zCheck that each hyperparam bound is of the form [low, high]. Raises: ValueError: if any of the hyperparam bounds are of an invalid format. r"zM`hyperparam_bounds` values must either be a list or tuple of size 2, but got z instead for the param ''zV`hyperparam_bounds` values must be of the form [low, high] where low <= high, but got z instead for param 'z'.N)r#r$r'r(r+ ValueError)rrkeyvaluer2r3s rrzPB2._validate_hyperparam_boundss ,1133  JCedE]33 s5zzQ 5:?55.1555 ICTzz U27UUMPUUU  r4statetimeresultc tt|||||}t|j}t |j fd|D}|||jg|z|gzg}ddg|zdgz} tj || } tj |j | g d|_ |j jd|j _dS) Nc g|] }| SrFrF)rGrflattened_configs rrJz)PB2._save_trial_state..s999C"3'999r4r rsrwrzTrur)rr_save_trial_stater'rrrr _time_attrrrrror}r r]) rrrrrscorenamesrMrrentryrrs @rrzPB2._save_trial_statesc4  225$NN T1668899' 5599995999vdo./&8E7BC 5(H:5 S$///Ity%011==4=HH )/0077 r4trial_to_cloner c L|jd|jkrd|_t |j|j|j||t |j\}||_fd|jD}tj |}| d|j }|jd|jkr>|jd|_||_nAtj |j|fd|_t|jt!}|jr!||}| Jd|ifS)aGets new config for trial by exploring trial_to_clone's config using Bayesian Optimization (BO) to choose the hyperparameter values to explore. Overrides `PopulationBasedTraining._get_new_config`. Args: trial: The current trial that decided to exploit trial_to_clone. trial_to_clone: The top-performing trial with a hyperparameter config that the current trial will explore. Returns: new_config: New hyperparameter configuration (after BO). operations: Empty dict since PB2 doesn't explore in easily labeled ways like PBT does. rsNc g|] }| SrFrF)rGrnew_config_flats rrJz'PB2._get_new_config..sLLLs#LLLr4r=rr;z5Custom explore function failed to return a new config)rorPrr7rrrrrr,rLrRrTrOr)r*r_custom_explore_fn)rrrrorrrs @r_get_new_configzPB2._get_new_configs$ 9V  " "T%? ? ?DL ( I  ( L   . / / ! ! IIKK MLLLt/KLLLhsmmkk!SX&& 9V  " "T%? ? ?)-6):)>)>)@)@D &88::DLL>4<*=AFFFDL LL & & &#O44  " G00<B333 3sm 3 % 3  U4u+<%= => 3!3333$HdVT\$:;333333j5,<5U555555 T(8#8+.88<8EJ888888*7U7E7eDRVJFW77777777r4r)8loggingrrtypingrrrrrr numpyr,pandasrray.tuner ray.tune.experimentr ray.tune.schedulersr ray.tune.schedulers.pbtr ray.tune.utils.utilrrray.util.debugr"ray.tune.execution.tune_controllerrr has_sklearnsklearn.gaussian_processrray.tune.schedulers.pb2_utilsrrrrrr getLoggerrr)rr%r'r(r&rLrndarrayrnrrrrrFr4rrsHHHHHHHHHHHHHHHH%%%%%%777777222222<<<<<<<<######BAAAAAA&%''  AAAAAA  8 $ $ %)#uT45F/G*G%H @Y (Y (YYh Y  Y  YZYYYYxK ,K eE5L)) *KK  K  K eE5L)) * K 4 KKKK\wwwww !wwwwwr4