Pi"ddZddlZGddZGddeZGddeZdS) z'Stochastic optimization methods for MLPNc,eZdZdZddZdZdZdZdS) BaseOptimizera9Base (Stochastic) gradient descent optimizer Parameters ---------- learning_rate_init : float, default=0.1 The initial learning rate used. It controls the step-size in updating the weights Attributes ---------- learning_rate : float the current learning rate 皙?c<||_t||_dSN)learning_rate_initfloat learning_rate)selfrs /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/sklearn/neural_network/_stochastic_optimizers.py__init__zBaseOptimizer.__init__s!"4"#566cz||}td|D|D] \}}||z } dS)aUpdate parameters with given gradients Parameters ---------- params : list of length = len(coefs_) + len(intercepts_) The concatenated list containing coefs_ and intercepts_ in MLP model. Used for initializing velocities and updating params grads : list of length = len(params) Containing gradients with respect to coefs_ and intercepts_ in MLP model. So length should be aligned with params c3K|]}|VdSr).0ps r z.BaseOptimizer.update_params..*s"!4!4!!4!4!4!4!4!4rN) _get_updateszip)r paramsgradsupdatesparamupdates r update_paramszBaseOptimizer.update_paramssX##E** !4!4V!4!4!4g>>  ME6 VOEE  rcdS)zhPerform update to learning rate and potentially other states at the end of an iteration Nrr time_steps r iteration_endszBaseOptimizer.iteration_ends-s  rc.|rt|dzdS)aHDecides whether it is time to stop training Parameters ---------- msg : str Message passed in for verbose output verbose : bool Print message to stdin if True Returns ------- is_stopping : bool True if training needs to stop Stopping.T)printr msgverboses r trigger_stoppingzBaseOptimizer.trigger_stopping3s$  & # $ % % %trN)r)__name__ __module__ __qualname____doc__r rr r'rrr rr s_  7777"   rrc@eZdZdZ d fd ZdZd Zd ZxZS) SGDOptimizeraStochastic gradient descent optimizer with momentum Parameters ---------- params : list, length = len(coefs_) + len(intercepts_) The concatenated list containing coefs_ and intercepts_ in MLP model. Used for initializing velocities and updating params learning_rate_init : float, default=0.1 The initial learning rate used. It controls the step-size in updating the weights lr_schedule : {'constant', 'adaptive', 'invscaling'}, default='constant' Learning rate schedule for weight updates. -'constant', is a constant learning rate given by 'learning_rate_init'. -'invscaling' gradually decreases the learning rate 'learning_rate_' at each time step 't' using an inverse scaling exponent of 'power_t'. learning_rate_ = learning_rate_init / pow(t, power_t) -'adaptive', keeps the learning rate constant to 'learning_rate_init' as long as the training keeps decreasing. Each time 2 consecutive epochs fail to decrease the training loss by tol, or fail to increase validation score by tol if 'early_stopping' is on, the current learning rate is divided by 5. momentum : float, default=0.9 Value of momentum used, must be larger than or equal to 0 nesterov : bool, default=True Whether to use nesterov's momentum or not. Use nesterov's if True power_t : float, default=0.5 Power of time step 't' in inverse scaling. See `lr_schedule` for more details. Attributes ---------- learning_rate : float the current learning rate velocities : list, length = len(params) velocities that are used to update params rconstant?T?ct|||_||_||_||_d|D|_dS)Nc6g|]}tj|Srnp zeros_likerrs r z)SGDOptimizer.__init__..s"DDDE2=//DDDr)superr lr_schedulemomentumnesterovpower_t velocities)r rrr9r:r;r< __class__s r r zSGDOptimizer.__init__xsT +,,,&     DDVDDDrcn|jdkr)t|j|dz|jzz |_dSdS)aPerform updates to learning rate and potential other states at the end of an iteration Parameters ---------- time_step : int number of training samples trained on so far, used to update learning rate for 'invscaling' invscalingN)r9r rr<r rs r r zSGDOptimizer.iteration_endssF  | + +d-..)a-DL1PP     , +rc|jdkr|rt|dzdS|jdkr|rt|dzdS|xjdzc_|rt|d|jzzdS) Nadaptiver"Tgư>z# Learning rate too small. Stopping.g@z Setting learning rate to %fF)r9r#r r$s r r'zSGDOptimizer.trigger_stoppings  z ) ) *cL()))4   % % CcAABBB4 c!  M #69KKK L L Lurcfdtj|D}|_jr!fdtj|D}|S)Get the values used to update params with given gradients Parameters ---------- grads : list, length = len(coefs_) + len(intercepts_) Containing gradients with respect to coefs_ and intercepts_ in MLP model. So length should be aligned with params Returns ------- updates : list, length = len(grads) The values to add to params c@g|]\}}j|zj|zz Srr:r rvelocitygradr s r r7z-SGDOptimizer._get_updates..sA   $ MH $t'9D'@ @   rc@g|]\}}j|zj|zz SrrGrHs r r7z-SGDOptimizer._get_updates..sA"Hd (4+=+DDr)rr=r;r rrs` r rzSGDOptimizer._get_updatess    "%dou"="=   " = &)$/5&A&AG r)rr.r/Tr0) r(r)r*r+r r r'r __classcell__r>s@r r-r-Hs--dEEEEEE"    rr-c,eZdZdZ dfd ZdZxZS) AdamOptimizeraStochastic gradient descent optimizer with Adam Note: All default values are from the original Adam paper Parameters ---------- params : list, length = len(coefs_) + len(intercepts_) The concatenated list containing coefs_ and intercepts_ in MLP model. Used for initializing velocities and updating params learning_rate_init : float, default=0.001 The initial learning rate used. It controls the step-size in updating the weights beta_1 : float, default=0.9 Exponential decay rate for estimates of first moment vector, should be in [0, 1) beta_2 : float, default=0.999 Exponential decay rate for estimates of second moment vector, should be in [0, 1) epsilon : float, default=1e-8 Value for numerical stability Attributes ---------- learning_rate : float The current learning rate t : int Timestep ms : list, length = len(params) First moment vectors vs : list, length = len(params) Second moment vectors References ---------- :arxiv:`Kingma, Diederik, and Jimmy Ba (2014) "Adam: A method for stochastic optimization." <1412.6980> MbP?r/+?:0yE>ct|||_||_||_d|_d|D|_d|D|_dS)Nrc6g|]}tj|Srr3r6s r r7z*AdamOptimizer.__init__.."<<.rVr)r8r beta_1beta_2epsilontmsvs)r rrrXrYrZr>s r r zAdamOptimizer.__init__sh +,,,   <.sC   4 K!Oq4;$6 6   rcLg|] \}}j|zdjz |dzzz!S)rA)rY)rvrJr s r r7z.AdamOptimizer._get_updates..sG   4 K!Oq4;47; ;   rcfg|]-\}}j |ztj|jzz .Sr)r r4sqrtrZ)rr`rcr s r r7z.AdamOptimizer._get_updates..sL   1  ! #rwqzzDL'@ A   r) r[rr\r]rr4rerYrXr rLs` r rzAdamOptimizer._get_updatess !     tw..       tw..     #ga$+tv--.. /4;&& (      DGTW--   r)rQr/rRrS)r(r)r*r+r rrMrNs@r rPrPs^++\SW = = = = = =       rrP)r+numpyr4rr-rPrrr rgs-- <<<<<<<<~zzzzz=zzzzZZZZZMZZZZZr