`i_ d Z ddlZddlZddlZddlmZ ddlmZmZ ddlZ ddl
m Z ddl m
Z
mZmZmZ ddlmZ dd lmZ ej e Ze G d
d Z G d d
Ze G d de Z G d d Z G d de Z G d de Z G d de Z G d de Z G d dee Z dS ) zJ
Callbacks to use with the Trainer class and customize the training loop.
N) dataclass)OptionalUnion)tqdm )HPSearchBackendIntervalStrategy SaveStrategy
has_length)TrainingArguments)loggingc e Zd ZU dZdZee ed< dZe ed< dZ
e ed< dZ e ed< dZ e ed < dZ
e ed
< dZee ed < dZe ed < dZe ed
< dZeed< dZeeeef ed< dZee ed< dZee ed< dZee ed< dZeed< dZeed< dZ eed< dZ ee ed< dZ eeee eee ef f ed< dZ eed ed< d Z!d efd Z"e#d efd Z$d Z%d! Z&dS )" TrainerStatea
A class containing the [`Trainer`] inner state that will be saved along the model and optimizer when checkpointing
and passed to the [`TrainerCallback`].
In all this class, one step is to be understood as one update step. When using gradient accumulation, one update
step may require several forward and backward passes: if you use `gradient_accumulation_steps=n`, then one update
step requires going through *n* batches.
Args:
epoch (`float`, *optional*):
Only set during training, will represent the epoch the training is at (the decimal part being the
percentage of the current epoch completed).
global_step (`int`, *optional*, defaults to 0):
During training, represents the number of update steps completed.
max_steps (`int`, *optional*, defaults to 0):
The number of update steps to do during the current training.
logging_steps (`int`, *optional*, defaults to 500):
Log every X updates steps
eval_steps (`int`, *optional*):
Run an evaluation every X steps.
save_steps (`int`, *optional*, defaults to 500):
Save checkpoint every X updates steps.
train_batch_size (`int`, *optional*):
The batch size for the training dataloader. Only needed when
`auto_find_batch_size` has been used.
num_input_tokens_seen (`int`, *optional*, defaults to 0):
When tracking the inputs tokens, the number of tokens seen during training (number of input tokens, not the
number of prediction tokens).
total_flos (`float`, *optional*, defaults to 0):
The total number of floating operations done by the model since the beginning of training (stored as floats
to avoid overflow).
log_history (`list[dict[str, float]]`, *optional*):
The list of logs done since the beginning of training.
best_metric (`float`, *optional*):
When tracking the best model, the value of the best metric encountered so far.
best_global_step (`int`, *optional*):
When tracking the best model, the step at which the best metric was encountered.
Used for setting `best_model_checkpoint`.
best_model_checkpoint (`str`, *optional*):
When tracking the best model, the value of the name of the checkpoint for the best model encountered so
far.
is_local_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on
several machines) main process.
is_world_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the global main process (when training in a distributed fashion on several
machines, this is only going to be `True` for one process).
is_hyper_param_search (`bool`, *optional*, defaults to `False`):
Whether we are in the process of a hyper parameter search using Trainer.hyperparameter_search. This will
impact the way data will be logged in TensorBoard.
stateful_callbacks (`list[StatefulTrainerCallback]`, *optional*):
Callbacks attached to the `Trainer` that should have their states be saved or restored.
Relevant callbacks should implement a `state` and `from_state` function.
Nepochr global_step max_stepsi
logging_steps
eval_steps
save_stepstrain_batch_sizenum_train_epochsnum_input_tokens_seen
total_flos log_history best_metricbest_global_stepbest_model_checkpointTis_local_process_zerois_world_process_zeroFis_hyper_param_search
trial_name trial_paramsTrainerCallbackstateful_callbacksc | j g | _ | j i | _ d S t | j t rd S i }| j D ]}t |t s t dt
| |j j }||v rUt || t s || g||< ||
| | ||< || _ d S )NzNAll callbacks passed to be saved must inherit `ExportableState`, but received ) r r$
isinstancedictExportableState TypeErrortype __class____name__listappendstate)selfr$ callbacknames q/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/trainer_callback.py
__post_init__zTrainerState.__post_init__u s, # !D " *&(D # # #
/
6
6 9 D "$ 3
@
@!(_>> #yimnviwiwyy )2--- &&8&>EE N4Ft4L3M*40&t,33HNN4D4DEEEE/7~~/?/?&t,,&8D # # # json_pathc t j t j | dd dz }t |dd 5 }| | ddd dS # 1 swxY w Y dS ) zDSave the content of this instance in JSON format inside `json_path`. T)indent sort_keys
wutf-8encodingN)jsondumps dataclassesasdictopenwrite)r0 r6 json_stringfs r3 save_to_jsonzTrainerState.save_to_json s j!3D!9!9!tTTTW[[
)S7
3
3
3 !q
GGK ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !s A""A&)A&c t |d 5 }| }ddd n # 1 swxY w Y | di t j | S )z3Create an instance from the content of `json_path`.r= r> N )rD readr@ loads)clsr6 rG texts r3 load_from_jsonzTrainerState.load_from_json s )g
.
.
. !6688D s&&TZ%%&&&s 377c dD ]H}t || d }|1|dk rt j ||z }t | | d| IdS )z
Calculates and stores the absolute value for logging,
eval, and save steps based on if it was a proportion
or not.
)r
evalsave_stepsNr )getattrmathceilsetattr)r0 argsr step_kind num_stepss r3
compute_stepszTrainerState.compute_steps st 5 ? ?I &:&:&:;;I$ q== $ )i*? @ @I 2 2 2I>>> ? ?r5 c Z |j &|j | |j | _ d| _ |4ddlm} |j t j k r|j n|} || | _ || _
|| _ | | _ |
| _
dS )zI
Stores the initial training references needed in `self`
Nr ) hp_params)hp_name_trialr! r" transformers.integrationsr] hp_search_backendr SIGOPT assignmentsr r r r )r0 trainerr r trialr] rc s r3 init_training_referencesz%TrainerState.init_training_references s ? &7>+E &oogn = =DO ; ; ; ; ; ;/6/HOLb/b/b%++hmK ) + 6 6D " 0 %,%B%B%D%D"%,%B%B%D%D"""r5 )'r,
__module__ __qualname____doc__r r float__annotations__r intr r r r r r r r r r- r' strr r r r boolr r r! r" r r$ r4 rH classmethodrO r[ rf rJ r5 r3 r r # s. 9 9v "E8E?!!!KIs M3 JJ&*hsm*** c !"3"""J*.Kd3:&'...#'K% '''&*hsm***+/8C=///"&4&&&"&4&&&"'4''' $J
$$$FJL(4U3sD+@%A ABCJJJ<@&7!89@@@9 9 96!c ! ! ! ! 's ' ' ' [' ? ? ?E E E E Er5 r c 4 e Zd ZdZdefdZed ZdS )r( aj
A class for objects that include the ability to have its state
be saved during `Trainer._save_checkpoint` and loaded back in during
`Trainer._load_from_checkpoint`.
These must implement a `state` function that gets called during the respective
Trainer function call. It should only include parameters and attributes needed to
recreate the state at a particular time, to avoid utilizing pickle/maintain standard
file IO writing.
Example:
```python
class EarlyStoppingCallback(TrainerCallback, ExportableState):
def __init__(self, early_stopping_patience: int = 1, early_stopping_threshold: Optional[float] = 0.0):
self.early_stopping_patience = early_stopping_patience
self.early_stopping_threshold = early_stopping_threshold
# early_stopping_patience_counter denotes the number of times validation metrics failed to improve.
self.early_stopping_patience_counter = 0
def state(self) -> dict:
return {
"args": {
"early_stopping_patience": self.early_stopping_patience,
"early_stopping_threshold": self.early_stopping_threshold,
},
"attributes": {
"early_stopping_patience_counter": self.early_stopping_patience_counter,
}
}
```returnc t d )Nz e Zd ZdZdededefdZdededefdZdededefdZ dededefdZ
dededefd Z dededefd
Z dededefd Z
dededefd Zdededefd
ZdededefdZdededefdZdededefdZdededefdZdededefdZdededefdZdS )r# a
A class for objects that will inspect the state of the training loop at some events and take some decisions. At
each of those events the following arguments are available:
Args:
args ([`TrainingArguments`]):
The training arguments used to instantiate the [`Trainer`].
state ([`TrainerState`]):
The current state of the [`Trainer`].
control ([`TrainerControl`]):
The object that is returned to the [`Trainer`] and can be used to make some decisions.
model ([`PreTrainedModel`] or `torch.nn.Module`):
The model being trained.
tokenizer ([`PreTrainedTokenizer`]):
The tokenizer used for encoding the data. This is deprecated in favour of `processing_class`.
processing_class ([`PreTrainedTokenizer` or `BaseImageProcessor` or `ProcessorMixin` or `FeatureExtractionMixin`]):
The processing class used for encoding the data. Can be a tokenizer, a processor, an image processor or a feature extractor.
optimizer (`torch.optim.Optimizer`):
The optimizer used for the training steps.
lr_scheduler (`torch.optim.lr_scheduler.LambdaLR`):
The scheduler used for setting the learning rate.
train_dataloader (`torch.utils.data.DataLoader`, *optional*):
The current dataloader used for training.
eval_dataloader (`torch.utils.data.DataLoader`, *optional*):
The current dataloader used for evaluation.
metrics (`dict[str, float]`):
The metrics computed by the last evaluation phase.
Those are only accessible in the event `on_evaluate`.
logs (`dict[str, float]`):
The values to log.
Those are only accessible in the event `on_log`.
The `control` object is the only one that can be changed by the callback, in which case the event that changes it
should return the modified version.
The argument `args`, `state` and `control` are positionals for all events, all the others are grouped in `kwargs`.
You can unpack the ones you need in the signature of the event using them. As an example, see the code of the
simple [`~transformers.PrinterCallback`].
Example:
```python
class PrinterCallback(TrainerCallback):
def on_log(self, args, state, control, logs=None, **kwargs):
_ = logs.pop("total_flos", None)
if state.is_local_process_zero:
print(logs)
```rX r/ controlc dS )zS
Event called at the end of the initialization of the [`Trainer`].
NrJ r0 rX r/ r kwargss r3 on_init_endzTrainerCallback.on_init_end^
r5 c dS )z<
Event called at the beginning of training.
NrJ r s r3 on_train_beginz TrainerCallback.on_train_begind r r5 c dS )z6
Event called at the end of training.
NrJ r s r3 on_train_endz TrainerCallback.on_train_endj r r5 c dS )z<
Event called at the beginning of an epoch.
NrJ r s r3 on_epoch_beginz TrainerCallback.on_epoch_beginp r r5 c dS )z6
Event called at the end of an epoch.
NrJ r s r3 on_epoch_endz TrainerCallback.on_epoch_endv r r5 c dS )z
Event called at the beginning of a training step. If using gradient accumulation, one training step might take
several inputs.
NrJ r s r3
on_step_beginz TrainerCallback.on_step_begin|
r5 c dS )zv
Event called before the optimizer step but after gradient clipping. Useful for monitoring gradients.
NrJ r s r3 on_pre_optimizer_stepz%TrainerCallback.on_pre_optimizer_step r r5 c dS )z}
Event called after the optimizer step but before gradients are zeroed out. Useful for monitoring gradients.
NrJ r s r3 on_optimizer_stepz!TrainerCallback.on_optimizer_step r r5 c dS )zU
Event called at the end of an substep during gradient accumulation.
NrJ r s r3 on_substep_endz TrainerCallback.on_substep_end r r5 c dS )z
Event called at the end of a training step. If using gradient accumulation, one training step might take
several inputs.
NrJ r s r3 on_step_endzTrainerCallback.on_step_end r r5 c dS )z9
Event called after an evaluation phase.
NrJ r s r3 on_evaluatezTrainerCallback.on_evaluate r r5 c dS )z=
Event called after a successful prediction.
NrJ )r0 rX r/ r metricsr s r3
on_predictzTrainerCallback.on_predict r r5 c dS )z7
Event called after a checkpoint save.
NrJ r s r3 on_savezTrainerCallback.on_save r r5 c dS )z;
Event called after logging the last logs.
NrJ r s r3 on_logzTrainerCallback.on_log r r5 c dS )z7
Event called after a prediction step.
NrJ r s r3 on_prediction_stepz"TrainerCallback.on_prediction_step r r5 N)r, rg rh ri r r r} r r r r r r r r r r r r r r r rJ r5 r3 r# r# ) s 1 1 f
1
,
Q_
#4
\
Tb
!2
<
R`
#4
\
Tb
!2
<
R`
"3
L
Sa
*;
L
[i
&7
We
#4
\
Tb
1
,
Q_
1
,
Q_
0
P^
-
l
^
,
\
N
'8
Xf
r5 r# c r e Zd ZdZd Zd Zd Zd Zed Z de
de d e fd
Z
de
de d e fd Zde
de d e fd Zde
de d e fd
Zde
de d e fdZde
de d e fdZde
de d e fdZde
de d e fdZde
de d e fdZde
de d e fdZde
de d e fdZde
de d e fdZde
de d e fdZde
de d e fdZde
de d e fdZd Z dS )CallbackHandlerz>Internal class that just calls the list of callbacks in order.c g | _ |D ]}| | || _ || _ || _ || _ d | _ d | _ t d | j D s$t
d| j z d S d S )Nc 3 @ K | ]}t |t V d S N)r& DefaultFlowCallback.0cbs r3 z+CallbackHandler.__init__.. s- PP2:b"566PPPPPPr5 zThe Trainer will not work properly if you don't have a `DefaultFlowCallback` in its callbacks. You
should add one before training with `trainer.add_callback(DefaultFlowCallback). The current list ofcallbacks is
:) callbacks add_callbackmodelprocessing_class optimizer lr_schedulertrain_dataloadereval_dataloaderanyloggerwarning
callback_list)r0 r r r r r r s r3 __init__zCallbackHandler.__init__ s " "B b ! ! ! !
0 " ( $ # PP PPPPP NN$ $%
r5 c 0 t |t r
| n|}t |t r|n|j }|d | j D v r)t d| ddz | j z | j | d S )Nc g | ] }|j
S rJ )r+ )r cs r3
z0CallbackHandler.add_callback.. s <<< <<>N))"--- III "
r5 c t |t r9| j D ]/}t || r | j | d S 0d S | j | d S r r r s r3 remove_callbackz CallbackHandler.remove_callback s h % % ,n
b(++ N))"---FF
N ! !( + + + + +r5 c J d d | j D S )Nr; c 3 . K | ]}|j j V d S r )r+ r, r s r3 r z0CallbackHandler.callback_list.. s' HH2 .HHHHHHr5 )joinr rt s r3 r z CallbackHandler.callback_list s% yyHH HHHHHHr5 rX r/ r c 2 | d||| S )Nr
call_eventr0 rX r/ r s r3 r zCallbackHandler.on_init_end }dE7CCCr5 c @ d|_ | d||| S )NFr )r~ r r s r3 r z CallbackHandler.on_train_begin s# ',$ /ugFFFr5 c 2 | d||| S )Nr r r s r3 r z CallbackHandler.on_train_end ~tUGDDDr5 c @ d|_ | d||| S )NFr )r r r s r3 r z CallbackHandler.on_epoch_begin s# $)! /ugFFFr5 c 2 | d||| S )Nr r r s r3 r z CallbackHandler.on_epoch_end r r5 c \ d|_ d|_ d|_ | d||| S )NFr )r r r r r s r3 r z CallbackHandler.on_step_begin s2 ""' #eWEEEr5 c 2 | d||| S )Nr r r s r3 r z%CallbackHandler.on_pre_optimizer_step s 6eWMMMr5 c 2 | d||| S )Nr r r s r3 r z!CallbackHandler.on_optimizer_step s 2D%IIIr5 c 2 | d||| S )Nr r r s r3 r z CallbackHandler.on_substep_end s /ugFFFr5 c 2 | d||| S )Nr r r s r3 r zCallbackHandler.on_step_end r r5 c D d|_ | d|||| S )NFr r )r r r0 rX r/ r r s r3 r zCallbackHandler.on_evaluate s& "' }dE7GTTTr5 c 6 | d|||| S )Nr r r r s r3 r zCallbackHandler.on_predict s |T5'7SSSr5 c @ d|_ | d||| S )NFr )r r r s r3 r zCallbackHandler.on_save s" #y$w???r5 c D d|_ | d|||| S )NFr )logs)r r )r0 rX r/ r r s r3 r zCallbackHandler.on_log# s& "xugDIIIr5 c 2 | d||| S )Nr r r s r3 r z"CallbackHandler.on_prediction_step' s 3T5'JJJr5 c | j D ]D} t || |||f| j | j | j | j | j | j d|}||}E|S )N)r r r r r r )r rT r r r r r r )r0 eventrX r/ r r r1 results r3 r zCallbackHandler.call_event* s ! !H-WXu-- j!%!6 . ! .!%!6 $ 4 F ! r5 N) r, rg rh ri r r r r propertyr r r r} r r r r r r r r r r r r r r r r rJ r5 r3 r r so HH & " " "
, , , I I XID 1 D, DQ_ D D D DG#4 G\ GTb G G G GE!2 E< ER` E E E EG#4 G\ GTb G G G GE!2 E< ER` E E E EF"3 FL FSa F F F F N*; NL N[i N N N NJ&7 J JWe J J J JG#4 G\ GTb G G G GD 1 D, DQ_ D D D DU 1 U, UQ_ U U U UT 0 T TP^ T T T T@ - @l @^ @ @ @ @J, J\ JN J J J JK'8 K KXf K K K K r5 r c : e Zd ZdZdededefdZdededefdZdS )r zx
A [`TrainerCallback`] that handles the default flow of the training loop for logs, evaluation and checkpoints.
rX r/ r c |j dk r|j rd|_ |j t j k r|j |j z dk rd|_ |j t j k r*|j |j z dk r|j |j k rd|_
|j t j k r%|j
dk r|j |j
z dk rd|_ |j |j k r#d|_ |j t j k rd|_ |S )Nr Tr )r logging_first_stepr logging_strategyr STEPSr
eval_strategyr
eval_delayr
save_strategyr
r r r r~ r s r3 r z DefaultFlowCallback.on_step_endC s
! !d&= !!%G $4$: : :u?PSXSf?fjk?k?k!%G
"2"8 8 8!E$44995#444&*G #
,"4 4 4 1$$!E$4499"&G / /+/G (!\%777&*#r5 c |j t j k rd|_ |j t j k r|j |j k rd|_ |j t j k rd|_
|S )NT) r r EPOCHr r r r r r r
r r s r3 r z DefaultFlowCallback.on_epoch_endc sf $4$: : :!%G !1!7 7 7DOu{ su 1 , Q_ @
!2
<
R`
r5 r c N e Zd ZdZd
defdZd Zd ZddZd Z d
Z
dd Z d Z dS )ProgressCallbackz
A [`TrainerCallback`] that displays the progress of training or evaluation.
You can modify `max_str_len` to control how long strings are truncated when logging.
d max_str_lenc 0 d| _ d| _ || _ dS )a!
Initialize the callback with optional max_str_len parameter to control string truncation length.
Args:
max_str_len (`int`):
Maximum length of strings to display in logs.
Longer strings will be truncated with a message.
N) training_barprediction_barr )r0 r s r3 r zProgressCallback.__init__y s" ! "&r5 c X |j rt |j d | _ d| _ d S )NT)total
dynamic_ncolsr )r r r r current_stepr s r3 r z ProgressCallback.on_train_begin s3 & P $5?$ O O OD r5 c ~ |j r5| j |j | j z
|j | _ d S d S r )r r updater r r s r3 r z ProgressCallback.on_step_end sJ & 2 $ $U%69J%J K K K % 1D 2 2r5 Nc |j r]t | rP| j +t t | | j d u d | _ | j d d S d S d S )NT)r leaver r )r r r r lenr r )r0 rX r/ r r r s r3 r z#ProgressCallback.on_prediction_step s & *:o+F+F *"*&* o..d6G46O_c' ' '#
& &q ) ) ) ) ) * * * *r5 c f |j r)| j | j d | _ d S d S r r r closer s r3 r z ProgressCallback.on_evaluate C & '".#))+++"&D ' 'r5 c f |j r)| j | j d | _ d S d S r r
r s r3 r zProgressCallback.on_predict r r5 c |j r| j i }| D ]V\ }}t |t r7t | | j k r dt | d| j d||< Q|||< W| dd } d|v rt |d d |d< | j t | d S d S d S )Nz%[String too long to display, length: z > z/. Consider increasing `max_str_len` if needed.]r r r8 )
r r rw r& rm r r poproundrE )
r0 rX r/ r r r shallow_logsry rz _s
r3 r zProgressCallback.on_log s & 74+<+H L
(
(1 a%% (#a&&43C*C*CHA H H4K[ H H H !OO
'(LOO t44A,&&(-l7.CQ(G(G W% # #C $5$5 6 6 6 6 6! 7 7+H+Hr5 c X |j r"| j d | _ d S d S r )r r r r s r3 r z ProgressCallback.on_train_end s: & % # # % % % $D % %r5 )r r )
r, rg rh ri rl r r r r r r r r rJ r5 r3 r r s s
' 'C ' ' ' '
2 2 2
* * * *' ' ' ' ' ' 7 7 7 7&% % % % %r5 r c e Zd ZdZddZdS )PrinterCallbackz?
A bare [`TrainerCallback`] that just prints the logs.
Nc b | dd }|j rt | d S d S )Nr )r r print)r0 rX r/ r r r r s r3 r zPrinterCallback.on_log s9 HH\4 ( ( & $KKKKK r5 r )r, rg rh ri r rJ r5 r3 r r s2 r5 r c N e Zd ZdZd
dedee fdZd Zd Z d Z
d
e fd Z d S )EarlyStoppingCallbacka1
A [`TrainerCallback`] that handles early stopping.
Args:
early_stopping_patience (`int`):
Use with `metric_for_best_model` to stop training when the specified metric worsens for
`early_stopping_patience` evaluation calls.
early_stopping_threshold(`float`, *optional*):
Use with TrainingArguments `metric_for_best_model` and `early_stopping_patience` to denote how much the
specified metric must improve to satisfy early stopping conditions. `
This callback depends on [`TrainingArguments`] argument *load_best_model_at_end* functionality to set best_metric
in [`TrainerState`]. Note that if the [`TrainingArguments`] argument *save_steps* differs from *eval_steps*, the
early stopping will not occur until the next save step.
r early_stopping_patienceearly_stopping_thresholdc 0 || _ || _ d| _ d S )Nr r r early_stopping_patience_counter)r0 r r s r3 r z EarlyStoppingCallback.__init__ s '>$(@%/0,,,r5 c |j r t j n t j }|j 1 |||j r)t ||j z
| j k r d| _ d S | xj dz
c_ d S )Nr r )greater_is_betternpgreaterlessr absr r" )r0 rX r/ r metric_valueoperators r3 check_metric_valuez(EarlyStoppingCallback.check_metric_value s} !%!7D2::RW $ H\5#4 5 5 %L5#44558UUU34D 0 0 0 0 0A 5 0 0 0 0r5 c |j st d |j
J d |j t
j k s
J d d S )NzUsing EarlyStoppingCallback without load_best_model_at_end=True. Once training is finished, the best model will not be loaded automatically.zBEarlyStoppingCallback requires metric_for_best_model to be definedzAEarlyStoppingCallback requires IntervalStrategy of steps or epoch)load_best_model_at_endr r metric_for_best_modelr r NOr s r3 r z$EarlyStoppingCallback.on_train_begin sp * NN^
)55 P 655 !%5%8888 O 98888r5 c |j }| d sd| }| | }| t d| d d S | |||| | j | j k r d|_ d S d S )Neval_z@early stopping required metric_for_best_model, but did not find z so early stopping is disabledT) r.
startswithgetr r r+ r" r r~ )r0 rX r/ r r r metric_to_checkr) s r3 r z!EarlyStoppingCallback.on_evaluate s 4 ))'22 8 7o 7 7O {{?33 NN Sb
F eWlCCC /43O O O+/G ( ( ( P Or5 rq c 4 | j | j dd| j idS )N)r r r" r r! rt s r3 r/ zEarlyStoppingCallback.state s8 ,0+G,0,I
243W
r5 N)r r )
r, rg rh ri rl r rj r r+ r r r' r/ rJ r5 r3 r r s 1 1 1S[\aSb 1 1 1 1 6 6 6
0 0 0"
t
r5 r )!ri rB r@ rU r typingr r numpyr% tqdm.autor
trainer_utilsr r r
r
training_argsr utilsr
get_loggerr, r r r( r} r# r r r r r rJ r5 r3 r= s ! ! ! ! ! ! " " " " " " " " V V V V V V V V V V V V , , , , , ,
H % % XE XE XE XE XE XE XE XEv) ) ) ) ) ) ) )X :
:
:
:
:
_ :
:
:
zO
O
O
O
O
O
O
O
d@ @ @ @ @o @ @ @F2 2 2 2 2/ 2 2 2jG% G% G% G% G% G% G% G%T o I
I
I
I
I
O_ I
I
I
I
I
r5