&`i^:ddlmZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z m Z mZmZmZmZmZmZddlZddlZddlZddlmZddlmZddlmZmZddl m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'm(Z(m)Z)dd l*m+Z+dd l,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8ddl9m:Z:ddl;mZ>m?Z?m@Z@ddlAmBZBmCZCddlDmEZEmFZFddlGmHZH ddlImJZJmKZKn#eL$r ddlmJZJmKZKYnwxYwejMjNOZPde8e1e/hZQeCGddZReBGddeRZSeBGddZTeCGddeSeTZUeCGddeSZVd eeWeWeeXffd!ZYd"eWd#eWd eeXeXffd$ZZd"eWd#eWd eXfd%Z[d&ee(fd'Z\ dWd&ee(d+eeeXeeXeXffd,eeeeXeeXeXffd-e]d.e^d/eXd0ee]d1e]d2e^d3eeXd4eeXd5e^fd6Z_ dXd7e d8e]d9e^d:e^d e f d;Z` dYd&ee(d+eeeXeeXeXffd,eeeeXeeXeXffd0ee]d3eeXd4eeXd5e^d1e]d eeeeXee^eXfffd<Za dZd&ee(d+eeeXeeXeXffd,eeeeXeeXeXffd/eXd0ee]d3eeXd4eeXd5e^d1e]d eeXfd=Zbd eXfd>Zc d[d&ee(d/eXd0ee]fd?Zd d\d@e(d3eXd,eeeeXeeXeXfffdAZe d]dBeeXee(fdCe]d5e^fdDZfd@e(dEegd e)fdFZh d^d@e(dGeeXdHeeXd1e]fdIZieBGdJdKe%Zjd]dLe^d eSfdMZkdNeedOe fd eeeXfdPZl d\dQeRdRee]e>fdSeeFd eeReFffdTZmdUejndQeRdSeFd e fdVZodS)_)print_functionN)Path)AnyCallable CollectionDictListOptionalTupleUnion) flatten_dict)tabulate)EXPR_ERROR_FILETRAINING_ITERATION)_force_on_current_node safe_print)Callback)DEBUG_PRINT_INTERVALTrial _Location) pretty_print) AUTO_RESULT_KEYSDEFAULT_METRICDONEEPISODE_REWARD_MEANEXPERIMENT_TAG MEAN_ACCURACY MEAN_LOSSNODE_IPPID TIME_TOTAL_STIMESTEPS_TOTALTRIAL_ID) Trainableunflattened_lookup) Verbosity has_verbosity set_verbosity) DeveloperAPI PublicAPI)EmptyQueue)Template)MappingMutableMappingconfigc eZdZdZ ddeedeedeedeefdZdd e e d e fd Z d e e d e d e fd ZdS)ProgressReporterzAbstract class for experiment progress reporting. `should_report()` is called to determine whether or not `report()` should be called. Tune will call these functions after trial state transitions, receiving training results, and so on. N start_time total_samplesmetricmodec dS)arSetup progress reporter for a new Ray Tune run. This function is used to initialize parameters that are set on runtime. It will be called before any of the other methods. Defaults to no-op. Args: start_time: Timestamp when the Ray Tune run is started. total_samples: Number of samples the Ray Tune run will run. metric: Metric to optimize. mode: Must be one of [min, max]. Determines whether objective is minimizing or maximizing the metric attribute. **kwargs: Keyword arguments for forward-compatibility. Nselfr5r6r7r8kwargss n/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/tune/progress_reporter.pysetupzProgressReporter.setupBs . Ftrialsdonect)zReturns whether or not progress should be reported. Args: trials: Trials to report on. done: Whether this is the last progress report attempt. NotImplementedErrorr<rArBs r> should_reportzProgressReporter.should_report[s "!r@sys_infoct)zReports progress across trials. Args: trials: Trials to report on. done: Whether this is the last progress report attempt. sys_info: System info. rDr<rArBrHs r>reportzProgressReporter.reportds "!r@NNNNF)__name__ __module__ __qualname____doc__r floatintstrr?r rboolrGrrKr:r@r>r4r49s'+'+ $"   UO  }   sm     2""DK"t"""""T%[""""""""r@r4ceZdZdZejededede de de diZ e eejejejejedhZdddd d d d d dddd d deeeeeeeffdeeeeeeeffdee de de de de de deedeedeedefdZ d3deedee deedeefdZdeedeefdZde fdZd4deefd Z d5d!ee!d"efd#Z"d4ded$eefd%Z# d4d&ed$eefd'Z$d(d)d*d!ee!d"ed+ed,ed-ef d.Z%d6d!ee!d0e fd1Z&d!ee!fd2Z'dS)7TuneReporterBaseaAbstract base class for the default Tune reporters. If metric_columns is not overridden, Tune will attempt to automatically infer the metrics being outputted, up to 'infer_limit' number of metrics. Args: metric_columns: Names of metrics to include in progress table. If this is a dict, the keys should be metric names and the values should be the displayed names. If this is a list, the metric name is used directly. parameter_columns: Names of parameters to include in progress table. If this is a dict, the keys should be parameter names and the values should be the displayed names. If this is a list, the parameter name is used directly. If empty, defaults to all available parameters. max_progress_rows: Maximum number of rows to print in the progress table. The progress table describes the progress of each trial. Defaults to 20. max_error_rows: Maximum number of rows to print in the error table. The error table lists the error file, if any, corresponding to each trial. Defaults to 20. max_column_length: Maximum column length (in characters). Column headers and values longer than this will be abbreviated. max_report_frequency: Maximum report frequency in seconds. Defaults to 5s. infer_limit: Maximum number of metrics to automatically infer from tune results. print_intermediate_tables: Print intermediate result tables. If None (default), will be set to True for verbosity levels above 3, otherwise False. If True, intermediate tables will be printed with experiment progress. If False, tables will only be printed at then end of the tuning run for verbosity levels greater than 2. metric: Metric used to determine best current trial. mode: One of [min, max]. Determines whether objective is minimizing or maximizing the metric attribute. sort_by_metric: Sort terminated trials by metric in the intermediate table. Defaults to False. acclossiterztotal time (s)tsrewardNF metric_columnsparameter_columnsr6max_progress_rowsmax_error_rowsmax_column_lengthmax_report_frequency infer_limitprint_intermediate_tablesr7r8sort_by_metricrarbr6rcrdrerfrgrhr7r8ric ||_|du|_i|_|p|j|_|pg|_||_||_||_ ||_ | ttj |_n| |_||_d|_t#j|_| |_| |_| |_dSNr)_total_samples_metrics_override_inferred_metricsDEFAULT_COLUMNScopy_metric_columns_parameter_columns_max_progress_rows_max_error_rows_max_column_length _infer_limitr)r(V3_TRIAL_DETAILS_print_intermediate_tables_max_report_freqency_last_report_timetime _start_time_metric_mode_sort_by_metric) r<rarbr6rcrdrerfrgrhr7r8ris r>__init__zTuneReporterBase.__init__s ,!/t!;!#-L1E1J1J1L1L"3"9r"3-"3' $ ,.;Ir?zTuneReporterBase.setupsK J''' }--- ""&t"<<<<rz&TuneReporterBase.set_search_propertiessv L V   N   "!DL  DJ < DJ )DLtr@c||_dSN)rl)r<r6s r>rz"TuneReporterBase.set_total_sampless+r@ timestampcL|tj|_dS||_dSr)r{r|)r<rs r>rzTuneReporterBase.set_start_times*  #y{{D   (D   r@rArBctj|jz |jkrtj|_dS|S)NT)r{rzryrFs r>rGzTuneReporterBase.should_reports6 9;;/ /$2K K K%)Y[[D "4 r@representationc d|_||jvr"td|t |jt r|p|}||j|<dS|||krtd|j|dS)aAdds a metric to the existing columns. Args: metric: Metric to add. This must be a metric being returned in training step results. representation: Representation to use in table. Defaults to `metric`. TColumn {} already exists.Nzl`representation` cannot differ from `metric` if this reporter was initialized with a list of metric columns.)rmrqrformat isinstancer1append)r<r7rs r>add_metric_columnz"TuneReporterBase.add_metric_columns"& T) ) )8??GGHH H d*N ; ; 0+5vN+9D  ( ( ()n.F.F )  ' ' / / / / /r@ parameterc||jvr"td|t|jtr|p|}||j|<dS|||krtd|j|dS)aAdds a parameter to the existing columns. Args: parameter: Parameter to add. This must be a parameter specified in the configuration. representation: Representation to use in table. Defaults to `parameter`. rNzo`representation` cannot differ from `parameter` if this reporter was initialized with a list of metric columns.)rrrrrr1r)r<rrs r>add_parameter_columnz%TuneReporterBase.add_parameter_columns / / /8?? JJKK K d-~ > > 6+8yN1?D #I . . .)n .I.I )  # * *9 5 5 5 5 5r@psql )fmtdelimrHrrc6|jr)|j|jd|_tjd|js5|||j}|j |dt|j tj g|}|rd}d} n|j }|j} ||\} } | r)|t#| | |jt't(jr{|t-||j|j|j|j|||j||j|j|j |t5||| |||zS)aReturns full progress string. This string contains a progress table and error table. The progress table describes the progress of each trial. The error table lists the error file, if any, corresponding to each trial. The latter only exists if errors have occurred. Args: trials: Trials to report on. done: Whether this is the last progress report attempt. fmt: Table format. See `tablefmt` in tabulate API. delim: Delimiter between messages. NFz[Both 'metric' and 'mode' must be set to be able to sort by metric. No sorting is performed.z == Status ==) rarbr6 force_tablermax_rowsrerBr7r8rirr)rr}r~warningswarnrm_infer_user_metricsrvrqupdate_time_passed_strr|r{rsrt_current_best_trialr_best_trial_strrrr)r( V1_EXPERIMENT_trial_progress_strrlrxru_trial_errors_strjoin) r<rArBrrrH user_metricsmessages max_progress max_errorcurrent_best_trialr7s r> _progress_strzTuneReporterBase._progress_str9s*   T\%9TZ=O#(D M>   % 633FDrz$TuneReporterBase._infer_user_metricss t% & &% / /) )!# 2 2A= !"!4!4!6!6 2 2 !555%555;;$*BBB=CD26:t-..%771111118  2%%r@c|sdS|j|j}}|sDt|jdkr,t |jd}|r|sd|fS|dkrdnd}t d}d}|D]H}|js t||jd}tj |r6|r ||z|kr||z}|}I||fS) NNNrmaxg?gz-infdefault) r}r~rrnlistkeysrRrr'pdisnull) r<rAr7r8 metric_op best_metric best_trialr metric_values r>rz$TuneReporterBase._current_best_trials :|TZ @4)**a//d499;;<rWrWos''T.k- 5 v  * T     O       T FJHL'+!# !#$%48 $"$&.&.&.!tCy$sCx.'@!AB&.$E$s)T#s(^*C$DE &.  } &.  &.&.&."&.&.$,D>&. &.sm&.&.&.&.&.T'+'+ $" = =UO = } = = sm = = = =HSM#(,s,,,,)))))) DKt 000Xc]00004?C666.6sm6666> D,D,D,U D,D, D,  D,  D,D,D,D,L&&$u+&c&&&&$"$u+""""""r@rWcdeZdZdZedefdZejdefdZdeddfdZ dS) RemoteReporterMixinzRemote reporter abstract mixin class. Subclasses of this class will use a Ray Queue to display output on the driver side when running Ray Client.returnc$t|ddS)N _output_queuegetattr)r<s r> output_queuez RemoteReporterMixin.output_queuest_d333r@rc||_dSr)r)r<rs r>rz RemoteReporterMixin.output_queues"r@stringNct)z[Display the progress string. Args: string: String to display. rD)r<rs r>displayzRemoteReporterMixin.displays "!r@) rNrOrPrQpropertyr.rsetterrTrr:r@r>rrs33 4e444X4#%####"c"d""""""r@rcfeZdZdZdddddddddddddd d ed eeeee eeffd eeeee eeffd ee d e de de de de deedeedeedeffdZ dee dede fdZ deddfdZdee dedefdZxZS)JupyterNotebookReporterafJupyter notebook-friendly Reporter that can update display in-place. Args: overwrite: Flag for overwriting the cell contents before initialization. metric_columns: Names of metrics to include in progress table. If this is a dict, the keys should be metric names and the values should be the displayed names. If this is a list, the metric name is used directly. parameter_columns: Names of parameters to include in progress table. If this is a dict, the keys should be parameter names and the values should be the displayed names. If this is a list, the parameter name is used directly. If empty, defaults to all available parameters. max_progress_rows: Maximum number of rows to print in the progress table. The progress table describes the progress of each trial. Defaults to 20. max_error_rows: Maximum number of rows to print in the error table. The error table lists the error file, if any, corresponding to each trial. Defaults to 20. max_column_length: Maximum column length (in characters). Column headers and values longer than this will be abbreviated. max_report_frequency: Maximum report frequency in seconds. Defaults to 5s. infer_limit: Maximum number of metrics to automatically infer from tune results. print_intermediate_tables: Print intermediate result tables. If None (default), will be set to True for verbosity levels above 3, otherwise False. If True, intermediate tables will be printed with experiment progress. If False, tables will only be printed at then end of the tuning run for verbosity levels greater than 2. metric: Metric used to determine best current trial. mode: One of [min, max]. Determines whether objective is minimizing or maximizing the metric attribute. sort_by_metric: Sort terminated trials by metric in the intermediate table. Defaults to False. TNr]r^r_F) overwriterarbr6rcrdrerfrgrhr7r8rirrarbr6rcrdrerfrgrhr7r8ric tt||||||||| | | | |  tst jd||_d|_|ddS)Nr`a!You are using the `JupyterNotebookReporter`, but not IPython/Jupyter-compatible environment was detected. If this leads to unformatted output (e.g. like ), consider passing a `CLIReporter` as the `progress_reporter` argument to `tune.RunConfig()` instead.) superrr IS_NOTEBOOKrr _overwrite_display_handler)r<rrarbr6rcrdrerfrgrhr7r8ri __class__s r>rz JupyterNotebookReporter.__init__s" %t,,55)/'/)/!5#&?) 6   M1   $# Rr@rArBrHc|j||g|R}|j|j|dS||dSr)_progress_htmlrputr)r<rArBrHprogresss r>rKzJupyterNotebookReporter.reports^&4&vt?h???   (   ! !( + + + + + LL " " " " "r@rrcddlm}m}m}|js0|jr |d|||d|_dS|j||dS)Nr)HTML clear_outputrT)wait display_id)IPython.displayrrrrrr)r<rrrrs r>rzJupyterNotebookReporter.display&s??????????# 6 ( $''''#*744<rz&JupyterNotebookReporter._progress_html0s % 633FD ,/-w777778      4/"5!>TTt'></"5    -Q/"5abb"9( )O4;O     7F G G 9::AA-(?)BCC C-..55%3X>) 6   r@)rNrOrPrQrUr r r rTrrSrrrKrr __classcell__rs@r>rrs$$REIHL'+!# !#$%48 $"$,,,,!tCy$sCx.'@!AB , $E$s)T#s(^*C$DE ,  } ,,,,",,$,D>, ,sm,,,,,,,\#T%[######6c6d6666? T%[? ? C? ? ? ? ? ? ? ? r@rc<eZdZdZddddddddddddd deeeeeeeffd eeeeeeeffd ee d e d e d e de de dee deedeede ffdZ defdZ dee de defdZxZS) CLIReporteraCommand-line reporter Args: metric_columns: Names of metrics to include in progress table. If this is a dict, the keys should be metric names and the values should be the displayed names. If this is a list, the metric name is used directly. parameter_columns: Names of parameters to include in progress table. If this is a dict, the keys should be parameter names and the values should be the displayed names. If this is a list, the parameter name is used directly. If empty, defaults to all available parameters. max_progress_rows: Maximum number of rows to print in the progress table. The progress table describes the progress of each trial. Defaults to 20. max_error_rows: Maximum number of rows to print in the error table. The error table lists the error file, if any, corresponding to each trial. Defaults to 20. max_column_length: Maximum column length (in characters). Column headers and values longer than this will be abbreviated. max_report_frequency: Maximum report frequency in seconds. Defaults to 5s. infer_limit: Maximum number of metrics to automatically infer from tune results. print_intermediate_tables: Print intermediate result tables. If None (default), will be set to True for verbosity levels above 3, otherwise False. If True, intermediate tables will be printed with experiment progress. If False, tables will only be printed at then end of the tuning run for verbosity levels greater than 2. metric: Metric used to determine best current trial. mode: One of [min, max]. Determines whether objective is minimizing or maximizing the metric attribute. sort_by_metric: Sort terminated trials by metric in the intermediate table. Defaults to False. Nr]r^r_Fr`rarbr6rcrdrerfrgrhr7r8ric ptt|||||||||| | | |  dS)Nr`)rr r)r<rarbr6rcrdrerfrgrhr7r8rirs r>rzCLIReporter.__init__sZ k4  )))/'/)/!5#&?) * r@rc$t|dSrrr<rs r>_printzCLIReporter._print3r@rArBrHcJ||j||g|RdSr)rrrJs r>rKzCLIReporter.reports2 &D&vt?h???@@@@@r@)rNrOrPrQr r r rTrrSrUrrrrKrr s@r>r r rs##PFJHL'+!# !#$%48 $"$   !tCy$sCx.'@!AB $E$s)T#s(^*C$DE   }      "  $,D>   sm       >#AT%[AAAAAAAAAAr@r rc: ddl}ddl}|jdz }||jdz z }||dzkrd}nd}t |dt |d|fS#t $rtjtjdfcYSwxYw)zGet the current memory consumption. Returns: Memory used, memory available, and optionally a warning message to be shown to the user when memory consumption is higher than 90% or if `psutil` is not installed rNi@g?a : ***LOW MEMORY*** less than 10% of the memory on this node is available for use. This can cause unexpected crashes. Consider reducing the memory used by your application or reducing the Ray object store size by setting `object_store_memory` when calling `ray.init`.rz@Unknown memory usage. Please run `pip install psutil` to resolve) raypsutilvirtual_memorytotal availableround ImportErrorrnan)rrrrmessages r>rrs   ((**0G<V2244>'JJ X^ # #A GGWa  %!"4"4g==    F F N     sA1A44#BBr5rcHtj|}tj|}||z }|}|dz}||dzz}|dz}||dzz}|dz}||dzz } |dkr|dd} nd} | |dd |dd | d z } |d | fS) a Get strings representing the current and elapsed time. Args: start_time: POSIX timestamp of the start of the tune run current_time: POSIX timestamp giving the current time Returns: Current time and elapsed time for the current run iQi<rz.0fz days, rz02.0f:z05.2fz%Y-%m-%d %H:%M:%S)datetime fromtimestamp total_seconds) r5rcurrent_time_dt start_time_dtdeltarestdayshoursminutessecondsrunning_for_strs r>rrs'55lCCO%33J??M /- ?E    D L !DDL !!D W EEW DbjGWr\!G axx!....%GGGGGGGGGGO 1 1? BBr@c:t||\}}d|d|dS)aaGenerate a message describing the current and elapsed time in the run. Args: start_time: POSIX timestamp of the start of the tune run current_time: POSIX timestamp giving the current time Returns: Message with the current and elapsed time for the current tune run, formatted to be displayed to the user zCurrent time: z (running for ))r)r5rcurrent_time_strr*s r>rrs3)6j,(O(O%o R, R R R R RRr@rActjt}|D]"}||j|#|Sr)r defaultdictrstatusr)rAtrials_by_staters r>_get_trials_by_stater2sE!-d33O ,,!((++++ r@Frr]rarbr6rrrrerBr7r8ric g} |dkrdnd} t|dkr| | St|}t|td|DD]*}| d|+fdtD}|r|t jkrd}| d ||rd |nd d ||sttj r|r| t|||||| | | | z } | | S)aReturns a human readable message for printing to the console. This contains a table where each row represents a trial, its parameters and the current values of its metrics. Args: trials: List of trials to get progress string for. metric_columns: Names of metrics to include. If this is a dict, the keys are metric names and the values are the names to use in the message. If this is a list, the metric name is used in the message directly. parameter_columns: Names of parameters to include. If this is a dict, the keys are parameter names and the values are the names to use in the message. If this is a list, the parameter name is used in the message directly. If this is empty, all parameters are used in the message. total_samples: Total number of trials that will be generated. force_table: Force printing a table. If False, a table will be printed only at the end of the training for verbosity levels above `Verbosity.V2_TRIAL_NORM`. fmt: Output format (see tablefmt in tabulate API). max_rows: Maximum number of rows in the trial table. Defaults to unlimited. max_column_length: Maximum column length (in characters). done: True indicates that the tuning run finished. metric: Metric used to sort trials. mode: One of [min, max]. Determines whether objective is minimizing or maximizing the metric attribute. sort_by_metric: Sort terminated trials by metric in the intermediate table. Defaults to False. r
rrch|] }|j Sr:)local_experiment_path.0rs r> z&_trial_progress_str..QsEEEQ4EEEr@zResult logdir: {}cbg|]+}dt||,S){} {})rr)r8stater1s r> z'_trial_progress_str..TsD  s?5122E::r@infinitezNumber of trials: {}{} ({})rr, r) rrr2sortedrrsysmaxsizer)r( V2_TRIAL_NORMr)rArarbr6rrrrerBr7r8rirr num_trials local_dirnum_trials_strsr1s @r>rrsZHVmmFFE 6{{Qzz(###VJ*622OEEfEEEFF?? +229==>>>>O,,O ##+55"  OO%,, #0 8    b IIo & &    }Y%<==  $  ))/)/     ::h  r@rmax_lenadd_addrwrapc0|-t|tttjt fr|St |}t||kr|S|rTt||dzkrd|d|dzz dz}tj ||}d |S|rXt|ttt fs6|d|dz dtt|d d}|Sd|d|z dz}|S) aAbbreviate a string representation of an object to `max_len` characters. For numbers, booleans and None, the original value will be returned for correct rendering in the table formatting tool. Args: value: Object to be represented as a string. max_len: Maximum return string length. add_addr: If True, will add part of the object address to the end of the string, e.g. to identify different instances of the same class. If False, three dots (``...``) will be used instead. Nz...r_)widthrr^_) rrSrRnumbersNumberrUrTrtextwraprIrhexid)rrGrHrIrwrappedresults r>_max_lenrVts! } 53w~t*LMM} ZZF 6{{g  " u::! # #FA1$5#8#899E-W555yy!!! 53t*<==?gk?+CCc"U))nnRSS.ACC VQ[OO, ,F Mr@c>t}t} |r6t| tj|dkfd| tj<tjtjtjtjtjg} |ptd}||krt| ||} gg} | D]l} | | vr| | z t| | t| | z }|dkr)| d || m||z }d | }nd}d }g| D]} | | vr| | z ttr"t!nfd Ds.tt%jd Dn9ttr"t!nfd D}ttrfd D}nfdD}ttrfdD}nfdD}gd|z|z}||||ffS)aGenerate a table showing the current progress of tuning trials. Args: trials: List of trials for which progress is to be shown. metric_columns: Metrics to be displayed in the table. parameter_columns: List of parameters to be included in the data max_rows: Maximum number of rows to show. If there's overflow, a message will be shown to the user indicating that some rows are not displayed metric: Metric which is being tuned mode: Sort the table in descending order if mode is "max"; ascending otherwise sort_by_metric: If true, the table will be sorted by the metric max_column_length: Max number of characters in each column Returns: - Trial data - List of column names - Overflow tuple: - boolean indicating whether the table has rows which are hidden - string with info about the overflowing rows rc2t|jdSNrr'r)rr7s r>z*_get_progress_table_data..s,VQ]DQQQr@reversekeyinfrr;r?FrcLg|]tfdD S)c3JK|]}t|jdduVdSrYrZ)r8rks r> z6_get_progress_table_data...sM   q!- > > >d J      r@)r)r8rbrAs @r>r=z,_get_progress_table_data..s`           r@cg|] }|j Sr:)evaluated_paramsr7s r>r=z,_get_progress_table_data..s-Q-Q-QQa.@-Q-Q-Qr@c6g|]}t|S))re)_get_trial_info)r8trialre metric_keysparameter_keyss r>r=z,_get_progress_table_data..sD   >;BS   r@cBg|]}t|ddSFT)rGrHrIrV)r8rbreras r>r=z,_get_progress_table_data..sJ$ $ $  q!+r=z,_get_progress_table_data..s;$ $ $  Q 1E M M M$ $ $ r@cBg|]}t|ddSrlrm)r8rbrerbs r>r=z,_get_progress_table_data..sK' ' '  !!$)    ' ' ' r@c6g|]}t|ddSrlrmros r>r=z,_get_progress_table_data..s;' ' '  Q 1E M M M' ' ' r@) Trial namer0loc)rr2r@r TERMINATEDRUNNINGPAUSEDPENDINGERRORrR_fair_filter_trialsrrrrr0rrsetunion)rArarbrr7r8rirerDr1state_tbl_ordertrials_by_state_trunc overflow_strsr<numoverflow overflow_str trial_tableformatted_metric_columnsformatted_parameter_columnscolumnsrirjs``` ` ` @@r>_get_progress_table_datars@VJ*622O ,2 E, -U]QQQQ- - - ()      O'5<..0011 $ K +  -Q-Q&-Q-Q-Q RSS %w / /+/446677* K.'**  $ $ $ $ $ ! $ $ $   $ $ $ $  $ $ $  #W-- ' ' ' ' ' $' ' ' ##' ' ' ' #' ' ' # ('' % & " # (L!9 99r@c t||||||||\} } \} } t| | |dg} | r| d| d| d| S)aqGenerate a list of trial progress table messages. Args: trials: List of trials for which progress is to be shown. metric_columns: Metrics to be displayed in the table. parameter_columns: List of parameters to be included in the data fmt: Format of the table; passed to tabulate as the fmtstr argument max_rows: Maximum number of rows to show. If there's overflow, a message will be shown to the user indicating that some rows are not displayed metric: Metric which is being tuned mode: Sort the table in descenting order if mode is "max"; ascending otherwise sort_by_metric: If true, the table will be sorted by the metric max_column_length: Max number of characters in each column Returns: Messages to be shown to the user containing progress tables F)headersr showindexz... z more trials not shown (r,)rrr)rArarbrrr7r8riredatarrrrs r>rr*s</G  / /+D'+HlwNNNOHRPxPPPPPQQQ Or@c\|r)d|ddSdS)zFormat system info into a string. *sys_info: System info strings to be included. Returns: Formatted string containing system information. r4rr)rreplace)rHs r>rrXs3;{{8$$,,T6::: 2r@c Bg}d|D}t|}|dkr|d|||ptdkr,|d|||z gd}d|d|D}|t |||d d t d |d|Dr|d |dkrdnd}||S)aReturns a readable message regarding trial errors. Args: trials: List of trials to get progress string for. fmt: Output format (see tablefmt in tabulate API). max_rows: Maximum number of rows in the error table. Defaults to unlimited. c g|] }|j | Sr:) error_filer7s r>r=z%_trial_errors_str..ps 0 0 0A1< 0a 0 0 0r@rzNumber of errored trials: {}r_z(Table truncated to {} rows ({} overflow))rrz # failuresz error filecg|]J}t|t|jj|jtjkrdndz|jgKS)r*)rT run_metadata num_failuresr0rrxrr8rhs r>r=z%_trial_errors_str..|sf    E E&3444422#?     r@NF)leftrightr)rrrcolalignc3@K|]}|jtjkVdSr)r0rrtrs r>rcz$_trial_errors_str..s,OOEu|u//OOOOOOr@z3* The trial terminated successfully after retrying.rr4r)rrrrRrrr) rArrrfailed num_failed fail_headerfail_table_datars r>rrdskH 0 0 0 0 0FVJA~~6==jIIJJJ 1U5\\ 2 2 OO:AAj83    A@@     *     #2        OOVIXI=NOOO O O S OOQ R R RVmmFFE ::h  r@rhcBt||jd}|jdi|p t}t |t r|}fd|D}d|jd|d|d|S) z:Returns a readable message stating the current best trial.Nrr2c2i|]}|t|Sr:r&)r8pr2s r> z#_best_trial_str..s& J J J1a#Av.. J J Jr@zCurrent best trial: z with =z and parameters=)r'rgetrrrr0trial_id)rhr7rbvalparamsr2s @r>rrs VU%6 E E EC   " "8R 0 0F)@T&++---@-@#W--5-2244 J J J J8I J J JF u~  V  c    r@r1 max_trialsctjtd}|dkrT|sRd}t|D]8}|t ||krd}|dz}|xxdz cc<9|dkr|Rt t|D]<}|t jkr|r |||< t||dd|<=fdt|D}|S)aFilters trials such that each state is represented fairly. The oldest trials are truncated if necessary. Args: trials_by_state: Maximum number of trials to return. Returns: Dict mapping state to List of fairly represented trials. FrTrc|jSr)r)rs r>r[z%_fair_filter_trials..sQZr@r\c>i|]}||d|Srr:)r8r<num_trials_by_statesorted_trials_by_states r>rz'_fair_filter_trials..sB  %e,-I/B5/I-IJr@)rr/rSr@rdictrrt)r1rri no_changer<filtered_trialsrrs @@r>ryrysO&1#66I q... O,, 0 0E"5)C0F,G,GGG! a #E***a/*** q..."VV(( E$ $ $ $,;E,B "5 ) ),2&;O;O--- "5 ) )O,,O r@rUc|td|td}}|r|rt||}n |jj}|Sr)rr r!rtemporary_statelocation)rhrUnode_ippidrs r>_get_trial_locationrs[::gt,,fjjd.C.CSG232Wc**(1 Or@ parametersmetricsc|j|jt|}t||jt|g}|fd|Dz }|fd|Dz }|S)aIReturns the following information about a trial: name | status | loc | params... | metrics... Args: trial: Trial to get information for. parameters: Names of trial parameters to include. metrics: Names of metrics to include. max_column_length: Maximum column length (in characters). c Tg|]$}tt|dd%SNrT)rGrHrVr')r8paramr2res r>r=z#_get_trial_info..sR   ufd ; ; ;%   r@c Tg|]$}tt|dd%Srr)r8r7rerUs r>r=z#_get_trial_info..sR   vvt < < <%   r@)rr2rrTr0)rhrrrer trial_infor2rUs ` @@r>rgrgs F \F"5&11He**elCMM:J  J  J r@c<eZdZdZ ddeedeeefdZdefdZde d ed d d d e fd Z de d ed d d fdZ de d ed d d fdZ dd d d e defdZd ed e dedefdZd e ee fdeedefdZd ed e dedefdZd e fdZdS)TrialProgressCallbackaReports (prints) intermediate trial progress. This callback is automatically added to the callback stack. When a result is obtained, this callback will print the results according to the specified verbosity level. For ``Verbosity.V3_TRIAL_DETAILS``, a full result list is printed. For ``Verbosity.V2_TRIAL_NORM``, only one line is printed per received result. All other verbosity levels do not print intermediate trial progress. Result printing is throttled on a per-trial basis. Per default, results are printed only once every 30 seconds. Results are always printed when a trial finished or errored. Nr7progress_metricscbtjt|_tjt|_t |_i|_||_ t |pg|_ |j r&|j r|j |j i|_ d|_ dSr)rr/rR _last_printrS_last_print_iterationrz_completed_trials_last_result_strr}_progress_metricsadd _last_resultr)r<r7rs r>rzTrialProgressCallback.__init__s'2599%0%rzTrialProgressCallback._print%rr@ iterationrArrhrUc 6|||ddS)NFerror) log_result)r<rrArhrUinfos r>on_trial_resultz%TrialProgressCallback.on_trial_result(s" vU33333r@c @|||jddS)NTr)rr)r<rrArhrs r>on_trial_errorz$TrialProgressCallback.on_trial_error2s% u0=====r@c :||jvr|j|||j}|j|d}||kr|||jddS|d|d|dSdS)NrFrTrial z completed. Last result: )rr _print_resultrrrrr)r<rrArhrprint_result_strlast_result_strs r>on_trial_completez'TrialProgressCallback.on_trial_complete7s . . .  " & &u - - -#11%2CDD "377rBBO?22u'8FFFFF WUWWEUWWXXXXX / .r@Frc|dddu}|j|}|p |ptj|z tk}|r#||jvr|j||rst r|||||n|||||tj|j|<t|vr|t|j |<dSdSdS)NrBFT) rrr{rrrrdisplay_result print_resultrr)r<rhrUrrB last_print should_prints r>rz TrialProgressCallback.log_resultFszz&%((D0%e, WuW j(@CW(W  .E!777  " & &u - - -  O >##E65$????!!%===&*ikkD U #!V++4:;M4N*5111 O O,+r@rBc Z|j|}ttjr|t |krb|d|d|dt| dd|r|d|ddSdSttj r|j pd}||d }t|j t}d } |rd } ||} | |j|<|rd |d |jd|} n2|j rd|d|d|dd|jd| } nd|d| d|jd| } || dSdS)a=Print the most recent results for the given trial to stdout. Args: trial: Trial for which results are to be printed result: Result to be printed error: True if an error has occurred, False otherwise done: True if the trial is finished, False otherwise z Result for rz {}rz rz completed.r}gXrz This trial completed.z The trial z errored with parameters=z. Error file: z reported rz.2fz with parameters=.N)rr)r(rwrrrrrrrCr}r local_pathras_posixrrr2) r<rhrUrrBlast_print_iteration metric_namerrrrrs r>rz"TrialProgressCallback.print_resultXsJ $9%@ 3 4 4' !zz,--1EEE 2%222333 FMM,v*>*>*F*FtV*T*TUUVVV 9 7U77788888 9 992 3 3 !,3)K!::k599Le.@@IIKKJD 0/#11&99 +;D !% ( 000"',00#-00  =U=="==%18==',|==6:===U=='==',|==6:== KK A ! !r@rrcg}t|}tjt|jd}|D]7\}|t |gfd|Dz8tdt|ddg|zd S) aGenerate an HTML table of trial progress info. Trials (rows) are sorted by name; progress stats (columns) are sorted as well. Args: trials: Trials and their associated latest results columns: Columns to show in the table; must be a list of valid keys for each Trial result Returns: HTML template containing a rendered table of progress info c,t|dSrk)rT)items r>r[z.ss47||r@)r^c<g|]}|dS)r)r)r8colrUs r>r=z>TrialProgressCallback.generate_trial_table..s''O'O'O 3(;(;'O'O'Or@ztrial_progress.html.j2rrrF)rrr)table) r@rrrrrrTr/rr)r<rArr sorted_trialsrhrUs @r>generate_trial_tablez*TrialProgressCallback.generate_trial_tables //#/ 4$**,,2K2K L L L  +0022 Q QME6 KKU 'O'O'O'Ow'O'O'OO P P P P01188v ~/GSX9   r@cddlm}m}||j|<t t jrddh}n5t t jrhdttz}ndS| |jt| |z }|j s|||d|_ dS|j ||dS) aDisplay a formatted HTML table of trial progress results. Trial progress is only shown if verbosity is set to level 2 or 3. Args: trial: Trial for which results are to be printed result: Result to be printed error: True if an error has occurred, False otherwise done: True if the trial is finished, False otherwise r)rrr2 hist_stats>rBr2rrexperiment_tagNTr)rrrrr)r(rwrCrzrrrrr) r<rhrUrrBrr ignored_keysrs r>rz$TrialProgressCallback.display_results 21111111#)% 3 4 4 LL 92 3 3  $%% &LL F))  s6;;==11L@  # 5#*744;;4#H#H#HD  ' 'U 4 4 4 4 4r@c|jr4t|}i}|jD]}||||<nT|}tD]}||dt D]}||ddd|D}|S)N,c&g|]\}}||d|S)Nrr:)r8rbvs r>r=z7TrialProgressCallback._print_result..s' J J JDAqAMZZAZZMMMr@) rr rrpSKIP_RESULTS_IN_REPORTpoprrr)r<rU flat_resultrr7 skip_result auto_resultrs r>rz#TrialProgressCallback._print_results  ! 4&v..KL0 ? ?'2v'>'> V$$ ? ";;==L5 4 4   d3333/ 4 4   d333388 J JL$6$6$8$8 J J J   r@rrM)rNrOrPrQr rTr rrrSrrrrrUrrrrrrr:r@r>rrs(UY$$sm$>FtCy>Q$$$$ #44W 4 4  4444>>&*7m>>>> Y Y&*7m Y%5E%54%5%5D%5%5%5%5N D      r@r _trainer_apic tr=|s;|dttj t di|}n t di|}|S)zDetect progress reporter class. Will return a :class:`JupyterNotebookReporter` if a IPython/Jupyter-like session was detected, and a :class:`CLIReporter` otherwise. Keyword arguments are passed on to the reporter class. rr:)r setdefaultr)r(rCrr )rr=progress_reporters r>_detect_reporterrsh2<2+=9P+Q+Q'QRRR3==f=='11&11 r@ trainabler%c,|sdSt|ddS)z"Detect progress metrics to report.Nrr)rs r>_detect_progress_metricsrs$ t 914 8 88r@r verbosity string_queuect||p t}t|tr*|!t ddit d}||_||fS)zzPrepares progress reported for Ray Client by setting the string queue. The string queue will be created if it's None.Nnum_cpusr) actor_options)r*rrrr.rr)rrrs r>)_prepare_progress_reporter_for_ray_clientrs))?-=-?-? #%8996   )1M0Ft0L0LML*6& l **r@ remote_futurecdSfdfd}tj|gddr'|tj|gdd'|dS)zZ Stream items from string queue to progress_reporter until remote_future resolves NcT dS#t$rYdSwxYw)NF)block)rr-)rsr>get_next_queue_itemz2_stream_client_output..get_next_queue_item)sA ##%#00 0   44 s  ''cf}|#|}|!dSdSr)r) string_itemr rs r>_handle_string_queuez3_stream_client_output.._handle_string_queue/sM))++ %  % %k 2 2 2--//K%%%%%r@g?)timeoutr)rr)rrrrr s `` @r>_stream_client_outputrs 000000 (M?C 0 0 0 3 (M?C 0 0 0 3 r@) NrFrNr]FNNF)r]FF)NNNNFr])NrNNNFr])rNrrM)r])p __future__rrrrOrArQr{rpathlibrtypingrrrrr r r r numpyrpandasrrray._private.dictr )ray._private.thirdparty.tabulate.tabulaterray.air.constantsrrray.air.util.noderray.experimental.tqdm_rayrray.tune.callbackrray.tune.experiment.trialrrrray.tune.loggerrray.tune.resultrrrrrrrr r!r"r#r$ray.tune.trainabler%ray.tune.utilsr'ray.tune.utils.logr(r)r*ray.util.annotationsr+r,ray.util.queuer-r. ray.widgetsr/collections.abcr0r1rwidgetsutil in_notebookrrr4rWrrr rRrTrrrr2rSrUrrVrrrrrryrrrgrrrr ObjectRefrr:r@r>r*s %%%%%%  PPPPPPPPPPPPPPPPPPPP ******>>>>>>AAAAAAAA444444000000&&&&&&LLLLLLLLLL((((((                            )(((((------FFFFFFFFFF88888888'''''''' 477777777744433333333334k**,, "HndC 2"2"2"2"2"2"2" 2"j{"{"{"{"{"'{"{"{"| """""""". h h h h h .0Ch h  h V IAIAIAIAIA"IAIA IAX! 5x}!<=! ! ! ! H Ce C5 CU38_ C C C CF S Se S S S S SeEI"  U U KU $s)T#s(^34U  d3ic3h&? @AU  U  U U smU U  U  SMU  3-U U U U U rIN$$ $$-1$AE$$$$$TEI"  L:L: KL:$s)T#s(^34L: d3ic3h&? @AL:sm L: SM L: 3- L:L:L: 4cE$), ,-L:L:L:L:dEI"  ++ K+$s)T#s(^34+ d3ic3h&? @A+ + sm + SM + 3-+++ #Y++++\     GK. . K. !. 6>sm. . . . hEI   d3ic3h&? @A*!&&#tE{*+&&&&&&R u d y    WY!! !"3i!26s)!PS!!!!Hg g g g g Hg g g T  4 >N     9k83459 jo9999%)++'+S)^$+5/+ U "# ++++2=' sC C! C!