&`i+ dZddlZddlZddlZddlZddlZddlmZddlZddl m cm Z ddl m Z mZddlmZmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(m)Z)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z1ddl2m3Z3 ddl4Z4n #e5$rdZ4YnwxYwej6e7Z8GddZ9deddfdZ:e7dkrej;dZd e<=d!de>d" e<=d#de>e j?e j@e jA$e<=d%de>e jBe jC&e<=d'de>e jDd(e jDd)&e<=d*d+e>d, e<=d-deEed.ed/&e<=d0deEed1ed2&e<=d3de>dd4&e<=d5de>d6d7&e<=d8de>d6d9&eejVe8Td?ejWe8Td@ejXeGjYZYeY eZdAeGj[s!e8TdBe*eYZ\n,ej]^eGj[Z[e(e[d+CZ\e9eYe\eGjOeGj_DZ`e`adSdS)EzAutoscaler monitoring loop daemon. See autoscaler._private/monitor.py for the legacy implementation. All the legacy flags are supported here, but the new implementation uses the new autoscaler v2. N)Optional) build_address parse_address)LOGGING_ROTATE_BACKUP_COUNTLOGGING_ROTATE_BYTES)record_extra_usage_tag) logging_utils)get_event_logger)setup_component_logger) SCRIPT_MODE) GcsClient)AUTOSCALER_METRIC_PORTAUTOSCALER_UPDATE_INTERVAL_S)AutoscalerPrometheusMetrics) Autoscaler)AutoscalerEventLogger)FileConfigReader IConfigReaderReadOnlyProviderConfigReader)AutoscalerMetricsReporter)AutoscalingState)Event)TagKeyc eZdZdZ ddededeedeefdZede d eefd Z ede d e fd Z d Z dZdS)AutoscalerMonitoraAutoscaling monitor. This process periodically collects stats from the GCS and triggers autoscaler updates. TODO: We should also handle autoscaler failures properly in the future. Right now, we don't restart autoscaler if it fails (internal reconciliation however, should not fail the autoscaler process). With the Reconciler able to handle extra cloud instances, we could in fact recover the autoscaler process from reconciliation. Naddress config_readerlog_dir monitor_ipctt|||_tjjj}t|j|_|rDt|t}|j d| dd| |j|_ td|j |t"t%|jd}d|_|rL t)t*jj|}t1||_n#t4$r d|_Yn wxYwd|_t7|j } t9| |_|rt<r tdt|dkrd dini} t=j dt| j!d | nK#t4$rt"d Yn%wxYwt<st#d tI|j ||j|j|j |_dS)N)rsAutoscalerMetricsAddressTzsession_name: r) session_namez-Starting autoscaler metrics server on port {}z 127.0.0.1addr)portregistryz8An exception occurred while starting the metrics server.z?`prometheus_client` not found, so metrics will not be exported.)r!r gcs_client event_loggermetrics_reporter)%record_autoscaler_v2_usager gcs_addressray_privateworker global_workerr%rrinternal_kv_putencode_get_session_name _session_nameloggerinfoset_moder r autoscalerr RayEvent SourceType AUTOSCALERrr& Exceptionrrmetric_reporterprometheus_clientformatstart_http_serverr$ exceptionwarningr) selfrrrrr- monitor_addr head_node_ipray_event_logger prom_metricskwargss m/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/autoscaler/v2/monitor.py__init__zAutoscalerMonitor.__init__Cs #9W#5#5666"$2#D,<===  (5KLLL O + ++\-@-@-B-BD$   "33DODD 9T%799::: $$$$T%566q9   % )#3'2G$$ %::J$K$K!! ) ) )$(!!! )!%D 2@RSSS 8FF  +   CJJ. 3?+2M2M&+..SU!3/)2      N #  NNQ   %+'*!1    s%3E E! E!AG;;$H"!H"r%returncv|dtjd}|r|}|S)zObtain the session name from the GCS. If the GCS doesn't respond, session name is considered None. In this case, the metrics reported from the monitor won't have the correct session name. s session_name )timeout)internal_kv_get ray_constantsKV_NAMESPACE_SESSIONdecode)r%r!s rGr1z#AutoscalerMonitor._get_session_namesK"11   .2   1'..00Lautoscaling_statec ||dS#t$rtdYdSwxYw)z(Report the autoscaling state to the GCS.z)Error reporting autoscaling state to GCS.N)report_autoscaling_stateSerializeToStringr:r3r?)r%rRs rG_report_autoscaling_statez+AutoscalerMonitor._report_autoscaling_statesl  J  / /0A0S0S0U0U V V V V V J J J   H I I I I I I Js'+$AAc |j}|r||j|ntdt jtk)zRun the monitor loop.TzNo autoscaling state to report.) r6update_autoscaling_staterVr%r3r@timesleepr)rArRs rG_runzAutoscalerMonitor._runsk 5 $ H H J J   B..t@QRRRR@AAA J3 4 4 4 5rQc |dS#t$rtdwxYw)NzError in monitor loop)r[r:r3r?)rAs rGrunzAutoscalerMonitor.runsK  IIKKKKK      4 5 5 5  s&>)NN)__name__ __module__ __qualname____doc__strrrrH staticmethodr r1rrVr[r]r(rQrGrr5s  ""&$( G G G %G # G SM G G G G RiHSM\$JJ2BJJJ\J 5 5 5rQrr%rIc ttjd|dS#t$rtdYdSwxYw)z) Record usage for autoscaler v2. v2z(Error recording usage for autoscaler v2.N)rrAUTOSCALER_VERSIONr:r3r?)r%s rGr)r)s`Ev8$ KKKKK EEECDDDDDDEs$AA__main__z/Parse GCS server for the monitor to connect to.) descriptionz --gcs-addressFzThe address (ip:port) of GCS.)requiredtypehelpz--autoscaling-configz'the path to the autoscaling config filez--logging-level)rirjdefaultchoicesrkz--logging-format)rirjrlrkz--logging-filenamezFSpecify the name of log file, log to stdout if set empty, default is ""z --logs-dirTzBSpecify the path of the temporary directory used by Ray processes.z--logging-rotate-bytesz8Specify the max bytes for rotating log file, default is z bytes.z--logging-rotate-backup-countz9Specify the backup count of rotated log file, default is .z --monitor-ipz:The IP address of the machine hosting the monitor process.z--stdout-filepathz$The filepath to dump monitor stdout.z--stderr-filepathz$The filepath to dump monitor stderr.win32) logging_levellogging_formatrfilename max_bytes backup_countz7Starting autoscaler v2 monitor using ray installation: z Ray version: z Ray commit: z(AutoscalerMonitor started with command: z--gcs-address must be set!zrs    222222222BBBBBBBB?>>>>>&&&&&&<<<<<<;;;;;;++++++!!!!!!MLLLLL333333@@@@@@ IHHHHH>>>>>>:::::://////  8 $ $HHHHHHHHVE9EEEEE z $X $FF %c8W   6    *2  ,   +  -   33  /333    Q     $) )))  ' +* &***    I    3    3      D;>,':Q:QT66WX,/LG,C,C(("(* &(2 3M2  %  KKP#,PP KK111222 KK/s~//000 KKE38EEFFF"Kj5666  "  RSSS44[AA W//0GHH((*d     ? G KKMMMMMMsB B*)B*