&`i;E<dZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl Z ddl Z ddlZddlmZddlZddlmZdZdZdZd Zd Zd Zd Zd ZdZdZdZdZ d5dZ e!dkr#e\Z"Z#Z$Z%Z&Z'e e"Z"ee"e(de"e(de#e(de$e(de%ej)e"*Z+e+ddze,e-ejze+d<e(de(de'e'r ee+e'e(de(de&ee&Z.e(de.e.r ee+e.e+/di/d Z0d!e+dd"<e0d#kr en0e0d$kr(ee+d1d%dne0d&kreej2gd'ej3d()j45d*6Z7e(d+e7ej2gd,ej3d()j45d*6Z8e(d-e8d.e8e+dd/<nCe0d0kre(de(d1n&e(de(d2ej9dej:d345Z;e;<ej=e+>d*e;?e e;j@Z"e e+e"e#e$e%ddddS#1swxYwYdSdS)6a This script automates the process of launching and verifying a Ray cluster using a given cluster configuration file. It also handles cluster cleanup before and after the verification process. The script requires one command-line argument: the path to the cluster configuration file. Usage: python launch_and_verify_cluster.py [--no-config-cache] [--retries NUM_RETRIES] [--num-expected-nodes NUM_NODES] [--docker-override DOCKER_OVERRIDE] [--wheel-override WHEEL_OVERRIDE] N)Path)storage)RAYc4tjd}|ddd|dtdd |d td d |dgddd|dtdd |dtd|}|jdkr|jdkr Jd|j|j |j |j |j|jfS)z Check command line arguments and return the cluster configuration file path, the number of retries, the number of expected nodes, and the value of the --no-config-cache flag. zLaunch and verify a Ray cluster) description--no-config-cache store_truez3Pass the --no-config-cache flag to Ray CLI commands)actionhelpz --retriesz;Number of retries for verifying Ray is running (default: 3))typedefaultr z--num-expected-nodesz9Number of nodes for verifying Ray is running (default: 1)z--docker-override)disablelatestnightlycommitrzAOverride the docker image used for the head node and worker nodes)choicesrr z--wheel-overridez:Override the wheel used for the head node and worker nodescluster_configz&Path to the cluster configuration file)r r z%Cannot override both docker and wheel) argparseArgumentParser add_argumentintstr parse_argsdocker_overridewheel_overriderretriesno_config_cachenum_expected_nodes)parserargss |/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/autoscaler/launch_and_verify_cluster.pycheck_argumentsr%!s  $1R S S SF  B    J     H   ::: P     I   s)Q     D  ))d.AR.G.G.G./H.G         c|dkrdS|dkrdS|dkrttjdtjr%dtjdtjd d d St d tjt jd d S)a  Get the docker image to use for the head node and worker nodes. Args: docker_override: The value of the --docker-override flag. Returns: The docker image to use for the head node and worker nodes, or None if not applicable. rzrayproject/ray:latest-py310rzrayproject/ray:nightly-py310rz^[0-9]+.[0-9]+.[0-9]+$zrayproject/ray:.Nz-py310zGError: docker image is only available for release version, but we get: r)rematchray __version__ __commit__printsysexit)rs r$get_docker_imager2Xs("",, I % %-- H $ $ 8,co > > QS_QQs~bqb7IQQQ Q B03BB    HQKKK 4r&c|rtj|tjs(t d|t jddSdS)z Check if the provided file path is valid and readable. Args: file_path: The path of the file to check. Raises: SystemExit: If the file is not readable or does not exist. z/Error: Cannot read cluster configuration file: rN)is_fileosaccessR_OKr/r0r1) file_paths r$ check_filer9ss_     bi 27&C&C K KKLLL  r&cn|dg}|d|d||d<dS)Nsetup_commandsz9pip3 uninstall -y ray && pip3 install -U "ray[default] @ ")getappend) config_yaml wheel_urlr;s r$override_wheels_urlrAsO __%5r::NPIPPP%3K !!!r&c|di}||d<d|d<|d Jd|d Jd||d<dS) Ndockerimage ray_containercontainer_name head_imagezCannot override head_image worker_imagezCannot override worker_image)r=)r? docker_image docker_configs r$override_docker_imagerKsOOHb11M)M'&5M"#   \ * * 2 24P 2 2 2   ^ , , 4 46T 4 4 4)Kr&c htdtdtjdd}|d}|d}t j|d }t j|d }|d }|d}t j}tj |d }t|d 5} | |dddn #1swxYwYtjdddd|d|d|g dS)zRGet Azure service principal credentials from AWS Secrets Manager and authenticate.&======================================z5Getting Azure credentials from AWS Secrets Manager...secretsmanager us-west-2 region_namez#azure-service-principal-oss-release)SecretId SecretString client_id tenant_idz#azure-service-principal-certificatezazure_cert.pemwNazloginz--service-principalz --usernamez --certificatez--tenant)r/boto3clientget_secret_valuejsonloadstempfilemkdtempr5pathjoinopenwrite subprocess check_call) secrets_clientsecret_responsesecretrTrU cert_responsecerttmp_dir cert_pathfs r$azure_authenticaterns 2333 ABBB\"2 LLLN%5566O^ ,F 6"";/I 6"";/I#3364M  (D  G W&677I i       !            s(D  DDctdtdtjdd}d}d}tjd|}tjtj|s1tjtj|| |||tj |d d S) z=Download the ssh key from the S3 bucket to the local machine.rMzDownloading ssh key...s3rOrPzaws-cluster-launcher-testzray-autoscaler_59_us-west-2.pem~/.ssh/N) r/rYrZr5r` expanduserexistsdirnamemakedirs download_filechmod) s3_client bucket_namekey_namelocal_key_paths r$download_ssh_key_awsr}s 2333 "### T{;;;I.K0HW''(<((<(<==N 7>>"'//.99 : :5 BGOON33444 K>BBBH^U#####r&c6tdtdtj}d}d}||}||}t jd|}t jt j |s1t j t j || |t j |ddS)zGDownload the ssh key from the google cloud bucket to the local machine.rMzDownloading ssh key from GCP...z*gcp-cluster-launcher-release-test-ssh-keyszAray-autoscaler_gcp_us-west1_anyscale-bridge-cd812d38_ubuntu_0.pemrqrrN) r/rClient get_bucketget_blobr5r`rsrtrurvdownload_to_filenamerx)rZrzr{bucketblobr|s r$download_ssh_key_gcprs 2333 +,,,^  F?KRH  { + +F ??8 $ $DW''(<((<(<==N 7>>"'//.99 : :5 BGOON33444n---H^U#####r&c tdtdd}d}t|D]8} tj}|ddt jdddd t|gd d | t|dS#t j $r}td |d zd|dt|j dtjtd|j dtd|j d|}Yd}~2d}~wwxYw|)z Clean up the cluster using the given cluster configuration file. Args: cluster_config: The path of the cluster configuration file. rMzCleaning up cluster...Nr PYTHONPATHr,down-v-yTcheckcapture_outputenvzray down fails[r/z]: utf-8stdout: stderr: )r/ranger5environcopypoprdrunrcleanup_security_groupsCalledProcessErroroutputdecode traceback print_excstdoutstderr)r?r last_error num_triesires r$cleanup_clusterrs 2333 "### JI 9   *//##C GGL$ ' ' ' NdC,?,?@#      $K 0 0 0 FF,    8AaC88)888 9 9 9 !(//'** + + +   ! ! ! 8ahoog6688 9 9 9 8ahoog6688 9 9 9JJJJJJ  sA-B&&E)5B)E$$E)c`d}|dkr ||dS#tjj$rm}|jdddkr4d|z}t d|d |t j||d z }nt d |Yd}~dSYd}~nd}~wwxYw|dkdSdS) Nr )GroupIdErrorCodeDependencyViolationzWaiting zGs for the instance to be terminated before deleting the security group rzError deleting security group: )delete_security_groupbotocore exceptions ClientErrorresponser/timesleep) ec2_clientidretryr sleep_times r$cleanup_security_grouprs E "**   , ,R , 8 8 8 F".   z'"6*.CCCX vzvvrtvv :&&& ;;;<<<  "******s"B#ABB#c tdtdtjd}tjd}tj|r0tj|rtddStdtj|}tj|stj|d  tjd d d d dd|ddddg ddtddS#tj $r}tdtd|j dtd|j dtjdYd}~dSd}~wwxYw)zX Ensure that the SSH keys for Azure tests exist, and create them if they don't. rMz Ensuring Azure SSH keys exist...z#~/.ssh/ray-autoscaler-tests-ssh-keyz'~/.ssh/ray-autoscaler-tests-ssh-key.pubzAzure SSH keys already exist.Nz.Azure SSH keys not found. Creating new keys...T)exist_okz ssh-keygenz-trsaz-b4096z-fz-Nrz-Czray-autoscaler-azure)rrz$Successfully created Azure SSH keys.zError creating SSH keys:rrrr)r/r5r`rsrtrurvrdrrrrrr0r1)private_key_pathpublic_key_pathssh_dirrs r$ensure_ssh_keys_azurer/s 2333 ,---w))*OPPg(()RSSO w~~&''BGNN?,K,K -... :;;;goo.//G 7>>' " ", Gd++++ &     " 455555  ( ())) 4!(//'2244555 4!(//'2244555  s1EGA7GGc|did}|dkrdS tjdd}|dtgdd |d gdg }|d D]}t ||d dS#t $r}td|Yd}~dSd}~wwxYw)Nproviderr awsec2rOrPztag-key)NameValuesztag:ray-cluster-name cluster_name)FiltersSecurityGroupsrz#Error cleaning up security groups: )r=rYrZdescribe_security_groupsrr Exceptionr/)config provider_typerrsecurity_grouprs r$rr[sJJz2..226::M9\%[AAA 66&"e 3%n56 7   ''78 J JN ":~i/H I I I I J J 999 7A778888888889sA"B B>!B99B>rc|did}|dkrttdtdgd}|r|d|t |t j}|dd  tj |d d | n#tj $ro}t|j td |j d td|jd |d }~wwxYwtdtdd} d} | |kr dddt |dd|dzg}|r|dtj |d |d } nJ#tj $r2| dz } td| d|dt!jdYnwxYw| |k| sdtdtd|dt%||tdtd t'jdtdtd!t%||tdtd"d S)#ak Run the necessary Ray commands to start a cluster, verify Ray is running, and clean up the cluster. Args: cluster_config: The path of the cluster configuration file. retries: The number of retries for the verification step. no_config_cache: Whether to pass the --no-config-cache flag to the ray CLI commands. rr azurerMzStarting new cluster...)r,uprrrrNTrrrrzVerifying Ray is running...Frr,execrz2python -c 'import ray; ray.init("localhost:6379");z assert len(ray.nodes()) >= ')rrrz#Verification failed. Retry attempt z of z...<z!Error: Verification failed after z0 attempts. Cleaning up cluster before exiting...zExiting script.zRay verification successful.z'Finished executing script successfully.)r=rnr/r>rr5rrrrdrrrrrrrrrr0r1) r?rrr r!rcmdrrsuccesscounts r$run_ray_commandsrts:" OOJ3377??M 2333 #$$$ # # #C( &'''JJs>""### *//  CGGL$s$tEEEEE  ( ah 4!(//'2244555 4!(//'2244555  2333 '(((G E '// N##IJ5GJJJK C 0 ./// N3d 4 4 4 4G ,    QJE OOO7OOO P P P JrNNNNN # '//,   6777          ^444 6777       2333 ()))K000 2333 344444s,C''E%6A*E  E%A G>HH__main__z"Using cluster configuration file: z4Number of retries for 'verify ray is running' step: zUsing --no-config-cache flag: z6Number of expected nodes for 'verify ray is running': r-rMzOverriding ray wheel...: zOverriding docker image...: zUsing docker image: rr Fcache_stopped_nodesrravailability_zonegcp)gcloudr get-valueaccountT)rrrzActive account email:)rrrprojectzInjecting GCP project 'z$' into cluster configuration file... project_idvspherezVSPHERE provider detected.z-Provider type not recognized. Exiting script.z.yaml)suffix)r)A__doc__rr\r5r*rdr0r^rrpathlibrrYryaml google.cloudrr,"ray.autoscaler._private.aws.configrr%r2r9rArKrnr}rrrrrr__name__rrr r!rrr/ safe_load read_textr?rrdocker_override_imager=rrrPIPErrstrip account_emailrr1NamedTemporaryFiletemprcdumpencodeflushnamer&r$rs#       222222444n6   333******Z$$$*$$$4***Z&)))X999< Q5Q5Q5Q5h z T.))NJ~ E ?~ ? ?@@@ E J J JKKK E >>> %   JN<<<!    FF7OO UWW  %}555 JN<<<!    FF7OO UWW   Vj V V V   1; J -- ) # # 6777 *++++ 6777 =>>>  % $G 4 4 4  949[))0099::: di /CU                     ]sA*NNN