&`i`a zddlZddlZddlZddlZddlmZddlmZmZm Z m Z m Z ddl Z ddl Z ddlmZddlmZejeZdZdZdZd Zd Zd Zd d iZdZdZdZdZdZ dZ!dZ"dZ#dZ$dZ%dZ&dZ'dZ(dZ)dZ*dZ+de,de,de,de e-fd Z.hd!hd"hd#/e.d$d$d%hd#/e.d%d%d&hd'hd'dZ0d(e-de e-fd)Z1d*e-fd+Z2d*e-de,fd,Z3d*e-de,fd-Z4d.e-d*e-de e-fd/Z5d0Z6d.e-d1e-de,fd2Z7d.e-d*e-de e-fd3Z8Gd4d5eZ9dS)6N) lru_cache)DictListOptionalSetTuple)AcceleratorManager) PlacementGroupSchedulingStrategy)TPU_ACCELERATOR_TYPE TPU_TOPOLOGY TPU_WORKER_IDTPU_NAMEzGhttp://metadata.google.internal/computeMetadata/v1/instance/attributes/zMetadata-FlavorGooglezaccelerator-typeztpu-envz instance-idzagent-worker-numberTPU_VISIBLE_CHIPS(RAY_EXPERIMENTAL_NOSET_TPU_VISIBLE_CHIPSTPU_CHIPS_PER_HOST_BOUNDSz1,1,1z1,2,1TPU_HOST_BOUNDSr r ) v5litepodv6e)v2v3v4v5prrmax_xmax_ymax_zreturnc "t}tt|dztD]a}tt|dztD]@}tt|dztD]}||d|d| Ab|S)zvReturns a set of larger 3D TPU topologies given the max x,y,z value. Using DEFAULT_TPU_NUM_CHIPS_PER_HOST as incrementr x)setrangeDEFAULT_TPU_NUM_CHIPS_PER_HOSTadd)rrr topologiesr#yzs q/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/_private/accelerators/tpu.py_get_larger_3d_topologiesr,DsJ & 3Q / / *EAI7U   / /A. . / / !~~a~~!~~....  / / >4x44x88x88x1616x16>r.r/r0r1r216x3232x32>2x2x12x2x22x2x42x4x4 >2x8r.r/r0r1r2keycb tjtjt |t }|jdkr|jr|jStj d|jd|jn1#tj $r}tj d|Yd}~nd}~wwxYwdS)zPoll and get TPU metadata.)headersz2Unable to poll TPU GCE Metadata. Got status code: z and content: z'Unable to poll the TPU GCE Metadata: %sN) requestsgetospathjoinGCE_TPU_ACCELERATOR_ENDPOINTGCE_TPU_HEADERS status_codetextloggingdebugRequestException)r=accelerator_type_requestes r+_get_tpu_metadatarOisD#+< GLL5s ; ;#$ $ $  % 0C 7 7(- 8,0 0 M< 8 D<<49<<      $DDD ?CCCCCCCCD 4sAA>$A>>B, B''B,accelerator_typecn|tstd|dtdS)NzInvalid accelerator type: z. Must start with one of: ) startswithVALID_TPU_TYPES ValueErrorrPs r+_accelerator_type_checkrVsI  & & 7 7  f)9 f fUd f f     r-cft||trdStS)Nr)rVrRSINGLE_HOST_8_CHIPS_TPU_TYPESr&rUs r+"get_num_tpu_visible_chips_per_hostrYs3,---""#@AAq ))r-cft||trdStS)Nr )rVrRSINGLE_CORE_TPU_TYPESDEFAULT_TPU_NUM_CORES_PER_CHIPrUs r+get_tpu_cores_per_chipr]s3,---""#899q ))r-topologyc`|r|sdS d}|dD]}|t|z}|dd}|d|S#t $r}t d|d|d |d}~wwxYw) zGInfer the TPU pod type (e.g. v4-32) from topology and accelerator type.Nr r#ztpu--z(Failed to infer pod type from topology 'z ' and type '')striplowersplitintreplace ExceptionrT)r^rP num_chipsvalue generationrNs r+ infer_tpu_pod_type_from_topologyrls +t  ^^%%++--33C88 $ $E U #II%++--55fbAA **y***  -x - -) - - -   sA>B B-B((B-ctjdd}|t|d}tj|S)Nr)num_cpusc4tSN)TPUAcceleratorManagerget_current_node_tpu_namer-r+_get_tpu_slice_namez9fetch_tpu_slice_name_from_pg.._get_tpu_slice_names$>>@@@r-)placement_groupplacement_group_bundle_index)scheduling_strategy)rayremoteoptionsr rB)pgrt tpu_name_refs r+fetch_tpu_slice_name_from_pgr}sZAAA'..<Q   / fhh  7<  r-accelerator_versionct}d}|dD]}|t |z}|dkr/|t vr|S|S)aGet the number of chips per host (aka VMs) based on topology and accelerator version. The current rule is as follows: Default chips per host is 4. If accelerator_version is v5e or v6e AND topology product <= 8, the chips per host will just be the proudct. i.e. 1, 4, or 8 If accelerator_version is v5e or v6e AND topology product > 8, the chips per host will be 4 If accelerator_version is v5p or other versions, the chips per host will be 4 Args: topology: The TPU topology string (e.g. "2x2x2"). accelerator_version: The accelerator version of the node (e.g. "V4", "v4"). Returns: A int representing the number of chips per host (aka VM) r r#r)r&rcrdrerfrX)r^r~chips_per_host total_chipsrjs r+get_chips_per_hostrs4NK!!''))//44""s5zz!  q  % % ' ' - - / /3P P P r-ct||}|dSd|d}tjd|ddig|g}tdd }tj|g| \}}|s5td ||tj t|}|td |S) aReserves a TPU slice using its head resource and returns the slice name. This enables gang scheduling of training workers with multi-host TPUs. This is used by JaxTrainer with TPUs in Ray Train. Args: topology: The TPU topology string (e.g. "2x2x2"). accelerator_type: The accelerator type of the node (e.g. "TPU-V4"). Returns: A string representing a unique TPU slice name. N0)zray.io/tpu-worker-idzray.io/tpu-pod-typeTPU--headr )bundlesbundle_label_selectorz)Waiting to reserve multi-host slice head.d)timeoutzFailed to reserve TPU head for slice with shape: {}. Ensure your cluster has sufficient resources. Requesting TPU head node with labels: {}. Current resources: {}zFailed to retrieve TPU slice name after reserving head placement group. Ensure that TPU slice metadata is available and correctly configured on multi-host nodes.) rlrxutilruloggerrKwaitready TimeoutErrorformatavailable_resourcesr} RuntimeError) r^rPpod_typehead_label_selectorhead_placement_grouprr_ slice_names r+reserve_tpu_slicers(0:JKKHt!$'833((((!,-234  LL<===Gx-33556HHHHE1   ??Ev-s/F/H/H@@   ..BCCJ h   r-ceZdZdZedefdZedefdZedee efdZ ee de fdZ ededefdZed ed edefd Zed edeeeeffd Zede eddfdZedeefdZedeefdZedee fdZedee fdZedeefdZedeefdZedeeeeffdZedeeeffdZdS)rqzGoogle TPU accelerators.r!cdS)NTPUrsrsr-r+get_resource_namez'TPUAcceleratorManager.get_resource_namesur-ctSrp)TPU_VISIBLE_CHIPS_ENV_VARrsr-r+#get_visible_accelerator_ids_env_varz9TPUAcceleratorManager.get_visible_accelerator_ids_env_vars((r-ctjtd}|dS|dkrgSt |dS)Nr`,)rCenvironrBrqrlistre)tpu_visible_chipss r++get_current_process_visible_accelerator_idszATPUAcceleratorManager.get_current_process_visible_accelerator_idsscJNN ! E E G G    $4  " "I%++C00111r-ctjd}|rt|S tjd}d|D}t|S#t$r&}t d|Yd}~dSd}~wwxYw)aAttempt to detect the number of TPUs on this machine. TPU chips are represented as devices within `/dev/`, either as `/dev/accel*` or `/dev/vfio/*`. Returns: The number of TPUs if any were detected, otherwise 0. z /dev/accel*z /dev/vfiocTg|]%}|t|&Srs)isdigitrf).0entrys r+ zKTPUAcceleratorManager.get_current_node_num_accelerators..5s+WWWeu}}Ws5zzWWWr-z#Failed to detect number of TPUs: %sNr)globlenrClistdirFileNotFoundErrorrrK) accel_files vfio_entriesnumeric_entriesrNs r+!get_current_node_num_acceleratorsz7TPUAcceleratorManager.get_current_node_num_accelerators$si ..  ${## # :k22LWW|WWWO'' '     LL> B B B11111 s.A B BBtpu_accelerator_typec\tjd}||sdSdS)aCheck whether the tpu accelerator_type is formatted correctly. The accelerator_type field follows a form of v{generation}-{cores/chips}. See the following for more information: https://cloud.google.com/sdk/gcloud/reference/compute/tpus/tpu-vm/accelerator-types/describe Args: tpu_accelerator_type: The string representation of the accelerator type to be checked for validity. Returns: True if it's valid, false otherwise. z^v\d+[a-zA-Z]*-\d+$FT)recompilematch)rexpected_patterns r+is_valid_tpu_accelerator_typez3TPUAcceleratorManager.is_valid_tpu_accelerator_type;s6 :&<==%%&:;; 5tr-tpu_accelerator_version tpu_topologyc$|dd}|tvs3|t|vrdSdS)a*Check whether the tpu topology is valid. The accelerator_type field follows a form of v{generation}. The accelerator_topology field follows either the form {A}x{B} or {A}x{B}x{C} depending on the v{generation} Args: tpu_accelerator_version: The string representation of the accelerator version. (e.g. v6e, V5P) tpu_topology: The string representation of the accelerator topology to be checked for validity Returns: True if it's valid topology, false othrwise rarFT)rcrdreVALID_TPU_TOPOLOGY)rrtpu_version_formatteds r+!is_valid_tpu_accelerator_topologyz7TPUAcceleratorManager.is_valid_tpu_accelerator_topologyPs"!8 = = ? ? E E G G M Mc R RST U ! ' ' ) )1C C C!!##))++%&;<==5tr-quantityc6|tvrdd|dtfSdS)NFz)The number of requested 'TPU' was set to zA which is not a supported chip configuration. Supported configs: )TN)TPU_VALID_CHIP_OPTIONS)rs r+"validate_resource_request_quantityz8TPUAcceleratorManager.validate_resource_request_quantityjsB 1 1 1,H,,),,  .s888c!ff888r-r r )rCrrBNOSET_TPU_VISIBLE_CHIPS_ENV_VARrrqrpop!TPU_CHIPS_PER_HOST_BOUNDS_ENV_VARTPU_HOST_BOUNDS_ENV_VARrEr'TPU_CHIPS_PER_HOST_BOUNDS_1_CHIP_CONFIGTPU_SINGLE_HOST_BOUNDS'TPU_CHIPS_PER_HOST_BOUNDS_2_CHIP_CONFIG)rnum_visible_tpu_chipsnum_accelerators_on_nodes r++set_current_process_visible_accelerator_idszATPUAcceleratorManager.set_current_process_visible_accelerator_idsxs4" :>>9 : :  F #$5 6 6 ! C C E E ! !$< < < JNN)Q..q   MA B B B4r-c< tjtx}r|St t }|r,t jd|}|r|dSdSdS#t$r }tj d|Yd}~dSd}~wwxYw)NrzTOPOLOGY:\s*'([^']+)'r zCould not get TPU topology: %s) rCrrBGKE_TPU_TOPOLOGY_ENV_VARrOGCE_TPU_ENV_KEYrsearchgrouprTrJrK)r^tpu_envrNs r+get_current_node_tpu_topologyz3TPUAcceleratorManager.get_current_node_tpu_topologys :>>*BCCCx 'O<<>!,,, - ---    M:A > > >44444 s'A1AA11 B;BBcdtdttfd}d}t}|+||}|td||t jd|S)aAttempt to detect the TPU accelerator type. The output of this function will return the "ray accelerator type" resource (e.g. TPU-V4) that indicates the TPU version. We also expect that our TPU nodes contain a "TPU pod type" resource, which indicates information about the topology of the TPU pod slice. We expect that the "TPU pod type" resource to be used when running multi host workers, i.e. when TPU units are pod slices. We expect that the "ray accelerator type" resource to be used when running single host workers, i.e. when TPU units are single hosts. Returns: A string representing the TPU accelerator type, e.g. "TPU-V2", "TPU-V3", "TPU-V4" if applicable, else None. rr!c|dt|ddzS)Nrrar)rreupperrs r+$tpu_pod_type_to_ray_accelerator_typezeTPUAcceleratorManager.get_current_node_accelerator_type..tpu_pod_type_to_ray_accelerator_type(s6C 2 23 7 7 : @ @ B BCCC Cr-NrzLWhile trying to autodetect a TPU type, received malformed accelerator_type: zFailed to auto-detect TPU type.)rrrqrrinforJ)rray_accelerator_typers r+!get_current_node_accelerator_typez7TPUAcceleratorManager.get_current_node_accelerator_types. D D c] D D D D $,JJLL  ##G#G)$$$ $+ K ?)HHJJ  ADYJs{= >,JJLL  M@LJs{< =(FFHH  I@HJs{< =r-)__name__ __module__ __qualname____doc__ staticmethodrrrrrrrrfrboolrrfloatrrrrrrrrrrrrrrsr-r+rqrq s""s\))))\) 2$s)9L 2 2 2\ 2Y[[s[\*CD\(!$47 \2    tXc]" #    \  (I9(I (I(I(I\(IT8C=\@x}\2HSM\  \" 8C=   \ +$x}+$+$+$\+$Z?8Dersm  33333333333333 DDDDDDKKKKKK  8 $ $&#9 )+!N%h/,'-/"L %@!*1'*1'+ "#!"!5-@SSSX* 1 0 0 B B B . . . 4 4!!"b"--       e % %b"b 1 122>>> 8 8 8"38C=. c    *******S*S****%( c]& ! ! !33:333c]3333lUUUUU.UUUUUr-