&`i" d dl Z d dlZd dlZd dlZddlmZ ddlmZ e j e Z
dZ dZ dZ
dZd Zd
Zd Zd Zd
ZdZdZdZ G d de ZdS ) N )RayOnSparkStartHook)get_spark_sessionDATABRICKS_HOSTDATABRICKS_TOKENDATABRICKS_CLIENT_IDDATABRICKS_CLIENT_SECRETc t t j v rt t j v p8t t j v o%t t j v ot
t j v S N)r osenvironr r r r/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/util/spark/databricks_hook.pyverify_databricks_auth_envr sK rz ) L.>"*.L 2:% 3 BJ . 3 $
2r c z dd l }| }|t d |j d | S )Nr zNo IPython environment. user_global)IPython get_ipython RuntimeErrorns_table) func_namer ip_shells r get_databricks_functionr sD NNN""$$H4555 ] +I 66r c t d S )N displayHTML)r r r r $get_databricks_display_html_functionr " s "= 1 11r c ^ ddl m} | }| S )zo
Return databricks entry_point instance, it is for calling some
internal API in databricks runtime
r )UserNamespaceInitializer) dbruntimer getOrCreateget_spark_entry_point)r user_namespace_initializers r get_db_entry_pointr$ & s:
322222!9!E!E!G!G % ; ; = ==r c | j j j j j j j }|
d }|
d }|
d }d| d| d| d}d| d| }t d t d | t d
| d | d d
S )aE
This helper function create a proxy URL for databricks driver webapp forwarding.
In databricks runtime, user does not have permission to directly access web
service binding on driver machine port, but user can visit it by a proxy URL with
following format: "/driver-proxy/o/{orgId}/{clusterId}/{port}/".
tagsorgId clusterIdz/driver-proxy/o//zhttps://dbc-dp-z.cloud.databricks.comz@To monitor and debug Ray from Databricks, view the dashboard at zM
N)
_jvmcom
databricksbackenddaemondriver DriverLocalcommandContextget toStringMapapplyprintr )
spark_contextporttitle driverLocalcommandContextTagsr' r(
proxy_link proxy_urls r #display_databricks_driver_proxy_urlr> 1 s $(3;BIUK$335599;;GGIIOOPVWW $ $W - -E"((55I?E??I?????JJ%JJjJJI
LMMM /i//*(** r ,DATABRICKS_RAY_ON_SPARK_AUTOSHUTDOWN_MINUTESz/local_disk0/tmpc 6 e Zd Zd Zd Zd Zd Z fdZ xZS )$DefaultDatabricksRayOnSparkStartHookc t S r ) _DATABRICKS_DEFAULT_TMP_ROOT_DIR)selfs r get_default_temp_root_dirz>DefaultDatabricksRayOnSparkStartHook.get_default_temp_root_dirX s //r c J t t j |d d S )NzRay Cluster Dashboard)r> r sparkContext)rE r8 s r on_ray_dashboard_createdz=DefaultDatabricksRayOnSparkStartHook.on_ray_dashboard_created[ s/ + ,d4K
r c L t | j rdn2t t j t d dk r t d d S dk rt dt d
n+# t $ r t d Y d S w xY wt d dt d fd
}t j |d d S )
Nr 30zThe Ray cluster will keep running until you manually detach the Databricks notebook or call `ray.util.spark.shutdown_ray_cluster()`.zYou must set 'z' to a value >= 0.a4 Failed to retrieve idle time since last notebook execution, so that we cannot automatically shut down Ray cluster when Databricks notebook is inactive for the specified minutes. You need to manually detach Databricks notebook or call `ray.util.spark.shutdown_ray_cluster()` to shut down Ray cluster on spark.ziThe Ray cluster will be shut down automatically if you don't run commands on the Databricks notebook for z? minutes. You can change the auto-shutdown minutes by setting 'z' environment variable, setting it to 0 means that the Ray cluster keeps running until you manually call `ray.util.spark.shutdown_ray_cluster()` or detach Databricks notebook.c dz dz } j rd S }|| k rDddlm} |j 5 |j u r| d d d n # 1 swxY w Y d S t j t )N< i Tr ) cluster_init)
is_shutdown+getIdleTimeMillisSinceLastNotebookExecutionray.util.sparkrN _active_ray_cluster_rwlock_active_ray_clustershutdown_ray_clustertimesleep.DATABRICKS_AUTO_SHUTDOWN_POLL_INTERVAL_SECONDS)auto_shutdown_millis idle_timerN auto_shutdown_minutes db_api_entryray_cluster_handlers r auto_shutdown_watcherzVDefaultDatabricksRayOnSparkStartHook.on_cluster_created..auto_shutdown_watcher s #82#=#D
K&2 F ( T T V V 333;;;;;;%@ @ @.,2RRR ( = = ? ? ?@ @ @ @ @ @ @ @ @ @ @ @ @ @ @ F
IJJJ
Ks A%%A),A)T)targetr/ )r$ is_globalfloatr r
r3 r@ _loggerinfo
ValueErrorrP Exceptionwarning threadingThreadstart)rE r\ r] rZ r[ s ` @@r on_cluster_createdz7DefaultDatabricksRayOnSparkStartHook.on_cluster_created` s )++ > %& ! !$)
KTRR% % ! !A % % LL;
F 1 $ $ #@# # #
D D F F F F OO(
FF
*$
*
* =
*
*
*
K K K K K K K& 5dCCCIIKKKKKs B$ $$C C c L t }| | d S r )r$ registerBackgroundSparkJobGroup)rE job_group_idr[ s r on_spark_job_createdz9DefaultDatabricksRayOnSparkStartHook.on_spark_job_created s' )++ 44\BBBBBr c i t ddd}t rt j t
|t
< t t j v r t j t |t < n\t j t |t < t j t |t < n d} t d| d |S )Neth0TRUE)GLOO_SOCKET_IFNAMEDISABLE_MLFLOW_INTEGRATIONa~ MLflow support is not correctly configured within Ray tasks.To enable MLflow integration, you need to set environmental variables DATABRICKS_HOST + DATABRICKS_TOKEN, or set environmental variables DATABRICKS_HOST + DATABRICKS_CLIENT_ID + DATABRICKS_CLIENT_SECRET before calling `ray.util.spark.setup_ray_cluster`, these variables are used to set up authentication with Databricks MLflow service. For details, you can refer to Databricks documentation at Databricks PAT auth or Databricks OAuth.zz
)
super custom_environment_variablesr r r
r r r r r )rE confwarn_msg __class__s r rt zADefaultDatabricksRayOnSparkStartHook.custom_environment_variables s
gg2244
#)
+1
& ' ' $&J$?D ! 2:--)+4D)E%&& .0Z8L-M)*13 rW r@ rD rB r r r r s 0 0 0 0 0 0 $ $ $ $ $ $
'
H
%
%#% - 5 7 7 72 2 2> > > : 23 .2 -
$6 F F F F F+> F F F F Fr