&`i1ddlmZmZmZddlmZddlmZm Z m Z m ZddlmZddlmZddlmZddlmZerddlmZed Gd d e Z ed Gd deZed GddeZ ed GddeZed Gdde Z edddGddeZedddGddeZed ddddd d!d d"eed#eed$eed%eeed&d'f d(Zed d)ed&d'fd*Z ed d+ed&d'fd,Z!ed ddd-d!d d"eed$eed&d'fd.Z"ed d/ed&d'fd0Z#gd1Z$dS)2) TYPE_CHECKINGOptionalType) Deprecated)CloudMirrorConfig LLMConfig LoraConfigModelLoadingConfig)LLMServingArgs) OpenAiIngress) LLMServer) PublicAPI) Applicationalpha) stabilityceZdZdZdS)rz1The configuration for starting an LLM deployment.N__name__ __module__ __qualname____doc__j/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/serve/llm/__init__.pyrr s;;DrrceZdZdZdS)r z=The configuration for starting an LLM deployment application.Nrrrrr r 'sGGDrr ceZdZdZdS)r z+The configuration for loading an LLM model.Nrrrrr r .s55Drr ceZdZdZdS)rz@The configuration for mirroring an LLM model from cloud storage.Nrrrrrr5sJJDrrceZdZdZdS)r z5The configuration for loading an LLM model with LoRA.Nrrrrr r <s??Drr zray.serve.llm.LLMServerz"ray.serve.llm.deployment.LLMServerF)oldnewerrorceZdZdS)r Nrrrrrrr r Hs Drr zray.serve.llm.LLMRouterz#ray.serve.llm.ingress.OpenAIIngressceZdZdS) LLMRouterNr#rrrr%r%Os  Drr%N) name_prefix bind_kwargsoverride_serve_optionsdeployment_cls llm_configr&r'r(r)returnrc.ddlm}||||||S)aqHelper to build a single vllm deployment from the given llm config. Examples: .. testcode:: :skipif: True from ray import serve from ray.serve.llm import LLMConfig, build_llm_deployment # Configure the model llm_config = LLMConfig( model_loading_config=dict( model_id="llama-3.1-8b", model_source="meta-llama/Llama-3.1-8b-instruct", ), deployment_config=dict( autoscaling_config=dict( min_replicas=1, max_replicas=2, ) ), accelerator_type="A10G", ) # Build the deployment llm_app = build_llm_deployment(llm_config) # Deploy the application model_handle = serve.run(llm_app) # Querying the model handle import asyncio model_handle = model_handle.options(stream=True) async def query_model(model_handle): from ray.serve.llm.openai_api_models import ChatCompletionRequest request = ChatCompletionRequest( model="qwen-0.5b", messages=[ { "role": "user", "content": "Hello, world!" } ] ) resp = model_handle.chat.remote(request) async for message in resp: print("message: ", message) asyncio.run(query_model(model_handle)) Args: llm_config: The llm config to build vllm deployment. name_prefix: Optional prefix to be used for the deployment name. bind_kwargs: Optional kwargs to pass to the deployment. override_serve_options: Optional serve options to override the original serve options based on the llm_config. deployment_cls: Optional deployment class to use. Returns: The configured Ray Serve Application for vllm deployment. r)build_llm_deployment)r*r&r'r(r))+ray.llm._internal.serve.core.server.builderr-)r*r&r'r(r)r-s rr-r-]sLN 5%    rllm_serving_argsc&ddlm}||S)a Helper to build an OpenAI compatible app with the llm deployment setup from the given llm serving args. This is the main entry point for users to create a Serve application serving LLMs. Examples: .. code-block:: python :caption: Example usage in code. from ray import serve from ray.serve.llm import LLMConfig, LLMServingArgs, build_openai_app llm_config1 = LLMConfig( model_loading_config=dict( model_id="qwen-0.5b", model_source="Qwen/Qwen2.5-0.5B-Instruct", ), deployment_config=dict( autoscaling_config=dict( min_replicas=1, max_replicas=2, ) ), accelerator_type="A10G", ) llm_config2 = LLMConfig( model_loading_config=dict( model_id="qwen-1.5b", model_source="Qwen/Qwen2.5-1.5B-Instruct", ), deployment_config=dict( autoscaling_config=dict( min_replicas=1, max_replicas=2, ) ), accelerator_type="A10G", ) # Deploy the application llm_app = build_openai_app( LLMServingArgs( llm_configs=[ llm_config1, llm_config2, ] ) ) serve.run(llm_app) # Querying the model via openai client from openai import OpenAI # Initialize client client = OpenAI(base_url="http://localhost:8000/v1", api_key="fake-key") # Basic completion response = client.chat.completions.create( model="qwen-0.5b", messages=[{"role": "user", "content": "Hello!"}] ) .. code-block:: yaml :caption: Example usage in YAML. # config.yaml applications: - args: llm_configs: - model_loading_config: model_id: qwen-0.5b model_source: Qwen/Qwen2.5-0.5B-Instruct accelerator_type: A10G deployment_config: autoscaling_config: min_replicas: 1 max_replicas: 2 - model_loading_config: model_id: qwen-1.5b model_source: Qwen/Qwen2.5-1.5B-Instruct accelerator_type: A10G deployment_config: autoscaling_config: min_replicas: 1 max_replicas: 2 import_path: ray.serve.llm:build_openai_app name: llm_app route_prefix: "/" Args: llm_serving_args: A dict that conforms to the LLMServingArgs pydantic model. Returns: The configured Ray Serve Application router. r)build_openai_appbuilder_config),ray.llm._internal.serve.core.ingress.builderr1)r/r1s rr1r1s8D  +; < < <ray.llm._internal.serve.serving_patterns.data_parallel.builderr:)r*r&r(r:s rr:r:tsE$  5   rdp_serving_argsc&ddlm}||S)a@Build an OpenAI compatible app with the DP attention deployment setup from the given builder configuration. Args: dp_serving_args: The configuration for the builder. It has to conform to the DPOpenAiServingArgs pydantic model. Returns: The configured Ray Serve Application. r)build_dp_openai_appr2)r;r>)r<r>s rr>r>s6  o > > >>r) rr r rr r-r1r7r:r>r r%)%typingrrrray._common.deprecationr/ray.llm._internal.serve.core.configs.llm_configr_CloudMirrorConfigr _LLMConfigr _LoraConfigr _ModelLoadingConfigr4r _LLMServingArgs,ray.llm._internal.serve.core.ingress.ingressr _OpenAiIngress.ray.llm._internal.serve.core.server.llm_serverr _LLMServerray.util.annotationsrray.serve.deploymentrr%strdictr-r1r7r:r>__all__rrrrPs0000000000......  +*****1000000 W         W     _    W     ,    W     *    W         !'KSX         !-            W"&"&-104 PPPP#P$ P %TN P T)_- PPPPPf We=te= e=e=e=e=P WV@V@-V@V@V@V@r W"&-1 #%TN   8 W??-????$   r