.`i8+jddlmZddlmZmZddlmZddlmZee Z GddZ dS))product) CUDAGraphMode VllmConfig)BatchDescriptor) init_loggerc eZdZdZdefdZddZdeded ede fd Z d e d e fd Z dde defdZ ddeded ededee e ff dZdeee ee ffdZdS)CudagraphDispatchera Runtime cudagraph dispatcher to dispatch keys for multiple set of cudagraphs. The dispatcher stores two sets of dispatch keys, one for PIECEWISE and one for FULL cudagraph runtime mode. The keys are initialized depending on attention support and what cudagraph mode is set in CompilationConfig. The keys stored in dispatcher are the only source of truth for valid cudagraphs that can be dispatched at runtime. At runtime, the dispatch method generates the runtime cudagraph mode (FULL, PIECEWISE, or NONE for no cudagraph) and the valid key (batch descriptor) based on the input key. After dispatching (communicated via forward context), the cudagraph wrappers will trust the dispatch key to either capture or replay (if the mode matches), or pass through to the underlying runnable without cudagraph (if the mode does not match or mode is NONE). vllm_configc||_|j|_|jjsdnd|jjjz|_t jtt jti|_ |jj rJ|j s1Jd|jj d|jj d|jjd|_t j|_ dS)NzCompilation mode should be CompilationMode.VLLM_COMPILE when cudagraph_mode piecewise cudagraphs is used, and attention should be in splitting_ops or inductor splitting should be used. cudagraph_mode=z, compilation_mode=z, splitting_ops=F)r compilation_configspeculative_confignum_speculative_tokensuniform_decode_query_lenr PIECEWISEsetFULLcudagraph_keyscudagraph_moderequires_piecewise_compilationis_attention_compiled_piecewisemode splitting_opskeys_initializedNONE)selfr s p/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/v1/cudagraph_dispatcher.py__init__zCudagraphDispatcher.__init__s&"-"@#6 PAAT%8OO %  #SUU  J  '6UUWW &FFHH  E#5D  E E !% 7 <  E E "4B  E E  I!&+0returnNc|jj}|jj}dg|dzz|_t ||dzgzdg|zD]3\}}t ||D]}||kr ||j|<||j|<4|jjr]|jtj krJ|jjD]?}||jjkr)|j|}||krtd|d|ds radd_cudagraph_keyz%CudagraphDispatcher.add_cudagraph_keytsY 79KLLLL F F FMLL L)--.>?????rr rrc F||_|tjkr d|_dS||jjr|jjrddg}ndg}ndg}| tjkrot|jj |D]T\}}| | | |d|U|tjkr|rx|jjjzfd|jj D}t||D]:\}}| tj| |d|;d|_dS)NTFc,g|]}|k|k|SrD).0xmax_num_tokensrs r zACudagraphDispatcher.initialize_cudagraph_keys..s9222&&10H+H+H+H+H+Hr)rrrrr0r lora_configr cudagraph_specialize_lora mixed_moderr#rAr< relax_for_mixed_batch_cudagraphs decode_moderseparate_routiner7r8)rrr lora_casesr-r3"cudagraph_capture_sizes_for_decoderGs ` @rinitialize_cudagraph_keysz-CudagraphDispatcher.initialize_cudagraph_keys|s - ]/ / /$(D ! F --///   ' !&@ $"E] "V J  $ $ & &-*< < < ''?!!   H&&"--//88E86688   & & ( (M,> > >//11 ?)"3@A 222220H222 . !((JJ W W   H&&!&88T8LL !%rF disable_fullc |jr%|jtjks||jjkrtjt |fS||||}|}|sN||j tj vrtj |fS||j tj vrtj |fS||j tj vrtj |fStjt |fS)a! Given conditions(e.g.,batch descriptor and if using piecewise only), dispatch to a cudagraph runtime mode and the valid batch descriptor. A new batch descriptor is returned as we might dispatch a uniform batch to a graph that supports a more general batch (uniform to non-uniform). Args: num_tokens: Number of tokens in the batch. uniform_decode: Whether the batch is uniform decode (i.e. uniform and query length is uniform_decode_query_len). has_lora: Whether LoRA is active. disable_full: If True, skip FULL cudagraph checks and return PIECEWISE or NONE only. (can be used for features like cascade attention that are not supported by full cudagraphs) ) rrrrr r"rr<rLrrr)rr1r2r3rR batch_descrelaxed_batch_descs rdispatchzCudagraphDispatcher.dispatchs .% C"m&888D3NNN %z'B'BB B99   (HHJJ >T01CDDD$):55"T%89K%LLL$)+=== !4]5L!M M M *,>> >!?:#>#>>>rc|jr|jtjkrgSg}tjtjfD]M}t |j|}|r/|dd| ||fN|S)a Returns capture descriptors for cudagraph capturing. Returns: List of (runtime_mode, batch_descriptors) tuples, ordered PIECEWISE first then FULL. Batch descriptors are sorted largest-first for memory efficiency. c|jS)N)r1)ds rz7CudagraphDispatcher.get_capture_descs..srT)keyreverse) rrrrrrlistrsortappend)rresultrdescss rget_capture_descsz%CudagraphDispatcher.get_capture_descss$ (;}?Q(Q(QI",m.@A - -D,T233E - 55t DDD tUm,,, r)r N)r )FFF)__name__ __module__ __qualname____doc__rrr0intboolrr<rrArQtuplerVr]rbrDrrr r sl$1J1111@@  /3 ?C     *@)@=L@@@@NO:%:%+:%GJ:%:%:%:%~ %" 1?1?1?1? 1?  1? }o- . 1?1?1?1?f4mT/=R.R(S#Trr N) itertoolsr vllm.configrrvllm.forward_contextr vllm.loggerrrcloggerr rDrrros11111111000000###### X  ttttttttttr