`iZ!ddlZddlmZmZddlmZddlmZddlZddl m Z ddl m Z e j dZd eejeeeffd ZGd d eZeGd dZeGddZdS)N) dataclassfield)Enum)Optional)logging)tracedContinuousBatchingLoggerreturnctjrtjd}tjtjtj|j}tj|}tj |}ntj j rytj j rVtjd}tj }|tj z }d}ntjd}d}d}d}||||fS)Ncudampsrcpu)torchr is_availabledevice empty_cache synchronizeget_device_properties total_memorymemory_reservedmemory_allocatedbackendsris_builtdriver_allocated_memoryrecommended_max_memory)rrreserved_memoryallocated_memorys /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/generation/continuous_batching/requests.pyget_device_and_memory_breakdownr s7 z  f%%        z77??L *44V<< :66v>>   ( ( * * u~/A/J/J/L/L e$$y88:: '%)*J*J*L*LLe$$  <2B BBc.eZdZdZdZdZdZdZdZdZ dZ d S) RequestStatusz5Status of a generation request through its lifecycle.pending prefillingprefilling_splitsplit_pending_remainderdecodingfinishedfailedN) __name__ __module__ __qualname____doc__PENDING PREFILLINGPREFILLING_SPLITSPLIT_PENDING_REMAINDERDECODINGFINISHEDFAILEDr!rr#r#5s9??GJ)7HH FFFr!r#ceZdZUdZeed<eeZee ed<eeZ ee ed<eeZ ee ed<dZ eeed<ejZeed <eejZe ed <dS) GenerationOutputa5Tracks the output of a generation request. Attributes: request_id (str): The ID of the generation request. prompt_ids (list[int]): The IDs of the prompt tokens. generated_tokens (list[int]): The generated tokens. logprobs (list[float]): The log probabilities of the generated tokens. error (Optional[str]): Any error message associated with the request. When None, the request was successful. status (RequestStatus): The status of the request. created_time (float): The time the request was created. request_iddefault_factory prompt_idsgenerated_tokenslogprobsNerrorstatus created_time)r+r,r-r.str__annotations__rlistr<intr=r>floatr?rr#r/r@timerAr6r!rr8r8As  OOO!E$777JS 777"'%"="="=d3i===!E$777Hd5k777E8C=)1FM111% :::L%:::::r!r8ceZdZUdZeed<dZeee ed<dZ eee ed<e eZ ee ed<e eZ ee ed<d Ze ed <d Ze ed <ejZeed <d Ze ed<dZe ed<e ejZeed<dZeeed<dZeeefed<edefdZejdefdZdZde fdZde fdZ e!de de"fdZ#dZ$dZ%dS) RequestStateayTracks the state of a generation request through its lifecycle. Attributes: request_id (str): The ID of the generation request. full_prompt_ids (list[int] | None): The tokens IDs of the full prompt. prompt_ids (list[int] | None): The tokens IDs currently being processed. remaining_prompt_ids (list[int]): The tokens IDs remaining to be processed (for split requests). static_outputs (list[int]): The generated tokens. allocated_blocks (int): The number of blocks allocated to the request. position_offset (int): The current position in the sequence for position_ids. status (RequestStatus): The status of the request: can be one of PENDING, PREFILLING, PREFILLING_SPLIT, SPLIT_PENDING_REMAINDER, DECODING, FINISHED, FAILED max_new_tokens (int): The maximum number of new tokens to generate. eos_token_id (int): The ID of the end-of-sequence token. created_time (float): The time the request was created. error (Optional[str]): Any error message associated with the request. When None, has had no error yet. r9Nfull_prompt_idsr<r:remaining_prompt_idsstatic_outputsrallocated_blocksposition_offset_statusmax_new_tokens eos_token_idrAr?)rRrRlifespanr c|jS)N)rOselfs rr@zRequestState.status{s |r!valuec|jtjkrtjdf|_nI|tjkr9|jdtjf|_|||_dS)NrRr)rOr#r/rGrTr4log_end_of_request)rWrXs rr@zRequestState.statussj <=0 0 0!Y[["-DMM m, , ,!]1-ty{{;DM  # # % % % r!c t|j}|}|jd|jz }|jd|jz }t d|jd|d|d|d| dS)NrzRequest z finished: prefill_len = z decode_len = z start_time = z end_time = )lenrJ generated_lenrTrAloggerinfor9)rW prefill_len decode_len start_timeend_times rrZzRequestState.log_end_of_requests$.// '')) ]1%(99 =#d&77 qt q q; q qJ q qT^ q qdl q q     r!c|jS)zCGet the current length of the sequence (prompt + generated tokens).)rNrVs r current_lenzRequestState.current_lens ##r!c*t|jS)z*Get the number of tokens generated so far.)r]rLrVs rr^zRequestState.generated_lens4&'''r!token_idc|jtjkrdS||jko |jdk}||jk}|r|r|j|g|s|rtj|_dSdS)zUpdate the request with a newly generated token and check for completion. Args: token_id: The token ID to add to the output sequence Returns: bool: True if the request is now complete, False otherwise FrRT) r@r#r3rSr^rQrLextendr4)rWrhis_eos is_max_lens rupdate_with_tokenzRequestState.update_with_tokens ;-0 0 05T..J43D3J''))T-@@  36 3   & &z 2 2 2  Z '0DK4ur!c Fd|jd|jd|dt|jdt|jd|jdt|jd|jd |j g }d d |zd zS) Nz request_id=zstatus=z out_tokens=z query_length=zremaining_tokens=z kv_length=zfull_prompt_length=zallocated_blocks=zgenerated_tokens=zRequestState( z, z )) r9rOr^r]r<rKrNrJrMrLjoin)rWmsgs r__repr__zRequestState.__repr__s +$/ + + $dl $ $ 0$,,.. 0 0 2C00 2 2 @D$= > > @ @ /- / / =#d&:";"; = = 7 5 7 7 5 3 5 5  #W\\#%6%66>>r!c^t|j|j|j|jg|jS)z7Convert the request state to a GenerationOutput object.)r9r<r@r=r>r?)r8r9rJr@rLr?rVs rto_generation_outputz!RequestState.to_generation_outputs7+;!0*     r!)&r+r,r-r.rBrCrJrrDrEr<rrKrLrMrNr#r/rOrQrSrGrArFr?rTtuplepropertyr@setterrZrfr^r boolrmrqrsr6r!rrIrIXs&OOO+/OXd3i(///&*Jc#***&+eD&A&A&A$s)AAA %d ; ; ;NDI;;;cOS*2G]222NCL#% :::L%:::E8C=$,HeE5L!,,,  X ]M]   $S$$$$(s((((  #$ V4 ? ? ?      r!rI)rG dataclassesrrenumrtypingrr utils.loggingr utils.metricsr getLoggerr_rtrrEr r#r8rIr6r!rr~sd (((((((( $$$$$$######  5 6 6Cu|S#s/J)KCCCC,     D    ;;;;;;; ;, s s s s s s s  s s s r!