`itdZddlZddlmZmZddlZddlmZddlmZddl m Z ddl m Z m Z mZdd lmZdd lmZmZdd lmZdd lmZmZdd lmZddlmZmZddlmZddl m!Z!ej"e#Z$Gddej%Z&Gddej%Z'Gddej(Z)Gddej(Z*GddeZ+eGddeZ,Gdde,Z-ed !Gd"d#e,Z.ed$!Gd%d&e,eZ/d&dgZ0dS)'z/PyTorch TrOCR decoder model (based on RoBERTa).N)OptionalUnion)nn)CrossEntropyLoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)_prepare_4d_attention_mask!_prepare_4d_causal_attention_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModel)auto_docstringlogging)deprecate_kwarg) TrOCRConfigcheZdZdZdedeffd Z d dejded eejffd Z xZ S) TrOCRLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcjd|_t||jz|dS)N)offsetsuper__init__)selfrr __class__s |/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/trocr/modeling_trocr.pyr z(TrOCRLearnedPositionalEmbedding.__init__0s3  $+5}EEEEErN input_idspast_key_values_length position_idsc0|V|jdd\}}tj|||ztj|jj|d}n|d}t ||j zS)z3`input_ids' shape is expected to be [bsz x seqlen].Nr)dtypedevicer) shapetorcharangelongweightr*expand unsqueezerforwardr)r!r%r&r'bszseq_lenr"s r#r3z'TrOCRLearnedPositionalEmbedding.forward6s  $?2A2.LC <&(>(HPUPZcgcncufS"oo L(11!44Lww|dk9:::r$)rN) __name__ __module__ __qualname____doc__intr r-Tensorrr3 __classcell__r"s@r#rr+sFsF3FFFFFFpt ; ; ;?B ;V^_d_kVl ; ; ; ; ; ; ; ; ; ;r$rc \eZdZdZd dedededeeffd Zdej ffd Z xZ S) TrOCRScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. ?rr padding_idx embed_scalec\t|||||_dSN)rr rB)r!rrrArBr"s r#r z!TrOCRScaledWordEmbedding.__init__Ls-  DDD&r$r%cVt||jzSrD)rr3rB)r!r%r"s r#r3z TrOCRScaledWordEmbedding.forwardPs!wwy))D,<<===CCEEOOPQRRUXUbUbcdUeUeei338a@@@EEnVXYY 1  ! !)S%+na"@"@AqIIIC  ""#C QQQ vve-//000r$rr%r&c>|\}}|||j||j}|jdz|z}|j||jdkr&|||j|j|_|j|j|_|j d| d ||d }|S)Nrrr+) size"create_position_ids_from_input_idsrAr\r*rMrLrrK index_selectrZdetach)r!r%r&r4r5r'max_posxs r#r3z*TrOCRSinusoidalPositionalEmbedding.forwardrs ~~'' W>>y$JZ\rssvv    "Q&0 < 7T\->->q-A-A#A#A--gt7I4K[\\DL|t'9:: L % %a):):2)>)> ? ? D DS'SU V V ] ] _ _r$c||}tj|d||z|z}||zS)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. rrQ)ner:r-cumsumtype_asr/)r!r%rAr&maskincremental_indicess r#rbzETrOCRSinusoidalPositionalEmbedding.create_position_ids_from_input_idssg||K((,,..$|Da888@@FFI__cgg"''))K77r$rD)r)r6r7r8r9r:rr staticmethodrLr-no_gradr;r3rbr<r=s@r#rHrHTs(NNDDcD#DHUXMDDDDDD11c1#1HUXM111\1$U]__s_&bc 8 8 847 8QYZ]Q^ 8 8 8 8 8 8 8 8r$rHceZdZdZ ddededeed eed eed eed eed eedeeffd Ze ddd dde j dee j dee dee j dee j deedee j de e j ee j ee e j ffdZxZS)TrOCRAttentionz>Multi-headed attention from 'Attention Is All You Need' paper.NFT embed_dim num_headskdimvdimdropout is_decoderbiasis_cross_attention layer_idxc <t||_||n||_||n||_||_||_||z|_|j|z|jkstd|jd|d|jdz|_ ||_ | |_ tj |j|||_tj |j|||_tj ||||_tj ||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rx)rr rrrtrursrvhead_dim ValueErrorscalingrwrzrLineark_projv_projq_projout_proj) r!configrrrsrtrurvrwrxryrzr"s r#r zTrOCRAttention.__init__s- " ,DD)  ,DD) " !Y.  )T^;;"dn""""" }d* $"i 94@@@ i 94@@@ i 94@@@  )YTBBB r$past_key_valuepast_key_values4.58new_nameversion hidden_stateskey_value_statesattention_masklayer_head_maskoutput_attentionscache_positionreturncd |du}|\} } } |||jz} d} |Ht|tr1|j|j} |r|j}n |j }n|}|r|n|}|r3|1| r/|j |jj }|j |jj }n| |}||}|| d|j|jdd}|| d|j|jdd}|N|s|nd}||||jd|i\}}|r$t|trd|j|j<| |jzd|jf}| | | |j|jdd} | j|} |j|}|j|}|d}t+j| |dd}|| |jz| |fkr2t/d| |jz| |fd |||| d| |fkr+t/d | d| |fd ||| |j| ||z}|| |jz| |}t0j|d }|||jfkr-t/d |jfd ||dddd|| |j| |z}|| |jz| |}|r=|| |j| |}|| |jz| |}nd}t0j||j|j }t+j||}|| |jz| |jfkr5t/d| |j| |jfd ||| |j| |j}|dd}|| | | }||}||fS)z#Input shape: Batch x Time x ChannelNFr+rrrTz$Attention weights should be of size z , but is z!Attention mask should be of size rQz/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )rarr isinstancer is_updatedgetrzcross_attention_cacheself_attention_cachelayerskeysvaluesrrrZrsr} transposeupdatereshaper-bmmr~r functionalsoftmaxrvrr)r!rrrrrrrryr4tgt_lenrr query_statesrcurr_past_key_valuecurrent_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r#r3zTrOCRAttention.forwards.T9"/"4"4"6"6Wi{{=11DL@   &/+>?? 6,7;;DNKK %O*9*O''*9*N''&5#-?R))]  F/"=*"=,3DNCHJ.5dnELLL^44J;;~66L#b$.$-PPZZ[\^_``J',,S"dndmTT^^_`bcddL*7I!St+>+E+E dn?OQ_>`,,( L&F*_FY*Z*ZFAEO.t~>DN*B > #((gt~t}UU__`acdee +|+Z8 'Z'4 +|+Z8 //!$$yz/C/CAq/I/IJJ     3#7'"J J J*dn8LgW^7_** %%''**   %""$$a'(BBB ta'8Rtt]k]p]p]r]rtt(,,S$.'7SSVddL',,S4>-A7GTTL},,\r,BB  &##%%$.)::: 1t~FW11',,..11+//2q!<<|?P?PQTVZVdfmov?w?wwL',,S4>-A7GTTL  ) %1$5$5c4>7T[$\$\ !055cDN6JGU\]]LL$( !]**<4!C!C!C!C!C!CF_%0A6RRR48+/1526,115q2q2|q2#5<0q2"% q2 !. q2 "%,/ q2$D>q2!.q2 u|Xel3XeEL>Q5RR Sq2q2q2SRq2q2q2q2q2r$rpcDeZdZddeffd Zeddd dd ejd eejd eejd eejdeejdeejdee dee dee deejfdZ xZ S)TrOCRDecoderLayerNrc t|j|_t ||j|j|jd||_|j|_t|j |_ |j |_ tj|j|_|jrTt ||j|j|j|j|jdd| |_tj|j|_tj|j|j|_tj|j|j|_tj|j|_dS)NT)rrrsrvrwrz)rrrsrtrurvrwryrz)rr hidden_sizerrrpdecoder_attention_headsattention_dropout self_attnrvractivation_function activation_fnactivation_dropoutr LayerNormself_attn_layer_normrwcross_attention_hidden_size encoder_attnencoder_attn_layer_normrdecoder_ffn_dimfc1fc2final_layer_norm)r!rrzr"s r#r zTrOCRDecoderLayer.__init__.s9 +' n4,    ~ #F$>?"(";$&L$@$@!   H .. 8770#'# ! ! !D ,.<+G+GD (9T^V-CDD9V3T^DD " T^ < <r$rrrrFTrrencoder_hidden_statesencoder_attention_maskrcross_attn_layer_head_maskr use_cacherc "|} ||||||| \}} tj||j|j}| |z}||}d} |g|} |||||||| \}} tj||j|j}| |z}||}|} || |}tj||j |j}| |}tj||j|j}| |z}| |}|f}|r|| | fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size *(decoder_attention_heads,)*. past_key_values (`Cache`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrrrrrN)rrrrrrr) rrrrvrrrrrrrrr)r!rrrrrrrrrrresidualself_attn_weightscross_attn_weightsoutputss r#r3zTrOCRDecoderLayer.forwardRs@!,0>>'+)+/) ,:, , ( ( --mt|VZVc-dd  =0 11-@@ " ,$H040A0A+!65 : /"3-1B11 -M-M11-4 >Gr$rD) NNNNNNFTN) r6r7r8rr rr-r;rr rr3r<r=s@r#rr-sN"="={"="="="="="=H_%0A6RRR268<9=26=A+/,1$(15QQ|Q!.Q ( 5 Q !) 6 Q "%,/ Q%-U\$:Q"%Q$D>QD>Q!.QQQSRQQQQQr$rc.eZdZUeed<dZdZdgZdZdS)TrOCRPreTrainedModelrmodelTrc|jj}t|tjtjfrJ|jjd||j |j j dSdSt|tj rS|jjd||j -|jj|j  dSdSdS)Nrq)meanstd) rinit_stdrrrConv1dr0datanormal_rxzero_ EmbeddingrA)r!modulers r# _init_weightsz"TrOCRPreTrainedModel._init_weightssk" fry")4 5 5 ? M  & &CS & 9 9 9{&  &&((((('&  - - ? M  & &CS & 9 9 9!- "6#56<<>>>>> ? ?--r$N) r6r7r8r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesrr$r#rrsF &*#,- ? ? ? ? ?r$rcJeZdZdZdeffd Z ddZxZS) TrOCRDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`TrOCRDecoderLayer`] Args: config: TrOCRConfig rctj|_j|_j|_jrtj j nd}tj j |j||_ jr tjj |_n0t%j|jzdzj |j|_jrt)jj |_nd|_t)jfdt/jD|_d|_|dS)Nr@)rBrc2g|]}t|S))rz)r).0irs r# z)TrOCRDecoder.__init__..s($r$r$rPQ%6v%K%K%K$r$r$rr$F)rr rvdecoder_layerdrop layerdrop pad_token_idrAscale_embeddingrSsqrtrr? vocab_size embed_tokensuse_learned_position_embeddingsrmax_position_embeddingsembed_positionsrHlayernorm_embeddingrr ModuleListrangedecoder_layersrgradient_checkpointing post_init)r!rrBr"s ` r#r zTrOCRDecoder.__init__s\    ~ 1!.7=7MVdi 2333SV 4  v143CQ\     1 #B6Cacicu#v#vD #E.1AAAE" $$D  % ,')|F4F'G'GD $ $'+D $m$r$r$r$rUZ[a[pUqUq$r$r$rss &+# r$Nc | | n |jj} | | n |jj} | | n |jj} | | n |jj} ||t d|$|}|d|jd}n=|,|dd}|dddddf}nt d|j r%|j r| rt dd} | rO|M|6tt|jt|jnt|j}| rCt|t r.t dtj|}||nd }|||}|jjr||| }n||| }||z}|j||}t.j||j|j }|j}t5||||}||t7||j|d }| rd nd}| rd nd}| r|d nd}t;||gddgD]z\}}|s|d t=|jkrCt d|dt=|jd|d d{tA|jD]\}}| r||fz }|j r tCj"g}||j#kr4||||||||nd|||nd|| | |  }|d }| r||dfz }| ||dfz }| r||fz }| st!d|||||fDStI|||||S)a; Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in encoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer+zEYou have to specify either decoder_input_ids or decoder_inputs_embedsz^`use_cache = True` is incompatible with gradient checkpointing. Setting `use_cache = False`...F)rzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.r)r&r)rr head_maskcross_attn_head_maskzThe `z` should be specified for z layers, but it is for .)rrrrrrrrrc3K|]}||V dSrDr)rvs r# z'TrOCRDecoder.forward..s0= ===r$)last_hidden_staterr attentionscross_attentions)%rroutput_hidden_statesruse_return_dictr~rZr,rarrlogger warning_oncer r rrfrom_legacy_cacheget_seq_lengthrrrrrrrvrr r)ziplenr enumerater-randrr)r!r%rrrrrr inputs_embedsrrr return_dictrinput input_shaper& embed_posrall_hidden_statesall_self_attnsall_cross_attentions attn_mask mask_nameidx decoder_layerdropout_probability layer_outputss r#r3zTrOCRDecoder.forwards:^2C1N--TXT_Tq$8$D $+Jj "+!6IIDKstt t  "E!r5;r?;;II  &',,..ss3K!!!!QQQ(+EEdee e  & "4= " "##t"  0)4$L $D$D$DlZ^ZeFfFfFfggg!555   UOU;; U   \    2COTTOETE`!?!?!A!A!Afg   --i88M ; 6 g,,UKa,bbII,,YOe,ffI% 1  # / 44]CCM --mt|VZVc-dd k : K8N   ! ,1G1S%?& (;[QS_&&& " #7@BBD0:d&7h>##A&3t{+;+;<<$3 33SEUEU33%NN,,Q/333#,DK"8"8 @ @ C# 6!m%55!} &+jnn#&77)M%'=3<3H3dI]Ii,@,E,Eos /"3#-   M*!,M  @=#3"55(4(]1-=,??(  2 -!1 1  ':K^]qr  9+++%1     r$) NNNNNNNNNNNNN)r6r7r8r9rr r3r<r=s@r#rrs{B"#!!O O O O O O O O r$ra The TrOCR Model with a language modeling head. Can be used for summarization. This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. ) custom_introc$eZdZfdZdZxZS)TrOCRDecoderWrappercrt|t||_dSrD)rr rdecoderr!rr"s r#r zTrOCRDecoderWrapper.__init__s.    #F++ r$c|j|i|SrD)r)r!argskwargss r#r3zTrOCRDecoderWrapper.forwardst|T,V,,,r$)r6r7r8r r3r<r=s@r#rrsG,,,,,-------r$rzy The TrOCR Decoder with a language modeling head. Can be used as the decoder part of [`EncoderDecoderModel`] and c"eZdZdgZfdZdZdZdZdZdZ dZ e dd e e jd e e jd e e jd e e jde e jde e jde ede e jde e jde ede ede ede ede e jdeeeffdZxZS)TrOCRForCausalLMzoutput_projection.weightcd|_d|_t|t ||_t j|j|j d|_ | dS)NTFr|) rwis_encoder_decoderrr rrrrrroutput_projectionrr s r#r zTrOCRForCausalLM.__init__sr $)!    (00 !#6+=v?PW\!]!]!] r$c$|jjjSrDrrrr!s r#get_input_embeddingsz%TrOCRForCausalLM.get_input_embeddingssz!..r$c(||jj_dSrDr*)r!values r#set_input_embeddingsz%TrOCRForCausalLM.set_input_embeddingss*/ '''r$c|jSrDr(r+s r#get_output_embeddingsz&TrOCRForCausalLM.get_output_embeddingss %%r$c||_dSrDr1)r!new_embeddingss r#set_output_embeddingsz&TrOCRForCausalLM.set_output_embeddingss!/r$c||j_dSrDrr)r!rs r# set_decoderzTrOCRForCausalLM.set_decoders$ r$c|jjSrDr7r+s r# get_decoderzTrOCRForCausalLM.get_decoders z!!r$Nr%rrrrrrr labelsrrrrrrc | | n |jj} | | n |jj} | | n |jj} |j||||||||| | | | | }||d}d}| Kt}||d|jj | d}| s|f|ddz}||f|zn|St|||j |j |j |jS)a cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import ( ... TrOCRConfig, ... TrOCRProcessor, ... TrOCRForCausalLM, ... ViTConfig, ... ViTModel, ... VisionEncoderDecoderModel, ... ) >>> import requests >>> from PIL import Image >>> # TrOCR is a decoder model and should be used within a VisionEncoderDecoderModel >>> # init vision2text model with random weights >>> encoder = ViTModel(ViTConfig()) >>> decoder = TrOCRForCausalLM(TrOCRConfig()) >>> model = VisionEncoderDecoderModel(encoder=encoder, decoder=decoder) >>> # If you want to start from the pretrained model, load the checkpoint with `VisionEncoderDecoderModel` >>> processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten") >>> model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten") >>> # load image from the IAM dataset >>> url = "https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg" >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB") >>> pixel_values = processor(image, return_tensors="pt").pixel_values >>> text = "industry, ' Mr. Brown commented icily. ' Let us have a" >>> # training >>> model.config.decoder_start_token_id = processor.tokenizer.eos_token_id >>> model.config.pad_token_id = processor.tokenizer.pad_token_id >>> model.config.vocab_size = model.config.decoder.vocab_size >>> labels = processor.tokenizer(text, return_tensors="pt").input_ids >>> outputs = model(pixel_values, labels=labels) >>> loss = outputs.loss >>> round(loss.item(), 2) 5.30 >>> # inference >>> generated_ids = model.generate(pixel_values) >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> generated_text 'industry, " Mr. Brown commented icily. " Let us have a' ```N) r%rrrrrrr rrrrrrr+r)losslogitsrrrr)rrrrrrr(rrZrrrrrr)r!r%rrrrrrr r;rrrrrrr>r=loss_fctoutputs r#r3zTrOCRForCausalLM.forwardsWZ2C1N--TXT_Tq$8$D $+Jj &1%u $D>u 'tnu d^u !.u u77 8!u u u ^u u u u u r$r%)1r9rStypingrrr-rtorch.nnr activationsr cache_utilsr r r generationr modeling_attn_mask_utilsr rmodeling_layersrmodeling_outputsrrmodeling_utilsrutilsrrutils.deprecationrconfiguration_trocrr get_loggerr6rrrr?ModulerHrprrrrr%__all__rr$r#rRsg65 """""""" %%%%%%!!!!!!CCCCCCCCCC)))))):99999llllllll------,,,,,,,,000000,,,,,,  H % %;;;;;bl;;;8 = = = = =r| = = =;8;8;8;8;8;8;8;8|X2X2X2X2X2RYX2X2X2vwwwww2wwwt?????????$v v v v v 'v v v r-----.--- V V V V V +_V V  V r 5 6r$