`i(dZddlZddlmZmZddlZddlmZddlmZm Z m Z ddl m Z ddl mZmZmZmZmZmZmZdd lmZdd lmZmZdd lmZmZd d lmZd dlm Z m!Z!m"Z"m#Z#m$Z$m%Z%ej&e'Z(Gddej)Z*Gddej)Z+Gddej)Z,Gddej)Z-Gddej)Z.Gddej)Z/Gddej)Z0Gddej)Z1Gdd ej)Z2eGd!d"eZ3eGd#d$e3Z4eGd%d&e3Z5Gd'd(ej)Z6ed)*Gd+d,e3Z7eGd-d.e3Z8eGd/d0e3Z9Gd1d2ej)Z:eGd3d4e3Z;d7d5Z4?[[['(>4?[[[&  %) .    "*$,4?!S!S!Sz&"<== Nrc|F|/t||j||j}n||}||}n|dd}|+t j|t j|j j}|| |\}}nd}| |\}} | |||| \} } |j dkr4||\} } | | | | | \} } || | \} } || } || | \} } | | fS)Nr*dtypedeviceidentityidentity_scaling_factorr-)"create_position_ids_from_input_idsr&torX&create_position_ids_from_inputs_embedssizerAzeroslongr)r=r?rGr,rFrJrNrK)rP input_idstoken_type_idsr) inputs_embedspast_key_values_length input_shapeinputs_embeds_scaling_factorr?$token_type_embeddings_scaling_factor embeddingsembeddings_scaling_factorrF"position_embeddings_scaling_factors rSforwardzIBertEmbeddings.forwardhs  $At/1G  "Y%&&  $JJ=YY  #..**KK',,..ss3K  !"[EJtO`OghhhN  :>:N:Ny:Y:Y 7M77+/ (FJF`F`aoFpFpCC040D0D  (*$H 1E1 1 - -  ': 5 5FJF^F^_kFlFl C !C484H4H),(J 5I55 1J115zKd0e0e- -\\*-- 040F0FzSl0m0m- -444rTc|dd}|d}tj|jdz||jzdztj|j}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr*rrVr)r_rArBr&rarX unsqueezerD)rPrdrfsequence_lengthr)s rSr^z6IBertEmbeddings.create_position_ids_from_inputs_embedss$((**3B3/ %a.|  q /D4D"Dq"HPUPZcpcw   %%a((// <<2>2>2>2>jrs-5-5-5-5^=======rTr c,eZdZfdZ ddZxZS)IBertSelfAttentionc t|j|jzdkr0t |ds t d|jd|jd|j|_d|_d|_d|_ |j|_t|j|jz |_ |j|j z|_ t|j|j d|j|j|jd |_t|j|j d|j|j|jd |_t|j|j d|j|j|jd |_t#|j |j |_t#|j |j |_t#|j |j |_t#|j |j |_t-j|j|_t5|d d |_|jd krt d t9|j |j|j|_dS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r"r%Tbiasr'bias_bitr( per_channelr.r,r-zDI-BERT only supports 'absolute' for `config.position_embedding_type`r(r2)r3r4r;num_attention_headshasattr ValueErrorr(r'r}r7intattention_head_size all_head_sizerquerykeyvaluerquery_activationkey_activationvalue_activationrKrrLattention_probs_dropout_probrNrEr,rr2softmaxrOs rSr4zIBertSelfAttention.__init__sO    : :a ? ?PVXhHiHi ?8F$688 4888 !+  #)#= #&v'9Fu>u;;:>((=Rn:o:o77>BjjXt>u>u;;372G2G ?3 3 / //3.A.A/Sq.r.r+ +262G2G ?3 3 / / %2$7! J!&&z2t7OQUQijjtt q  NN:r43KTMeffppqrtuvv !&&z2t7OQUQijjtt q  !< Y5H5HR5P5PQQ $233+e3 ? 3.HKc.cfk.k + +.2 +  %/.@ ;?,, =; ; 77 ,,77  - 9O _kBB ) 5+ILf+f ( (+/ (%--aAq99DDFF "/"4"4"6"6ss";t?Q>S"S* *,CD 7;6L6L 77 7 3 37H]=/22mM]! 1 )+I J J.0  ---rTNNFrprqrrr4rlrtrus@rSrwrwse8p8p8p8p8p| P.P.P.P.P.P.P.P.rTrwc$eZdZfdZdZxZS)IBertSelfOutputc .t|j|_d|_d|_d|_d|_d|_t|j |j d|j|j|jd|_ t|j|j|_ t|j |j|j|j|j|_t|j|j|_t%j|j|_dSNr"r%r$Tr{r.r/)r3r4r(r7r'r}r8r9rr;denser ln_input_actrrIr2rJrKrrLrMrNrOs rSr4zIBertSelfOutput.__init__8s  +      ]    %T%64?SSS%  %) .    "*$,4?!S!S!Sz&"<== rTc|||\}}||}|||||\}}|||\}}|||\}}||fSNrYrrNrrJrKrPrr input_tensorinput_tensor_scaling_factors rSrlzIBertSelfOutput.forwardU6:jjPl6m6m3 3 ]33 6:6G6G  (!$? 7H7 7 3 3 7;nn]Tp6q6q3 36:6L6L 77 7 3 3:::rTrrus@rSrr7G>>>>>:;;;;;;;rTrc2eZdZfdZdZ ddZxZS)IBertAttentionct|j|_t||_t ||_t|_dSN) r3r4r(rwrProutputset pruned_headsrOs rSr4zIBertAttention.__init__gsS  +&v.. %f-- EErTct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)Nrrdim)lenrrPrrrrrrrrrrunion)rPheadsindexs rS prune_headszIBertAttention.prune_headsns u::?? F7 490$)2OQUQb  u -TY_eDD *49=%@@ ,TY_eDD .t{/@%QOOO )- (EE (R %"&)"?$)B_"_  -33E::rTNFc||||||\}}||d|d||\}} |f|ddz} | f|ddz} | | fS)Nrr)rPr) rPrrrrr self_outputsself_outputs_scaling_factorattention_outputattention_output_scaling_factorroutputs_scaling_factors rSrlzIBertAttention.forwards59II  (    5 5 1 1=AKK O8;]Lh= = 99$% QRR(88"A!CFabcbdbdFe!e...rTr)rprqrrr4rrlrtrus@rSrrfsf""""";;;, ////////rTrc$eZdZfdZdZxZS)IBertIntermediatec t|j|_d|_d|_d|_t |j|jd|j|j|jd|_ |j dkrtdt|j|j |_t|j|j|_dS) Nr"r%Tr{r z3I-BERT only supports 'gelu' for `config.hidden_act`rr.)r3r4r(r7r'r}rr;intermediate_sizer hidden_actrrr2intermediate_act_fnrrKrOs rSr4zIBertIntermediate.__init__s  +      $]       & &RSS S#*doU[Ui#j#j#j !)$,4?!S!S!SrTc|||\}}|||\}}|||\}}||fSr)rrrK)rPrrs rSrlzIBertIntermediate.forwardsn6:jjPl6m6m3 36:6N6N 77 7 3 3 7;6L6L 77 7 3 3:::rTrrus@rSrrsLTTTTT( ; ; ; ; ; ; ;rTrc$eZdZfdZdZxZS) IBertOutputc .t|j|_d|_d|_d|_d|_d|_t|j |j d|j|j|jd|_ t|j|j|_ t|j |j|j|j|j|_t|j|j|_t'j|j|_dSr)r3r4r(r7r'r}r8r9rrr;rrrrrIr2rJrKrrLrMrNrOs rSr4zIBertOutput.__init__s  +    $  ]    %T%64?SSS%  %) .    "*$,4?!S!S!Sz&"<== rTc|||\}}||}|||||\}}|||\}}|||\}}||fSrrrs rSrlzIBertOutput.forwardrrTrrus@rSrrrrTrc2eZdZfdZ ddZdZxZS) IBertLayerctt|j|_d|_d|_t ||_t||_t||_ t|j|j|_ t|j|j|_ dS)Nr"rr.)r3r4r(r7 seq_len_dimr attentionr intermediaterrrpre_intermediate_actpre_output_actrOs rSr4zIBertLayer.__init__s  + '//-f55!&)) $,T\do$V$V$V!&t|PPPrTNFc||||||\}}|d}|d} |dd} ||| \} } | f| z} | S)N)rrr)rfeed_forward_chunk) rPrrrrrself_attention_outputs%self_attention_outputs_scaling_factorrrr layer_outputlayer_output_scaling_factors rSrlzIBertLayer.forwardsIM  (  / IWI I E E2!4*OPQ*R'(,484K4K =5 5 1 1 /G+rTc|||\}}|||\}}|||\}}|||||\}}||fSr)rrrr)rPrrintermediate_output"intermediate_output_scaling_factorrrs rSrzIBertLayer.feed_forward_chunks<@z)IBertEncoder.__init__..'s!#`#`#`1Jv$6$6#`#`#`rT) r3r4rQr(r ModuleListrangenum_hidden_layerslayerrOs `rSr4zIBertEncoder.__init__#s`   +]#`#`#`#`fF^@_@_#`#`#`aa rTNFTc8|rdnd}|rdnd} d} t|jD]>\} } |r||fz}||| nd} | |||| |}|d}|r | |dfz} ?|r||fz}|std||| | fDSt||| | S)Nrrrc3K|]}||V dSrr)rvs rS z'IBertEncoder.forward..Ms4   =!===  rT)last_hidden_stater attentionscross_attentions) enumeratertupler )rPrrrrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsall_cross_attentionsi layer_modulelayer_head_mask layer_outputss rSrlzIBertEncoder.forward)s;#7@BBD$5?bb4#(44 P POA|# I$58H$H!.7.CillO(L,! M*!,M  P&9]1=M ? ? % M  & &CT[5R & S S S!- "6#56<<>>>>>.- r| < = = % K  " " $ $ $ M  $ $S ) ) ) ) )  , , % K  " " $ $ $ $ $ % %rTNc td)Nz6`resize_token_embeddings` is not supported for I-BERT.)NotImplementedError)rPnew_num_tokenss rSresize_token_embeddingsz,IBertPreTrainedModel.resize_token_embeddingss!"Z[[[rTr)rprqrrr__annotations__base_model_prefixr/r3rrTrSr r osQ %%%$\\\\\\rTr cXeZdZdZdfd ZdZdZdZe dde e j d e e j d e e j d e e j d e e j d e e j de e de e de e deeee j ffdZxZS) IBertModela The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. Tct|||_|j|_t ||_t ||_|rt|nd|_ | dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r3r4rQr(r rirencoderrpooler post_init)rPrQadd_pooling_layerrRs rSr4zIBertModel.__init__s}      +)&11#F++ ->Hk&)))D  rTc|jjSrrir=rPs rSget_input_embeddingszIBertModel.get_input_embeddingss ..rTc||j_dSrr>)rPrs rSset_input_embeddingszIBertModel.set_input_embeddingss*/'''rTc|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr9rrr)rPheads_to_prunerrs rS _prune_headszIBertModel._prune_headssU +0022 C CLE5 L u % / ; ;E B B B B C CrTNrbrrcr)rrdrr r returnc r||n |jj}||n |jj}| | n |jj} ||t d|+||||} n.||dd} nt d| \} } ||jn|j} |tj | | f| }|!tj | tj | }| || }| ||jj}|||||\}}|||||||| }|d}|j||nd}| s||f|d dzSt%|||j|j|j S) NzDYou cannot specify both input_ids and inputs_embeds at the same timer*z5You have to specify either input_ids or inputs_embeds)rXrV)rbr)rcrd)rrrr r rr)r pooler_outputrrr )rQrr use_return_dictr%warn_if_padding_and_no_attention_maskr_rXrAonesr`raget_extended_attention_mask get_head_maskrrir9r:r rrr )rPrbrrcr)rrdrr r rfrrrXextended_attention_maskembedding_outputembedding_output_scaling_factorencoder_outputssequence_outputrs rSrlzIBertModel.forwards*2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U!, J%.%:!! @T  !"Z*j)A6RRRN  !"[EJvVVVN150P0PQ_al0m0m&&y$+2OPP <@OO%)' =L= = 99 ,,  +2/!5#'  *!,8< 8O O444UY  J#]3oabb6II I;-')7&1,=     rT)T) NNNNNNNNN)rprqrrrsr4r@rBrFrrrA LongTensor FloatTensorboolrr r rlrtrus@rSr7r7s}"///000CCC156:59371559,0/3&*J J E,-J !!23J !!12 J u/0 J E-. J   12J $D>J 'tnJ d^J  ;U5CT=UU VJ J J ^J J J J J rTr7cpeZdZddgZfdZdZdZe ddee j dee j d ee j d ee j d ee j d ee j d ee j dee dee dee de eee j ffdZxZS)IBertForMaskedLMzlm_head.decoder.biaszlm_head.decoder.weightct|t|d|_t ||_|dSNF)r<)r3r4r7r!r-lm_headr;rOs rSr4zIBertForMaskedLM.__init__sV    %@@@ "6**  rTc|jjSr)r[decoderr?s rSget_output_embeddingsz&IBertForMaskedLM.get_output_embeddingss |##rTc@||j_|j|j_dSr)r[r]r|)rPnew_embeddingss rSset_output_embeddingsz&IBertForMaskedLM.set_output_embeddingss- */ rTNrbrrcr)rrdlabelsrr r rGc | | n |jj} ||||||||| |  } | d} || } d}|Kt }|| d|jj|d}| s| f| ddz}||f|zn|St|| | j| j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrrcr)rrdrr r rr*rlosslogitsrr) rQrJr!r[rrr:r rr)rPrbrrcr)rrdrbrr r rrSprediction_scoresmasked_lm_lossloss_fctrs rSrlzIBertForMaskedLM.forwards(&1%1 'tn1 d^1  ~uU%677 81 1 1 ^1 1 1 1 1 rTrXc0eZdZdZfdZdZddZxZS)r-z)I-BERT Head for masked language modeling.cttj|j|j|_tj|j|j|_tj|j|j |_ tj tj |j |_|j|j _dS)N)r0)r3r4rrr;rrJrI layer_normr:r] ParameterrAr`r|rOs rSr4zIBertLMHead.__init__Ns Yv163EFF ,v'9v?TUUUy!3V5FGG LV->!?!?@@  I rTc ||}t|}||}||}|Sr)rr ror])rPfeatureskwargsxs rSrlzIBertLMHead.forwardWsE JJx  GG OOA   LLOOrTrGNc|jjjjdkr|j|j_dS|jj|_dS)Nmeta)r]r|rXtyper?s rS _tie_weightszIBertLMHead._tie_weightsas< <  # (F 2 2 $ DL    )DIIIrT)rGN)rprqrrrsr4rlrxrtrus@rSr-r-Ksa33&&&&&********rTr-z I-BERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introc\eZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd ee d ee d ee d e e e ejffdZ xZS)IBertForSequenceClassificationct||j|_t|d|_t ||_|dSrZ)r3r4 num_labelsr7r!IBertClassificationHead classifierr;rOs rSr4z'IBertForSequenceClassification.__init__qs`     +%@@@ 1&99 rTNrbrrcr)rrdrbrr r rGc | | n |jj} ||||||||| |  } | d} || } d}|Z|jjf|jdkr d|j_nN|jdkr7|jtjks|jtj kr d|j_n d|j_|jjdkrWt}|jdkr1|| | }n|| |}n|jjdkrGt}|| d|j| d}n*|jjdkrt}|| |}| s| f| d dz}||f|zn|St|| | j| j S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrdrr regressionsingle_label_classificationmulti_label_classificationr*rre)rQrJr!r problem_typer}rWrArarrsqueezerrrrrrrPrbrrcr)rrdrbrr r rrSrgrfrjrs rSrlz&IBertForSequenceClassification.forward{s (&1%B 'tnB d^B  'u/@)AA BB B B ^B B B B B rTr{c\eZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd ee d ee d ee d e e e ejffdZ xZS)IBertForMultipleChoicect|t||_t j|j|_t j|j d|_ | dS)Nr) r3r4r7r!rrLrMrNrr;rr;rOs rSr4zIBertForMultipleChoice.__init__sl    '' z&"<== )F$6:: rTNrbrcrrbr)rrdrr r rGc | | n |jj} | |jdn |jd} |)|d|dnd} |)|d|dnd} |)|d|dnd}|)|d|dnd}|=|d|d|dnd}|| | |||||| |  }|d}||}||}|d| }d}|t}|||}| s|f|ddz}||f|zn|St|||j |j S)a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nrr*r)r)rcrrrdrr r rre) rQrJrrr_r!rNrrrrr)rPrbrcrrbr)rrdrr r  num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsrrrgreshaped_logitsrfrjrs rSrlzIBertForMultipleChoice.forwards.X&1%,>???^bLXLdL--b,2C2CB2G2GHHHjnR`Rln11"n6I6I"6M6MNNNrvR`Rln11"n6I6I"6M6MNNNrv(   r=#5#5b#9#9=;M;Mb;Q;Q R R R  ** *..,/!5#      ]33 // ++b+66  '))H8OV44D F%''!""+5F)-)9TGf$$v E("!/)     rTrk)rprqrrr4rrrArTrUrVrrr rlrtrus@rSrrsT15596:-1371559,0/3&*W W E,-W !!12W !!23 W )* W u/0 W E-.W   12W $D>W 'tnW d^W  (%0A*BB CW W W ^W W W W W rTrc\eZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd ee d ee d ee d e e e ejffdZ xZS)IBertForTokenClassificationc:t||j|_t|d|_t j|j|_t j |j |j|_ | dSrZ) r3r4r}r7r!rrLrMrNrr;rr;rOs rSr4z$IBertForTokenClassification.__init__*s~     +%@@@ z&"<== )F$68IJJ rTNrbrrcr)rrdrbrr r rGc | | n |jj} ||||||||| |  } | d} || } || } d}|Ft }|| d|j|d}| s| f| ddz}||f|zn|St|| | j | j S)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrdrr*rre) rQrJr!rNrrrr}rrrrs rSrlz#IBertForTokenClassification.forward5s$&1%2 'tn2 d^2  $eE,=&>> ?2 2 2 ^2 2 2 2 2 rTrc(eZdZdZfdZdZxZS)r~z-Head for sentence-level classification tasks.cttj|j|j|_tj|j|_tj|j|j |_ dSr) r3r4rrr;rrLrMrNr}out_projrOs rSr4z IBertClassificationHead.__init__nsc Yv163EFF z&"<==  &"4f6GHH rTc |dddddf}||}||}tj|}||}||}|Sr)rNrrAtanhr)rPrrrsrs rSrlzIBertClassificationHead.forwardtss Aqqq)  ]33  =11  =11  ]33  m44 rT)rprqrrrsr4rlrtrus@rSr~r~ksR77IIIII rTr~cxeZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd eejd ee d ee d ee de e e ejffdZ xZS)IBertForQuestionAnsweringct||j|_t|d|_t j|j|j|_| dSrZ) r3r4r}r7r!rrr; qa_outputsr;rOs rSr4z"IBertForQuestionAnswering.__init__sj     +%@@@ )F$68IJJ rTNrbrrcr)rrdstart_positions end_positionsrr r rGc | | n |jj} |||||||| | |  } | d} || }|dd\}}|d}|d}d}||t|dkr|d}t|dkr|d}|d}| d|}| d|}t|}|||}|||}||zdz }| s||f| ddz}||f|zn|St|||| j | j S) Nrdrrr*r) ignore_indexr)rf start_logits end_logitsrr)rQrJr!rsplitrrrr_clamprrrr)rPrbrrcr)rrdrrrr r rrSrgrr total_loss ignored_indexrj start_lossend_lossrs rSrlz!IBertForQuestionAnswering.forwards&1% > E,-> !!23> !!12 > u/0 > E-. >   12> "%"23>   01> $D>> 'tn> d^>  +U53D-EE F> > > ^> > > > > rTrc||}tj|d||z|z}||zS)aM Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's *utils.make_positions*. Args: input_ids (`torch.LongTensor`): Indices of input sequence tokens in the vocabulary. Returns: torch.Tensor rr)nerrAcumsumtype_asra)rbr&remaskincremental_indicess rSr\r\sg << $ $ ( ( * *D <!444<rsrtypingrrrArtorch.nnrrr activationsr modeling_outputsr r r rrrrmodeling_utilsr pytorch_utilsrrutilsrrconfiguration_ibertr quant_modulesrrrrrr get_loggerrploggerModuler rwrrrrrrrr r7rXr-r{rrr~rr\__all__rrTrSrsa$ """""""" AAAAAAAAAA.-----QQQQQQQQ,,,,,,,,,,,,,,cccccccccccccccc  H % %w=w=w=w=w=biw=w=w=tK.K.K.K.K.K.K.K.\,;,;,;,;,;bi,;,;,;^./././././RY./././b;;;;; ;;;D,;,;,;,;,;"),;,;,;^7979797979797979t: : : : : 29: : : z     ")    \\\\\?\\\4t t t t t %t t t nE E E E E +E E E P*****")***> N N N N N %9N N  N bc c c c c 1c c c L? ? ? ? ? "6? ? ? Dbi&J J J J J 4J J J Z4444"   rT