`i$pdZddlZddlZddlmZmZmZddlZddlmZddl m Z ddl m Z ddl mZmZmZdd lmZdd lmZdd lmZmZmZdd lmZdd lmZmZmZddlm Z m!Z!m"Z"ddl#m$Z$e!j%e&Z'dZ(Gddej)Z*Gddej)Z+Gddej)Z,GddeZ-e GddeZ.e Gdde.Z/e dGd d!e.eZ0e d"Gd#d$e.Z1gd%Z2dS)&zPyTorch OpenAI ImageGPT model.N)AnyOptionalUnion)nn)CrossEntropyLoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions SequenceClassifierOutputWithPast)PreTrainedModel)Conv1D find_pruneable_heads_and_indicesprune_conv1d_layer)auto_docstringlogging torch_float)ImageGPTConfigc ddl}ddl}n)#t$rtdwxYwt j|}td||j |}g}g}|D]|\} } td| d| |j || } | | | | }t||D]=\} } | dd} | d} t!d | Ds | d d vrAtd d| |} | d d vrt'| d} | D]} |d| r|d| }n| g}|ddks |ddkrt'| d} nJ|ddkrt'| d} n,|ddks |ddkr't'| |d} t'| d} n|ddvr!t'| d} t'| d} nt+| dkr?| ddkr3|ddkr't'| |d} t'| d} np|ddkr!t'| d} t'| d} nC|dd kr!t'| d} t'| d} nt'| |d} t+|d!krt-|d}| |} t+| dkr | ddks$| d dks| d d ks | d dkrnC | j| jksJn/#t0$r"}|xj| j| jfz c_d}~wwxYwtd"| | d d#krLt5j| |j|jj| jddd|jf<|| d d$krTt5j| |j|jj| jdd|jd!|jzf<| d d%krOt5j| |j|jj| jddd!|jzdf<7t+| dkrQ| ddkrE| d!dkr9t5j| |j|j| _| d dkrt5j| | _| d dkr,t5j| | jd|j dz ddf<| d d krt5j| | jd <$t5j| | _?|S)&z0 Load tf checkpoints in a pytorch model rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape /c3K|]}|dvV dS))adam_vadam_mAdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepN).0ns /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/imagegpt/modeling_imagegpt.py z.load_tf_weights_in_imagegpt..Ps<   n n      )_stepz Skipping {})wtet transformerz [A-Za-z]+\d+z(\d+)wgweightbbiaswpewte)q_projk_projv_projc_attnrrattnc_projr,lm_headsoszInitialize PyTorch weight r5r6r7)!re tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variableappendsqueezezipsplitanyformatjoingetattr fullmatchlenintshapeAssertionErrorargstorch from_numpyreshapen_embdTdata vocab_size)modelconfigimagegpt_checkpoint_pathr>tftf_path init_varsnamesarraysnamerUarraypointerm_name scope_namesnumes r'load_tf_weights_in_imagegptrn/s     Q      goo677G KKBBBCCC''00I E F '' e BBB5BBCCC&&w55 T emmoo&&&&5&))L3L3 eABBxzz#         "X " " KK ,,SXXd^^<< = = =  88 # #g}55G ' 'F||OV44 ' hhx88 %h 1~$$ A#(=(=!'844Q3&&!'622Q5((KNe,C,C!';q>::!'844Q#AAA!'844!'844TaDGv$5$5+a.H:T:T!';q>::!'844Q6))!'955!'844Q5((!'511!'844!';q>::;1$$+a.))!#, t99q==T!W..$r(f2D2DRTYHYHY]abd]ein]n]n  } 33333!   7=%+66   777888 8x  /4/? fm]c]j@k@k/l/l/nGLOfmO+ , , "X ! !AFAQ fmV];;BB LFMA ,=== > >"X ! !383CEMMRXR_aganDoDo3p3p3rGLA -/// 0 0 YY!^^Q6 1 1d1g6I6I +EMM&-,W,WXXGLL "X   +E22GLL "X  7<7G7N7NGL06,q00!!!3 4 4 "X  $/66GL   +E22GLL Ls  &1"O55 P!?PP!cZeZdZddeedeffd ZdejdejfdZ xZ S) ImageGPTLayerNormh㈵> hidden_sizeepsct||_tjt j||_dSN)super__init__rsr ParameterrXTensorr0)selfrrrs __class__s r'rwzImageGPTLayerNorm.__init__s? l5< #<#<== r)tensorreturnc|tjtjtj|dd|jzz }||jz}|S)Nr*T)axiskeepdim)rXsqrtmeansquarersr0)rzr|s r'forwardzImageGPTLayerNorm.forwardsL%*UZ V0D0D2W[%\%\%\_c_g%ghhh$+% r))rq) __name__ __module__ __qualname__tuplerTfloatrwrXryr __classcell__r{s@r'rprpsz>>E#J>U>>>>>> elu|r)rpcBeZdZddeedeeffd ZdZddZddZ d Z d Z dd e j d eed ee j dee j dee j dee j deedeedee j defdZxZS)ImageGPTAttentionFNis_cross_attention layer_idxc `t|j}|dt jt j||ftjdd||d|dt j dd|j |_ |j |_ |j |j z|_|j |_|j|j z|j kr t!d|j d |j d |j|_||_|j|_||_|j|_|jrBt-d |j z|j |_t-|j |j |_n"t-d |j z|j |_t-|j |j |_t5j|j|_t5j|j|_tA|_!dS) Nr2dtyperF) persistent masked_biasgz=`embed_dim` must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r=r)"rvrwmax_position_embeddingsregister_bufferrXtrilonesboolviewr|rr embed_dimnum_attention_heads num_headshead_dim split_size ValueErrorscale_attn_weightsrscale_attn_by_inverse_layer_idxrreorder_and_upcast_attnrr8q_attnr:rDropout attn_pdrop attn_dropout resid_pdrop resid_dropoutset pruned_heads)rzr`rr max_positionsr{s r'rwzImageGPTAttention.__init__s 6    Juz=-"@ SSS T T Y Y1m]       ]EL,>,>5QQQ+3$.8 . =4> )T^ ; ;'PTP^''N'''  #)";"406/U,"'-'E$  " E T^!3T^DDDK @@DKK T^!3T^DDDKT^T^<< Jv'899Z(:;;EEr)c t|dkrdSt||j|j|j\}}t j|||jz|d|jzzg}t|j |d|_ t|j |d|_ |j|jz|jt|z z|_|jt|z |_|j ||_dS)Nrr=rdim) rSrrrrrXcatrrr8r:union)rzheadsindex index_attns r' prune_headszImageGPTAttention.prune_headss u::?? F7t~t}^b^opp uYut'>T_I\@]^__ )jaHHH (eCCC  ?dn<RUV[R\R\A\]#e**4 -33E::r)c tj||dd}|jr(|t |ddzz }|jr|t|jdzz }|j s|d|d}}|j dddd||z |d|f} tj |j j } tj| |j |j} tj| || }|||z}t#jd|}||j }||}|||z}tj||} | |fS)Nr*?rrdevicer)rXmatmul transposerrsizerrrrr2finforminr|rwhererSoftmaxtyper) rzquerykeyvalueattention_mask head_mask attn_weights query_length key_length causal_mask mask_value attn_outputs r'_attnzImageGPTAttention._attns|E3==R+@+@AA  " M'+ejjnn6K*L*LLL  / D'%0B*C*CCL& N',zz"~~sxx||*L)AAAqqq*|*Cj*PR]S]R]$]^K\%788'-Wa  !*&9:: 1'266t~FF %J*4*J''*4*I''&0#2DW..-  W4**  t %*% M22)0@E+24>BI M22![[88>>tTU>VV UhhsB FFPPQRTUVV 3DNDMJJTTUVXYZZ $ N ; ; A A$/WX A Y Y E3((3DNDMBBLLQPQRRCJJsB FFPPQRTUVVE  !3EO^^4N,33CQacqPrssJC! =8< %dn5 3GGQQRSUVWW  ' a(,(G(GsTY[ikt(u(u %K(, 5#unV_(`(` %K'' T^T]SS kk+.. ((55 L((r))FN)NNNNNNNFFN)rrrrrrTrwrrrrrrXryr rrrrs@r'rrs)")"8D>)"V^_bVc)")")")")")"V ; ; ;$)$)$)$)L2)2)2)2)h***&&&'+15,08<9=$),115D)D)|D)UOD)!. D) EL) D) ( 5 D)!) 6D)D>D)$D>D)!.D) D)D)D)D)D)D)D)D)r)rcBeZdZfdZdejdejfdZxZS) ImageGPTMLPct|j}t|||_t|||_t |j|_tj |j |_ dSru) rvrwrrrc_fcr:r activation_functionactrrrdropout)rzintermediate_sizer`rr{s r'rwzImageGPTMLP.__init__sl & ,i88 Y(9:: &45z&"455 r)rr}c||}||}||}||}|Sru)rrr:r)rzrs r'rzImageGPTMLP.forwardsL -00 //  M22  ]33 r))rrrrwrXryrrrs@r'rrs^66666U\elr)rceZdZdfd Z ddejdeedeejdeejdeejd eejd eed eed eejd e fdZ xZ S) ImageGPTBlockNct|j}|j|jnd|z}t ||j|_t|||_t ||j|_ |j r2t|d||_ t ||j|_ t|||_dS)NrsrT)rr)rvrwrrn_innerrplayer_norm_epsilonln_1rr9ln_2add_cross_attentioncrossattention ln_cross_attnrmlp)rzr`rrr inner_dimr{s r'rwzImageGPTBlock.__init__s ( &,n&@FNNa+o %kv7PQQQ %f BBB %kv7PQQQ  % _"3Ft_h"i"i"iD !2;FD]!^!^!^D y&11r)Frrrrrrrrrr}c |} ||}|||||||| } | d} | dd} | | z}|qt|dstd|d|} ||}||||||||| }|d} | | z}| |ddz} |} ||}||}| |z}|f| zS)N)rrrrrrrrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`)rrrrrrr)rr9rrrrrr)rzrrrrrrrrrresidual attn_outputsroutputscross_attn_outputsfeed_forward_hidden_statess r'rzImageGPTBlock.forwardsg! -00 yy !)/)!  #1o qrr"#h. ,4!122  ZdZZZ%H ..}==M!%!4!4%-#&;'="3-"5 " " -Q/K${2M 2122 66G  -00 %)XXm%<%<" #== '))r)rur) rrrrwrXryrr rrrrrs@r'r r s 2 2 2 2 2 2$'+15,08<9=$),115:*:*|:*UO:*!. :* EL) :* ( 5 :*!) 6:*D>:*$D>:*!.:* :*:*:*:*:*:*:*:*r)r cFeZdZUeed<eZdZdZdZ dgZ fdZ dZ xZ S)ImageGPTPreTrainedModelr`r- input_idsTr c:tj|i|dSru)rvrw)rzinputskwargsr{s r'rwz ImageGPTPreTrainedModel.__init__s%&+F+++++r)c t|tjtfrQ|jjd|jj|j |j j nt|tj r\|jjd|jj|j )|jj|j  n4t|tr|jjd|D]U\}}d|vrLd|vrH|jd|jjt!jd|jjzz VdS)zInitialize the weights.g)rstdNrr:r0r=)rrLinearrr0r]normal_r`initializer_ranger2zero_ Embedding padding_idxrpfill_named_parametersmathrn_layer)rzmodulergps r' _init_weightsz%ImageGPTPreTrainedModel._init_weightssj fry&1 2 2 * M  & &CT[5R & S S S{&  &&(((  - - * M  & &CT[5R & S S S!- "6#56<<>>>  1 2 2 * M  $ $S ) ) )..00 s sGD!4H$4$4Cdk.KdiXY\`\g\oXoNpNp.prrr s sr))rrrr__annotations__rnload_tf_weightsbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modulesrwr3rrs@r'r r s{ 1O%!O&*#(),,,,,sssssssr)r c$eZdZdeffd ZdZdZdZe dde e j de e d e e j d e e j d e e j d e e j d e e j de e j de e j de e de e de e de e de e j dedeeeff dZxZS) ImageGPTModelr`c0tj|_t jj|j|_t jj|j|_ t j j |_ t j fdtjD|_t#|jj|_d|_d|_d|_|dS)Nc2g|]}t|S)r)r )r%ir`s r' z*ImageGPTModel.__init__..$s&lllq f B B Blllr)rF)rvrwrrrrr+r^r4rr3r embd_pdropdrop ModuleListrangenum_hidden_layershrprln_fmodel_parallel device_mapgradient_checkpointing post_initrzr`r{s `r'rwzImageGPTModel.__init__s    +< 14>BB< >OOJv011 llllERXRjLkLklllmm%dn&:STTT $&+# r)c|jSrur4)rzs r'get_input_embeddingsz"ImageGPTModel.get_input_embeddings.s xr)c||_dSrurM)rznew_embeddingss r'set_input_embeddingsz"ImageGPTModel.set_input_embeddings1s !r)c|D]*\}}|j|j|+dS)zv Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsrEr9r)rzheads_to_prunelayerrs r' _prune_headszImageGPTModel._prune_heads4sM+0022 2 2LE5 F5M  * *5 1 1 1 1 2 2r)Nr!past_key_valuesrtoken_type_ids position_idsr inputs_embedsrrrroutput_hidden_states return_dictrr$r}c | | n |jj} | | n |jj} | | n |jj} | | n |jj} ||t d|T||||}|d|d}|j d}n;|*|dd}|j d}nt d||j n|j }|j r%|j r| rtdd} | r8|6tt!|jt!|j}| rCt#|t$r.td tj|}||n|}||d|d}|@t+j||d|zt*j| }|d}|z|dkrt d ||d}|ddddddf}||j }d |z t+j|jjz}|jjrL|J|\}}}||f}| t+j||} || } nd} | ||jj!}||"|}|#|}|||j z}||"|}||z}|$|}||dfz}| rdnd}| r|jjrdnd}| rdnd}tK|j&D]F\}} |j'rtt*j()|j |||j }t#|t*j*r||j }| r||fz}| ||||||| | | | }!|!d}| r$||!dfz}|jjr ||!dfz}|j'rn|j+,D]T\}"}#||#dkrCdt[|"z|j.kr(|dt[|"dzz}UH|/|}|j|}| r||fz}| st%d|||||fDSta|||||S)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTModel >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTModel.from_pretrained("openai/imagegpt-small") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer*rz5You have to specify either input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)r`zPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.rz$batch_size has to be defined and > 0rr)rr$)rrrrrr=zcuda:c3K|]}||V dSrur$)r%vs r'r(z(ImageGPTModel.forward..s0= ===r))last_hidden_staterWr attentionscross_attentions)1r`rr[ruse_return_dictr%warn_if_padding_and_no_attention_maskrrrUrrItrainingrA warning_oncer r rrfrom_legacy_cacheget_seq_lengthrXarangelong unsqueezetorrrrrinvert_attention_mask get_head_maskr0r4r3rA enumeraterErGcuda set_deviceryrHrSstr last_devicerFr)$rzr!rWrrXrYrrZrrrrr[r\rr$ input_shape batch_sizer past_lengthencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeposition_embedsrtoken_type_embeds output_shapeall_self_attentionsall_cross_attentionsall_hidden_statesr>blockrrr_s$ r'rzImageGPTModel.forward;sv`2C1N--TXT_Tq$8$D $+Jj "+!6IIDKcdd d  "  6 6y. Q Q Q#..**K!r;r?;;I"+JJ  &',,..ss3K&,Q/JJTUU U%.%:!! @T  & "4= " "##p"  v01,dk2R2R2RT`hlhsTtTtTtuuO  UOU;; U   \    2COTTO:I:Uo44666[j  %+00[_EEN   < [_{5RZ_ZdmstttL'11!44L  %Q !GHHH+00R@@N ,AAAtT111,<=N,..TZ.@@N!N2ek$*6M6M6QQN ; * */D/P=R=W=W=Y=Y :  7$68O#P %-).4HQW)X)X)X&%)%?%?@V%W%W " "%) " &&y$+2EFF   HHY//M((<00%(:(:=;O(P(PP  % $ 8 8 ),==M -00 "m&8&8&<&<%>> $5?bb4%6d4;;Zdrr`d"6@BBD!$&))" O" OHAu" C %%m&:;;;!-%3%6%6}7K%L%LNi66C ) ]-A B BI# I$58H$H!e! %'=#"3-   G$AJM  P&9WQZM&I#;2P+?71:-+O(" O O1133OODAqAbEzzgA&6$:J&J&J(5(8(83q1u::9M(N(N  -00 * *L9   E 1]4D D  ':KM`bvw  9+++*1     r))NNNNNNNNNNNNNN)rrrrrwrNrQrVrrrXryr rrrrrrrrs@r'r;r;s~&"""222-1+/1515/3,0048<9=$(,0/3&*15K K EL)K "%K !. K !. K u|, K EL)K   -K  ( 5K !) 6K D>K $D>K 'tnK d^K !.K !K " u?? @#K K K ^K K K K K r)r;z The ImageGPT Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc&eZdZdgZdeffd Ze ddeej dee deej deej d eej d eej d eej d eej d eej deej dee dee dee dee deej de de eeff"dZxZS)ImageGPTForCausalImageModelingzlm_head.weightr`ct|t||_t j|j|jdz d|_d|_ d|_ | dS)NrFr2) rvrwr;r-rr'r[r^r;rGrHrJrKs r'rwz'ImageGPTForCausalImageModeling.__init__st    (00y0AA0EERRR $ r)Nr!rWrrXrYrrZrrlabelsrrr[r\rr$r}c V||n |jj}|||||||||| | | | ||}|d}||}d}| |dddddf}| dddf}t }||d|d|d}|s|f|ddz}||f|zn|St|||j |j |j |j S)a& input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size, input_ids_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTForCausalImageModeling >>> import torch >>> import matplotlib.pyplot as plt >>> import numpy as np >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTForCausalImageModeling.from_pretrained("openai/imagegpt-small") >>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu") >>> model.to(device) # doctest: +IGNORE_RESULT >>> # unconditional generation of 8 images >>> batch_size = 4 >>> context = torch.full((batch_size, 1), model.config.vocab_size - 1) # initialize with SOS token >>> context = context.to(device) >>> output = model.generate( ... input_ids=context, max_length=model.config.n_positions + 1, temperature=1.0, do_sample=True, top_k=40 ... ) >>> clusters = image_processor.clusters >>> height = image_processor.size["height"] >>> width = image_processor.size["width"] >>> samples = output[:, 1:].detach().cpu().numpy() >>> samples_img = [ ... np.reshape(np.rint(127.5 * (clusters[s] + 1.0)), [height, width, 3]).astype(np.uint8) for s in samples ... ] # convert color cluster tokens back to pixels >>> f, axes = plt.subplots(1, batch_size, dpi=300) >>> for img, ax in zip(samples_img, axes): # doctest: +IGNORE_RESULT ... ax.axis("off") ... ax.imshow(img) ```N) rWrrXrYrrZrrrrr[r\rr.r*r)losslogitsrWrrarb) r`rcr-r;rrrrrrWrrarb)rzr!rWrrXrYrrZrrrrrr[r\rr$transformer_outputsr lm_logitsr shift_logits shift_labelsloss_fctoutputs r'rz&ImageGPTForCausalImageModeling.forwardszN&1%>L!#qrr'?5577L'))H8L--b,2C2CB2G2GHH,J[J[\^J_J_``D F\$7$;;F)-)9TGf$$v E0/?-;*50A     r))NNNNNNNNNNNNNNN)rrr_tied_weights_keysrrwrrrXryr rrrrrrrrs@r'rr s++ ~      -1+/1515/3,0048<9=)-$(,0/3&*15!p p EL)p "%p !. p !. p u|, p EL)p   -p  ( 5p !) 6p &p D>p $D>p 'tnp d^p !.!p "#p $ u77 8%p p p ^p p p p p r)rz The ImageGPT Model transformer with an image classification head on top (linear layer). [`ImageGPTForImageClassification`] average-pools the hidden states in order to do the classification. c teZdZdeffd Ze ddeejdee deejdeejdeejd eejd eejd eejd ee d ee dee dee de de e effdZxZS)ImageGPTForImageClassificationr`ct||j|_t||_t j|j|jd|_| dS)NFr) rvrw num_labelsr;r-rr'r[scorerJrKs r'rwz'ImageGPTForImageClassification.__init__si     +(00Yv}doEJJJ  r)Nr!rWrrXrYrrZrrrr[r\r$r}c | | n |jj} ||||||||| | | |  }|d}|d}||}d}|||||j}| s|f|ddz}||f|zn|St |||j|j|j S)ay input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTForImageClassification >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTForImageClassification.from_pretrained("openai/imagegpt-small") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits ```N) rWrrXrYrrZrrr[r\rrr)rrrWrra) r`rcr-rr loss_functionrrWrra)rzr!rWrrXrYrrZrrrr[r\r$rrpooled_hidden_statesrrrs r'rz&ImageGPTForImageClassification.forwardsd&1%T $D>T 'tnT d^T T  u66 7T T T ^T T T T T r)r)rrr;r rn)3__doc__r/rCtypingrrrrXrtorch.nnr activationsr cache_utilsr r r generationr modeling_layersrmodeling_outputsrrrmodeling_utilsr pytorch_utilsrrrutilsrrrconfiguration_imagegptr get_loggerrrArnModulerprrr r r;rr__all__r$r)r'rs%$ '''''''''' %%%%%%!!!!!!CCCCCCCCCC))))))999999 .-----YYYYYYYYYY 322222  H % %iiiX        i)i)i)i)i) i)i)i)X")"J*J*J*J*J*.J*J*J*Z#s#s#s#s#so#s#s#sLm m m m m +m m m `      %