o 2h4@sXddlmZddlmZddlmZddlmZddlmZedGdddeZ d S) )backend)ops) keras_export) KerasTensor)Layerzkeras.layers.AttentioncseZdZdZ    dfdd Zdd Zd d Zdd d ZddZ    dddZ dddZ ddZ    d ddZ dddZ fddZZS)! Attentiona Dot-product attention layer, a.k.a. Luong-style attention. Inputs are a list with 2 or 3 elements: 1. A `query` tensor of shape `(batch_size, Tq, dim)`. 2. A `value` tensor of shape `(batch_size, Tv, dim)`. 3. A optional `key` tensor of shape `(batch_size, Tv, dim)`. If none supplied, `value` will be used as a `key`. The calculation follows the steps: 1. Calculate attention scores using `query` and `key` with shape `(batch_size, Tq, Tv)`. 2. Use scores to calculate a softmax distribution with shape `(batch_size, Tq, Tv)`. 3. Use the softmax distribution to create a linear combination of `value` with shape `(batch_size, Tq, dim)`. Args: use_scale: If `True`, will create a scalar variable to scale the attention scores. dropout: Float between 0 and 1. Fraction of the units to drop for the attention scores. Defaults to `0.0`. seed: A Python integer to use as random seed in case of `dropout`. score_mode: Function to use to compute attention scores, one of `{"dot", "concat"}`. `"dot"` refers to the dot product between the query and key vectors. `"concat"` refers to the hyperbolic tangent of the concatenation of the `query` and `key` vectors. Call arguments: inputs: List of the following tensors: - `query`: Query tensor of shape `(batch_size, Tq, dim)`. - `value`: Value tensor of shape `(batch_size, Tv, dim)`. - `key`: Optional key tensor of shape `(batch_size, Tv, dim)`. If not given, will use `value` for both `key` and `value`, which is the most common case. mask: List of the following tensors: - `query_mask`: A boolean mask tensor of shape `(batch_size, Tq)`. If given, the output will be zero at the positions where `mask==False`. - `value_mask`: A boolean mask tensor of shape `(batch_size, Tv)`. If given, will apply the mask such that values at positions where `mask==False` do not contribute to the result. return_attention_scores: bool, it `True`, returns the attention scores (after masking and softmax) as an additional output argument. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). use_causal_mask: Boolean. Set to `True` for decoder self-attention. Adds a mask such that position `i` cannot attend to positions `j > i`. This prevents the flow of information from the future towards the past. Defaults to `False`. Output: Attention outputs of shape `(batch_size, Tq, dim)`. (Optional) Attention scores after masking and softmax with shape `(batch_size, Tq, Tv)`. FdotNc s`tjdi|||_||_||_|jdkrtjj|d|_|jdvr+t d|d|_ dS)Nrseed)rconcatz_Invalid value for argument score_mode. Expected one of {'dot', 'concat'}. Received: score_mode=F) super__init__ use_scale score_modedropoutrrandom SeedGeneratorseed_generator ValueError_return_attention_scores)selfrrrr kwargs __class__r _/var/www/html/chatgem/venv/lib/python3.10/site-packages/keras/src/layers/attention/attention.pyrBs   zAttention.__init__cCs^||d|_d|_|jr|jddd|jdd|_|jdkr-|jddd|jdd|_dSdS)Nscaler onesT)nameshape initializerdtype trainabler concat_score_weight)_validate_inputsrr$r add_weightr"r)r input_shaper r rbuildZs(   zAttention.buildcCs|jdkrt|tj|gdd}|jdur||j9}|S|jdkr[tj|dd}tj|dd}|jdurJ|jtjt|j||d d}|S|jtjt||d d}|St d ) aCalculates attention scores as a query-key dot product. Args: query: Query tensor of shape `(batch_size, Tq, dim)`. key: Key tensor of shape `(batch_size, Tv, dim)`. Returns: Tensor of shape `(batch_size, Tq, Tv)`. r)r)axesNr axiszscores not computed) rrmatmul transposer expand_dimsr$sumtanhr)rquerykeyscores q_reshaped k_reshapedr r r_calculate_scoresos$      zAttention._calculate_scorescCs~|durt|}|jdkrdnd}||tj||jd8}tj|dd}|r7|jdkr7tjj||j|jd }t |||fS) aApplies attention scores to the given value tensor. To use this method in your attention layer, follow the steps: * Use `query` tensor of shape `(batch_size, Tq)` and `key` tensor of shape `(batch_size, Tv)` to calculate the attention `scores`. * Pass `scores` and `value` tensors to this method. The method applies `scores_mask`, calculates `attention_distribution = softmax(scores)`, then returns `matmul(attention_distribution, value). * Apply `query_mask` and return the result. Args: scores: Scores float tensor of shape `(batch_size, Tq, Tv)`. value: Value tensor of shape `(batch_size, Tv, dim)`. scores_mask: A boolean mask tensor of shape `(batch_size, 1, Tv)` or `(batch_size, Tq, Tv)`. If given, scores at positions where `scores_mask==False` do not contribute to the result. It must contain at least one `True` value in each line along the last dimension. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). Returns: Tensor of shape `(batch_size, Tq, dim)`. Attention scores after masking and softmax with shape `(batch_size, Tq, Tv)`. Nfloat16g@geAr"r0r-rr ) r logical_notr"castsoftmaxrrrrr1)rr8value scores_masktraining padding_mask max_valueweightsr r r _apply_scoress zAttention._apply_scoresc Cs|r>t|}d|d|df}tj|dd}tj|dd}tj|dd}t||} |durt|d|dt|dksJt|dkrYt|d|d |ddSdS) z'Validates arguments of the call method.zj layer must be called on a list of inputs, namely [query, value] or [query, value, key]. Received: inputs=.r)zl layer accepts inputs list of length 2 or 3, namely [query, value] or [query, value, key]. Received length: NzL layer mask must be a list, namely [query_mask, value_mask]. Received: mask=z< layer accepts mask list of length 2 or 3. Received: inputs=z, mask=)r__name__ isinstancelistrrX)rrVrW class_namer r rr%'s<  zAttention._validate_inputscs(t}|j|j|jd}i||S)N)rrr)r get_configrrr)r base_configconfigrr rrsBs  zAttention.get_config)Frr N)NF)NFFF)N)NFNF)ro __module__ __qualname____doc__rr(r;rGrTr`rbrhrlr%rs __classcell__r r rrrs2: !.  "rN) keras.srcrrkeras.src.api_exportrkeras.src.backendrkeras.src.layers.layerrrr r r rs