o 2h@s@ddlmZddlmZddlmZedGdddeZdS))ops) keras_export) Attentionzkeras.layers.AdditiveAttentioncsBeZdZdZ  d fdd ZddZdd Zfd d ZZS) AdditiveAttentiona[ Additive attention layer, a.k.a. Bahdanau-style attention. Inputs are a list with 2 or 3 elements: 1. A `query` tensor of shape `(batch_size, Tq, dim)`. 2. A `value` tensor of shape `(batch_size, Tv, dim)`. 3. A optional `key` tensor of shape `(batch_size, Tv, dim)`. If none supplied, `value` will be used as `key`. The calculation follows the steps: 1. Calculate attention scores using `query` and `key` with shape `(batch_size, Tq, Tv)` as a non-linear sum `scores = reduce_sum(tanh(query + key), axis=-1)`. 2. Use scores to calculate a softmax distribution with shape `(batch_size, Tq, Tv)`. 3. Use the softmax distribution to create a linear combination of `value` with shape `(batch_size, Tq, dim)`. Args: use_scale: If `True`, will create a scalar variable to scale the attention scores. dropout: Float between 0 and 1. Fraction of the units to drop for the attention scores. Defaults to `0.0`. Call arguments: inputs: List of the following tensors: - `query`: Query tensor of shape `(batch_size, Tq, dim)`. - `value`: Value tensor of shape `(batch_size, Tv, dim)`. - `key`: Optional key tensor of shape `(batch_size, Tv, dim)`. If not given, will use `value` for both `key` and `value`, which is the most common case. mask: List of the following tensors: - `query_mask`: A boolean mask tensor of shape `(batch_size, Tq)`. If given, the output will be zero at the positions where `mask==False`. - `value_mask`: A boolean mask tensor of shape `(batch_size, Tv)`. If given, will apply the mask such that values at positions where `mask==False` do not contribute to the result. return_attention_scores: bool, it `True`, returns the attention scores (after masking and softmax) as an additional output argument. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). use_causal_mask: Boolean. Set to `True` for decoder self-attention. Adds a mask such that position `i` cannot attend to positions `j > i`. This prevents the flow of information from the future towards the past. Defaults to `False`. Output: Attention outputs of shape `(batch_size, Tq, dim)`. (Optional) Attention scores after masking and softmax with shape `(batch_size, Tq, Tv)`. Tc stjd||d|dS)N) use_scaledropout)super__init__)selfrrkwargs __class__r h/var/www/html/chatgem/venv/lib/python3.10/site-packages/keras/src/layers/attention/additive_attention.pyr <szAdditiveAttention.__init__cCsD|||dd}d|_|jr |jd|gd|jdd|_dSdS)Nrscaleglorot_uniformT)nameshape initializerdtype trainable)_validate_inputsrr add_weightr)r input_shapedimr r rbuildDs   zAdditiveAttention.buildcCsHtj|dd}tj|dd}|jr|jnd}tj|t||ddS)aCalculates attention scores as a nonlinear sum of query and key. Args: query: Query tensor of shape `(batch_size, Tq, dim)`. key: Key tensor of shape `(batch_size, Tv, dim)`. Returns: Tensor of shape `(batch_size, Tq, Tv)`. )axisg?r)r expand_dimsrrsumtanh)r querykey q_reshaped k_reshapedrr r r_calculate_scoresQs z#AdditiveAttention._calculate_scorescst}|d=|S)N score_mode)r get_config)r base_configrr rr*cs zAdditiveAttention.get_config)Tr) __name__ __module__ __qualname____doc__r rr(r* __classcell__r r rrrs6 rN) keras.srcrkeras.src.api_exportr$keras.src.layers.attention.attentionrrr r r rs