o 2h@@sbdZddlZddlmZddlmZddlmZddlmZddlm Z ddlm Z dd lm Z dd lm Z dd lm Z dd lmZdd lmZddlmZddlmZedejej d!ddZedddZedejd"ddZedddgdejedd!ddZedejd"ddZejdddedgdejd!d dZdS)#zEOperations for clipping (gradient, weight) tensors to min/max values.N) constant_op)dtypes)indexed_slices)ops) array_ops)array_ops_stack) gen_array_ops) gen_nn_ops)math_ops) deprecation)dispatch)collections_abc) tf_export clip_by_valuecCst|d|||gL}tjt|tjr|jn|dd}t||}|j |j tj |||d}|j |j t|tjrNt||j |j }Wd|SWd|S1sYwY|S)aq Clips tensor values to a specified min and max. Given a tensor `t`, this operation returns a tensor of the same type and shape as `t` with its values clipped to `clip_value_min` and `clip_value_max`. Any values less than `clip_value_min` are set to `clip_value_min`. Any values greater than `clip_value_max` are set to `clip_value_max`. Note: `clip_value_min` needs to be smaller or equal to `clip_value_max` for correct results. For example: Basic usage passes a scalar as the min and max value. >>> t = tf.constant([[-10., -1., 0.], [0., 2., 10.]]) >>> t2 = tf.clip_by_value(t, clip_value_min=-1, clip_value_max=1) >>> t2.numpy() array([[-1., -1., 0.], [ 0., 1., 1.]], dtype=float32) The min and max can be the same size as `t`, or broadcastable to that size. >>> t = tf.constant([[-1, 0., 10.], [-1, 0, 10]]) >>> clip_min = [[2],[1]] >>> t3 = tf.clip_by_value(t, clip_value_min=clip_min, clip_value_max=100) >>> t3.numpy() array([[ 2., 2., 10.], [ 1., 1., 10.]], dtype=float32) Broadcasting fails, intentionally, if you would expand the dimensions of `t` >>> t = tf.constant([[-1, 0., 10.], [-1, 0, 10]]) >>> clip_min = [[[2, 1]]] # Has a third axis >>> t4 = tf.clip_by_value(t, clip_value_min=clip_min, clip_value_max=100) Traceback (most recent call last): ... InvalidArgumentError: Incompatible shapes: [2,3] vs. [1,1,2] It throws a `TypeError` if you try to clip an `int` to a `float` value (`tf.cast` the input to `float` first). >>> t = tf.constant([[1, 2], [3, 4]], dtype=tf.int32) >>> t5 = tf.clip_by_value(t, clip_value_min=-3.1, clip_value_max=3.1) Traceback (most recent call last): ... TypeError: Cannot convert ... Args: t: A `Tensor` or `IndexedSlices`. clip_value_min: The minimum value to clip to. A scalar `Tensor` or one that is broadcastable to the shape of `t`. clip_value_max: The maximum value to clip to. A scalar `Tensor` or one that is broadcastable to the shape of `t`. name: A name for the operation (optional). Returns: A clipped `Tensor` or `IndexedSlices`. Raises: `tf.errors.InvalidArgumentError`: If the clip tensors would trigger array broadcasting that would make the returned tensor larger than the input. TypeError: If dtype of the input is `int32` and dtype of the `clip_value_min` or `clip_value_max` is `float32` rtnameN)r name_scopeconvert_to_tensor isinstancer IndexedSlicesvaluesr minimumshapeassert_is_compatible_withmaximumindices dense_shape)rclip_value_minclip_value_maxrrt_mint_maxr"Y/var/www/html/chatgem/venv/lib/python3.10/site-packages/tensorflow/python/ops/clip_ops.pyr"s*F     ClipByValuecCs|jd}|jd}|jd}|j}t|}t|}t|}t|} t| |} t||} t||} t ||\} }t ||\} }t t | | | |}t | || }t | || }t t |||}t t |||}|||fS)zReturns grad of clip_by_value.r)inputsdtyperrzerosr lessgreaterrbroadcast_gradient_argswhere logical_orreshape reduce_sum)opgradxyzgdtypesxsysz gradshaper)xymaskxzmask_ryrzxgradygradzgradgygzr"r"r#_clip_by_value_grads&           rE clip_by_normc Cs*t|d||g}tjt|tjr|jn|dd}t|r&|dkr%d}n t j t |d|j d}t j |||dd}|dk}t||t|}t|t ||}||} |j| jtj| t |||d} t|tjrt| |j|jWdS| WdS1swYdS) aClips tensor values to a maximum L2-norm. Given a tensor `t`, and a maximum clip value `clip_norm`, this operation normalizes `t` so that its L2-norm is less than or equal to `clip_norm`, along the dimensions given in `axes`. Specifically, in the default case where all dimensions are used for calculation, if the L2-norm of `t` is already less than or equal to `clip_norm`, then `t` is not modified. If the L2-norm is greater than `clip_norm`, then this operation returns a tensor of the same type and shape as `t` with its values set to: `t * clip_norm / l2norm(t)` In this case, the L2-norm of the output tensor is `clip_norm`. As another example, if `t` is a matrix and `axes == [1]`, then each row of the output will have L2-norm less than or equal to `clip_norm`. If `axes == [0]` instead, each column of the output will be clipped. Code example: >>> some_nums = tf.constant([[1, 2, 3, 4, 5]], dtype=tf.float32) >>> tf.clip_by_norm(some_nums, 2.0).numpy() array([[0.26967996, 0.5393599 , 0.80903983, 1.0787199 , 1.3483998 ]], dtype=float32) This operation is typically used to clip gradients before applying them with an optimizer. Most gradient data is a collection of different shaped tensors for different parts of the model. Thus, this is a common usage: ``` # Get your gradients after training loss_value, grads = grad(model, features, labels) # Apply some clipping grads = [tf.clip_by_norm(g, norm) for g in grads] # Continue on with training optimizer.apply_gradients(grads) ``` Args: t: A `Tensor` or `IndexedSlices`. This must be a floating point type. clip_norm: A 0-D (scalar) `Tensor` > 0. A maximum clipping value, also floating point. Note: If a negative clip_norm is provided, it will be treated as zero. axes: A 1-D (vector) `Tensor` of type int32 containing the dimensions to use for computing the L2-norm. If `None` (the default), uses all dimensions. name: A name for the operation (optional). Returns: A clipped `Tensor` or `IndexedSlices`. Raises: ValueError: If the clip_norm tensor is not a 0-D scalar tensor. TypeError: If dtype of the input is not a floating point or complex type. rFrrrr(T)keepdimsN)rrrrrrrnpisscalarr castrr(r0rr- ones_likesqrtrridentityrr) r clip_normaxesrrl2sumpred l2sum_safel2norm intermediate values_clipr"r"r#rFs4=  $zlinalg.global_norm global_norm)v1c Cst|tjr t|trtdt|dt|}t|d|Q}ddt |D}g}|D]#}|durQt || t |Wdn1sLwYq.tt|}tj|tjd|jddd }Wd|S1stwY|S) aDComputes the global norm of multiple tensors. Given a tuple or list of tensors `t_list`, this operation returns the global norm of the elements in all tensors in `t_list`. The global norm is computed as: `global_norm = sqrt(sum([l2norm(t)**2 for t in t_list]))` Any entries in `t_list` that are of type None are ignored. Args: t_list: A tuple or list of mixed `Tensors`, `IndexedSlices`, or None. name: A name for the operation (optional). Returns: A 0-D (scalar) `Tensor` of type `float`. Raises: TypeError: If `t_list` is not a sequence. 3`t_list` should be a sequence of tensors. Received .rWcS@g|]\}}|durtjt|tjr|jn|d|dn|qSNzt_%drrrrrrr.0irr"r"r# zglobal_norm..Ng@rGr)rr Sequencestr TypeErrortypelistrr enumerate colocate_withappendr l2_lossr r0rstackrMrconstantr()t_listrrhalf_squared_normsvhalf_squared_normnormr"r"r#rWs>    clip_by_global_normc CsZt|tjr t|trtdt|dt|}|dur"t||}t |d||gu}|t d|t j d|jd|}|||}ddt|D}g}t|D]8\}} | durb|dqTt| |tj| t || jd ||fd Wdn1swYqTd dt||D} Wd| |fS1swY| |fS) aClips values of multiple tensors by the ratio of the sum of their norms. Given a tuple or list of tensors `t_list`, and a clipping ratio `clip_norm`, this operation returns a list of clipped tensors `list_clipped` and the global norm (`global_norm`) of all tensors in `t_list`. Optionally, if you've already computed the global norm for `t_list`, you can specify the global norm with `use_norm`. To perform the clipping, the values `t_list[i]` are set to: t_list[i] * clip_norm / max(global_norm, clip_norm) where: global_norm = sqrt(sum([l2norm(t)**2 for t in t_list])) If `clip_norm > global_norm` then the entries in `t_list` remain as they are, otherwise they're all shrunk by the global ratio. If `global_norm == infinity` then the entries in `t_list` are all set to `NaN` to signal that an error occurred. Any of the entries of `t_list` that are of type `None` are ignored. This is the correct way to perform gradient clipping (Pascanu et al., 2012). However, it is slower than `clip_by_norm()` because all the parameters must be ready before the clipping operation can be performed. Args: t_list: A tuple or list of mixed `Tensors`, `IndexedSlices`, or None. clip_norm: A 0-D (scalar) `Tensor` > 0. The clipping ratio. use_norm: A 0-D (scalar) `Tensor` of type `float` (optional). The global norm to use. If not provided, `global_norm()` is used to compute the norm. name: A name for the operation (optional). Returns: list_clipped: A list of `Tensors` of the same type as `list_t`. global_norm: A 0-D (scalar) `Tensor` representing the global norm. Raises: TypeError: If `t_list` is not a sequence. References: On the difficulty of training Recurrent Neural Networks: [Pascanu et al., 2012](http://proceedings.mlr.press/v28/pascanu13.html) ([pdf](http://proceedings.mlr.press/v28/pascanu13.pdf)) rYrZNrs?rGcSr[r\r]r^r"r"r#raorbz'clip_by_global_norm..z%s_%drcSs2g|]\}}t|tjrt||j|jn|qSr")rrrrr)r_c_vrr"r"r#ras  )rr rcrdrerfrgrWrrr rrrmr(rhrjrirrNrKzip) rnrOuse_normrscale_for_finitescalervalues_clippedr`rp list_clippedr"r"r#rs*sP 3      ##zclip_by_average_norm is deprecated in TensorFlow 2.0. Please use clip_by_norm(t, clip_norm * tf.cast(tf.size(t), tf.float32), name) instead.)date instructionsclip_by_average_normc Cst|d||gA}tj|dd}tt|tj}t t ||t t |}tj ||t||td||d}Wd|S1sMwY|S)a]Clips tensor values to a maximum average L2-norm. Given a tensor `t`, and a maximum clip value `clip_norm`, this operation normalizes `t` so that its average L2-norm is less than or equal to `clip_norm`. Specifically, if the average L2-norm is already less than or equal to `clip_norm`, then `t` is not modified. If the average L2-norm is greater than `clip_norm`, then this operation returns a tensor of the same type and shape as `t` with its values set to: `t * clip_norm / l2norm_avg(t)` In this case, the average L2-norm of the output tensor is `clip_norm`. This operation is typically used to clip gradients before applying them with an optimizer. Args: t: A `Tensor`. clip_norm: A 0-D (scalar) `Tensor` > 0. A maximum clipping value. name: A name for the operation (optional). Returns: A clipped `Tensor`. r~rrrtN)rrrr rKrsizerfloat32rsqrtr0rangerankrNrrrm)rrOr n_element l2norm_invtclipr"r"r#r~s      )N)NN) __doc__numpyrItensorflow.python.frameworkrrrrtensorflow.python.opsrrrr r tensorflow.python.utilr r tensorflow.python.util.compatr tensorflow.python.util.tf_exportrregister_unary_elementwise_apiadd_dispatch_supportrRegisterGradientrErFdeprecated_endpointsrWrs deprecatedr~r"r"r"r#sN             [ [2_