o 2h@sJdZddlZddlmZddlZddlmZddlmZddlm Z ddlm Z ddlm Z dd lm Z dd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlm Z ddlm!Z!ddlm"Z#ddlm$Z%ddl&m'Z'ddl(m)Z*ddl+m,Z,ddl-m.Z.ddl-m/Z0dd l-m1Z1d!d"Z2d#d$Z3Gd%d&d&e0j4Z4Gd'd(d(e4Z5Gd)d*d*e4ej6e0j7Z7dRd,d-Z8e9e7e8Gd.d/d/ej:Z;e'j?Gd0d1d1e5e0j@Z@Gd2d3d3eAZBGd4d5d5e1jCZDGd6d7d7e5e%jEe.jFZGGd8d9d9e,jHZIGd:d;d;e,jHZJGdd?d?e,jHZLGd@dAdAeGZM   +dRdBdCZNe9eGeNdRdDdEZOe9eKeOdRdFdGZPe9e@ePdRdHdIZQe9eMeQGdJdKdKeAZRGdLdMdMeRZSGdNdOdOeRZTGdPdQdQZUdS)Sz0Various classes representing distributed values.N)Optional) struct_pb2) device_util)distribute_lib)packed_distributed_variable) reduce_util) values_util)context)record)composite_tensordevice)dtypes)ops)tensor)tensor_conversion_registry) tensor_shape) tensor_util) type_spec) array_ops)control_flow_ops)math_ops)resource_variable_ops)variable_scope) variables)nested_structure_coder)base)saveable_object)core) distribute)tracec sjtjjkr|fi|StjsEjtjj kr-j j s-t |r-tdt|j}ttj|f|ddSfdd}tj||f|dS)zCUpdates variables with ON_WRITE synchronization in replica context.Cannot update non-float variables with tf.VariableAggregation.MEAN aggregation in replica context. Either change the variable dtype to float or update it in cross-replica context.Targskwargsgroupcs\jtjjkrjjst|trtd|j ksJt ||j}j |fi|S)zCAggregate values and update all variables in cross replica context.r!) aggregationvsVariableAggregationMEANdtype is_floating isinstance PerReplica ValueErrordistribute_strategyrapply_aggregation_update_cross_replica)strategyvaluer$v update_fnvar^/var/www/html/chatgem/venv/lib/python3.10/site-packages/tensorflow/python/distribute/values.pymerge_fnRs z*_on_write_update_replica..merge_fn)r#r$)r&r'r(NONE_get_on_device_or_primaryr get_strategyextended_use_merge_callr)r*r+r is_tf_typer.!apply_aggregation_replica_contextrmark_as_unsaveableget_replica_context_update merge_call)r7r6r3r$aggregated_valuer:r8r5r9_on_write_update_replica4s4 rGcsnt|tr tdt|s|S|tjjkr&fdd}t j ||fdSt j |}tj||}|S)zBAggregate `value` to `destinations` as specified by `aggregation`.zDCannot use DistributedValues to update variables in replica context.cs|jj||ddS)Nr destinations)r> broadcast_toexperimental_local_results)r2r3rHr8r9r:vs z3apply_aggregation_replica_context..merge_fn)r#)r,DistributedValues TypeErrorrr@r'r(ONLY_FIRST_REPLICArrCrErReduceOpfrom_variable_aggregationr=r>_replica_ctx_all_reduce)r3r&rIr: reduce_oprFr8rHr9rAks      rAc@sXeZdZdZddZddZddZdd Zed d Z ed d Z ddZ ddZ dS)rLz/Base class for representing distributed values.cCst||_dS)z+Should only be called by subclass __init__.Ntuple_values)selfvaluesr8r8r9__init__zDistributedValues.__init__cCs"t}|dur |S|j|Sz@Returns the value for the current device or raises a ValueError.N)rget_current_replica_id_as_int_get_cross_replicarUrV replica_idr8r8r9_gets zDistributedValues._getcCtdt|)Nz}DistributedValues._get_cross_replica should be implemented by sub-classes which support cross-replica accesses. Type name is NotImplementedErrortyperVr8r8r9r\s z$DistributedValues._get_cross_replicacCsRt}|dur$tt}|jD]}t|j|kr |Sq|jS|j|S)GReturns value in same replica or device if possible, else the _primary.N)rr[r canonicalizecurrentrUr _primary)rVr^current_devicer3r8r8r9r<s  z+DistributedValues._get_on_device_or_primarycCs |jdS)z#Returns a representative component.rrUrdr8r8r9rhs zDistributedValues._primarycCstdd|jDS)Ncs|]}|jVqdSNr .0r4r8r8r9 z-DistributedValues._devices..rSrdr8r8r9_devicesszDistributedValues._devicescC*dddt|jD}d|jj|fS)N, cs |] \}}d||fVqdS)z %d: %sNr8rnir4r8r8r9ro z,DistributedValues.__str__.. %s:{ %s }join enumeraterU __class____name__)rV debug_strr8r8r9__str__ zDistributedValues.__str__cCrr)Nrscsrt)z %d: %rNr8rur8r8r9rorwz-DistributedValues.__repr__..rxry)rV debug_reprr8r8r9__repr__rzDistributedValues.__repr__N) r} __module__ __qualname____doc__rXr_r\r<propertyrhrqrrr8r8r8r9rLs   rLcs6eZdZdZfddZeddZddZdd Zd d Z d d Z ddZ ddZ ddZ ddZddZddZddZddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0d1Zd2d3Zd4d5ZdId7d8Zd9d:Z d;d<Z!d=d>Z"d?d@Z#dAdBZ$dCdDZ%dEdFZ&dGdHZ'Z(S)JDistributedDelegatezAA map from device to values; acts as the same type as the values.cs>|ds |dvrtt||S|dkrtt||S)N_self_)_use_resource_variables_attribute_sentinel_distributed_containerrU) startswithsuperr __getattr__AttributeErrorgetattrr_rVnamer|r8r9rs  zDistributedDelegate.__getattr__cC|jSzReturns the per replica values.rjrdr8r8r9rWzDistributedDelegate.valuescC|S)aReturns the value for operations for the current device. Some implementations, e.g. `TPUMirroredVariable`, are not able to return the value type within a replica context. They can, however, return a value that can be used by the operations below. )r_rdr8r8r9_get_as_operandsz#DistributedDelegate._get_as_operandcCs ||SrlrrVor8r8r9__add__ zDistributedDelegate.__add__cCs ||Srlrrr8r8r9__radd__rzDistributedDelegate.__radd__cCs ||Srlrrr8r8r9__sub__rzDistributedDelegate.__sub__cCs ||Srlrrr8r8r9__rsub__rzDistributedDelegate.__rsub__cCs ||Srlrrr8r8r9__mul__rzDistributedDelegate.__mul__cCs ||Srlrrr8r8r9__rmul__rzDistributedDelegate.__rmul__cCs ||Srlrrr8r8r9 __truediv__rzDistributedDelegate.__truediv__cCs ||Srlrrr8r8r9 __rtruediv__rz DistributedDelegate.__rtruediv__cCs ||Srlrrr8r8r9 __floordiv__ rz DistributedDelegate.__floordiv__cCs ||Srlrrr8r8r9 __rfloordiv__ rz!DistributedDelegate.__rfloordiv__cCs ||Srlrrr8r8r9__mod__rzDistributedDelegate.__mod__cCs ||Srlrrr8r8r9__rmod__rzDistributedDelegate.__rmod__cCs ||kSrlrrr8r8r9__lt__rzDistributedDelegate.__lt__cCs ||kSrlrrr8r8r9__le__rzDistributedDelegate.__le__cCs ||kSrlrrr8r8r9__gt__rzDistributedDelegate.__gt__cCs ||kSrlrrr8r8r9__ge__rzDistributedDelegate.__ge__cCs ||@Srlrrr8r8r9__and__!rzDistributedDelegate.__and__cCs ||@Srlrrr8r8r9__rand__$rzDistributedDelegate.__rand__cCs ||BSrlrrr8r8r9__or__'rzDistributedDelegate.__or__cCs ||BSrlrrr8r8r9__ror__*rzDistributedDelegate.__ror__cCs ||ASrlrrr8r8r9__xor__-rzDistributedDelegate.__xor__cCs ||ASrlrrr8r8r9__rxor__0rzDistributedDelegate.__rxor__cCs ||Srlrrr8r8r9 __getitem__3rzDistributedDelegate.__getitem__NcCst|||Srlpowr)rVrmodulor8r8r9__pow__6zDistributedDelegate.__pow__cCst||Srlrrr8r8r9__rpow__9zDistributedDelegate.__rpow__cCs |Srlrrdr8r8r9 __invert__< zDistributedDelegate.__invert__cCs | Srlrrdr8r8r9__neg__?rzDistributedDelegate.__neg__cCs t|Srl)absrrdr8r8r9__abs__BrzDistributedDelegate.__abs__cC(z||WStytYSwrl)r__div__rNotImplementedrr8r8r9rE  zDistributedDelegate.__div__cCrrl)r__rdiv__rrrr8r8r9rLrzDistributedDelegate.__rdiv__cCrrl)r __matmul__rrrr8r8r9rSrzDistributedDelegate.__matmul__cCrrl)r __rmatmul__rrrr8r8r9rZrzDistributedDelegate.__rmatmul__rl))r}rrrrrrWrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr __classcell__r8r8rr9rsL    rc@s(eZdZdZeddZeddZdS)r-z2Holds a map from replica to unsynchronized values.cCstdd|jDS)Ncss|]}t|VqdSrl)rtype_spec_from_valuermr8r8r9roksz(PerReplica._type_spec..)PerReplicaSpecrUrdr8r8r9 _type_spechszPerReplica._type_speccCrrrjrdr8r8r9rWmrzPerReplica.valuesN)r}rrrrrrWr8r8r8r9r-ds  r-FcCsb~|dur||jstd|j|jj|rtdts$ts(tdt }|j |S)z&Converts a `PerReplica` to a `Tensor`.NzMIncompatible type conversion requested to type {!r} for variable of type {!r}z5PerReplica doesn't support being used as a reference.zIt looks like you are using a PerReplica object while not inside a replica context, which is not supported. Try running your op or function inside a replica context by using `strategy.run`) is_compatible_withr*r.formatrrbrin_cross_replica_context has_strategyrr[rW)r7r*ras_refr^r8r8r9_per_replica_to_tensorss  rc@sNeZdZdZdgZeddZddZddZed d Z d d Z d dZ dS)rz&Type specification for a `PerReplica`. _value_specscCstSrlr-rdr8r8r9szPerReplicaSpec.cGst||_dSrl)rTr)rV value_specsr8r8r9rXrzPerReplicaSpec.__init__cCrrlrrdr8r8r9 _serializezPerReplicaSpec._serializecCrrlrrdr8r8r9_component_specszPerReplicaSpec._component_specscCs(t}|dur|jdkrtd|jS)NzJFlattening a PerReplica to components is not supported in replica context.)rrCnum_replicas_in_syncr.rU)rVr3replica_contextr8r8r9_to_componentss zPerReplicaSpec._to_componentscCst|Srlr)rV tensor_listr8r8r9_from_componentszPerReplicaSpec._from_componentsN) r}rrr __slots__r value_typerXrrrrr8r8r8r9rs   rc@(eZdZdZddZddZddZdS) Mirroredz:Holds a map from replica to values which are kept in sync.cCrrlr<rdr8r8r9r\rzMirrored._get_cross_replicacCs*|}t|dd}|rt|r|S|S)N_as_graph_element)r_rcallable)rVobjconv_fnr8r8r9rs   zMirrored._as_graph_elementcCdSNTr8rdr8r8r9 _is_mirroredzMirrored._is_mirroredN)r}rrrr\rrr8r8r8r9rs  rc@r) DistributedVarOpz'A class that looks like `tf.Operation`.cCs||_||_||_||_dSrl)rgraph tracebackrc)rVrrrtypr8r8r9rXs zDistributedVarOp.__init__cCs@t||jst|j|jko|j|jko|j|jko|j|jkSrl)r,r|rbrrrrcrr8r8r9__eq__s   zDistributedVarOp.__eq__cCst|j|jt|j|jfSrl)hashrrrTrrcrdr8r8r9__hash__szDistributedVarOp.__hash__N)r}rrrrXrrr8r8r8r9rs  rc@s^eZdZdZddZddZddZdd d Zd d Zd dZ de fddZ de fddZ dS)DistributedVariableTraceTypez)TraceType of DistributedVariable objects.cCs ||_t|j|jf|_dSrl)distributed_variablerTshapeas_listr* components)rVrr8r8r9rXs  z%DistributedVariableTraceType.__init__cCs||kSrlr8rVotherr8r8r9 is_subtype_ofrz*DistributedVariableTraceType.is_subtype_ofcstfdd|Dr SdS)Nc3s|]}|kVqdSrlr8)rnrrdr8r9roszNDistributedVariableTraceType.most_specific_common_supertype..)all)rVothersr8rdr9most_specific_common_supertypesz;DistributedVariableTraceType.most_specific_common_supertypeNcCrrl)r)rVplaceholder_contextr8r8r9placeholder_valuerz.DistributedVariableTraceType.placeholder_valuecCsgSrlr8)rVr3r8r8r9 to_tensorsrz'DistributedVariableTraceType.to_tensorscCs|Srlr8)rVr3_r8r8r9castrz!DistributedVariableTraceType.castreturncCs t|jSrl)rrrdr8r8r9rrz%DistributedVariableTraceType.__hash__cCst|tsdS|j|jkSNF)r,rrrr8r8r9rs  z#DistributedVariableTraceType.__eq__rl)r}rrrrXrrrrrintrboolrr8r8r8r9rs rcseZdZdZdfdd ZddZddZdd d Zed d Z d dZ ddZ eddZ eddZ eddZeddZeddZeddZeddZedd Zed!d"Zed#d$Zed%d&Zed'd(Zed)d*Zdd+d,Zed-d.Zd/d0Zd1d2Zed3d4Zed5d6Zed7d8Zd9e j!fd:d;Z"ddd?Z&ed@dAZ'dBdCZ(dDdEZ)dFdGZ*dHdIZ+dJdKZ,dLdMZ-ddPdQZ.ddRdSZ/ddTdUZ0ddVdWZ1ddXdYZ2ddZd[Z3dd\d]Z4dd^d_Z5dd`daZ6ddbdcZ7dddeZ8dfdgZ9dhdiZ:djdkZ;dldmZdrdsZ?ddtduZ@  ddveAeBjCdweAeDd9eEjFfdxdyZG   ddzd{ZHd|d}ZId~dZJeddZKddZLZMS)DistributedVariablez&Holds a map from replica to variables.Nc s|tjjkr|djjstd||_||_tt | ||j j dd|_|D]}t|tjr?t|dr?t||j_q*t||_q*trt|ddrd|j}t|ddrwtjtd d |Dg|d }|D]}||_qmd|_n tj||d |_nd|_d|_d|_||_ dS) Nrzcreating distributed tf.Variable with aggregation=MEAN and a non-floating dtype is not supported, please use a different aggregation or dtype:handle%_enable_packed_variable_in_eager_modeFz %s/packed/_varscsrkrl)r )rnr3r8r8r9ro rpz/DistributedVariable.__init__..r)! variables_libr(r)r*r+r._distribute_strategy _aggregationrr rXrhrsplit _common_namer,r CompositeTensorhasattrweakrefrefr rr#executing_eagerly_outside_functionsrpackedPackedDistributedVariablesum _packed_var_keras_initialized_initializer_op_policy) rVr2rWr& var_policyr4r packed_varr3rr8r9rXs@     zDistributedVariable.__init__c Cst|j/g}|jD]!}t|j|t||Wdn1s(wYq Wdn1s8wYt ||j||j t|j |d}||t |<|S)abPerform a deepcopy of the `DistributedVariable`. Unlike the deepcopy of a regular tf.Variable, this keeps the original strategy and devices of the `DistributedVariable`. To avoid confusion with the behavior of deepcopy on a regular `Variable` (which does copy into new devices), we only allow a deepcopy of a `DistributedVariable` within its originating strategy scope. Args: memo: The memoization object for `deepcopy`. Returns: A deep copy of the current `DistributedVariable`. Raises: RuntimeError: If trying to deepcopy into a different strategy. N)r2rWr&r ) renter_or_assert_strategyrrUrr appendcopydeepcopyrcrrid)rVmemo new_valuesr3copied_variabler8r8r9 __deepcopy__9s"   z DistributedVariable.__deepcopy__cCs|jduo t Srl)rris_saving_non_distributedrdr8r8r9_use_packed_variable\s z(DistributedVariable._use_packed_variablecCsptr |jS|r|jS|j}|jddD] }t||}qtj||jd|d}|S)zIdentifies if all the component variables are initialized. Args: name: Name of the final `logical_and` op. Returns: The op that evaluates to True or False depending on if all the component variables are initialized. rr) rr+rhis_initializedr,rrUr logical_and)rVrresultr4r8r8r9r.bs   z"DistributedVariable.is_initializedcCs>tr|jjS|jr|j}|Sttdd|jD}|S)Ncsrkrl) initializerrmr8r8r9rorpz2DistributedVariable.initializer..) rr+rhr1rrr%rTrU)rVinit_opr8r8r9r1{szDistributedVariable.initializercC |Srl)r<initialized_valuerdr8r8r9r4rz%DistributedVariable.initialized_valuecCs|jduo |jSrl)rrrdr8r8r9rz DistributedVariable._is_mirroredcC |jSrl)r< initial_valuerdr8r8r9r7 z!DistributedVariable.initial_valuecC|jjSrl)rh constraintrdr8r8r9r:zDistributedVariable.constraintcCr9rl)rhrrdr8r8r9rr;zDistributedVariable.graphcCrrl)rrdr8r8r9 _shared_namerz DistributedVariable._shared_namecCr9rl)rh _unique_idrdr8r8r9r=r;zDistributedVariable._unique_idcCr9)z7Lets Optimizers know which graph this variable is from.)rh _graph_keyrdr8r8r9r>szDistributedVariable._graph_keycCr9rl)rhrrdr8r8r9rr;zDistributedVariable.namecCr9rl)rhr*rdr8r8r9r*r;zDistributedVariable.dtypecCr9rl)rhrrdr8r8r9rr;zDistributedVariable.shapecCr9rl)rhsynchronizationrdr8r8r9r?r;z#DistributedVariable.synchronizationcCrrlrrdr8r8r9r&rzDistributedVariable.aggregationcCs|r|jSdSrl)r,rrdr8r8r9_packed_variablesz$DistributedVariable._packed_variablecCsDtr|jjSt}|durtd|r|jjS|j|jS)NztDistributedVariable.handle is not available outside the replica context or a `tf.distribute.Strategy.update()` call.) rr+rhr r[r.r,rrUr]r8r8r9r s zDistributedVariable.handlecCs||Srl)r<eval)rVsessionr8r8r9rBrzDistributedVariable.evalcCr9rl)rh_save_slice_infordr8r8r9rDr;z$DistributedVariable._save_slice_infocC |jSrl)rh_get_save_slice_infordr8r8r9rFrz(DistributedVariable._get_save_slice_infocCs|jD]}||qdSrl)rU_set_save_slice_info)rVsave_slice_infor4r8r8r9rGs  z(DistributedVariable._set_save_slice_infocCr6rl)r<r rdr8r8r9r r8zDistributedVariable.devicecCr9rl)rh trainablerdr8r8r9rIr;zDistributedVariable.trainablecCrrl)rrdr8r8r9r/rz'DistributedVariable.distribute_strategyrcCrErl)rh get_shaperdr8r8r9rJrzDistributedVariable.get_shapecCs|jj|dS)N) export_scope)rhto_proto)rVrKr8r8r9rLrzDistributedVariable.to_protocCsHtr|jjStrt|jjj|jjj|jjj |jjj S| jSrl) rr+rhoprrrrrrrcr_rdr8r8r9rMs zDistributedVariable.opcCr9rl)rh_in_graph_moderdr8r8r9rNr;z"DistributedVariable._in_graph_modecCs$|j|}|r|j|jS|S)z8Returns the value on a device with the given replica_id.)rUr,r on_devicer )rVr^r3r8r8r9 _get_replicas z DistributedVariable._get_replicacCs0tr|jSt}|dur|S||SrZ)rr+rhr[r\rPr]r8r8r9r_s  zDistributedVariable._getcCsrtr|jSt}|dur4tt}t|jD]\}}t|j |kr.| |Sq| dS| |S)reNr) rr+rhr[rrfrgr{rUr rP)rVr^rirvr3r8r8r9r<s  z-DistributedVariable._get_on_device_or_primarycCsPtr |jSt|jt| WdS1s!wYdSrl) rr+rh read_valuerr"rridentityr_rdr8r8r9rQ s   $zDistributedVariable.read_valuecCs0tr |jS|jr|j|S|Srl)rr+rhr3rr<rdr8r8r9r3&s    zDistributedVariable.valuecCstr |Std)NzNDistributedVariable.numpy() is only available when eager execution is enabled.)r executing_eagerlyrQnumpyrbrdr8r8r9rT-s zDistributedVariable.numpyFTcCJtr |j||||S|jr|jj|||||dStj|||||dSN use_lockingrrQ)rr+rh assign_subron_write_assign_subrVr3rXrrQr8r8r9rY4 zDistributedVariable.assign_subcCrUrV)rr+rh assign_addron_write_assign_addr[r8r8r9r]Ar\zDistributedVariable.assign_addcCrUrV)rr+rhassignron_write_assignr[r8r8r9r_Nr\zDistributedVariable.assigncCDtr |j|||S|jr|jj||||dStj||||dSNrXr)rr+rh scatter_subrrV sparse_deltarXrr8r8r9rd[zDistributedVariable.scatter_subcCrarb)rr+rh scatter_addrrer8r8r9rhdrgzDistributedVariable.scatter_addcCrarb)rr+rh scatter_mulrrer8r8r9rimrgzDistributedVariable.scatter_mulcCrarb)rr+rh scatter_divrrer8r8r9rjvrgzDistributedVariable.scatter_divcCrarb)rr+rh scatter_minrrer8r8r9rkrgzDistributedVariable.scatter_mincCrarb)rr+rh scatter_maxrrer8r8r9rlrgzDistributedVariable.scatter_maxcCrarb)rr+rhscatter_updaterrer8r8r9rmrgz"DistributedVariable.scatter_updatecCst|Srl)rrVrr8r8r9__tf_tracing_type__rz'DistributedVariable.__tf_tracing_type__cjffdd }tj|iS)zOverrides Trackable method. This allows both name-based and object-based save and restore of DistributedVariables. Returns: A dictionary mapping attribute names to `SaveableObject` factories. ctj|Srl)_DistributedVariableSaveablerhrrdr8r9_saveable_factoryrzODistributedVariable._gather_saveables_for_checkpoint.._saveable_factoryr trackableVARIABLE_VALUE_KEYrVrsr8rdr9 _gather_saveables_for_checkpoint z4DistributedVariable._gather_saveables_for_checkpointcCs6tr |jS|jr|j|Stdt|)NzDistributedVariable._as_graph_element requires a valid VariablePolicy. Please set the policy via the `var_policy` argument in the constructor, or override this method in sub-classes which support cross-replica accesses. Type name is )rr+rhrrrbrcrdr8r8r9rs  z%DistributedVariable._as_graph_elementcCs2tr|jS|jr|j|Stdt|)NzDistributedVariable._get_cross_replica requires a valid VariablePolicy. Please set the policy via the `var_policy` argument in the constructor, or override this method in sub-classes which support cross-replica accesses. Type name is )rr+rhrr\rbrcrdr8r8r9r\s z&DistributedVariable._get_cross_replicacKs"t|jjj|||f|ddS)agApplies updates across replicas. Args: update_fn: A callable to pass to `strategy.extended.update` to update the variable. It should has the same signature as `Variable.assign()`. value: value to be passed to `update_fn`. **kwargs: remaining arguments to `update_fn`. Returns: Updated variable or `tf.Operation`. Tr")rrBr/r>updaterVr6r3r$r8r8r9r1s  z)DistributedVariable._update_cross_replicacKs0|jr|jj|||fi|Stdt|)a@Applies updates in one replica. Args: update_fn: A callable to update the variable. It should has the same signature as `Variable.assign()`. value: value to be passed to `update_fn`. **kwargs: remaining arguments to `update_fn`. Returns: Updated variable or `tf.Operation`. zDistributedVariable._update_replica requires a valid VariablePolicy. Please set the policy via the `var_policy` argument in the constructor, or override this method in sub-classes which support cross-replica accesses. Type name is )r_update_replicarbrcr{r8r8r9r|s z#DistributedVariable._update_replicacKstr||j|fi|St|jJtrGt}|dur6||}|||fi|WdS|j ||fi|WdSt |j|j ||fi|WdS1sawYdS)aApplies updates depending on the context. The method calls `_update_replica` in replica context, `_update_cross_replica` in cross replica context, and `update_fn` in update context. If `read_value` is True, the method returns the updated Variable. If `read_value` is False, the method returns the update `tf.Operation`. Args: update_fn: A callable to pass to `strategy.extended.update` to update the variable. It should have the same signature as `Variable.assign()`. value: value to be passed to `update_fn`. **kwargs: keyword arguments to `update_fn`. Returns: Updated variable or `tf.Operation`. N) rr+rhrr"r/rget_update_replica_idrPr1assert_replica_contextr|)rVr6r3r$update_replica_id replica_valuer8r8r9rDs  $zDistributedVariable._updatecCr)z6Pass resource_variable_ops.is_resource_variable check.Nr8rdr8r8r9 _should_act_as_resource_variablez4DistributedVariable._should_act_as_resource_variablecCsbtrtj|j|||dSt|jtj||||dWdS1s*wYdS) Converts a variable to a tensor.r*rrN) rr+rconvert_to_tensorrhrr"rr_rVr*rrr8r8r9_dense_var_to_tensors  $z(DistributedVariable._dense_var_to_tensorr*rcCs |||Srlr)rVr*rr8r8r9 __tf_tensor__s z!DistributedVariable.__tf_tensor__c sjjd|||d|}fddjDD]-}|jr.||jd|||d|q|j||<|jj||j<||jq|j|<|jj|<|jduro|jj|jj <|jj |S)N) object_map tensor_mapoptionscsg|] }|jkr|qSr8)rhrmrdr8r9 -szDDistributedVariable._export_to_saved_model_graph..r8) rh_export_to_saved_model_graphrUexperimental_variable_policy_expand_distributed_variablesextendr r#r packed_handle)rVrrrr$ resource_listr4r8rdr9r!s@   z0DistributedVariable._export_to_saved_model_graphc Cs||vr.cSrrlr8rr8r8r9r|r)backward_functionforward_function)rreplace_capturer record_operation)rVconcrete_functioninternal_capturerr8r8r9#__tf_experimental_restore_capture__ts  z7DistributedVariable.__tf_experimental_restore_capture__rlFNTFNNNF)NN)NNN)Nr}rrrrXr*r,r.rr1r4rr7r:rr<r=r>rr*rr?r&rAr rBrDrFrGr rIr/r TensorShaperJrLr OperationrMrNrPr_r<rQr3rTrYr]r_rdrhrirjrkrlrmrorxrr\r1r|rDrrrrDTypestr tensor_libTensorrrrrrrrr8r8rr9r s;#                           !    " r c(eZdZdZfddZddZZS)rrz8Class for defining how to restore a DistributedVariable.csB||_|jjs td|j|||\}}tt||||dS)NzThe VariablePolicy of the argument `distributed_variable` must be set to create a _DistributedVariableSaveable. Please set it via the `var_policy` argument in the constructor of DistributedVariable.)_distributed_variablerr. get_saveablerrrrX)rVrprimary_variablerrspecrr8r9rXsz%_DistributedVariableSaveable.__init__cCs|\}|jj|j|Sz*Restore the same value into all variables.)rrget_restore_opsrVrestored_tensorsrestored_shapesrr8r8r9restoresz$_DistributedVariableSaveable.restorer}rrrrXrrr8r8rr9rrs  rrcr)_MirroredSaveablez5Class for defining how to restore a MirroredVariable.cs2||_t|j||\}}tt||||dSrl)_mirrored_variablerget_on_write_saveablerrrX)rVmirrored_variablerrrrrr8r9rXs z_MirroredSaveable.__init__cCs|\}t|j|Sr)rget_on_write_restore_opsrrr8r8r9rsz_MirroredSaveable.restorerr8r8rr9rs rcsjeZdZdZddZddZfddZfdd Zfd d Zd d Z ddZ ddZ dddZ Z S)MirroredVariablezDHolds a map from replica to variables whose values are kept in sync.cCs t|Srl)rrrdr8r8r9rrzMirroredVariable._is_mirroredcKst|||fi|Srl)rGr{r8r8r9r|r5z MirroredVariable._update_replicacbtr |jj|i|S|jtjjkr&|jtjjkr&t tj j d|jdt t |j|i|S)Nrkop_namer&)rr+rhrkrr'r(rNr;rbscatter_error_msgrrrrVr#r$rr8r9rkzMirroredVariable.scatter_mincr)Nrlr)rr+rhrlrr'r(rNr;rbrrrrrrr8r9rlrzMirroredVariable.scatter_maxcr)Nrmr)rr+rhrmrr'r(rNr;rbrrrrrrr8r9rmrzMirroredVariable.scatter_updatecCstt|Srl)rrRrr\rdr8r8r9r\sz#MirroredVariable._get_cross_replicacCr3rlr<rrdr8r8r9rrz"MirroredVariable._as_graph_elementcrp)zOverrides Trackable method. This allows both name-based and object-based save and restore of MirroredVariables. Returns: A dictionary mapping attribute names to `SaveableObject` factories. crqrl)rrhrrdr8r9rsrzLMirroredVariable._gather_saveables_for_checkpoint.._saveable_factoryrtrwr8rdr9rxryz1MirroredVariable._gather_saveables_for_checkpointNFcCs"|rtdtj||||dS)rzYou may be using variable created under distribute strategy in TF 1.x control flows. Try explicitly converting the variable to Tensor using variable.read_value(), or switch to TF 2.x.r)r.rrr_rr8r8r9rs z%MirroredVariable._dense_var_to_tensorr)r}rrrrr|rkrlrmr\rrxrrr8r8rr9rs   rcr)_SyncOnReadSaveablez7Class for defining how to restore a SyncOnReadVariable.cs2||_t||j|\}}tt||||dSrl)_sync_on_read_variablerget_on_read_saveablerhrrrX)rVsync_on_read_variablerrrrr8r9rXs z_SyncOnReadSaveable.__init__cCs|\}t|j||jjSr)rget_on_read_restore_opsrr&rr8r8r9rs z_SyncOnReadSaveable.restorerr8r8rr9rs rcseZdZdZddZfddZd+fd d Zd+fd d Zd+fd d ZddZ ddZ ddZ ddZ ddZ ddZddZddZdd Zfd!d"Zd#d$Zd%d&Zd'd(Zd,d)d*ZZS)-SyncOnReadVariablezGHolds a map from replica to variables whose values are reduced on save.cKs|||fi|Srlrr{r8r8r9r|z"SyncOnReadVariable._update_replicacs>t|jtt|WdS1swYdS)aReturns the value of SyncOnReadVariable based on surrounding context. If called under a non-default replica-context, returns the corresponding variable on that replica. If called under default replica-context or cross-replica context, returns the synced value. N)rr"rrrr_rdrr8r9r_s $zSyncOnReadVariable._getFNTctr |j||||St|j.tr/ts/t tj |||dWdSt t |||||WdS1sDwYdSNrQ) rr+rhrYrr"rrin_replica_update_contextrB on_read_assign_sub_cross_replicarrr[rr8r9rY!  $zSyncOnReadVariable.assign_subcrr) rr+rhr]rr"rrrrB on_read_assign_add_cross_replicarrr[rr8r9r].rzSyncOnReadVariable.assign_addcrr) rr+rhr_rr"rrrrBon_read_assign_cross_replicarrr[rr8r9r_;s$zSyncOnReadVariable.assigncCtd|d)Nz:Variables with `synchronization=ON_READ` doesn't support ``rbrVmethodr8r8r9_scatter_not_implementedHs z+SyncOnReadVariable._scatter_not_implementedcO(tr |jj|i|S|ddSNrd)rr+rhrdrrr8r8r9rdLzSyncOnReadVariable.scatter_subcOrNrh)rr+rhrhrrr8r8r9rhQrzSyncOnReadVariable.scatter_addcOrNri)rr+rhrirrr8r8r9riVrzSyncOnReadVariable.scatter_mulcOrNrj)rr+rhrjrrr8r8r9rj[rzSyncOnReadVariable.scatter_divcOrNrk)rr+rhrkrrr8r8r9rk`rzSyncOnReadVariable.scatter_mincOrNrl)rr+rhrlrrr8r8r9rlerzSyncOnReadVariable.scatter_maxcOrNrm)rr+rhrmrrr8r8r9rmjrz!SyncOnReadVariable.scatter_updatecCstrtdtr|jSt|j6t r@t s@|j t j jkr5|dWdS|WdS|WdS1sPwYdS)NzMcall `variable.value()` inside variable_sync_on_read_context is not supportedr)r in_variable_sync_on_read_contextrbrr+rhr3r"rrrrr'r(rNrPr\r<rdr8r8r9r3os"   $zSyncOnReadVariable.valuecstrtdtS)NzRcall `variable.read_value()` inside variable_sync_on_read_context is not supported)rrrbrrQrdrr8r9rQs  zSyncOnReadVariable.read_valuecCsz|jtjjkr |dS|jtjjkrtt |j |j j t j |j|ddWdS1s6wYdSNr)axis)rr'r(rNrPSUMrrBrr"rreducerrOrPrdr8r8r9r\s  $z%SyncOnReadVariable._get_cross_replicacCsntr |jSt|jtr"t | WdSWdn1s,wY| Srl) rr+rhrrr"rrrrr\r_rdr8r8r9rs   z$SyncOnReadVariable._as_graph_elementcrp)zOverrides Trackable method. This allows both name-based and object-based save and restore of `SyncOnReadVariable`s. Returns: A dictionary mapping attribute names to `SaveableObject` factories. cs t|Srl)rrrdr8r9rsrzNSyncOnReadVariable._gather_saveables_for_checkpoint.._saveable_factoryrtrwr8rdr9rxryz3SyncOnReadVariable._gather_saveables_for_checkpointcCstrtj|j|||dSt|jet}|durgt rg|j t j j kr;tj|d|||dWdS|j t j jkrFt|jjtj|j |}tj||||dWdStj||||dWdS1s|wYdS)z*Converts a SyncOnReadVariable to a tensor.rNr)rr+rrrhrr"rrCrrr'r(rNrPrrBr2r>rQrrOrPr_rQ)rVr*rrrreducedr8r8r9rs>   $z'SyncOnReadVariable._dense_var_to_tensorrr)r}rrrr|r_rYr]r_rrdrhrirjrkrlrmr3rQr\rrxrrr8r8rr9rs*        rcC|j|||dSNrrr7r*rrr8r8r9"_tensor_conversion_distributed_varsrcCrrrrr8r8r9_tensor_conversion_mirroredrrcCstj||||dSr)rrr_)r3r*rrr8r8r9_tensor_conversion_mirrored_vals rcCrrrrr8r8r9_tensor_conversion_sync_on_readrrc@s@eZdZdZddZddZddZdd Zd d Zd d Z dS)VariablePolicyaPolicy defining synchronization and aggregation of a distributed variable. Given `synchronization` and `aggregation` parameters set on a `tf.Variable` during variable creation within `tf.distribute` scope, `tf.distribute` creates an appropriate policy object and assigns it to the distributed variable. All variable operations are delegated to the respective policy object. cCs ||_dSrlr@)rVr&r8r8r9rXrzVariablePolicy.__init__cCr`)NzGVariablePolicy.value should be overridden by sub-classes. Type name is rardr8r8r9r3 zVariablePolicy.valuecCr`)NzNVariablePolicy._is_mirrored should be overridden by sub-classes. Type name is rardr8r8r9rrzVariablePolicy._is_mirroredcCr`)NzSVariablePolicy._as_graph_element should be overridden by sub-classes. Type name is rarnr8r8r9r rz VariablePolicy._as_graph_elementcCr`)NzTVariablePolicy._get_cross_replica should be overridden by sub-classes. Type name is rarVr7r8r8r9r\rz!VariablePolicy._get_cross_replicacKr`)NzQVariablePolicy._update_replica should be overridden by sub-classes. Type name is rarVr7r6r3r$r8r8r9r|rzVariablePolicy._update_replicaN) r}rrrrXr3rrr\r|r8r8r8r9rs rc@seZdZdZddZddZddZdd Zd d Zd d Z   d)ddZ   d)ddZ d)ddZ ddZ ddZddZddZdd Zd!d"Zd#d$Zd%d&Zd'd(ZdS)* OnReadPolicya{Policy defined for `tf.VariableSynchronization.ON_READ` synchronization. This policy is created when `synchronization` is set to `tf.VariableSynchronization.ON_READ` and `aggregation` is set to any of the values allowed by the `tf.VariableAggregation` enum such as `NONE`, `SUM`, `MEAN` or `ONLY_FIRST_REPLICA`when creating a `tf.Variable` in `tf.distribute` scope. cCrrr8rdr8r8r9r&rzOnReadPolicy._is_mirroredcCst|j6tr/ts/|jtjj kr$| d WdS| WdS| WdS1s?wYdS)Nr)rr"r/rrrrr'r(rNrPr3r\r<rr8r8r9r3)s  $zOnReadPolicy.valuecCs\t|jtrt|WdSWdn1s#wY|Srl) rr"r/rrrr\r_rrr8r8r9r3s  zOnReadPolicy._as_graph_elementcCsz|jtjjkr |dS|jtjjkrtt |j |j j t j |j|ddWdS1s6wYdSr)rr'r(rNrPrrrBrr"r/rrrOrPrr8r8r9r\9s  $zOnReadPolicy._get_cross_replicacKs|||fi|Srlrrr8r8r9r|DrzOnReadPolicy._update_replicacCr)Nz#ON_READ variables doesn't support `z` in cross replica contextrrr8r8r9rGrz%OnReadPolicy._scatter_not_implementedFNTcCzt|j-tr"ts"ttj|||dWdStj|||||dWdS1s6wYdS)z%Subtracts a value from this variable.rNrW) rr"r/rrrrBrrZrVr7r3rXrrQr8r8r9rYK"$zOnReadPolicy.assign_subcCr)zAdds a value to this variable.rNrW) rr"r/rrrrBrr^rr8r8r9r]`rzOnReadPolicy.assign_addcCszt|j-tr"ts"ttj|||dWdStj|||||dWdS1s6wYdS)NrrW) rr"r/rrrrBrr`rr8r8r9r_us"$zOnReadPolicy.assigncO~~|ddSrrrr8r8r9rdzOnReadPolicy.scatter_subcOrrrrr8r8r9rhr zOnReadPolicy.scatter_addcOrrrrr8r8r9rir zOnReadPolicy.scatter_mulcOrrrrr8r8r9rjr zOnReadPolicy.scatter_divcOrrrrr8r8r9rkr zOnReadPolicy.scatter_mincOrrrrr8r8r9rlr zOnReadPolicy.scatter_maxcOrrrrr8r8r9rmr zOnReadPolicy.scatter_updatecCt|||S)z0Create a saveable object for the given variable.)rrrVr7 primary_varrr8r8r9rrYzOnReadPolicy.get_saveablecCst|||jSr)rrrrVr7rr8r8r9rszOnReadPolicy.get_restore_opsr)r}rrrrr3rr\r|rrYr]r_rdrhrirjrkrlrmrrr8r8r8r9rs4      rc@seZdZdZddZddZddZdd Zd d Zd'ddZ d'ddZ d'ddZ d(ddZ d(ddZ d(ddZd(ddZd(ddZd(dd Zd(d!d"Zd#d$Zd%d&Zd S)) OnWritePolicyaaPolicy defined for `tf.VariableSynchronization.ON_WRITE` synchronization. This policy is created when the following `synchronization` and `aggregation` parameters are specified when creating a `tf.Variable` in `tf.distribute` scope and `synchronization` is equal to `tf.VariableSynchronization.ON_WRITE` or `tf.VariableSynchronization.AUTO`. cCrrr8rdr8r8r9rrzOnWritePolicy._is_mirroredcC |Srl)r<r3rr8r8r9r3rzOnWritePolicy.valuecCrrlrrr8r8r9rrzOnWritePolicy._as_graph_elementcCst|Srl)rrRr<rr8r8r9r\sz OnWritePolicy._get_cross_replicacKs8|jtjjkr|||fi|St|||fi|Srl)r&rr(r;r<rGrr8r8r9r|szOnWritePolicy._update_replicaFNTcCtj|||||dSrV)rr`rr8r8r9r_s zOnWritePolicy.assigncCrrV)rr^rr8r8r9r] zOnWritePolicy.assign_addcCrrV)rrZrr8r8r9rYrzOnWritePolicy.assign_subcCtj||||dSrb)rrdrVr7rfrXrr8r8r9rdzOnWritePolicy.scatter_subcCrrb)rrhrr8r8r9rhrzOnWritePolicy.scatter_addcCrrb)rrirr8r8r9rirzOnWritePolicy.scatter_mulcCrrb)rrjrr8r8r9rjrzOnWritePolicy.scatter_divcCD|jtjjkr|jtjjkrttjjd|jdtj ||||dS)Nrkrrc) rr'r(rNr;rbrrrrkrr8r8r9rkzOnWritePolicy.scatter_mincCr)Nrlrrc) rr'r(rNr;rbrrrrlrr8r8r9rlrzOnWritePolicy.scatter_maxcCr)Nrmrrc) rr'r(rNr;rbrrrrmrr8r8r9rmrzOnWritePolicy.scatter_updatecCr )z Saveable ops for AUTO variables.)rrr r8r8r9rrYzOnWritePolicy.get_saveablecCs t||Srl)rrr r8r8r9r rzOnWritePolicy.get_restore_opsrr)r}rrrrr3rr\r|r_r]rYrdrhrirjrkrlrmrrr8r8r8r9rs2         rcs<eZdZdZddZfddZfddZdd ZZS) PerWorkerResourceaA per-worker CapturableResource class for non-ParameterServer strategy. Resources that populate `host_to_resources` should be instances of classes subclassing CapturableResource, although currently it's only used and tested for StaticHashTable with TPUStrategy. cCs$tjddd||_||_dS)NrTPUDistributedLookupTabler)r'distribution_strategy_input_api_counterget_cell increase_by _strategy_host_to_resources)rVr2host_to_resourcesr8r8r9rXs  zPerWorkerResource.__init__cs&|dvr t||Stt||S)N)rX__getattribute__rrlocal_resource)rr rrrrrr8r9rsz"PerWorkerResource.__getattribute__cs*|dvr t|||Stt|||S)N)rr)setattrr rr __setattr__)rVrr3rr8r9r""szPerWorkerResource.__setattr__cCs<tt}tt|}|j||jtt|jS)z)Returns the resource on the local worker.)rrfrgget_host_for_devicergetnextiter)rVri host_devicer8r8r9r 'sz PerWorkerResource.local_resource) r}rrrrXrr"r rr8r8rr9rs   rr)Vrr$typingrrtensorflow.core.protobufrtensorflow.python.distributerrrrrrtensorflow.python.eagerr r tensorflow.python.frameworkr r rrrrrrrrrtensorflow.python.opsrrrrrr'rrtensorflow.python.saved_modelrtensorflow.python.trackablerru!tensorflow.python.training.savingrtensorflow.python.typesrrds_typesr rGrArLrrr-r#register_tensor_conversion_functionTypeSpecrregister_codecBuiltInTypeSpecCodec TypeSpecProtoPER_REPLICA_SPECrobjectr TraceTyperVariablerr SaveableObjectrrrrrrrrrrrrrrr8r8r8r9s                              7@   !P>    +e