o 2hW!@sdZddlZddlZddlmZddlmZddlm Z ddlm Z dddZ d d Z d d Z d dZddZddZddZddZddZddZddZdS)z5Utilities for distribution strategy with JAX backend.N) global_state)seed_generator) jax_utils) rng_utilscCs*|r|nd}tj|d}dd|DS)aReturn all the available devices based on the device type. Note that this should return the global devices in a distributed setting. Args: device_type: string of `"cpu"`, `"gpu"` or `"tpu"`. Defaults to `"gpu"` or `"tpu"` if available when device_type is not provided. Otherwise will return the `"cpu"` devices. Return: List of devices that are available for distribute computation. NbackendcSsg|] }|jd|jqS):)platformid).0devicer a/var/www/html/chatgem/venv/lib/python3.10/site-packages/keras/src/backend/jax/distribution_lib.py sz list_devices..)lowerjaxdevices) device_type jax_devicesr r r list_devices s rcCs t||S)aCreate a distributed variable for JAX. Since JAX doesn't have a variable class, this will just return a `jax.Array` with the corresponding layout/sharding specified. Note that this function should be used in eager context, not in jitted function. Args: value: the initial value of the variable. layout: `TensorLayout` for the created variable, or a JAX-supported layout instance (e.g. `jax.sharding.Sharding`). Returns: jax.Array which is the distributed variable. )distribute_tensor)valuelayoutr r rdistribute_variables rcCsddlm}t||r|j}trtj||St|tj rVt|tj j r3|j j |t |jdr3|St|drEt|dd}||krD|Snt|drVt|dd}||krV|St||S)aDistribute the tensor based on the layout. Note that this function can be used both in eager context, or within a jitted function. Args: tensor: `jax.Array` that need to be distributed. layout: `TensorLayout` for the created variable, or a JAX-supported layout instance (e.g. `jax.sharding.Sharding`). Returns: Distributed value. r TensorLayout)ndimrNformat)keras.src.distributionr isinstancebackend_layoutris_in_jax_tracing_scoperlaxwith_sharding_constraintArrayshardingShardingis_equivalent_tolenshapehasattrgetattr device_put)tensorrrcurrent_layoutr r rr2s,        rcCs(ddlm}t||r|j}t||S)aDistribute the input data with the corresponding layout. Note that the inputs here is a local worker batch. Within the local worker, the data need to be further partitioned to map to each of the devices. Args: inputs: `jax.Array` that is already sharded to a local process size. layout: `TensorLayout` for the distribution information, or a `jax.sharding.Sharding` instance. Returns: A global batch distributed according to `layout`. rr)rrrr r"make_array_from_process_local_data)per_process_batchrbatch_dim_namerr r rdistribute_data_input_s   r2cCst}|dur5td}td}tjjtg|tjj d}tj ddd|d| d}t |t d }|durX|d }|durZt d tj||j|jd dSdSdS) zInitializes the global random number generator across processes. This is required for consistent initialization in multi-host settings. Ncpu)dtypecSstj|dS)Nall)rr"psum)xr r rsz initialize_rng..r5) axis_namerrglobal_seed_generatorseed)r;namer)rget_random_seedrrlocal_device_countnumpyasarrayrmake_default_seeduint32pmapitemset_random_seedrget_global_attribute get_configset_global_attribute SeedGeneratorr<r) global_seed cpu_devicesnum_local_cpu_devices local_seedr:r;r r rinitialize_rngvsD     rNcCsr|r)d|vr)|d}|dur$|t|kr$td|dt|d||d}n|}tjj|||dtdS)N,zThe provided job_addresses z has z jobs, but num_processes is r)coordinator_address num_processes process_id)splitr( ValueErrorr distributed initializerN) job_addressesrQrRrPr r rrVs$    rVcCtS)zDReturn the number of processes for the current distribution setting.)r process_countr r r rrQrQcCrX)z;Return the current process ID for the distribution setting.)r process_indexr r r rrRrZrRcCsbt|tjr|S|d\}}tj|d}|D]}|j|kr)|jt|kr)|Sqtd|)NrrzDevice not found: ) rrDevicerSrr r intrT) device_namer device_idrr r r r_to_backend_devices  r`cCs<|jj}dd|jD}t||}tj||j S)zConvert the DeviceMesh to JAX backend specific Mesh. Args: device_mesh: DeviceMesh instance to convert. Returns: A `jax.sharding.Mesh` instance. cSsg|]}t|qSr )r`)r dr r rrsz$_to_backend_mesh..) rr)flattennparrayreshaperr%Mesh axis_names) device_meshr)rr r r_to_backend_meshs ricCs6|jdur tdtjj|j}|jj}tj||S)zConvert the TensorLayout to JAX backend specific Sharding. Args: tensor_layout: TensorLayout instance to convert. Returns: A `jax.sharding.NamedSharding` instance. NzDCannot create sharding when device mesh is not set for TensorLayout.)rhrTrr% PartitionSpecaxes backend_mesh NamedSharding) tensor_layoutpartition_specjax_meshr r r_to_backend_layouts rq)N)__doc__rr?rckeras.src.backend.commonrkeras.src.randomrkeras.src.utilsrrrrrr2rNrVrQrRr`rirqr r r rs$     -.