What is: Mesh-TensorFlow?
Source | Mesh-TensorFlow: Deep Learning for Supercomputers |
Year | 2000 |
Data Source | CC BY-SA - https://paperswithcode.com |
Mesh-TensorFlow is a language for specifying a general class of distributed tensor computations. Where data-parallelism can be viewed as splitting tensors and operations along the "batch" dimension, in Mesh-TensorFlow, the user can specify any tensor dimensions to be split across any dimensions of a multi-dimensional mesh of processors. A MeshTensorFlow graph compiles into a SPMD program consisting of parallel operations coupled with collective communication primitives such as Allreduce.