Ring AllReduce
Reduce-scatter followed by allgather across N = 4 GPUs in a ring topology
GPU 0
GPU 1
GPU 2
GPU 3
← Prev
Next →
or use
←
→
arrow keys
Chunk 0
Chunk 1
Chunk 2
Chunk 3