SoCC ’26 · Singapore
FluxReduce: Straggler-Resilient In-Network Aggregation for Distributed Machine Learning
George Karlos, Henri Bal, Lin Wang
Communication is a large and growing share of training time
¹ A. Sapio et al., “Scaling Distributed Machine Learning with In-Network Aggregation”, USENIX NSDI 2021
² G. He et al., “Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUs”, NeurIPS 2025
³ J. Yin et al., “Comparative Study of Large Language Model Architectures on Frontier”, IEEE IPDPS 2024
⁴ M. Liang et al., “Lumos: Efficient Performance Modeling and Estimation for Large-scale LLM Training”, MLSys 2025
⁵ NVIDIA H100 and Blackwell architecture documents; fastest NVIDIA NIC announced with each GPU generation (ConnectX-5/6/7/8 specifications)
2
Data-parallel training
Each worker trains a copy of the model on its own slice of the batch. Every step ends with an AllReduce that averages gradients.
Share of a training step spent on communication
0%
25%
50%
75%
100%
3–79%
Classic DNNs
(ResNet, BERT…)¹
30–50%
Production LLMs
pre-training²
≈40%
6.7B GPT-NeoX-style
data-parallel³
30–41%
GPT-3
15B to 175B⁴
Compute grows faster than the network⁵
■ FP16 TFLOP/s ■ Gb/s
1×
1×
V100
100G
2.5×
2×
A100
200G
8×
4×
H100
400G
18×
8×
B200
800G
Communication is a third to a half of every step today, and grows with every new GPU generation
In-network aggregation
Streaming aggregation transport
¹ A. Sapio et al., “Scaling Distributed Machine Learning with In-Network Aggregation”, USENIX NSDI 2021
3
Every gradient already crosses the network, so let the switch add them up and forward only the results
Training up to 2.3× faster compared to NCCL over RDMA, and 5.5× over TCP¹
Commercially available: NVIDIA SHARP · Broadcom Tomahawk Ultra · Ultra Ethernet specification
- Less traffic. Each worker sends its gradient once and receives the result once.
- No work on the hosts. The switch sums at line rate.
- Scales with the switch. Adding workers adds no latency, where a ring gets slower with every worker.
Stragglers
Workers can experience unpredictable slowdowns. A straggler is one significantly slower than the rest
Load imbalance, contention, congestion, OS jitter, a throttled or faulty GPU, ...
J. Lin et al., “Understanding Stragglers in Large Model Training Using What-if Analysis”, USENIX OSDI 2025
4
<5% of workers
straggle at any one time
42% of jobs
are affected, in production
1.8× slowdown
in step time, at worst
Stragglers
Workers can experience unpredictable slowdowns. A straggler is one significantly slower than the rest
… with in-network aggregation
5
Our approach: Reactive fastest-K in the dataplane
1
Detect inside the switch
The switch is the first to notice a delay
2
Keep the stream flowing
Slots complete partially w/o waiting
✓ Convergence under gradient drops¹ ²
3
Let the straggler back in
Resync and reset to fastest-N
Challenge
High performance switch ASICs restrict what protocols can do
Feed-forward RMT pipelines³. Limited operations and memory accesses.
No timers, no loops, no threads
¹ S. Dutta et al., “Slow and Stale Gradients Can Win the Race”, AISTATS 2018
² C. Yu et al., “Distributed Learning over Unreliable Networks”, ICML 2019
³ P. Bosshart et al., “Forwarding Metamorphosis: Fast Programmable Match-Action Processing in Hardware for SDN”, ACM SIGCOMM 2013
6
Detecting the straggler with retransmissions
To time out a slot, the switch needs a clock. It has none
No timers, no threads, nothing to raise an interrupt. Logic runs only while a packet passes
We do not need one
- Healthy workers retransmit every RTO
- Each duplicate is a tick
- RTO: a round trip plus margins
- STO: how far apart workers arrive
- Many RTOs fit in one STO
- Past the STO, ≥ K in: partial result
7
Making progress without the straggler
We cannot wait a full STO on every slot
A straggle event spans hundreds of slots. Each one would stall again
- Partial results tell healthy workers that a straggler is present
- They tag their packets fastest-K, and a tagged slot completes with K, no timeout
- One stall per straggle, about an STO. Then full rate at K of N
8
What happens to the straggler
The straggler is behind
It must catch up on its own, without putting a stale gradient into the stream
- Usable results still wait in its queue if it was not gone long
- A newer result in the queue means it was gone too long
- SYN handshake. Switch responds only when the slot completes. Straggler passively empties its queue during
- Reset to fastest-N when it rejoins
9
FluxReduce: inside the switch
10
In the paper: the full protocol, and how it fits the constraints of a programmable switch pipeline
Evaluation
11
Implementation
About 2000 LoC of P4 and C++ library on DPDK of about 7000 LoC. Custom PyTorch DDP backend of about 2,000 LoC.
Testbed
Six GPU servers on 100 Gbit/s Ethernet behind an Intel Tofino2 switch.
Workloads
Training ResNet-50 and GPT-2, fine-tuning RoBERTa and Qwen-2.5
Injected stragglers
Following prior work¹ ², a random victim sleeps for 0.5 to 2 steps with probability 0.15, at one or three points per step
GPU servers
Intel Tofino 2 switch
GPU servers
¹ M. Yang et al., “Using Trio: Juniper Networks’ programmable chipset for emerging in-network applications”, SIGCOMM 2022 · ² A. Harlap et al., “Addressing the straggler problem for iterative convergent parallel ML”, SoCC 2016
Training with stragglers
1.4× faster to hit metric
accuracy, perplexity or F1
1.45× faster for
the same steps
Same model quality
Final metrics within noise of the baseline
12
Three straggle points per step. Arrows mark the speedup to the target and to the same step count
Reactive vs preemptive fastest-K
Preemptive misses the target
on ResNet-50 and GPT-2, within the time budget
Reactive gives better accuracy
Far fewer dropped gradients
Preemptive drops 1/6 at every step. FluxReduce drops under 0.5%, about 4% on Qwen
13
Step time vs straggle severity
SO-INA degrades linearly
up to 1.66× step time
FluxReduce stays constant, within 1.02–1.18×
of the straggler-free baseline, across the whole range
No cost without stragglers
At p = 0, step times are within 0.995–1.0005× of the baseline
14
AllReduce performance
Higher throughput
than NCCL RDMA, ring or tree
Lower latency
than NCCL RDMA, ring or tree
Better scaling
Goodput keeps growing with tensor size where NCCL RDMA plateaus
Straggler resilience does not come at the cost of INA-grade performance
15
All NCCL runs use GPUDirect and 8 channels
Conclusion
16
In-network aggregation improves AllReduce performance, but suffers from stragglers
Reactive fastest-K aggregation in the dataplane retains most of the speedup under stragglers
Detect stragglers in the switch, using retransmissions of healthy workers as the clock
Up to ~1.4× faster training under stragglers
No cost in the absence of stragglers
github.com/net-compute/flux
g.karlos@{vu.nl,upb.de}
Thank you!
This work was supported by