ACM Symposium on Cloud Computing

SoCC ’26 · Singapore

FluxReduce: Straggler-Resilient In-Network Aggregation for Distributed Machine Learning

George Karlos, Henri Bal, Lin Wang

Vrije Universiteit Amsterdam Paderborn University

Communication is a large and growing share of training time

¹ A. Sapio et al., “Scaling Distributed Machine Learning with In-Network Aggregation”, USENIX NSDI 2021
² G. He et al., “Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUs”, NeurIPS 2025
³ J. Yin et al., “Comparative Study of Large Language Model Architectures on Frontier”, IEEE IPDPS 2024
⁴ M. Liang et al., “Lumos: Efficient Performance Modeling and Estimation for Large-scale LLM Training”, MLSys 2025
⁵ NVIDIA H100 and Blackwell architecture documents; fastest NVIDIA NIC announced with each GPU generation (ConnectX-5/6/7/8 specifications)

2

Data-parallel training

Each worker trains a copy of the model on its own slice of the batch. Every step ends with an AllReduce that averages gradients.

Share of a training step spent on communication

0%

25%

50%

75%

100%

3–79%

Classic DNNs
(ResNet, BERT…)¹

30–50%

Production LLMs
pre-training²

≈40%

6.7B GPT-NeoX-style
data-parallel³

30–41%

GPT-3
15B to 175B⁴

Compute grows faster than the network⁵

■ FP16 TFLOP/s    ■ Gb/s

1×

1×

V100
100G

2.5×

2×

A100
200G

8×

4×

H100
400G

18×

8×

B200
800G

Communication is a third to a half of every step today, and grows with every new GPU generation

In-network aggregation

Streaming aggregation transport

¹ A. Sapio et al., “Scaling Distributed Machine Learning with In-Network Aggregation”, USENIX NSDI 2021

3

Every gradient already crosses the network, so let the switch add them up and forward only the results

Training up to 2.3× faster compared to NCCL over RDMA, and 5.5× over TCP¹

Commercially available: NVIDIA SHARP · Broadcom Tomahawk Ultra · Ultra Ethernet specification

  • Less traffic. Each worker sends its gradient once and receives the result once.
  • No work on the hosts. The switch sums at line rate.
  • Scales with the switch. Adding workers adds no latency, where a ring gets slower with every worker.

Stragglers

Workers can experience unpredictable slowdowns. A straggler is one significantly slower than the rest

Load imbalance, contention, congestion, OS jitter, a throttled or faulty GPU, ...

time → overhead = W3's delay W3 forwardbackwardstraggling… W1 · W2 forwardbackwardGPU idle gradient exchange update

J. Lin et al., “Understanding Stragglers in Large Model Training Using What-if Analysis”, USENIX OSDI 2025

4

<5% of workers

straggle at any one time

42% of jobs

are affected, in production

1.8× slowdown

in step time, at worst

Stragglers

Workers can experience unpredictable slowdowns. A straggler is one significantly slower than the rest

time → overhead = W3's delay W3 forwardbackwardstraggling… W1 · W2 forwardbackwardGPU idle gradient exchange update

… with in-network aggregation

5

Our approach: Reactive fastest-K in the dataplane

1

Detect inside the switch

The switch is the first to notice a delay

2

Keep the stream flowing

Slots complete partially w/o waiting

✓ Convergence under gradient drops¹ ²

3

Let the straggler back in

Resync and reset to fastest-N

W3 is slow In-network aggregation fullstallfull FluxReduce fullstallK of Nfull one timeout time →

Challenge

parser match-action match-action match-action deparser

High performance switch ASICs restrict what protocols can do

Feed-forward RMT pipelines³. Limited operations and memory accesses. 
No timers, no loops, no threads

¹ S. Dutta et al., “Slow and Stale Gradients Can Win the Race”, AISTATS 2018 
² C. Yu et al., “Distributed Learning over Unreliable Networks”, ICML 2019
³ P. Bosshart et al., “Forwarding Metamorphosis: Fast Programmable Match-Action Processing in Hardware for SDN”, ACM SIGCOMM 2013

6

Detecting the straggler with retransmissions

To time out a slot, the switch needs a clock. It has none

No timers, no threads, nothing to raise an interrupt. Logic runs only while a packet passes

We do not need one

  • Healthy workers retransmit every RTO
  • Each duplicate is a tick
  • RTO: a round trip plus margins
  • STO: how far apart workers arrive
  • Many RTOs fit in one STO
  • Past the STO, ≥ K in: partial result

7

straggler timeout (STO): tens of ms one RTO, about 1 ms first tick past the STO retransmission timeout (RTO): about 1 ms W1 W2 W3 RTO 2 of 3 waiting

Making progress without the straggler

We cannot wait a full STO on every slot

A straggle event spans hundreds of slots. Each one would stall again

  • Partial results tell healthy workers that a straggler is present
  • They tag their packets fastest-K, and a tagged slot completes with K, no timeout
  • One stall per straggle, about an STO. Then full rate at K of N
after the partial result: every packet says K W1KKK W2KKK W3straggling, sends nothing target K = 2 2 in: complete

8

What happens to the straggler

The straggler is behind

It must catch up on its own, without putting a stale gradient into the stream

W3 is back, still at 5 W3 5 5 5 SYN 8K here is the result for 8 9N seq 8 target K = 2
  • Usable results still wait in its queue if it was not gone long
  • A newer result in the queue means it was gone too long
  • SYN handshake. Switch responds only when the slot completes. Straggler passively empties its queue during
  • Reset to fastest-N when it rejoins

9

FluxReduce: inside the switch

10

In the paper: the full protocol, and how it fits the constraints of a programmable switch pipeline

Evaluation

11

Implementation

About 2000 LoC of P4 and C++ library on DPDK of about 7000 LoC. Custom PyTorch DDP backend of about 2,000 LoC.

Testbed

Six GPU servers on 100 Gbit/s Ethernet behind an Intel Tofino2 switch. 

Workloads

Training ResNet-50 and GPT-2, fine-tuning RoBERTa and Qwen-2.5

Injected stragglers

Following prior work¹ ², a random victim sleeps for 0.5 to 2 steps with probability 0.15, at one or three points per step

The testbed rack: six Dell GPU servers, GNODE 1 to 6, with the Netberg Tofino2 switch and a second switch in the middle

GPU servers

Intel Tofino 2 switch

GPU servers

¹ M. Yang et al., “Using Trio: Juniper Networks’ programmable chipset for emerging in-network applications”, SIGCOMM 2022 · ² A. Harlap et al., “Addressing the straggler problem for iterative convergent parallel ML”, SoCC 2016

Training with stragglers

ResNet-50 time-to-accuracy, SO-INA versus FluxReduce, 3 straggle points GPT-2-124M time-to-perplexity RoBERTa time-to-F1 Qwen-2.5-0.5B time-to-perplexity

1.4× faster to hit metric

accuracy, perplexity or F1

1.45× faster for
the same steps

Same model quality

Final metrics within noise of the baseline

12

Three straggle points per step. Arrows mark the speedup to the target and to the same step count

Reactive vs preemptive fastest-K

ResNet-50, preemptive fastest-K versus FluxReduce, no injected stragglers GPT-2-124M, preemptive fastest-K versus FluxReduce RoBERTa, preemptive fastest-K versus FluxReduce Qwen-2.5-0.5B, preemptive fastest-K versus FluxReduce

Preemptive misses the target

on ResNet-50 and GPT-2, within the time budget

Reactive gives better accuracy

Far fewer dropped gradients

Preemptive drops 1/6 at every step. FluxReduce drops under 0.5%, about 4% on Qwen

13

Step time vs straggle severity

ResNet-50 step time versus straggle probability GPT-2-124M step time versus straggle probability RoBERTa step time versus straggle probability Qwen-2.5-0.5B step time versus straggle probability

SO-INA degrades linearly

up to 1.66× step time

FluxReduce stays constant, within 1.02–1.18×

of the straggler-free baseline, across the whole range

No cost without stragglers

At p = 0, step times are within 0.995–1.0005× of the baseline

14

AllReduce performance

Goodput (Gbit/s), 100 MB 65.5256/8 61.3256/6 47.664/8 36.864/6 47.7ring 17.5tree 1.37× FluxReduceNCCL RDMA Latency (ms), 100 MB 14.9256/8 17.9256/6 20.664/8 25.864/6 16.6ring 44.7tree 1.11× FluxReduceNCCL RDMA Goodput (Gbit/s), by tensor size 0 20 40 60 0.1110100500tensor size (MB) NCCL RDMA FluxReduce Latency (ms), by tensor size 0 30 60 90 0.1110100500tensor size (MB) NCCL RDMA FluxReduce

Higher throughput

than NCCL RDMA, ring or tree

Lower latency

than NCCL RDMA, ring or tree

Better scaling

Goodput keeps growing with tensor size where NCCL RDMA plateaus

Straggler resilience does not come at the cost of INA-grade performance

15

All NCCL runs use GPUDirect and 8 channels

Conclusion

16

In-network aggregation improves AllReduce performance, but suffers from stragglers

Reactive fastest-K aggregation in the dataplane retains most of the speedup under stragglers

Detect stragglers in the switch, using retransmissions of healthy workers as the clock

Up to ~1.4× faster training under stragglers

No cost in the absence of stragglers

github.com/net-compute/flux

g.karlos@{vu.nl,upb.de}

Thank you!

This work was supported by

NWO, Dutch Research Council DFG, Deutsche Forschungsgemeinschaft Google Intel