Expert Parallel Overlap

A serving system that overlaps expert routing, communication, and compute so sparse MoE execution does not stall on movement alone.

Opening summary

Sparse MoE can waste time moving token batches around even when the experts themselves are fast. Expert parallel overlap reduces that idle time by hiding communication behind useful compute wherever the runtime can.

At a glance

Released

June 2026

System type

Routing

Related modules

What It Is

This is a runtime scheduling pattern for sparse experts. It belongs to the serving stack that makes MoE practical, not to the logical definition of an MoE block by itself.

Where It Sits

It sits in the execution layer around expert dispatch, communication, and gather. The model asks for experts. The runtime decides how to keep those expert steps from idling each other out.

How It Works

Routing decisions, device-to-device movement, and expert compute are overlapped so the next useful action can begin before the previous communication phase has fully drained.
Expert Parallel Overlap System Flow
Request and weight flow
The goal is not a new expert function. It is better scheduling around expert work.

Practical Impact

This keeps sparse expert models closer to their theoretical compute savings. Without a system like this, MoE can lose too much time to transport overhead.
textstallfractionapproxfractextcommunicationlatency−textoverlappedworktexttotalsteptime\\text{stall fraction} \\approx \\frac{\\text{communication latency} - \\text{overlapped work}}{\\text{total step time}}

Tags

References

  1. DeepSeek-AI. "DeepSeek-V4 Technical Report." 2026.

On-Disk KV Cache

Previous Page

Routing

Next Page

On this page

What It IsWhere It SitsHow It WorksPractical ImpactRelated ToTagsReferences