Sparse MoE Expert-Credit Loss Reweighting Diagram

Overview

A three-panel diagram of robust training for a sparse mixture-of-experts language model. It is an editable PowerPoint diagram adapted from the original paper figure under CC BY 4.0.

What the diagram shows

(a) Sparse MoE Transformer: mixed-domain input tokens x_j → embedding → multi-head self-attention → Add & Norm → MoE layer l → Add & Norm (×L) → LM head → per-token losses ℓ_j. (b) Expert credit inside layer l: a token x, router φ(x) → Top-k with gate bars p_l,i, a shared expert FFN_s plus routed experts FFN₁ … FFN₈, the activation-weighted credit c̃_l,i = sg(p_l,i · ‖h_l,i‖₂) and the expert-attributed loss ℓ_l,i = (1/N) Σ_j c̃_l,i(x_j) ℓ_j 1[i ∈ 𝒯_j,l]. (c) Layer-wise DRO reweighting: per-expert loss bars, an EMA tracker ℓ̂_l,i ← βℓ̂_l,i + (1 − β) sg(ℓ_l,i), stacked per-layer weight updates μ_l ← softmax(μ_l + ηℓ̂_l) against the uniform 1/E line, and the objective ℒ_DRO = Σ_l Σ_i sg(μ_l,i) ℓ_l,i, whose gradient flows back to high-loss experts and hard tokens. A legend separates forward, detached-statistics and gradient arrows. Every box, arrow, label and math term is a native shape you can recolor, resize and relabel for your own slides, posters or papers.

File format

The download contains .pptx (1 slide, 13.28 × 7.42 in; opens in PowerPoint, Keynote, WPS and LibreOffice Impress; Google Slides works too, but complex freeforms may shift slightly), .svg and .drawio. Includes preview.png (full resolution) and a 1200 px web preview.

Shapes and editing

The file has 171 native editable shapes: 77 auto-shapes, 47 text boxes, 11 freeforms, 36 lines/connectors. No embedded images. Everything is vector. You can edit all text and math (the stacked 1/N and 1/E fractions, Σ limits, ℓ_l,i, c̃_l,i, ‖h_l,i‖₂, p_l,i, ℓ̂_l,i with native vector hats, μ_l, ℒ_DRO, ∇_θ), every loss, gate and weight bar, the token squares, expert columns, stacked per-layer cards, the dashed zoom-in panel, ⊕ node, self-loop β arrow, the three arrow styles and the legend.

Embedded images and fonts

This file has no embedded images; everything is vector. Body text uses Georgia and formulas use Times New Roman; both are preinstalled on Windows and macOS (LibreOffice substitutes the metric-compatible Gelasio and Liberation Serif). The calligraphic 𝒯 and ℒ use Cambria Math, which ships with Microsoft Office; other apps substitute a math font. The hats on ℓ̂ are small native line shapes placed over the letter, so nudge them if you retype the formula. Sub- and superscripts are real formatted text runs, not equation objects. The SVG keeps text as text; the draw.io file is a shape-by-shape conversion and may need small nudges. This is a faithful redraw for reuse, not the authors' original source file.