Mixture of Wide Experts.

Biological neural systems are extremely sparse: only a small subset of neurons is active for any input, and specialization emerges across circuits. Modern neural networks are the opposite — dense matrix multiplications activate every neuron on every token because that is what current hardware computes efficiently.

Mixture-of-Experts (MoE) models introduced conditional computation: route each input to a few specialists instead of using everyone. But there is a structural problem that MoE does not solve. Every active expert writes a full $d_{\text{model}}$ vector back into the same residual stream. The outputs are summed into shared coordinates, so co-active experts must learn compatible output conventions. This is inter-expert interference — a coordination tax that grows with the number of active experts.

MoWE: read globally, write locally

Mixture of Wide Experts (MoWE) changes one thing about MoE: the write side. Each expert still reads the full residual stream. But instead of writing a full $d_{\text{model}}$ update, each expert writes only to a disjoint slice of the residual. Co-active experts land in different coordinates. Their outputs are concatenated, not summed.

In standard MoE, experts share the residual write space:

Expert A: $d \to H \to d$ , Expert B: $d \to H \to d$ — outputs summed in the same $d$ coordinates

In Full MoWE, experts own their output coordinates:

Expert A: $d \to H \to d/G$ writes slice A, Expert B: $d \to H \to d/G$ writes slice B — concatenated into disjoint coordinates

Dense downstream layers and the final head can still recombine the full residual, so MoWE is not a hard ensemble. It is a structured residual partition — sparse across groups, dense within each group, with no output-coordinate conflict between experts.

Why not just use MoE?

MoE handles conditional computation but not residual write conflict. If two active experts both write full $d_{\text{model}}$, then every output coordinate becomes a negotiation between experts. Full MoWE removes this inter-expert write conflict. The cost is that each expert gets only a $d/G$ slice of the residual output. That is a real architectural constraint — but if specialization is the goal, the constraint is exactly the point: an expert owns its coordinates and can build a coherent local representation there.

The bet: specialization should not be an accident of optimization. The architecture should make specialization the easy solution.

The difference between MoE and MoWE is best understood as two kinds of interference that dense, MoE, and MoWE architectures handle differently.

Two kinds of interference

Intra-expert interference — features inside one expert share that expert's hidden space. This is superposition within a single subnetwork.

Inter-expert interference — co-active experts write into the same residual coordinates. This forces experts to coordinate their output conventions even though they are conceptually independent specialists.

Architecture	Expert read	Expert write	Inter-expert interference
Dense FFN	all neurons	full residual	everything always mixed
MoE	full residual	full residual	experts sum into shared coordinates
Full MoWE	full residual	disjoint slice	zero by construction

MoE reduces intra-expert interference (each expert has its own hidden space) but retains inter-expert interference (the residual stream is a shared write target). Full MoWE eliminates inter-expert interference entirely — any remaining superposition is local within each expert's own slice.

The architectural tradeoff

The cost is explicit: each expert gets only $d/G$ output dimensions instead of $d$. This is a capacity partition. For $G = 8$ experts in $d = 768$, each expert writes $96$ dimensions. If the task requires experts to coordinate output conventions, this constraint hurts. If specialization is the goal — different experts owning different kinds of information — the constraint is a feature.

The question is empirical: does the routing learn to exploit this partition, and does the interference reduction show up in the representations? The next posts examine this across controlled, vision, and language-model settings.

Toy sparse-feature tasks

The cleanest test is synthetic sparse data with clustered features. Inputs contain groups of co-occurring features; the model must reconstruct them with limited active neurons. MoWE discovers the cluster structure directly from reconstruction pressure alone — no explicit cluster labels.

When within-cluster correlation is high ($p_{\text{within}} = 1.0$), routing matrices become nearly diagonal: one model group specializes to one data cluster.

Routing alignment at $N = 200$, $G = 20$. As within-cluster correlation increases from $0.6$ to $1.0$, routing matrices sharpen toward diagonal structure — each expert learns to own one data cluster.

At medium scale ($N = 200$), MoWE reaches target accuracy with fewer active neurons than MoE when the data has real co-occurrence structure for the router to exploit.

MoWE vs MoE accuracy. When cluster structure exists, MoWE exploits it to achieve the same accuracy with fewer active neurons.

CIFAR-10: domain and class specialization

On an augmented 4-domain CIFAR-10 setup — color, grayscale, rotated $12°$, Gaussian-blurred — routing became domain-sensitive without explicit domain labels. In a $G = 4$, $k = 1$ model, the rotated domain strongly avoided one group: Group 2 received only $3.9\%$ of rotated examples versus $25\%$ under uniform routing.

Per-domain routing fractions. The rotated domain develops a strong avoidance of Group 2 — routing discovers domain structure without supervision.

Class routing also became semantically structured. Transport classes (airplane, ship, truck) concentrated in Group 2, while animal classes (cat, dog) concentrated in Group 0.

Per-class routing fractions. Semantic categories emerge in the routing pattern: vehicles cluster in Group 2, animals in Group 0.

Robustness at extreme expert counts

In large-$G$ scaling with fixed active neurons, MoWE was much more robust than MoE under extreme fragmentation. MoWE held steady at $\sim47$–$48\%$ accuracy from $G = 512$ to $G = 4096$, while MoE collapsed from $35.6\%$ to $29.1\%$ over the same range.

Per-class expert specialization at $G = 4096$, $k = 2048$. Each class develops strongly preferential routing to specific experts, well above the uniform baseline.

The caveat is important: dense still beats shallow MoWE on absolute CIFAR accuracy. The CIFAR result supports specialization and robustness, not an unconditional accuracy win.

The Full MoWE distinction matters most in deep networks, where routed residual blocks feed later layers and interference compounds.

Deep synthetic residual autoencoder

In a deep synthetic setting ($d = 64$, $G = 8$, $k = 2$, depth $= 3$), Full MoWE removes cross-expert residual interference exactly — measured inter-expert interference is $0.000$ by construction. MoE and half-MoWE retain $3$–$7\%$ cross-expert interference.

In the hard clustered regime ($p_{\text{within}} = 1.0$, $S = 0.7$), Full MoWE reaches $0.508$ exact-match accuracy with $66\text{K}$ params versus MoE at $0.245$ with $110\text{K}$ params.

Full MoWE (green) achieves zero cross-expert interference in all conditions while MoE (red) and half-MoWE (orange) retain measurable interference. In the hard-clustered regime, Full MoWE doubles MoE's accuracy with fewer parameters.

CIFAR residual-block experiment

On a real-data CIFAR-10 residual model ($d = 256$, $G = 8$, $k = 2$, depth $= 3$), all variants reach roughly the same test accuracy — but the interference decomposition tells a very different story:

Condition	Params	Test accuracy	Cross-expert	Intra-expert
Dense (FLOP-matched)	$757\text{K}$	$89.27\%$	—	$0.0131$
Dense (memory-matched)	$1.35\text{M}$	$89.22\%$	—	$0.0099$
MoE	$1.36\text{M}$	$89.22\%$	$0.0209$	$0.0009$
Full MoWE	$1.01\text{M}$	$89.36\%$	$0.0000$	$0.0112$

MoE drives intra-expert interference very low ($0.0009$) but still has cross-expert residual interference ($0.0209$). Full MoWE has zero cross-expert interference by construction, with moderate intra-expert interference. Total measured interference is roughly halved.

All architectures match on accuracy, but Full MoWE achieves exactly zero cross-expert interference while using fewer parameters than MoE. The interference has moved: from between experts (eliminated) to within experts (local and manageable).

Language models are the real test. They are deep, residual-stream dominated, and hard enough that architectural constraints show up in loss.

Native interpretability rivals post-hoc SAEs

Using Tier-3 GPT scoring over $200$ residual-stream units (fixed-RoPE runs only):

Representation	Mean correlation	Median	$> 0.5$ fraction
FullMoWE native residual	$0.583$	$0.671$	$64\%$
Dense native residual	$0.322$	$0.325$	$32\%$
Dense + SAE	$0.542$	$0.572$	$58\%$

FullMoWE native residual directions are far more interpretable than dense native directions, and are in the same range as a post-hoc SAE trained on dense activations — without any SAE training step.

MoWE residual dimensions stay interpretable across all layers, while dense residual dimensions collapse in mid-layers where superposition is strongest. MoWE produces SAE-like native features without a separate decomposition step.

The CE gap: not closed, but closing

The loss story is not finished. The dense fixed-RoPE baseline reaches CE $2.944$ at $5\text{B}$ tokens. The initial FullMoWE run reached CE $3.166$ — a significant interpretability win but a meaningful CE tax. A new stabilized FullMoWE run is tracking substantially better but still trails dense at $768d$ scale.

Dense vs FullMoWE cross-entropy over training. The stabilized run (green) is a large improvement over the initial FullMoWE trajectory (blue dashed), but has not yet reached dense parity at $768d$.

Router stabilization

A $512d$ router-stability sweep found a promising recipe: loss-free balancing, router z-loss, and tiny auxiliary load-balancing. This combination reached dense parity at $1\text{B}$ tokens while keeping routing healthier than the no-auxiliary variant.

CE loss curves for the $512d$ stability sweep. The bias+zloss+tinyaux recipe (green) reaches dense parity ($4.178$ vs $4.187$), validating the stabilization direction.

Current status

The mechanism is supported: MoWE routing discovers structure, Full MoWE removes cross-expert interference by construction, and LM native features are SAE-like. The main unresolved issue is optimization — closing the CE gap to dense at the full $768d$ / $5\text{B}$ token scale. The current paper shape is honest:

MoWE trades extra parameters and routing constraints for native specialization and interpretability. It structurally removes expert-output interference, learns meaningful routing, and produces SAE-like native language-model features. The remaining research problem is closing the CE gap to dense at scale.

That is a defensible progress story and a clear technical agenda.