eploying heavy language models directly onto local mobile silicon has long forced engineers into an ugly compromise: butcher the model with static pruning and stomach the resulting drop in reasoning quality, or burn millions on extended fine-tuning cycles just to patch baseline output back together. A joint research initiative published in Transactions on Machine Learning Research by researchers from Samsung Research America, Amazon AGI, Florida State University, and partner institutions introduces ToMoE, a dynamic framework that sidesteps weight retraining altogether.
The team proved that functional sparse expert networks already sit dormant inside standard dense weights rather than needing to be forged from scratch through resource-draining continued pretraining.
"ToMoE transforms dense LLMs into Mixture-of-Experts (MoE) models by uncovering experts inherently present within dense models, without requiring any weight updates."
By uniting dynamic structural pruning and router calibration into a single stage, ToMoE forces monolithic checkpoints to execute like conditional Mixture-of-Experts systems while preserving their frozen base parameters.
Dynamic Structural Pruning Across MHA and MLP Layers
Standard sparse MoE designs achieve efficient execution by activating only a fraction of their parameter mass per token. ToMoE ties this conditional behavior straight to input-dependent pruning. Inside Multi-Head self-Attention (MHA) blocks, the pipeline runs top-K routing paired with static pruning to trim computational waste. Across feed-forward MLP blocks, ToMoE slices dense layers into dedicated sub-networks governed by dynamic top-1 routing.
Across benchmark evaluations on Phi-2, LLaMA-2, LLaMA-3, and Qwen-2.5, ToMoE systematically outpaced conventional structural pruning and dense-to-MoE conversion baselines. Activating only token-critical parameters slashes active memory traffic and NPU compute cycles per pass, allowing mobile hardware to run previously unmanageable dense foundations without choking thermal limits.
What this means:
For engineering leads and platform architects evaluating edge deployments, ToMoE offers an immediate pipeline to test sparse on-device execution using off-the-shelf checkpoints without managing expensive retraining workflows. However, theoretical FLOP reductions do not automatically translate into real-world battery savings. The critical hurdle remains commercial hardware execution: dynamic top-1 routing and sparse attention kernels must still overcome memory bandwidth bottlenecks and rigid fixed-function NPU accelerators on consumer chipsets before this translates into true sub-millisecond latency on everyday devices.