Ai2 releases Olmo-core 3, open training code for trillion-parameter MoE models
Ai2 has released Olmo-core 3, an upgrade to its framework for building large language models with a redesigned mixture-of-experts (MoE) training system. Ai2 says it is designed to scale MoE training into the trillion-parameter range. In one benchmark, Ai2 grew the expert pool from 8 to 128 and total parameters from 4.6B to 47B, while training throughput fell by less than 5%. In a preliminary test on eight NVIDIA B300 GPUs, Ai2 reports 2.7 times the throughput of its earlier implementation, and says it benchmarked a 1.2-trillion-parameter model across 512 GPUs. The code is open on GitHub, and Ai2 says its next-generation Olmo will use an MoE architecture.