1 hour ago
25
Researchers from Microsoft and Cornell University have published a new training technique that makes large language models measurably better at predicting the next token, all while keeping inference speed essentially unchanged. The method, called Free Pause Tokens, represents one of those rare improvements in AI research where you get something for almost nothing.
The paper, titled “Free Pause Tokens” (also referred to as “Almost Free State Prediction Separation”), was submitted to arXiv on September 3. It describes a parallel prediction stream that shares weights with a model’s main state stream, adding extra computation near the end of training without the usual penalties that come with architectural changes.
What Free Pause Tokens actually do
Previous research from 2023 and 2024 tried to address transformer models’ fixed computation constraints by inserting explicit “pause tokens” into the input sequence. These gave the model extra steps to think, but they came with a cost: longer context lengths, bigger key-value caches, and slower inference.
The Free Pause Tokens approach sidesteps these problems entirely. Instead of adding tokens to the input sequence, it creates a lightweight parallel prediction stream during training. This stream shares weights with the main model through shared gated feedforward networks (FFNs), so the additional parameters and compute are minimal. During inference, the prediction stream can be essentially ignored, meaning the model runs at roughly the same speed as a standard transformer.
The performance gain on a 1 billion parameter model comes out to approximately 2 to 3 centinats improvement in next-token prediction. Centinats are a unit of information that measure how well a model predicts the next token.
The engineering details that make it work
Three specific technical choices keep the training overhead low. First, the team uses a technique called two-pass splitting that maintains compatibility with FlashAttention, the widely adopted memory-efficient attention algorithm. Second, they employ a prediction window of w=0, which means the prediction stream doesn’t need to look ahead in ways that would complicate the architecture. Third, the shared gated FFNs allow the prediction stream to piggyback on the main model’s existing parameters rather than requiring its own dedicated layers.
The result is training that takes only about 1.09 to 1.14 times as long as a standard optimized pipeline. The researchers report improvements across isoflop, isoparameter, and isotoken metrics compared to traditional transformers, meaning Free Pause Tokens produce better models whether you hold compute budget, parameter count, or training data size constant.
Why this matters beyond the lab
The practical appeal of Free Pause Tokens lies in what it doesn’t require. No additional context length means existing deployment infrastructure doesn’t need to be redesigned. No extra key-value cache overhead means memory-constrained serving environments can adopt the technique without hardware upgrades. No additional decode steps means latency-sensitive applications see no degradation in response time.
Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our Editorial Policy.
English (US) ·