The headline benchmark isn't the whole story
Hybrid architectures have been quietly climbing the LLM leaderboard, matching or beating pure transformers on standard evals. But aggregate perplexity scores hide something important: which specific tokens does a hybrid actually predict better?
AI2 just dropped a detailed token-level analysis comparing their 7B transformer (Olmo 3) against their hybrid (Olmo Hybrid) head-to-head. The results are specific, surprising, and useful if you're thinking about architecture tradeoffs for production.
The punchline: hybrids dominate on meaning-bearing content words—nouns, verbs, adjectives—but transformers still win when the answer is sitting right there to copy verbatim.
Attention vs. recurrence: the architectural split
A quick refresher on what makes a hybrid a hybrid.
Transformers use attention in every layer. Every token can look back at every prior token, weighting relevance on the fly. That's powerful for exact recall—grabbing a specific fact from 10,000 tokens back—but it's quadratic in cost. As context grows, so does compute.
Hybrids keep a few attention layers but swap most for recurrent layers. Recurrence processes tokens left-to-right with a fixed-size memory, folding each new token in as it goes. The cost per token stays flat no matter how long the input gets.
The tradeoff: recurrent memory is compressed and lossy. You can't reach back for an exact earlier token the way attention can. But recurrence excels at maintaining a running state—tracking entities, evolving sentiment, anything that changes sequentially as you read.
The experiment: same data, same tokens, different architectures
AI2's setup is clean. Olmo 3 and Olmo Hybrid were built to be as alike as possible outside their core architecture—same data, same tokenizer, same training recipe. Any prediction difference mostly reflects architecture, not confounds.
They fed both models the same passages: articles, Wikipedia, books, scientific papers, Python, HTML, LaTeX. For each token, they recorded the probability each model assigned to the ground-truth next token.
The key metric is the loss gap: the difference in loss between the two models, token by token. Positive gap means the hybrid predicted better. Negative gap means the transformer did.
Then they grouped tokens into categories—part of speech, syntactic role, whether the token repeats earlier text—and averaged the loss gap within each bucket. They also ran regressions to control for confounds like token rarity.
Where the hybrid wins: content words
The clearest pattern is in prose. The hybrid has lower loss than the transformer on most token types, but the advantage is biggest on content words—the nouns, verbs, adjectives, and adverbs that carry semantic meaning.
The loss gap on content words is around 0.04. On function words—"the," "of," "is"—it drops to around 0.02.
Some specific categories stand out:
- Adverbs and adjectives: hybrid advantage is especially pronounced
- Existentials ("there"): also show a large hybrid edge, even though they're grammatical rather than content-bearing
The interpretation: recurrent layers are well-suited to tracking what's happening in a passage—who's doing what, how entities evolve, what the sentence is about. Attention can aggregate information, but recurrence naturally represents sequential state.
Function words, by contrast, are often predictable from syntax alone. The model doesn't need deep context tracking to guess "the" after "of."
Where the transformer wins: exact copying
The hybrid's advantage shrinks—or disappears entirely—in two specific contexts.
Closing braces
First: closing brackets, braces, and parentheses. The loss gap nearly vanishes on }, ], ) tokens in code, markup, and structured text.
Why? Bracket matching is a known strength of attention. The model can look back directly to the opening brace and count intervening pairs. Recurrence has to track nesting depth in compressed state, which is harder.
Interestingly, the pattern holds for closing braces but not opening ones. Opening braces are predicted from context like any other token. Closing braces are lookups.
Repeated n-grams
Second, and more striking: when the next token simply repeats something verbatim from earlier in the passage.
AI2 measured this by looking for repeated n-grams—sequences where the completion token appeared, word-for-word, earlier in the same context. The longer the repeated run, the smaller the hybrid's advantage.
On long exact copies, the gap approaches zero.
Again, the reason is clear: attention can point back to the earlier occurrence and copy it directly. Recurrence has to reconstruct the token from compressed memory. When the answer is sitting right there to be looked up, attention wins.
Using filtered losses as an architecture eval
Here's where this gets practical for model developers.
AI2 took these insights and used filtered token losses—loss computed only on specific token types—as a fine-grained eval during pretraining.
They trained three 1B models: a transformer, a hybrid, and a pure recurrent model (no attention at all). Then they plotted loss curves on two subsets:
- Meaning-bearing tokens that aren't repeats: hybrid and pure recurrent overtake the transformer, with the hybrid best overall
- Repeated tokens: pure recurrent falls behind both hybrid and transformer (no attention to copy), while transformer and hybrid stay close
These differences show up early in training at 1B scale, in a way that wouldn't be visible in aggregate loss.
This is useful. If you're comparing architectures during ablations or scaling experiments, filtered losses let you see specific strengths and weaknesses before you've burned a full training budget.
What this means for hybrid modeling
Two takeaways from this work.
First: aggregate loss is too blunt to compare transformer vs. hybrid architectures. A single perplexity number hides architectural tradeoffs. Scoring loss on token subsets that test specific abilities—semantic tracking, exact recall, syntactic prediction—surfaces the real differences.
Second: the hybrid's advantage on open-class tokens (content words with flexible membership) likely relates to the state-tracking capabilities of recurrent layers. If your task is heavy on entity tracking, coreference, or evolving narrative state, recurrence might buy you something attention alone doesn't.
Of course, this is one model pair at 7B scale, trained on one dataset. The patterns might shift with scale, data distribution, or hybrid layer ratios. But the methodology is solid and the results are specific enough to be actionable.
Where to go from here
AI2 is taking these findings into their ongoing hybrid modeling work. The bet is that the best hybrid architectures will come from understanding, token by token, what each component does well—and composing them accordingly.
If you're working on architecture research, this kind of fine-grained analysis is worth stealing. Benchmarks are useful, but token-level loss gaps tell you why a model wins or loses, not just that it does.
The full tech report is up on arXiv. You can explore Olmo 3 and try Olmo Hybrid via their associated artifacts on Hugging Face.
And if you're building hybrids: test on repeated n-grams and bracket matching early. Those are your canaries for whether you've got enough attention in the stack.