The quantization tax just got a lot cheaper
Liquid AI dropped Q4_0 GGUFs for their LFM2.5 family that sidestep the usual quality-versus-size tradeoff. Instead of post-training quantization (PTQ)—where you take a trained model and crunch it down afterward—they used Quantization-Aware Distillation (QAD): distill a high-precision teacher into a student that's quantized from the start.
The result? The Q4_0 checkpoints recover 97% of the accuracy lost to quantization compared to the BF16 baseline. That's not a typo. They're shipping 4-bit models that perform almost as well as the full-precision originals, with the same memory footprint and speed you'd expect from native Q4_0.
This matters because edge deployment has always been a game of painful compromises. You want small, you want fast, you want accurate—pick two. QAD is the first approach I've seen that credibly delivers all three by baking quantization into the training process itself.
What they shipped
Four models got the QAD treatment:
LFM2.5-230MLFM2.5-350MLFM2.5-1.2B-InstructLFM2.5-2.6B
All four are available as Q4_0 GGUFs on Hugging Face today. They're drop-in compatible with llama.cpp or any runtime that speaks GGUF.
The baseline comparison is straightforward: Liquid AI benchmarked the QAD checkpoints against their own PTQ Q4_0 GGUFs and the BF16 reference. The eval suite spans reasoning (GPQA Diamond, MMLU-Pro), instruction-following (IFEval, IFBench, Multi-IF), tool use (BFCLv4), and math (GSM8K for the smaller models, AIME25 for the larger ones).
Across all four models, the QAD Q4_0 checkpoints retain 97.1%, 96.5%, 97.4%, and 96.6% of their respective BF16 performance. That's a substantial recovery compared to standard PTQ, which typically bleeds 5–15% of performance at Q4_0.
Speed: QAD Q4_0 beats higher-bit PTQ
Here's where it gets interesting. The QAD Q4_0 checkpoints don't just match PTQ Q4_0 quality—they beat higher quantization levels while running faster.
Liquid AI profiled decode throughput across four real-world targets:
- MacBook Pro (GPU)
- NucBox EVO-X2 (GPU)
- Samsung Galaxy S26 Ultra (Arm CPU)
- Raspberry Pi 5 (Arm CPU)
The 230M and 350M QAD Q4_0 models match Q5_K_M quality within eval variance while delivering 4–33% higher decode throughput. The 1.2B and 2.6B QAD Q4_0 models match Q4_K_M quality at 3–14% higher throughput.
Think about that trade: you get the quality of a 5-bit or mixed-precision quantization, but you pay the memory and latency cost of a pure 4-bit model. For edge deployment, that's the difference between "barely fits" and "runs comfortably with headroom."
They also benchmarked against Unsloth's UD-Q4_K_XL PTQ checkpoint (where available). The QAD models matched it. That's notable because Unsloth's quantization is widely regarded as best-in-class for PTQ.
Why QAD works: training the student in its final form
Quantization-aware training isn't new—Google's been doing it for mobile vision models since 2018. What's new here is combining it with distillation.
Standard PTQ is a lossy compression step applied after training. You're rounding weights and activations to lower precision without giving the model a chance to adapt. The model was never trained to operate in that regime, so it degrades.
QAD flips the script: the student model trains in its quantized form from the start. It learns to represent the teacher's behavior within the constraints of 4-bit precision. The quantization is part of the objective function, not an afterthought.
This is conceptually similar to how you'd train a model with dropout or weight decay—constrain the hypothesis space during training so the final artifact is robust to the constraint. In this case, the constraint is "4 bits per weight."
The teacher-student setup also lets you recover expressiveness that pure quantization would lose. The teacher (presumably BF16 or FP32) provides a rich training signal. The student learns to approximate that signal within its quantized budget. You're not just rounding—you're learning a compressed representation that preserves the teacher's decision boundaries.
Edge deployment is the real prize
This isn't an academic exercise. Liquid AI is explicitly targeting edge hardware: phones, embedded devices, local-first apps.
The Raspberry Pi 5 numbers are telling. That's a $60 board with 8GB RAM. Running a 1.2B model at 4 bits with competitive quality opens up a huge design space for on-device agents, local assistants, and privacy-preserving deployments.
The Samsung Galaxy benchmark is equally important. On-device inference on consumer phones is still mostly a novelty—Gemini Nano, Apple Intelligence, some limited Whisper deployments. But if you can fit a capable 1B+ model in under 1GB with decent throughput, you unlock persistent agents, offline-first tools, and workflows that don't round-trip to the cloud.
The QAD approach also matters for cost at scale. Cloud providers charge by compute and memory. If you can serve the same quality at half the memory footprint and higher throughput, your cost per token drops proportionally. For high-traffic applications, that's the difference between profitable and underwater.
Open questions and what I want to see next
The blog post is light on training details. I'd love to know:
- What's the training cost ratio? How much more expensive is QAD versus standard fine-tuning?
- Does this generalize to other architectures? LFM2.5 uses a "liquid" state-space backbone—does QAD work as well for standard Transformers?
- What about
Q3_KorQ2_K? Is there a floor where distillation can't recover the losses? - Can you stack QAD with other compression techniques (pruning, low-rank adapters) without collision?
The comparison to Unsloth's PTQ is encouraging, but I'd also want to see QAD benchmarked against other quantization-aware methods (QLoRA, GPTQ with fine-tuning, etc.). The landscape is crowded and evolving fast.
The broader pattern: training for deployment
QAD fits into a larger trend: training models in the configuration you'll actually deploy them.
We've seen this with mixed-precision training (train in FP16, deploy in FP16). We've seen it with knowledge distillation (train a small model to mimic a large one). We've seen it with quantization-aware training for vision models.
What's notable here is that Liquid AI is shipping production artifacts, not publishing a technique. The GGUFs are on Hugging Face. You can download them and run them today. That's a different posture than "we got good benchmark numbers in a controlled setting."
It also suggests that the quantization tax—historically 5–15% performance loss at 4 bits—is negotiable if you're willing to train for it. That changes the economics of edge AI in a meaningful way.
Try it yourself
The QAD GGUFs are live:
Usage is straightforward if you're already running llama.cpp:
llama-cli -hf LiquidAI/LFM2.5-350M \
--hf-file LFM2.5-350M-QAD-Q4_0.gguf \
-p "What is C. elegans?"
I'm curious to see how these perform on real-world tasks beyond the benchmark suite—especially agentic workflows, long-context summarization, and tool use. If the quality holds up in production, this could be the unlock for a lot of on-device use cases that weren't economically viable before.
Quantization has always been about compromise. QAD suggests the compromise might be a lot smaller than we thought.