Nunchaku Lite Lands in Diffusers: 4-Bit Diffusion Models Without the Custom Engine

Hugging Face just shipped something that makes high-end diffusion models significantly more accessible: native Nunchaku Lite support in Diffusers. If you've been following the quantization space, you know SVDQuant and its Nunchaku inference engine have been delivering impressive speedups with 4-bit weights and activations. The catch? Until now, using those checkpoints meant reaching for a separate inference library.

That friction is gone. Current Diffusers lets you load Nunchaku-style checkpoints with a standard from_pretrained() call, treat them like any other model, and get roughly 30% faster inference with 50% less VRAM—all without compiling custom CUDA kernels locally.

This is a textbook example of the quantization integration work that matters: taking research-grade techniques and making them disappear into the standard workflow.

What Makes SVDQuant Different

Most quantization backends in Diffusers—bitsandbytes, GGUF, torchao, Quanto—are weight-only. They store weights in low precision, dequantize at compute time, and save memory without necessarily improving speed. Sometimes they even add latency overhead.

SVDQuant, the method behind Nunchaku, takes a different path: it quantizes both weights and activations to 4 bits (W4A4) for the main transformer layers. That reduces memory and speeds up the denoising loop.

The trick is handling outliers. Diffusion transformers have large activation outliers that break naive 4-bit quantization. SVDQuant moves those outliers into the weights, represents the hardest part of each weight matrix with a small 16-bit low-rank branch, and quantizes the residual to 4 bits.

Nunchaku's fused kernels make this fast by eliminating memory-access overhead: the low-rank down projection fuses with the quantization kernel, and the up projection fuses with the 4-bit compute kernel. Clean approach, solid engineering.

Nunchaku Lite: The Diffusers Integration

The original Nunchaku engine gets much of its speed from model-specific fused execution paths—fused QKV projections, fused GELU/MLP kernels, custom module layouts. Supporting a new architecture often requires model-specific integration work.

Nunchaku Lite is the new integration path. Instead of requiring a custom pipeline or separate engine, it patches relevant nn.Linear modules of a stock Diffusers model with runtime SVDQ/AWQ linear layers before the checkpoint loads. The CUDA kernels come from the Hub through Hugging Face's kernels package.

Two kernel families do the work:

• svdq_w4a4: 4-bit weights and activations with the SVDQuant low-rank correction, used for attention and MLP projections where most compute happens. Available in INT4 and NVFP4 variants.
• awq_w4a16: 4-bit weights with 16-bit activations, used for adaptive normalization and modulation layers (FLUX's adanorm_single/adanorm_zero, Qwen-Image modulation). These layers are memory-bound and precision-sensitive, so AWQ preserves quality while saving space.

The trade-off: without architecture-specific fused kernels, Nunchaku Lite can't match the original engine's full speedup. But it still delivers around 30% faster inference while keeping the same VRAM reduction, and it works with the entire Diffusers ecosystem—schedulers, LoRA hooks, offloading, torch.compile—out of the box.

Loading a Nunchaku Checkpoint

If you've used bitsandbytes or torchao in Diffusers, the mechanics are familiar. A Nunchaku Lite repository is a standard Diffusers repo with one addition: a quantization_config block in the transformer's config.json.

That config tells Diffusers which modules were quantized, which scheme they use, and which runtime layer to instantiate. The quantized model keeps the exact module structure of the dense one, so everything downstream sees a normal Diffusers model.

Loading looks like this:

import torch
from diffusers import ErnieImagePipeline

pipe = ErnieImagePipeline.from_pretrained(
    "lite-infer/ERNIE-Image-Turbo-nunchaku-lite-nvfp4_r32-bnb4-text-encoder",
    torch_dtype=torch.bfloat16,
).to("cuda")

image = pipe(
    prompt="A cinematic portrait of a red fox in a misty forest at sunrise, detailed fur, volumetric light",
    height=1024,
    width=1024,
    num_inference_steps=8,
    guidance_scale=1.0,
    generator=torch.Generator("cuda").manual_seed(42),
).images[0]

No custom pipeline class. No local compilation. The NVFP4 kernels download from the Hub the first time they're used. This checkpoint pairs a Nunchaku NVFP4 transformer with a bitsandbytes NF4 text encoder and generates a 1024×1024 image in about 1.7 seconds on an RTX 5090, using roughly 12 GB peak memory versus 24 GB for BF16.

Hardware and Precision Support

Nunchaku Lite uses different kernel variants depending on GPU generation and checkpoint precision:

• NVFP4 (svdq_w4a4 with nvfp4 precision): Blackwell GPUs only (RTX 50 series, RTX PRO 6000, B200)
• INT4 (svdq_w4a4 and awq_w4a16 with int4 precision): Turing, Ampere, Ada (RTX 30 & 40 series, A100, L40S)

Volta and Hopper GPUs are currently unsupported by the 4-bit kernels. The quantizer validates CUDA capability at load time and raises a clear error instead of producing incorrect outputs.

For earlier GPU generations, use the INT4 checkpoint variants. They're slightly slower than NVFP4 on Blackwell but still deliver meaningful speedups on supported hardware.

Benchmarks: Speed and Memory

Hugging Face tested on an NVIDIA RTX PRO 6000 (Blackwell) at 1024×1024 using the ERNIE-Image-Turbo checkpoint. The numbers:

Nunchaku cuts peak VRAM by up to 50% while improving latency by roughly 30%. The remaining overhead comes from extra kernel launches, which torch.compile can mitigate—bringing the full pipeline down to 1.68 seconds, or 1.8× faster than BF16.

Further quantizing the text encoder with bitsandbytes NF4 reduces peak VRAM by about 22% in the benchmark, which is meaningful when text encoders like T5 or Qwen3 occupy several gigabytes on their own.

Quantizing Your Own Models

The companion diffuse-compressor toolkit provides an end-to-end SVDQuant workflow: calibrate, quantize, package, and publish. The process is architecture-agnostic.

The generic scanner walks the model and decides what to target: compatible linears inside the repeated transformer-block stack become SVDQ W4A4 targets, recognized modulation linears become AWQ W4A16 targets, and everything else stays dense.

After inspection, you run quantization, package the result as a Diffusers pipeline (optionally converting text encoders to NF4), then verify and push to the Hub. The full tutorial in the Hugging Face documentation covers every flag in detail, including handling models with structural rewrites.

Image Quality

The Hugging Face blog includes side-by-side comparisons of BF16 versus 4-bit outputs with identical seeds and settings. The visual fidelity is remarkably close—minor texture differences are visible on close inspection, but the overall composition, color, and detail preservation hold up well.

This is the payoff of SVDQuant's low-rank correction: it lets you run 4-bit inference without the quality collapse you'd see from naive quantization.

Why This Matters

Large diffusion transformers create stunning images, but loading a modern text-to-image model in BF16 often requires 20–30 GB of VRAM. That puts them out of reach for most consumer GPUs.

Nunchaku Lite brings those models onto more accessible hardware—RTX 30 and 40 series cards, RTX 50 series, even older datacenter GPUs—without sacrificing much speed or quality. And because it integrates natively into Diffusers, you don't need to learn a new inference stack or maintain custom forks.

The other win: diffuse-compressor makes quantization reproducible and shareable. Quantize once, push to the Hub, and anyone can load your checkpoint with from_pretrained(). That's the kind of composability that accelerates adoption.

Ready-to-Use Checkpoints

Hugging Face has published several Nunchaku Lite checkpoints covering FLUX, ERNIE-Image-Turbo, and other architectures. They're tagged on the Hub and ready to use with current Diffusers.

If you're working with diffusion models on consumer hardware, this is worth your attention. The integration is clean, the performance gains are real, and the workflow is familiar. No heroics required.