The diffusion model ecosystem has a scaling problem. FLUX.1-dev has 12 billion parameters. HunyuanVideo has 13 billion. You can run inference on these models from a laptop, but fine-tuning them on your own data? That's been a different story—one involving checkpoint conversions, custom training scripts, and a lot of prayer that your multi-GPU setup actually works.
NVIDIA and Hugging Face just shipped an integration that changes the calculus. NeMo Automodel now works natively with any Diffusers-format model on the Hub, bringing production-grade distributed training to the ecosystem without requiring you to leave the Diffusers world. No format conversions. No model rewrites when a new architecture drops. Just point at a model ID and start training.
This matters because the current fine-tuning story for large diffusion models has been messy. Built-in Diffusers training scripts get you started, but scaling beyond a few GPUs means stitching together your own parallelism, checkpointing, and data pipeline. NeMo Automodel packages the hard parts—FSDP2, tensor parallelism, context parallelism, multiresolution bucketing—into a configuration layer that doesn't require rewriting your models.
What NeMo Automodel actually does
NeMo Automodel is an open-source PyTorch training library built on two principles that matter for the Diffusers community. First, it's Hugging Face native—you point pretrained_model_name_or_path at any Diffusers model ID and train. Under the hood it uses Diffusers model classes like WanTransformer3DModel for loading and WanPipeline for generation. Your checkpoints round-trip cleanly back to the Hub.
Second, parallelism is a configuration choice, not a code rewrite. You can switch between FSDP2, tensor parallel, expert parallel, context parallel, and pipeline parallel by declaring configurations in YAML. The same recipe structure works whether you're running on a single node or a SLURM cluster.
Currently the library supports flow-matching models only. It uses flow matching as the training objective, with latent-space training via pre-encoded VAE outputs and multiresolution bucketed dataloading to speed up throughput.
Which models are supported
The integration ships with ready-to-use fine-tuning recipes for six major open diffusion families:
- Wan 2.1 T2V (1.3B and 14B): Text-to-video, both sizes supported, LoRA recipes available
- Wan 2.2 T2V A14B: 27B total parameters (MoE), 14B active per step, full fine-tuning only
- FLUX.1-dev: 12B parameter text-to-image, LoRA supported
- FLUX.2-dev: 32B parameter text-to-image, LoRA supported
- HunyuanVideo 1.5: 13B parameter text-to-video, LoRA supported
- Qwen-Image: 20B parameter MMDiT architecture, LoRA supported
The 1.3B Wan model fits on a single 40GB A100. Everything else benefits from multi-GPU setups, though LoRA lets you train on smaller clusters than full fine-tuning would require.
The workflow is simpler than you'd expect
The recommended installation path is the NeMo Automodel Docker container, which bundles PyTorch, TransformerEngine, and CUDA dependencies. Alternatively you can pip3 install nemo-automodel or install from source.
The typical workflow has three stages. First, you pre-encode your dataset. The diffusion recipe consumes cached VAE latents and text embeddings instead of encoding images during every training step. You can stream directly from Hugging Face datasets and distribute preprocessing across all visible GPUs.
Second, you launch training using existing YAML configs with command-line overrides for dataset-specific settings. The checked-in configs already select model architectures, parallelism strategies, and effective batch sizes. You just point them at your cache directory and specify resolution, learning rate schedule, and checkpoint frequency.
Third, you generate from the fine-tuned checkpoint using the existing generation YAML and a Diffusers pipeline. The checkpoint format is native Diffusers—no conversion step before you can load it into a DiffusionPipeline or push it to the Hub.
The tarot fine-tune walkthrough
The blog post includes a complete example fine-tuning FLUX.1-dev on the 78-card Rider–Waite tarot dataset. Preprocessing streams the images from Hugging Face and distributes across GPUs, producing .pt cache files and sharded metadata organized by resolution bucket.
Training uses the existing flux_t2i_flow.yaml config directly, with command-line overrides for the tarot-specific cache directory, base resolution, learning rate schedule, and checkpoint frequency. The run produces checkpoints every 50 steps up to 200 total steps.
Generation loads the final checkpoint and uses a trtcrd trigger token to invoke the learned tarot style. The results show the model acquired cream/red/black vintage palettes, heavy ink contours, flat color fields, and allegorical card composition—while untriggered prompts remained photographic, demonstrating the effect is associated with the trigger rather than replacing base model behavior globally.
Performance numbers from real hardware
All measurements come from one node with 8× NVIDIA H100 80GB GPUs, averaged over three steady-state 10-step windows.
For text-to-image at 512×512, FLUX.1-dev full fine-tuning with FSDP2 processes 35.51 images/second (4.44 per GPU) with 63.88 GiB peak allocated per GPU. FLUX.1-dev LoRA rank-64 with DDP hits 53.73 images/second (6.72 per GPU) at 67.43 GiB per GPU. Qwen-Image full fine-tuning runs 41.21 images/second, while LoRA reaches 46.63 images/second.
For text-to-video at 512×512×49 frames, Wan 2.1 1.3B processes 8.50 clips/second (1.06 per GPU) with activation checkpointing off, using just 6.09 GiB per GPU. The 14B variant runs 2.107 clips/second with activation checkpointing on, allocating 33.35 GiB per GPU.
These aren't synthetic benchmarks—they're sustained training throughput on production-scale models.
Why this integration matters
The immediate value is practical: you can fine-tune FLUX or HunyuanVideo on your own dataset without leaving the Diffusers ecosystem. No checkpoint surgery, no custom data loaders, no debugging why your eight GPUs aren't actually cooperating.
The longer-term value is architectural. When a new diffusion model lands in Diffusers, enabling it in NeMo Automodel requires a small, contained code addition—a data preprocessing handler and a model adapter. The rest of the recipe stack (FSDP2, bucketed dataloading, checkpointing, generation) carries over unchanged.
That's the difference between "we support these six models" and "we support the Diffusers ecosystem." Every new architecture that lands on the Hub inherits production-grade training infrastructure without waiting for custom script development.
What comes next
The current release uses YAML-driven configuration, which is tractable for recipes but verbose for experimentation. The roadmap includes Pythonic recipe APIs that let you compose training configurations programmatically.
Kubernetes orchestration is coming alongside the existing SLURM support, which matters for cloud-native deployments.
And the model support list will grow as new diffusion architectures ship. The integration is fully open source under Apache 2.0, so the community can contribute adapters for new models without waiting for NVIDIA or Hugging Face to add them.
The integration is documented in the Diffusers training guide and available today. If you've been putting off fine-tuning a large diffusion model because the infrastructure overhead looked prohibitive, that excuse just evaporated.