What Just Shipped
Hugging Face published a comprehensive fine-tuning guide for TRELLIS.2, the latest 3D generation model from the team behind TRELLIS. This isn't just documentation—they're shipping ready-to-run scripts, Docker containers, and concrete recipes for adapting a general-purpose 3D model to your specific domain.
The practical pitch: if you've got 3D assets on the Hub, you can spin up fine-tuning jobs with a few command-line arguments. No deep pipeline archaeology required.
But the interesting part isn't the convenience—it's the choices they made about which components to train and which to freeze. Those choices reveal something important about how multi-stage 3D pipelines actually work.
The Problem They're Solving
TRELLIS.2 is a general-purpose model. It generates plausible 3D objects from images across many categories. But "plausible" isn't the same as "correct for your use case."
The example they chose is telling: footwear from the Google Scanned Objects dataset. The base model produces shoes that look right—silhouette, laces, overall shape. But the soles? Consistently wrong. Not catastrophically wrong, just... off.
For a furniture catalog or a game asset library, that's not acceptable. You need geometric accuracy in the parts that matter for your domain. A general model won't give you that without adaptation.
This is the classic fine-tuning value prop, but in 3D it's harder. You're dealing with multiple representation formats (NeRFs, Gaussian splatting, meshes), multiple file formats (GLB, PLY), and a pipeline with distinct stages that handle structure, geometry, and materials separately.
How TRELLIS.2 Actually Works
The pipeline has five main stages:
- Background removal (BiRefNet)
- Image encoding (DINOv3-L produces feature tokens)
- Sparse structure prediction (a diffusion transformer generates a 16³ × 8 latent, decoded to a 64³ occupancy grid)
- Detailed geometry generation (shape flow DiT produces 512³ resolution O-Voxel geometry)
- Material generation (texture flow DiT outputs base color, metallic, roughness, alpha)
TRELLIS.2's key innovation is a unified internal representation that lets different decoders export to whatever format you need. You don't need separate models for different output types.
The O-Voxel format is worth understanding: it's a sparse voxel grid that only stores surface voxels, not the empty interior. Each voxel holds local geometry and material properties. This makes the representation compact—the model uses only ~10% of voxels in the grid. CPU-to-O-Voxel conversion takes seconds; GPU-to-mesh takes seconds.
The Critical Decision: What to Freeze
Here's where it gets interesting. The guide adds LoRA adapters to exactly two components:
- The sparse-structure flow DiT (rank 8 adapters in attention and MLP layers)
- The shape flow DiT (rank 16 adapters in attention layers)
They explicitly freeze the texture flow DiT. This isn't laziness—it's architectural insight.
Why Skip Texture Training?
The dataset has diffuse color maps with baked-in lighting. To properly train the texture stage, you need separated material maps: base color, metallic, roughness. The GSO dataset doesn't provide those.
If you train on diffuse maps anyway, the model learns to encode lighting as a surface property. It starts representing shadows and highlights as texture, not as the result of material interaction with light. Worse, it risks catastrophic forgetting—overwriting the base model's existing material understanding.
The shape is wrong. The materials are already good. So you train shape and freeze materials.
Why LoRA Instead of Full Fine-Tuning?
Their footwear subset has 188 objects from the 1,030-object GSO dataset. For a dataset this small, full fine-tuning is both expensive and risky. You can easily overfit or destroy the model's learned priors.
The LoRA configuration trains approximately 9.3 million adapter parameters total across both DiTs. The base model keeps its weights frozen. The adapters are small enough to train quickly, small enough to avoid overfitting, and preserve what the base model already does well.
This is textbook LoRA usage, but seeing it applied to a multi-stage 3D pipeline clarifies why low-rank adaptation matters. You're not just saving compute—you're protecting capabilities you want to keep.
What You Actually Run
They provide two Docker images: one for preprocessing (converting meshes to the internal training format), one for training itself. Both run on Hugging Face Jobs or locally.
The preprocessing pipeline converts GLB meshes to O-Voxel format and renders multi-view images for conditioning. The training pipeline handles the actual LoRA fine-tuning with the PEFT library.
You need:
- A Hugging Face token
- Access to DINOv3 (it's gated)
- Your dataset uploaded to the Hub
Then it's a few CLI arguments and you're training.
What This Means for 3D ML
The 3D generation field has been fragmented. Different representations, different formats, different models for different tasks. TRELLIS introduced a unified latent representation; TRELLIS.2 improved the structural understanding.
But research models and production models have different requirements. A general-purpose model generates plausible objects. A production model generates correct objects for a specific domain.
This guide bridges that gap. It's not just "here's how to fine-tune"—it's "here's which components to adapt, which to freeze, and why."
The architectural choices matter. Knowing that the texture stage needs separated material maps, not diffuse color, prevents a common failure mode. Knowing that sparse latents let you train efficiently on surface voxels without wasting compute on empty space explains why the approach scales.
The Broader Pattern
We're seeing this pattern across modalities: foundation models that are good-enough out of the box, plus practical recipes for domain adaptation. The innovation isn't the base capability—it's the composability of the pipeline and the clarity about which components to touch.
In 2D, we know how to fine-tune diffusion models. We know when to train text encoders versus UNets versus both. We know LoRA ranks, learning rates, dataset sizes.
In 3D, that knowledge is still consolidating. Guides like this—opinionated, specific, with working code—accelerate the transition from "research artifact" to "tool you can actually use."
The fact that they chose footwear soles as their example problem is perfect. It's not a benchmark. It's not AGI. It's a real person saying "the shoes look right but the bottoms are wrong" and the model not knowing how to fix it without help.
That's the use case. And now there's a recipe.