Training a 7B text-to-image model isn't just about architecture and compute—the real grind is the data pipeline. Photoroom's PRX Part 4 data strategy post is the rare blog that actually shows you the work: the pragmatic choices, the tooling tradeoffs, and the experiments that proved which corners were safe to cut.
This isn't a victory lap about data quality. It's a blueprint for the unsexy infrastructure decisions that quietly determine whether your pre-training run succeeds or wastes a month of cluster time.
Pre-Training Is For Breadth, Fine-Tuning Is For Taste
The guiding philosophy is clear: pre-training datasets should be large and diverse, not pristine. PRX's team argues that over-filtering for aesthetics at the pre-training stage actively hurts the model by narrowing the distribution and costing it compositional variety it can't recover later.
Ordinary snapshots, slightly compressed images, screenshots with text—all of that belongs in the corpus. The model needs to learn "how the world looks: the visual concepts, the objects and scenes, how things are composed and lit, and the sheer range of what images can contain."
Polish comes later. Fine-tuning on small, ruthlessly curated sets is where you teach taste. Pre-training is where you teach structure.
The Long Caption Thesis
What matters most for pre-training, according to Photoroom, is long captions that accurately describe everything in the image. They saw this directly in Part 2 of the series, where switching from short captions to long ones substantially improved sample quality.
Faithful captioning turns "noise" into conditioned, controllable attributes. If a caption describes the logo, the screenshot, or the text in an image, the model learns to treat those elements as promptable features rather than reproducing them unconditionally.
This is why the team's filtering is deliberately light. They remove what's genuinely unusable, not everything that's imperfect.
JPEG Quality 92: Safe or Reckless?
Photoroom encoded all images as JPEG at quality 92 instead of lossless PNG. For a project at this scale, that choice saves 3–10× in storage. But does it hurt model quality?
They didn't assume—they measured. Real-world images have usually already been JPEG-compressed multiple times, so the question is whether one more re-encode adds perceptible artifacts.
Across repeated decode/encode cycles on both high-resolution (1–2 MP) and lower-resolution (0.25–0.5 MP) images, the first re-encode at quality 92 is essentially imperceptible. Even after 10 cycles, images stay well within the imperceptible range.
Training on JPEG vs PNG: The Experiment
The real test is whether training on JPEGs changes what the model produces. Photoroom trained two identical PRX models at 1024px on the same images: one stored as PNG, one as JPEG at quality 92.
Both trained equally well on metrics. Generations were practically indistinguishable. Using a known technique for estimating JPEG quality by matching quantization tables, they found:
- JPEG-trained model: 12.0% detection rate, mean estimated quality 39.6, median 34.0
- PNG-trained model: 10.8% detection rate, mean estimated quality 42.1, median 45.0
Only about one generation in ten from either model shows any detectable quantization structure, and the differences are negligible. High-quality JPEG storage adds nothing measurable on top of what the sources already carry.
For artifact-sensitive tasks like Photoroom's custom AI Shadows model, they still use PNGs. But for large-scale text-to-image pre-training, JPEG at quality 92 is more than good enough.
Lance vs MDS: Two Formats, Two Jobs
Photoroom uses two dataset formats that play complementary roles: Lance for building and exploring, Mosaic Data Shards (MDS) for streaming during training.
Lance is a columnar data format with cheap predicate pushdown, scalar indexes, and vector search. It's the right tool for feature engineering and dataset curation—queryable, filterable, explorable.
MDS is rigid but optimized for distributed training. MDS datasets can be mixed, shuffled, and streamed directly from object storage like S3 or GCS. The tradeoff: adding a column or creating a subset means scanning and rewriting the whole dataset.
Lance lets you iterate. MDS lets you train.
Fragmentation: A Lesson Learned The Slow Way
One hard-won insight: Lance table fragmentation matters. A Lance table is split into fragments, and several operations scale with the total number of fragments rather than row count. Too many small fragments make queries crawl.
Photoroom's first ingest targeted only 100,000 rows per fragment, leaving thousands of tiny fragments and making even simple filters and full-text queries slow. After compacting to roughly a million rows per fragment (about a thousand fragments for several hundred million rows), queries were fast.
The Lance performance guidance rule of thumb is to keep fragment count on the order of a hundred even for a billion-row table. Lance supports distributed compaction via the Lance-Ray integration, so it's easy to fix after the fact—but better not to over-fragment in the first place.
Text Latents: On-The-Fly vs Pre-Computed
Photoroom previously pre-computed text latents using T5Gemma and stored them in MDS as bytes. This time, after switching to Qwen3-VL, they decided to compute text latents on the fly during training.
Running the text encoder inside the training loop costs throughput. For a small denoiser, that cost can be significant. But at PRX's 7B scale, the text encoder's compute is negligible next to the denoiser, and they measured only a roughly 3–4% throughput cost—about one extra day on a 30-day run.
The benefits:
- MDS shards stay much smaller, small enough to store the full pre-training dataset on SSD-backed shared filesystem instead of streaming over the network from object storage.
- Freedom to change the text encoder later without rewriting terabytes of stored latents.
When the text encoder is cheap relative to the denoiser, on-the-fly computation is the right call.
Re-Captioning Everything With a VLM
Photoroom chose to re-caption every image rather than rely on the captions some datasets shipped with. The reason: consistency. Across a mix of sources, caption length, style, and quality vary wildly.
Pre-existing captions and embeddings (like CLIP embeddings) are still valuable for a different purpose: making the dataset explorable in Lance right away. Full-text search over captions and nearest-neighbor search over embeddings let the team navigate the data and judge its quality before running their own captioning.
The post doesn't specify which VLM they used as the captioner, but the emphasis is on long, faithful descriptions that capture everything in the image.
The Pragmatic Take
This post is a masterclass in pragmatic data engineering. Photoroom didn't build the absolute best dataset one could build—they built a solid, lightweight starting point for pre-training a 7B model.
They leaned on existing curation work instead of redoing it at scale. They measured JPEG quality impacts instead of assuming. They picked tooling that let them iterate (Lance) and stream (MDS) without forcing one format to do both jobs badly.
The unsexy truth about training data: there's no one right answer, only tradeoffs you understand well enough to make intentionally. Photoroom's data pipeline isn't glamorous. It's just defensible, measurable, and built to ship.
And that's the kind of infrastructure thinking that actually moves models from experiments to production.