The biggest constraint in enterprise AI isn't intelligence anymore. It's utilization. And a new paper from Dharma-AI just proved that with the right scheduler, you can squeeze 33 percentage points more out of the same cluster—without buying a single additional GPU.
This is not a marginal gain. This is the GPU equivalent of finding a third of your compute budget sitting idle because you handed out capacity in the wrong order.
The Setup: Four Workloads, Two Incompatible Shapes
The problem statement is deceptively simple: which GPU runs which job, in which timestep, at what priority?
In practice, four workload types compete for the grid: training, real-time inference, batch inference, and quantization. They split into two allocation shapes, and that split is where everything breaks.
Batch-like jobs (training, batch inference, quantization) need contiguous blocks of GPUs held without interruption until completion. Real-time inference is the opposite: elastic, driven by a demand curve that changes every timestep, growing and shrinking with traffic.
Two incompatible shapes competing for the same hardware in the same timestep is the core problem. And it gets worse: even within a single type, training jobs for the same base model can range from a few hours to several days, from one GPU to dozens.
What FIFO Costs Under Contention
The comparison baseline is a FIFO-based scheduler: real-time inference gets a fixed reservation, everything else gets placed in arrival order, no regard for priority.
Under slack conditions, that's fine. When the cluster has headroom, allocation order costs nothing in utilization—everything fits regardless of sequence.
Contention is where FIFO becomes expensive, and it costs capacity in two separate ways.
The Reservation Tax
Real-time inference can't wait for capacity. The GPUs have to be there the moment traffic needs them. A FIFO scheduler has no mechanism to release GPUs during a trough and reclaim them before the next peak, so the only way to guarantee availability is to reserve each real-time application's maximum daily demand for the entire day.
An application needing six GPUs at midday and two at 4am holds all six for twenty-four hours. The four idle GPUs are unavailable to any batch job for the entire day—not being used, not free either.
That's why the baseline sits near half the cluster in the two scenarios where reservation dominates: 51.6% utilization in the mixed control scenario, 53.6% in the training-heavy case. Roughly half a pool, with much of the idle half reserved rather than free.
The Ordering Tax
Under real contention, which jobs fit at all depends on the order you place them, not just on how much capacity exists. Order is not a tiebreaker applied after the capacity question is settled. Order is a capacity decision.
FIFO places each job as it arrives, without weighing what that job is worth and without checking what else still has to fit inside the horizon. High-priority work waits behind whatever asked first. Capacity gets committed in placements that later jobs cannot use.
It's the GPU equivalent of an airline assigning aircraft to whichever charter called first, then finding nothing left to fly the route that actually pays.
The Results: 33 Points, Same Hardware
Across five benchmark scenarios built for genuine contention, the constraint-aware allocator improved both axes at once.
Utilization moved from a 52–85% band to a 72–88% band. Priority-weighted value rose between 24.6% and 105.1%, averaging 52%. Every scenario, both metrics, no tradeoff to explain away.
The strongest single case was a training-heavy workload on 8 GPUs: utilization went from 53.6% to 87.0%, and value more than doubled, up 105%. Thirty-three points of a fixed, already-depreciating asset, recovered by reclaiming reserved standby capacity and placing the rest in priority order.
Nothing about the hardware changed. What changed was the order in which allocation decisions get made.
Why Utilization Alone Isn't Enough
One scenario pulls utilization and value apart completely, and the gap runs in a direction that's easy to miss.
In the scale test—30 jobs across 64 GPUs—FIFO and the allocator produced identical utilization (44.9% each) and identical throughput (27 of 30 jobs completed). The allocator delivered 15.9% more priority-weighted value.
Every dashboard reads the same. The cluster produced materially different output.
An objective that doesn't price priority can fill the cluster to exactly the same level, finish exactly as many jobs, and still deliver less. Occupancy is a poor read on whether a cluster is earning.
How It Works: Writing the Problem Down
The alternative isn't a longer list of heuristic rules. Some constraints only mean anything globally, and no local rule can express them: contiguous blocks, a budget for how much GPU churn is acceptable across the entire horizon, a guarantee that running work is never preempted.
To honor those, the problem has to be written down as one thing.
Five Constraints, One Objective
Five constraints define a legal allocation:
- A GPU serves at most one job per timestep
- Every job respects its demand range; running work is inherited and held
- Batch-like jobs occupy contiguous blocks of GPUs, sized to a power of two
- Real-time jobs have a hard cap on how many GPUs they may swap between consecutive timesteps
- A job that has started cannot be interrupted
The objective function has two terms. Allocating a GPU to a batch-like job earns a reward equal to its priority multiplied by a time-decay weight. Failing to meet real-time demand incurs a penalty proportional to the size of the shortfall.
The relative size of those weights is the entire service-level policy, expressed as one number. The real-time penalty weight is 5 to 10 times greater than the allocation weight. One unit of unmet real-time demand costs what 5 to 10 GPU-timesteps of equal-priority batch work costs.
The asymmetry is deliberate. It means latency obligations are enforced inside the same optimization that places batch work, rather than by a separate autoscaler competing with the scheduler for the same GPUs.
The Architecture: Heuristic on the Hot Path, Model Behind It
This is NP-hard combinatorial allocation, and the scheduler is re-invoked every time a job arrives. The decision has to come back in the gap between two API requests.
That latency budget is the fixed constraint the architecture is designed around, which is why a heuristic sits on the hot path and the formal model sits behind it as the specification the heuristic is built to satisfy.
That heuristic isn't a generic greedy allocator. Its rules are the formal model's structural constraints, which means every grid it produces is a legal allocation by construction. Not usually valid. Valid by design.
It runs in 1 to 2 milliseconds on the five contended scenarios, and 15 milliseconds at 64 GPUs and 30 jobs—fast enough to run on every incoming request.
The system exposes two modes. Fast mode runs the allocator alone and returns its grid; this is the hot path. Full mode uses that grid as a starting point for the formal model, which attempts to improve on it—suited to periodic review rather than per-request decisions.
What This Generalizes To
The design here applies anywhere you have:
- Heterogeneous workloads competing for the same resource pool
- At least one latency-sensitive class and one throughput-oriented class
- Contention that makes allocation order materially affect capacity utilization
That's not just GPU clusters. It's Kubernetes pods competing for CPU, storage tiers balancing hot and cold access patterns, network bandwidth shared between interactive and batch traffic.
The lesson is structural: when reservation is the only tool you have to guarantee availability, you pay for peak demand around the clock. When you can price unmet demand inside the same objective that places batch work, you can treat reservations as elastic and reclaim the troughs.
Order is a capacity decision. And if you're still handing out capacity in arrival order, you're leaving a third of your cluster on the table.