The AsyncGRPOTrainer in TRL can now train a LoRA adapter and sync only the adapter to vLLM—not the full model. A rank-1 adapter for a 1.5B model is a few megabytes instead of ~3GB, which unlocks a clever architecture: the trainer and vLLM replicas can run as separate Hugging Face Jobs on separate machines, communicating through a Storage Bucket mounted as a FUSE filesystem in every container.
No NCCL. No shared node. Just a bucket, a proxy, and some surprisingly clean systems design.
This shipped in TRL v1.14, and the HuggingFace team documented a real-world setup that took the same recipe from 3 hours 27 minutes down to 53 minutes for 500 steps across five runs. Let's break down how it works and why LoRA makes this architecture possible.
Why LoRA is perfect for async RL
LoRA training is particularly well-suited for reinforcement learning. The key insight comes from Thinking Machines's "LoRA Without Regret" post: the advantage function in policy-gradient RL only provides roughly O(1) bits of information per episode.
There's not much to learn from each step from an information-theoretic perspective. A rank-1 adapter has enough capacity to absorb it.
Thinking Machines showed that LoRA can match full fine-tuning for policy-gradient RL, even with rank 1. For a 1.5B model, that's a few megabytes instead of gigabytes. This has a direct systems consequence: instead of syncing the full policy to inference workers after every update, you just send the adapter.
vLLM can keep several adapters loaded simultaneously. Old rollouts finish with the policy they started under. New rollouts use the latest. The trainer doesn't have to wait for all in-flight rollouts to drain before updating—it just publishes a new adapter version.
The constraint: HuggingFace Jobs can't talk to each other
A HuggingFace Job is one container running on one VM. One Job can't spawn multiple nodes, and Jobs can't form an NCCL group across machines. There's no shared localhost, no shared local disk, no direct network path between containers.
The AsyncGRPOTrainer is built for exactly the kind of scale where the trainer and a fleet of vLLM servers run on separate hardware. With full-weight syncs, you'd be stuck: every update would need to move gigabytes between machines with no fast interconnect.
With LoRA, a sync is only a few megabytes. That's small enough to route through HuggingFace's Storage Buckets, which mount as a FUSE filesystem inside every Job via hf-mount.
No network path needed. The adapter travels through the bucket.
The architecture: three Jobs and a bucket
The setup has four components:
- A trainer Job running
AsyncGRPOTrainerwith LoRA (and FSDP). - Two vLLM Jobs, each serving the base model plus whatever adapter the trainer last published.
- A Storage Bucket mounted at the same absolute path in all three Jobs.
- A proxy server that routes rollouts, broadcasts adapter loads, and adds auth headers.
Every few optimizer steps, the trainer saves the adapter under <output_dir>/.vllm_lora/trl-policy-v{N}, publishes it with an atomic rename, then sends its path to vLLM's /v1/load_lora_adapter endpoint.
vLLM loads the files from disk. The rollout worker requests model="trl-policy-v{N}". This is how runtime adapter loading already works in vLLM—the endpoint takes a path, not tensors, so the trainer and server are expected to share a filesystem.
On a Slurm cluster, that would be the network filesystem. On Jobs, it's the bucket:
hf jobs run ... -v hf://buckets/aminediroHF/asyncgrpo-lora-buckets:/lora ...
Nothing in TRL or vLLM had to change. The trainer writes to /lora/<run>/.vllm_lora/, and the servers read from the same path. The POST request contains a path that's already valid inside every container.
Why version the adapters instead of overwriting one name?
The trainer keeps max_staleness + 1 adapter versions registered. With max_staleness=4, a rollout sample can lag up to four versions behind the current policy before the trainer discards it.
A rollout that started under trl-policy-v3 must finish under v3. At any moment, vLLM serves the current policy plus the four before it. Each sync loads the new version before unloading the oldest, which needs one extra slot during the swap: --max-loras 6.
vLLM keys its prefix cache by adapter name. If you published every update under the same name, KV blocks computed under the old weights would still match after the swap. The prefill wouldn't be redone. A rollout could get its prefix from one policy version and its decode from the next.
The trainer would have no way to tell. It would show up as the importance sampling ratio drifting away from 1. Versioned names make this impossible: a name always means one set of weights.
The vLLM replicas: runtime LoRA and adapter slots
Each replica uses one GPU and the stock vllm/vllm-openai:v0.27.1 image. You enable runtime LoRA loading and reserve enough adapter slots:
hf jobs run --detach --flavor h200 --timeout 8h --secrets HF_TOKEN \
--expose 8000 \
-v "hf://buckets/${BUCKET}:/lora:ro" \
-e VLLM_ALLOW_RUNTIME_LORA_UPDATING=1 \
-e VLLM_SERVER_DEV_MODE=1 \
-- vllm/vllm-openai:v0.27.1 \
vllm serve Qwen/Qwen2.5-Math-1.5B --host 0.0.0.0 --port 8000 \
--max-model-len 4096 --enable-lora --max-lora-rank 1 --max-loras 6
The bucket is mounted read-only. The server only reads adapters.
VLLM_SERVER_DEV_MODE=1 enables /pause, /resume, and /server_info endpoints that TRL needs. With only five adapter slots instead of six, vLLM would silently evict a policy that still has rollouts in flight at every sync.
The proxy: routing, broadcasting, and auth
The proxy sits between TRL and the vLLM replicas. It does three things:
- Adds the auth header so the trainer doesn't need to manage secrets for every request.
- Routes each rollout to the replica most likely to hold its KV prefix, improving cache hit rates.
- Broadcasts adapter load requests to all replicas so both stay in sync.
TRL talks to http://localhost:8000. The proxy talks to the replicas over HTTPS at their https://<job_id>--8000.hf.jobs URLs.
The routing logic is key. vLLM's automatic prefix caching is extremely effective when rollouts with the same prompt prefix land on the same replica. The proxy can hash the prompt or use session affinity to keep related requests together.
Without the proxy, you'd need to hardcode secrets in the training script, manually load-balance across replicas, and send the same adapter-load POST twice. The proxy keeps the trainer script clean.
The dataset: Sanity-Test-R1D-1.5B
They chose sail/Sanity-Test-R1D-1.5B from the "Defeating the Training-Inference Mismatch via FP16" paper (Qi et al., 2025).
The authors generated 40 answers per MATH problem with DeepSeek-R1-Distill-Qwen-1.5B and kept problems with success rates between 20% and 80%. That yielded 1,460 questions.
This is a great validation set for RL. The questions are neither already solved nor hopeless for the model. The model gets a strong early signal and can improve.
It's also a robust end-to-end test. If one vLLM replica silently serves the base model under an adapter name, you'll see it in the reward curve within a few dozen steps. The dataset is small enough to cycle through in under two hours.
The hyperparameters came from the paper's LoRA scripts: Qwen/Qwen2.5-Math-1.5B, rank 1 with alpha 2, learning rate 4e-5, 8 samples per prompt, 128 completions per step, up to 3,000 generated tokens, and a 4,096-token context.
The results: 3h27m to 53m
Five runs took the same recipe from 3 hours 27 minutes to 53 minutes for 500 steps. The AsyncGRPO metrics show where the bottleneck sits at any moment.
The speedup comes from two things:
- The trainer and inference scale independently. The trainer doesn't wait for generation. Generation doesn't wait for the optimizer.
- Adapter syncs are nearly free. A few megabytes every few steps is invisible compared to gigabytes.
Checkpoints and the final adapter also live in the bucket. The Jobs are ephemeral, but a preempted trainer can resume training. The final adapter is never lost when the Job stops.
Why this matters
This setup proves you can run production-scale async RL training on ephemeral infrastructure with no NCCL and no shared node.
The architecture is clean: the trainer writes adapters, the bucket ferries them, the proxy routes traffic, and vLLM serves. Each piece does one thing. The fact that it works with stock vLLM, stock TRL, and a mount command is a testament to good API design.
LoRA makes it possible. Full-weight syncs would drown in cross-machine bandwidth. Rank-1 adapters are small enough to pass through a FUSE filesystem without thinking twice.
If you're running RL training on managed infrastructure—or want to scale training and inference separately without standing up a cluster—this is the blueprint.