#inference

#…

17 posts

VocabX: The 77% Inference Cost Cut You Get by Rethinking Tokenization

Dharma AI's VocabX rebuilds a model's vocabulary around cost per token rather than frequency—and slashes production inference costs by 77% without touching the base model.

#tokenization #inference #optimization #slms #production-ml

Liquid AI ships a DSpark drafter for vision models—and shows the limits of speculative decoding on edge

Liquid AI's new DSpark drafter speeds up LFM2.5-VL by up to 3x on device. But the bigger story is what it reveals about Amdahl's law in VLM inference: when prefill dominates, decode speedups matter less.

#vlm #inference #speculative-decoding #edge-deployment #benchmarks

Hugging Face Jobs now lets you spin up a private vLLM endpoint in one command

HF Jobs just turned into the fastest way to stand up a private LLM server. One CLI command gives you an OpenAI-compatible endpoint on H200s, pay-per-second, no Kubernetes required.

#inference #vllm #infrastructure #deployment #huggingface

Hugging Face Ships 207 WebGPU Kernels—And a Crowdsourced Benchmark Fleet

Hugging Face just released @huggingface/kernels: 207 optimized WebGPU operations for browser AI, each versioned and testable. Plus Fleet, a browser-based benchmarking tool that crowdsources performance data.

#webgpu #browser-ai #inference #benchmarking #kernels

Liquid AI Ships DSpark: Up to 3.2× Faster Inference Without Changing a Single Token

Liquid AI just released DSpark draft models for three LFM2.5 checkpoints, delivering GPU and on-device speedups via speculative decoding—without touching output quality. Here's how it works.

#inference #speculative-decoding #on-device #llms #performance

How Much Memory Does Your Agent Actually Need?

IBM's ALTK-Evolve reveals a counterintuitive finding: more agentic memory isn't always better. The right dose depends on the model—and sometimes the cheapest strategy wins.

#agents #memory #inference #benchmarks #altk

Baseten Joins the Hugging Face Inference Provider Ecosystem

Baseten is now live as a Hugging Face Inference Provider, bringing serverless access to frontier LLMs like DeepSeek V4 Flash and Kimi K3 directly through the Hub with zero markup pricing.

#llms #inference #huggingface #baseten #serverless

OpenAI's abundance thesis: the full-stack playbook that makes frontier AI economical

OpenAI just cut GPT-5.6 Luna pricing 80% and published the operating philosophy behind it: abundance through vertical integration, compound efficiency gains, and obsessive focus on useful work per dollar.

#openai #economics #inference #efficiency #vertical-integration

Nunchaku Lite Lands in Diffusers: 4-Bit Diffusion Models Without the Custom Engine

Hugging Face just made SVDQuant-style 4-bit diffusion inference native in Diffusers. Load Nunchaku checkpoints with from_pretrained(), no custom pipeline or local CUDA compilation required.

#diffusion #quantization #inference #huggingface #optimization

Hugging Face + Cerebras put Gemma 4 into sub-second voice AI—and 10k robots are already using it

The full voice loop—speech recognition, Gemma 4 reasoning, and TTS—now runs fast enough to feel natural. Plus: why the P95 latency tail matters more than median response time.

#voice-ai #inference #gemma #real-time #open-source

Hugging Face Jobs Just Made Standing Up vLLM Stupid Easy

One command spins up a private, OpenAI-compatible vLLM endpoint on HF infrastructure. Pay-per-second, no K8s, zero provisioning. Here's how it works and when to reach for it.

#vllm #inference #huggingface #deployment #llms

DiffusionGemma: Google's 4x speed boost trades autoregressive for diffusion

Google DeepMind just open-sourced a 26B MoE model that generates 256 tokens in parallel—4x faster on GPUs. It's lower quality than Gemma 4, but the architecture shift is fascinating.

#llms #diffusion #open-models #inference #google

JetBrains Mellum2: A 12B MoE Built for the Boring (But Critical) Parts of Your AI Stack

JetBrains just released Mellum2, a 12B-parameter MoE model that activates only 2.5B per token. It's not trying to be frontier—it's built for routing, RAG, and agent subtasks where speed matters.

#moe #code-models #inference #jetbrains #open-source

NVIDIA Nemotron-Labs Diffusion: The Speed-of-Light Text Generation Nobody Saw Coming

NVIDIA just open-sourced diffusion language models that generate multiple tokens in parallel at 6× the speed of autoregressive models—and they're actually good. Here's what changes.

#llms #diffusion #inference #nvidia #open-models

vLLM V1 and the Correctness-First Paradigm in RL

ServiceNow AI's deep dive into vLLM's V1 upgrade reveals why getting base correctness right matters more than chasing incremental RL gains—a lesson in engineering priorities.

#vllm #reinforcement-learning #inference #ml-infrastructure #correctness

Google's TPU v8 Splits Into Two: Training vs. Inference in the Agentic Era

Google just announced TPU v8, but instead of one chip, they're shipping two: v8T for training and v8I for inference. Here's why the bifurcation matters for AI's next phase.

#tpu #infrastructure #agents #google-cloud #inference

Hugging Face makes Transformers play nice with Apple Silicon—and it's kind of brilliant

Hugging Face just shipped MLX support in Transformers, letting you run models natively on Apple Silicon with zero code changes. It's the PR we all wanted to write.

#transformers #apple-silicon #mlx #inference #tooling

Loading…