DFlash Speculative Decoding

LLM inference suffers from a problem that can’t be solved by more compute. And that is the data bottleneck when delivering all of your previous input tokens to the GPU to predict the next token. Repeat this process for every token, and you are limited by the data transfer rate of your GPU. That’s where speculative decoding shines.

Speculative decoding is predicting multiple token ahead of time, so you can run cheaper forward passes through your LLM to simply verify that the predicted tokens are correct. These are still done with autoregressive methods, and DFlash “solves” this problem. Dflash is a large language diffusion model that predicts blocks of tokens at a time, leading to much faster inference on the same GPU.

The core of DFlash is a diffusion model. Originally designed for image generation, a diffusion model learns to predict coherent data from a noisy input. When training an image diffusion model, you place standard images into your model, slowly add noise to the images, and let the model learn to predict the original images from the noisy inputs. Given labeled images, you can apply this process to have a model learn what noise applies to each label. To create a new image, you start with pure noise, and let the model predict an image that fits your label. That’s why most generated images look pretty similar and can be spotted with a careful eye.

This process of training a diffusion model applies to LLMs as well. Given a previous history of tokens (like your query), can you turn a block of noise (the next 256 tokens) into a coherent sequence of tokens? That’s what DFlash does, and it does it well. Now, your main LLM only needs to verify that the predicted tokens are correct, rather than generating them from scratch, leading to drastic increases in speed.

Using Qwen 3 4B and Qwen 3 8B, DFlash was able to attain speedups of 2.5x-8x over standard autoregressive decoding, across a variety of benchmarks. They even showed that DFlash can be used to increase the speed of reasoning models, which might increase the speed of consumer level LLMs in the near future.