Perspectives on AI/ML from the trenches.https://micromodels.com/https://micromodels.com/news/dflash-decoding/https://micromodels.com/news/dflash-decoding/LLM inference suffers from a problem that can't be solved by more compute. And that is the data bottleneck when delivering all of your previous input tokens to the GPU to predict the next token. Repeat this process for every token, and you are limited by the data transfer rate of your GPU. That's where speculative decoding shines. Speculative decoding is predicting multiple token ahead of time, so you can run cheaper forward passes through your LLM to simply verify that the predicted tokens are correct. These are still done with autoregressive methods, and DFlash "solves" this problem. Dflash is a large language diffusion model that predicts blocks of tokens at a time, leading to much faster inference on the same GPU. The core of DFlash is a diffusion model. Originally designed for image generation, a diffusion model learns to predict coherent data from a noisy input. When training an image diffusion model, you place standard images into your model, slowly add noise to the images, and let the model learn to predict the original images from the noisy inputs. Given labeled images, you can apply this process to have a model learn what noise applies to each label. To create a new image, you start with pure noise, and let the model predict an image that fits your label. That's why most generated images look pretty similar and can be spotted with a careful eye. This process of training a diffusion model applies to LLMs as well. Given a previous history of tokens (like your query), can you turn a block of noise (the next 256 tokens) into a coherent sequence of tokens? That's what DFlash does, and it does it well. Now, your main LLM only needs to verify that the predicted tokens are correct, rather than generating them from scratch, leading to drastic increases in speed. Using Qwen 3 4B and Qwen 3 8B, DFlash was able to attain speedups of 2.5x-8x over standard autoregressive decoding, across a variety of benchmarks. They even showed that DFlash can be used to increase the speed of reasoning models, which might increase the speed of consumer level LLMs in the near future.Thu, 02 Jul 2026 00:00:00 GMThttps://micromodels.com/news/ibm-1nm-node/https://micromodels.com/news/ibm-1nm-node/IBM recently released a new one nanometer node, with transistors that are .7nm, or 7 angstroms wide. We won't see chips with transistors this small until maybe 2030, but it is an innovation that seems unique to IBM. With how much compute is necessary to train these LLMs, packing more compute in an equally sized chip is going to be great. The bottleneck in production of chips this small is from ASML, who makes the technology to produce the chips. The current frontier ASML machine, the high NA EUV TWINSCAN EXE:5000, will only be able to print down to 8nm in resolution. For reference Intel's 14A process has an back end of line pitch of 20-22 nm, while this new IBM stack drops it to below 18nm. IBM made this implementation by making a new nanostack architecture, where transistors scale vertically as well as horizontally. This builds on the ideas from gate all-around transistors, and should scale like how we saw FinFET and Gate All Around transistors scale. I am excited to see where this leads, but I have my doubts in the yield rates for these future processes. Intel's 14A nodes reportedly suffer from low yields of less than 60%, while mature processes often exceed 90%. If ASML can scale down and TSMC can scale up, we are in for a good extension of Moore's law.Thu, 02 Jul 2026 00:00:00 GMThttps://micromodels.com/news/mamba-3/https://micromodels.com/news/mamba-3/Quick front-warning: I am not an expert in either the math or architecture for LLMs. It is something I am interested in, but I am not an expert in either. The Mamba 3 paper could be the future of sub-quadratic LLM inference. Nvidia Nemotron already uses some Mamba2 layers with the classic transformer layers. The pitch of Mamba 3 is that it is designed inference-first. Most earlier sub-quadratic models (including Mamba 2) were designed training-first, and that left decoding memory-bound. Your GPU spends time moving memory around instead of computing. The Mamba 3 paper frames itself as a fix for exactly that, and the results are good enough to pay attention to. Three things are new in Mamba 3, and I will try to explain them like I understand them: - A more expressive recurrence by a discretization trick. The math is over my head, but the practical effect is that the recurrence itself can represent more than Mamba 2's recurrence could. - Storing the state in a complex value format, very similar to RoPE in transformers. Hopefully this makes an improvement in state tracking, which was a failure point in previous state space models. - Multi input multi output, which allows the state space model to apply the recurrence to a vector at a time instead of a state at a time. This increases training cost slightly, but gives the model more representational power. The numbers from the paper are decent. At the 1.5B scale, Mamba 3 improves downstream accuracy by about 0.6% over the next best sub-quadratic model, which is Gated DeltaNet. The MIMO variant adds another 1.2%, for a total gain of about 1.8%. Mamba 3 also matches Mamba 2 perplexity with about half the state size, which matters because state size is a big chunk of memory for these models. The big practical claim is that Mamba 3 decodes at roughly half the cost of a comparable transformer, while staying competitive on quality. That is the part that matters for inference-heavy workloads like agentic coding, long-context retrieval, and reasoning models. Nvidia has been shipping hybrid Mamba-Transformer models for a while, and we could likely see them ship Nemotron 4 with these Mamba3 layers. In my mind, these layers plus DFlash could release a very powerful combo of efficient inference in the near future.Thu, 02 Jul 2026 00:00:00 GMThttps://micromodels.com/blog/tiny-moe-from-scratch/https://micromodels.com/blog/tiny-moe-from-scratch/My ideas on what could be the future of distributed LLMs.Mon, 22 Jun 2026 00:00:00 GMThttps://micromodels.com/news/sakana-fugu/https://micromodels.com/news/sakana-fugu/Sakana Fugu released today and I think it is an exciting frontier of LLMs. A quick description of Sakana Fugu is a model that behaves like a single model, handling routing to the correct frontier model. I can see it as a future, where Anthropic or OpenAI release a similar product that routes your request to a specific LLM based on the complexity of the query. The performance of Fugu-Ultra really does seem to be innovative, with reportedly Mythos level performance, but I will need to see some third party benchmarks before I truly believe the claims. I am excited to see where this API level integration will go with teams of agents, but I will not subscrribe the the $20/month plan yet. The pricing follows standard LLM API rates, with \$5/Mtok input and \$30/Mtok output with increased rates for context larger than 272k tokens. There are also Subscription plans from \$20 a month to \$200 a month, with more usage the more you pay.Mon, 22 Jun 2026 00:00:00 GMThttps://micromodels.com/news/subquadratic-release/https://micromodels.com/news/subquadratic-release/Subquadratic released a model today, SubQ 1.1 Small, claiming near-frontier performance with a context window of a staggering 12M tokens. The API and plans are still in closed early-access, so don't get your hopes up for personal long context tasks. SubQ 1.1 Small appears around the Sonnet 4.6 and GPT-5.4-mini level of performance but is exceedingly good at needle in the haystack problems. I understand that Subquadratic can't release the math behind their Subquadratic Sparse Attention (SSA), but I would love to see the math behind this model. If open source models like DeepSeek or Qwen got their hands on this kind of technology, we could see a revolution in local LLMs, running ultra-long context LLMs for personal use. In the end, I would like to see how it fares against RAG, RLMs, and other long-context solutions for both needle in the haystack problems and reasoning over twelve million tokens worth of context.Mon, 22 Jun 2026 00:00:00 GMThttps://micromodels.com/news/openai-erdos-problem/https://micromodels.com/news/openai-erdos-problem/Apparently, an internal OpenAI model has disproved a discrete geometry problem that mathemeticians have spent the last 80 years studying. The problem is deceptively "simple": If you place n points in the the plane, how many pairs of points can be exactly distance 1 apart? The view of mathemeticians for the past 80 years has been a square grid is optimal for maximizing the number of pairs, but GPT took another approach. The LLM devised an "infinite family of examples that yield a polynomial improvement". I can't say I understand the math behind the conjecture, or what GPT did, but this feels like a breakthrough with LLMs that I have not seen before. If an LLM can tackle an abstract math problem better than 80 years of leading mathematicians, what will it do next?Wed, 20 May 2026 00:00:00 GMT