micromodels

LLM inference suffers from a problem that can't be solved by more compute. And that is the data bottleneck when delivering all of your previous input tokens to the GPU to predict the next token.

IBM recently released a new one nanometer node, with transistors that are .7nm, or 7 angstroms wide. We won't see chips with transistors this small until maybe 2030, but it is an innovation that seems unique to IBM.

Quick front-warning: I am not an expert in either the math or architecture for LLMs. It is something I am interested in, but I am not an expert in either.

Sakana Fugu released today and I think it is an exciting frontier of LLMs. A quick description of Sakana Fugu is a model that behaves like a single model, handling routing to the correct frontier model.

Subquadratic released a model today, SubQ 1.1 Small, claiming near-frontier performance with a context window of a staggering 12M tokens. The API and plans are still in closed early-access, so don't get your hopes up for personal long context tasks.

My ideas on what could be the future of distributed LLMs.