소비자 GPU 추론 최적화, Claude 워터마킹, 오픈 모델 성능 경쟁

UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: https:// arxiv.org/pdf/2608.16157 repo: https:// github.com/FlashML-org/Fr eeToken … almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.

GLM-5.3 on KernelBench-Mega. Kimi-Linear Decode for RTX PRO 6000 at 21.4x the PyTorch baseline. GLM-5.2 was 11.1x and failed the single-launch gate. One CUDA `load_inline` launch. Persistent 512-thread CTAs. 30 grid barriers. Int4 dequant stays inside the GEMV. MLA is absorbed so the fat KV is not materialized. https:// kernelbench.com/mega solution file below!

A couple of days ago, I did a quick explainer on Claude’s new watermarking process and implementation. Since it’s such a popular topic and sparked such a lively discussion, I thought it might be interesting to go into a bit more detail when explaining how it works. So, instead of the usual text article, I recorded a little lecture on the topic (to change it up a bit from my usual articles). It ended up a bit longer than intended, but I hope it clarifies a lot of things: - Sampling the next token in an LLM and pseudorandom number generators - How watermarking relates to the regular LLM sampling process - Whether watermarking makes text "worse" - How to remove watermarks - Tournament sampling - How new text is checked for watermarks without rerunning the LLM I ended up with ~50 slides, but I hope that these explain it well, though! Happy watching!

DeepSeek just dropped a quiet bombshell. V4 Pro hits 90.5% on ARC-AGI-1 ($0.18/task) and 61.3% on ARC-AGI-2 ($0.60/task). At a fraction of the usual cost. Same level as its Flash version but still crushing the cost-performance curve.

A recent article by @SemiAnalysis_ looked at the progress of open models vs. closed models. This chart is interesting. I remember just a few years ago, before the agentic and reasoning era of LLMs, a substantial portion of the AI community believed it was nearly impossible for open models to catch up. Closed models maintained a substantial lead for so long. But in this new era it is no longer true!! Here's a link to the post, it's a good summary of the progress: https:// newsletter.semianalysis.com/p/are-open-mod els-catching-up …

The idea of late interaction has enormous staying power! CC @lateinteraction

Grok Bot now runs on its own computer. Free trial is available right now. Here’s the exact 20-minute setup that lets it handle your emails, follow-ups, research, and content while you rest.

Previously, I spent days making class materials with back-and-forth reviews. Now it takes only a few minutes—better than anything I could find on the Internet. Amazing era we have.