CacheBlend의 문서 단위 캐시와 에이전트 설계 논쟁, AMD Helios 출하

Prompt caching only works in one direction. (3 ways it breaks, 1 fix for all of them) If your system prompt and tool definitions are stable, prompt caching is the single highest-leverage optimization available today. Cached input tokens get up to 90% cheaper, and hit rates of 60 to 85% are realistic. But it comes with one rigid rule. The cached portion must be an exact, byte-for-byte prefix of the new request. Change a single character in that region and you get a full cache miss. That rule breaks in three situations you hit constantly: → RAG with multiple documents. You cached document A alone and document B alone. A query now needs both. Document B's cached state was computed without any awareness of A, so it's invalid and gets recomputed from scratch. → Document order changes. The same three documents appear in a different order across requests. Every permutation is a cache miss, even though the content is identical. → Growing conversation history. Each new turn changes everything after the stable prefix, so earlier cached states beyond it become useless. Alibaba Cloud's production data shows how bad this gets: 10% of KV cache blocks serve 77% of all hits. The rest sits in storage and never gets reused, because prefix matching won't allow it. CacheBlend, a research paper from the LMCache team (EuroSys 2025 Best Paper Award), attacks exactly this. The insight is that in modern transformers, tokens overwhelmingly attend to their own local context. Only a small fraction of tokens carry real connections across document boundaries. So instead of recomputing everything after the first cached document, CacheBlend reuses every document's cache as-is and selectively recomputes just those few boundary tokens. Those are the small orange fixes between documents in the diagram, and they are the entire cost. The result is 2 to 4x faster processing on multi-document queries with no quality loss. The order problem disappears with it. Shuffle the same documents however you like, and every permutation stays cached, where prefix caching recomputes all of them every time. The bottom of the diagram shows that side by side. That's the real shift: from caching prefixes to caching knowledge. Every document in your knowledge base becomes a reusable cached asset, regardless of what order it appears in or what sits next to it. CacheBlend ships inside LMCache, the open-source cache management layer that runs outside the inference engine and integrates with vLLM, SGLang, and TensorRT-LLM, on both NVIDIA and AMD GPUs. Check it out on GitHub: https:// github.com/LMCache/LMCache (don't forget to star ) I wrote the full breakdown of the architecture, including why cache management should never live inside your inference engine. The article is quoted below. Stay tuned for more on this!

The tokenmaxxing culture is absolutely out of hand. You don't need an agent for everything. In fact, I'm 99% sure you don't need an agent for 99% of the problems you need to solve. Good engineering principles still apply. Simplicity still wins. A regular, boring ol' script that works every time is 1000 times better than a fancy agent that works sometimes. We need common sense back.

At Interrupt, @Coinbase Engineering Manager Evan Kormos took the stage to share how their developer support team built a self-improving agentic system on top of self-hosted LangSmith. Full session: https:// youtu.be/py9d6zTl4Dc

Join us for an upcoming webinar on “Why does my RAG-agent get worse? A live autopsy” As collections grow, documents change, chunks get duplicated, and retrieval pipelines evolve, Agent's quality can decline without an obvious breaking point. @DylanCouzon from Qdrant and Rishabh Hada from Future AGI will build, debug, and fix RAG agents through a single end-to-end workflow. Qdrant powers the retrieval layer, while Future AGI will trace and evaluate the agent throughout the same workflow. What we’ll cover: - Build a RAG agent and observe where the agent is losing quality - Migrate to a new embedding model with zero downtime - Remove duplicate and fragmented chunks from the collection - Add late interaction reranking - Compare evaluation scores before and after each change to verify what actually worked Date & Time: Thursday, August 6 at 6:30 PM CEST | 9:30 AM PDT | 10:00 PM IST Register for the webinar now: https:// luma.com/future-pq6q

AMD is launching Helios, its first rack-scale AI system built to rival NVIDIA, with shipments starting later this year. Microsoft will deploy the racks in its Azure data centers, joining Meta, OpenAI, and Oracle as customers.