재귀형 에이전트 확장, LLM 캐시 계층화, 코드 기반 그림 학습과 Qwen 성능 경쟁

Teknium 🪽

@Teknium

1일 전

Today I've been putting Hermes Agent to the ultimate test of dramatically cleaning up the Hermes Agent repo. One /goal and it's been working for ~15hrs now and has simplified, unified, optimized, removed, or in some way cleaned up the repo enough to shed 375,000 lines of code. it's had waves of ~120 subagents, 3x sets of 15 subagents with all of those subagents able to spawn their own subagents (yes, recursive subagents all the way down) for almost the entire time - and my single desktop machine has handled it like a champ. Pretty epic test of the limits of both the latest gen models and Hermes Agent Stay tuned for more!

1320 subagents lived and died in this session today

Four caches in LLM serving, clearly explained: Every LLM request reads the whole prompt and computes attention state for every token in it. This step is called prefill, and it impacts both the input bill and the time before the first token appears. In an agent loop, most of the prompt comprises text that the model already processed in the previous turn. There are four cache layers that prevent paying for the processed tokens at each turn. ↳ The KV cache holds the key and value tensors for every token at every layer, for one active request. ↳ Prefix caching keeps those tensors on the server instead. vLLM stores them in 16-token blocks and identifies each block by a hash that chains in the previous block's hash, so a block only matches if everything before it matched too. The scheduler stops at the first miss and prefills the suffix from there. ↳ Prompt caching is the same reuse that runs on a provider's hardware, with a price sheet attached. Anthropic charges 1.25x the base input rate to write an entry and 0.1x to read it. ↳ Semantic caching works differently. It embeds the incoming prompt, runs a similarity search over stored prompts, and returns a stored answer outright when the score is above a threshold. That's why it saves output tokens as well as input tokens. It's also why every request pays for an embedding round trip, including every miss. The first three match on exact tokens and cannot change what the model produces. This technique matches on similarity, which means it is quite susceptible to generating a wrong response since embeddings may match to a wrong prompt. The diagram below depicts all these techniques. To use these techniques, you don't need to build a custom serving stack. The transformers library already implements the cache as an object of KV vectors that you can preserve, so you can prefill a corpus once, retain the returned tensors, and reuse them across queries in about ten lines. And this KV cache is only one of four separate caching layers in an LLM stack. The other three are prefix caching on the server, prompt caching billed by a provider, and a semantic cache that skips the model entirely. I wrote a full breakdown of all four caches in LLM serving that you should know as an AI engineer, with code for each. Read it below.

full write-up is out with everything open > the env, the hand-rated pool, the 3 runs and every painting they made https:// huggingface.co/blog/train-to- paint-with-code …

If you want to learn how to engineer AI agents, these are the people you’d want to learn it from. They know the space inside and out and couldn’t be more excited to teach it. First time it’s running at Stanford this fall. @Diyi_Yang , @michaelryan207 , @jyangballin

From 2.4T parameters to 27B — innovation at every scale. Proud to see Qwen models pushing boundaries in both performance and accessibility. Thanks @arena for the in-depth technical review! 🚀 #BuildWithQwen API access · Qwen Cloud: https:// click.qwencloud.com/m/20000001102/ · Model Studio: https:// click.alibabacloud.com/m/20000001093/ #Qwen #AlibabaCloud #QwenCloud

Thanks for the shoutout, @arena ! Hitting #1 on WebDev and leading across categories reflects our deep focus on complex coding and multi-step workflows. Ready to experience the power of 2.4T parameters yourself? More API access： • Model Studio: https:// click.alibabacloud.com/m/20000002957/ • Qwen Cloud: https:// click.qwencloud.com/m/20000002949/

防衛AIに不可欠な「レジリエンス（回復力）」とは何か。核抑止の歴史がAIガバナンスに示すものとは。本編でさらに詳しく語っています。 インタビュー本編はYouTubeからご覧いただけます👇 https:// youtu.be/NIS8nqwI7C4 🐟