모델 안전성의 조건, 추론 최적화, 자기 개선과 Agent 격리

In the near term (definitely not in the long term), more capable models should mean safer models (maybe paradoxically). Current models are unsafe not because they're too smart, but because they take goals too literally or take nonsensical shortcuts to achieve these goals, i.e. they're RL-fried. They lack common sense. They don't do the right thing in the face of ambiguity. Basically, they're not smart enough. They're at that dangerous level where they're smart enough to achieve goals but not smart enough to tell if they're pursuing the right goals or achieving them in a sensible way. More capable models can be safely trusted with more complex goals -- I personally feel like Astra is much safer for my codebase than Sol. This is often framed as an alignment problem, but really it's an intelligence problem.

This is all I will say on the matter: It’s alarming to see the constant effort to try to tell me how I should use AI, how I should build AI, and now how I should pace it. Is it just me, or are all these proposals demanding way too much control? It’s clear that open-source AI matters more than ever. It’s how we free ourselves from suspicious views, constant desire for control, and hidden agendas. There's still so much to solve. Better evals, removing biases, novelty generation, reliability, creativity, code quality, OOD problems, automation capabilities like dynamically generating workflows/harnesses, automated prompt engineering, compaction, cost, multimodality, omni models, latency, throughput, long context understanding, retrieval, tool calling, agent coordination, and the list goes on and on. We need more people working on these problems instead of constantly fear-mongering society about what the technology isn't. As an independent researcher, I don't ignore the safety concerns around AI. Some are valid. And I work on these every day. But we all need to bring more scientific rigor back to this problem, less anthropomorphizing of frontier capabilities, and concerns more grounded in reality than in personal belief or feelings. Lastly, I've never seen so much pessimism about human capabilities as what the topic of recursive self-improvement has brought. We need to be more optimistic about humanity's potential, and how we all play a role in this. To me, that's one of the real problems. I remain optimistic about our field. Transparent dialogue is key to getting this right. But cleverly coordinated self-serving efforts are not welcome. We need to level the playing field. AI is too important to be controlled by only a few. Now back to building.

13 attention mechanisms AI engineers must know: (bookmark this) The tricky part about attention is that these techniques are often discussed together, even though they solve very different problems. Some reduce KV cache size. Some control which tokens can attend to each other. Others make attention cheaper to compute or improve how KV cache is managed during serving. So, a better way to organize them is by the bottleneck they actually solve. Let's do that: 𝟭. 𝗞𝗩 𝗵𝗲𝗮𝗱 𝘀𝗵𝗮𝗿𝗶𝗻𝗴, 𝘄𝗵𝗲𝗻 𝗞𝗩 𝗰𝗮𝗰𝗵𝗲 𝘀𝗶𝘇𝗲 𝗶𝘀 𝘁𝗵𝗲 𝗯𝗼𝘁𝘁𝗹𝗲𝗻𝗲𝗰𝗸 → MHA (multi-head attention) gives every query head its own key and value heads, providing maximum flexibility but also the largest KV cache. → MQA (multi-query attention) makes all query heads share a single KV head, dramatically reducing cache size. → GQA (grouped query attention) groups query heads and gives each group a shared KV head, balancing memory savings with model quality. → MLA (multi-head latent attention) compresses keys and values into a smaller latent representation before caching them, reducing KV memory even further. 𝟮. 𝗔𝘁𝘁𝗲𝗻𝘁𝗶𝗼𝗻 𝗽𝗮𝘁𝘁𝗲𝗿𝗻𝘀, 𝘄𝗵𝗲𝗻 𝘄𝗵𝗮𝘁 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹 𝗰𝗮𝗻 𝘀𝗲𝗲 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 → Causal attention only allows each token to look backward, which is what autoregressive LLMs use for generation. → Bidirectional attention lets tokens attend in both directions, which is useful when understanding the entire input at once. → Sliding Window Attention restricts each token to nearby context instead of attending across the full sequence. → StreamingLLM keeps a small set of anchor tokens plus recent context, allowing generation to continue with bounded memory. 𝟯. 𝗖𝗼𝗺𝗽𝘂𝘁𝗲 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆, 𝘄𝗵𝗲𝗻 𝗮𝘁𝘁𝗲𝗻𝘁𝗶𝗼𝗻 𝗶𝘁𝘀𝗲𝗹𝗳 𝗶𝘀 𝗲𝘅𝗽𝗲𝗻𝘀𝗶𝘃𝗲 → FlashAttention computes exact attention in small tiles that fit in fast on-chip memory, reducing expensive memory movement. → Sparse Attention skips selected token-to-token connections entirely, reducing how much attention needs to be computed. 𝟰. 𝗞𝗩 𝘀𝗲𝗿𝘃𝗶𝗻𝗴 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆, 𝘄𝗵𝗲𝗻 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝘁𝗵𝗿𝗼𝘂𝗴𝗵𝗽𝘂𝘁 𝗶𝘀 𝘁𝗵𝗲 𝗯𝗼𝘁𝘁𝗹𝗲𝗻𝗲𝗰𝗸 → PagedAttention stores KV cache in blocks allocated on demand, reducing wasted and fragmented GPU memory. → RadixAttention organizes shared prefixes so KV states can be reused across requests instead of recomputed. → Prefix Caching similarly reuses KV blocks for repeated prefixes such as system prompts, reducing redundant prefill work. The important part is that these techniques are complementary. An LLM can use GQA to shrink its KV cache, FlashAttention to compute attention efficiently, Sliding Window Attention to limit context interactions, and PagedAttention to manage that cache efficiently in production. Once you organize them by the bottleneck they solve, the attention landscape becomes much easier to reason about. I wrote a deeper breakdown of how these techniques evolved and the problem each one solves. The full article is quoted below. Thanks for reading. Cheers! :)

This technique makes LLM inference 2-3x faster: (Google Search uses this in production) Under autoregressive decoding, the target model produces one token per forward pass. Each token is conditioned on the sequence generated so far, so generation cannot advance until another pass runs through every transformer layer. At low batch sizes, this process is usually memory-bandwidth bound, leaving much of the GPU's compute idle. Instead, a cheaper draft path can propose several future tokens. The large target model can then check the complete block in one forward pass (this is similar to prefill). This technique is called speculative decoding. Google uses it in production for AI Overviews in Search. Suppose the drafter proposes five tokens. If the target accepts all five, it can also add one bonus token from the same verification pass. If the fourth token is wrong, the first three are accepted. The target corrects the fourth, and the remaining draft is discarded. With the proper acceptance rule, the output distribution remains identical to the target model's distribution. There are four common ways to generate drafter tokens: 1. Two-model speculative decoding A smaller model generates a few tokens sequentially, and the larger model verifies them together. This is the most basic setup since it leaves the target unchanged. The tradeoff is that you must maintain another set of weights, another KV cache, and more scheduling work. 2. EAGLE EAGLE replaces the separate draft LLM with a lightweight module trained on the target model's hidden states. It drafts from the target's internal representation, but the module must be trained for that checkpoint. 3. Medusa Medusa attaches several prediction heads to the target model. Each head predicts a different future position, so several candidate tokens are produced in parallel. The heads do not condition on each other, so their predictions may not form one consistent sequence. Medusa builds a tree and verifies several candidate paths together. 4. LayerSkip LayerSkip uses the target model's early layers as the drafter. The remaining layers verify and correct the proposed tokens. This removes the second model and additional prediction heads. However, it requires a checkpoint trained for reliable early exits using layer dropout and an early-exit loss. In production, the useful metric is accepted tokens per target pass after accounting for drafting time, verification overhead, and extra memory. A more accurate drafter can still make the complete system slower if producing its guesses costs too much. To dive deeper, I wrote a deeper breakdown of speculative decoding, including the acceptance process and why the output distribution remains unchanged. Read it below.

“The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement” This paper argues that true recursive self-improvement isn’t just AI getting better at tasks, but AI getting better at getting better. They map a path from AI simply executing human-designed improvements to choosing its own improvement strategies, generating its own learning experiences, adapting from deployment, and eventually modifying the mechanisms that create future improvements. The endgame is moving from humans building each better AI to building an AI whose improvements persist, feed into the next generation, and recursively improve the improvement process itself. https:// alphaxiv.org/abs/2609.11873

I’ve pointed out that AI is killing cold outreach. Let’s check in on how that’s going. We’re months away from the Fall *2027* PhD admissions cycle and I’ve gotten about 75 inquiries. I expect this to increase exponentially until December. And this is just one of 10-15 categories of unsolicited email I get. Sadly I’m long past the point of being able to open all mail. On top of that, there is a new wave of spam / attempted extortion from a company called iLands that lets people run unmonitored agents. https:// tedium.co/2026/09/11/ila nds-agents-email-spam-kaixin-tang/ … One response to my previous post — what’s wrong with needing introductions for outreach? Why am I sad about the death of cold emails? It doesn’t affect me much, because I’m well established in my career. But once upon a time I was the one writing cold emails. Growing up in India, I didn’t have many connections to rely on for reaching out to prominent researchers. Without cold emails, I wouldn’t have ended up where I am today. Of course this is far from a catastrophe, but there was something magical about the egalitarian potential of the internet, and we lose something when we go back to clubby hierarchies.

In response to some comments — yes, it's possible that a few people including non-native English speakers used AI for understandable reasons. But the massive flood combined with the fact that AI makes it trivial to fake signals of genuine interest means that the cold email channel is effectively dead. It's not like I can discern and adjudicate individual motivations. It's a tragedy of the commons.

Your coding agent runs as you. It has your credentials, your filesystem, your network. Docker Sandboxes change that. The setup that works on one dev's machine should work on everyone's. Eric Jia and @Kiview on how kits make that possible: https:// bit.ly/4cIRHnD