오픈 웨이트 확산과 Grok·Opus 경쟁, Kimi K3의 delta attention

Mark Zuckerberg

@finkd

1달 전

Open source is a positive and important force for both empowering people and preventing centralization. Proud to support this.

Elon Musk

@elonmusk

1달 전

Next month, every line of code touching the 𝕏 system will be open source and third-party audited. Only total transparency deserves trust.

Grok is now available directly inside Google Workspace One add-on brings Grok into Google Sheets, Slides and Docs, where it works beside the file you already have open In Sheets, Grok can: • Read selected ranges and explain the data • Cite the exact cells behind its answers • Write and fill real spreadsheet formulas • Update values and rebuild financial scenarios In Slides, it can: • Generate a complete presentation from an outline • Add new slides that match the existing theme • Restructure entire sections • Rewrite titles so every slide communicates a clear takeaway In Docs, it can: • Draft directly inside the document • Turn rough notes into formatted sections • Correct grammar and rewrite content • Apply consistent headings and styles across the file Everything happens through a side panel, without constantly copying information between tabs or rebuilding the output manually One installation works across all three apps, and organizations can deploy it across their entire workforce Grok is rapidly becoming a native intelligence layer for the tools people already use every day

Just over 6 months later, Opus 5 now produces near-superhuman level spreadsheets and slide decks that match what a consultant would make. Things are changing fast.

one matrix replaced the KV cache. (the technique is 100% open source) Kimi just dropped K3, an open model at frontier scale. it leans on a new mechanism called delta attention that does not keep a growing KV cache. that is how it holds a million tokens of context without the memory blowing up. before we can understand delta attention, we need to understand attention itself. it is a lookup. every token stores a key, which works like an address, and a value, which is the content at that address. to build its output, a token sends out a query, matches it against every key in the sequence, and pulls back a blend of the values whose keys matched. that is the picture at the top of the diagram. standard attention keeps every one of those key and value pairs as a list, one entry per token. that list is the KV cache, and it grows with the sequence, so each new token has to scan the whole thing to build its output. double the context and you double both the storage and the scanning, which is where the quadratic cost and the ballooning memory come from. delta attention keeps the lookup but throws away the list. the entire past collapses into one fixed-size matrix that still behaves like the lookup table. hand it a key, it returns the value the past associated with that key. one thousand tokens or one million, the matrix stays the same size. the hard part is writing into it. you cannot staple a new pair onto a fixed matrix, because new writes land on top of old ones and blur them together. the delta rule solves this in two moves per token, shown at the bottom of the diagram. → it reads before it writes. it hands the matrix the new token's key and sees what value the memory currently returns, its existing guess for that address. → it writes the difference, not the value. it compares that guess against the value it actually wants stored and writes only the gap. that gap is the delta, and it corrects the old association instead of stacking a new one on top. the matrix also lets old entries fade over time, so a fixed size can keep absorbing a long sequence without filling up. that is the shift the diagram draws. standard attention remembers by keeping everything and pays quadratic cost to re-scan it. delta attention remembers by rewriting one matrix and pays linear cost. you give up something real. a compressed matrix cannot store every token exactly, so recall of any single token becomes approximate. that is why production models interleave the two, a few full-attention layers for exact lookup and the rest running linear. the difference is one verb. standard attention appends, delta attention corrects. read more: https:// kimi.com/blog/kimi-k3 for anyone who wants the full picture of KV caching, i have written a detailed article, which is quoted below. it explains how KV caching works from first principles, with illustrations for every step.

Your Pet is ready to meet other builders. On ChatGPT web, create a shareable link for your custom pet and send it to a friend to adopt.

Tibo

@thsottiaux

1달 전

Show me your pet

Perplexity Developers

@perplexitydevs

1달 전

The Perplexity CLI is now available, giving coding agents the ability to search the web. Copy this to your agent to get set up: "Read: https:// github.com/perplexityai/a pi-platform-developers/blob/main/skills/pplx-cli/SKILL.md … and install this skill."

Teknium

@Teknium

1달 전

Introducing our implementation of IronProxy into Hermes Agent. Keep your secrets safe, even from your own agent.