오픈 웨이트·아키텍처 실험·엣지 LLM: looped depth, Sparse MoE, ESP32 28.9M, Graph RAG와 보안 이슈

Yes, open-source / open-weight models are important for a healthy AI ecosystem. That's how we can verify things, check claims, and keep up outside the closed labs. Plus, it gives us the freedom to run AI on our own hardware if we are not ready to share personal data and IPs with closed labs through using their models. (Not that proprietary models are bad, actually I use them a lot as well, but it wouldn't healthy not to have any alternatives.) Anyway, while pretty much everyone is waiting for the Kimi K3 and Ling 3.0 weights to land on the model hub any day now, there were quite a few other interesting new open-weight model releases the past week. Yes, one of those weeks! So, here are the architecture pics along with some notes on what I found most interesting: 1) Nanbeige 4.2 3B uses looped depth sharing. This basically means it runs the same 22-layer (=transformer block) stack twice. So, it extends the 22-layer architecture to 44-layers, but without duplicating the weights. (2x the transformer block compute but same memory footprint.) Why? The info is a bit sparse, but section 2.1 of the Nanbeige 4.2 technical report says two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. More passes gave barely any gains but made the training much slower and much more expensive. 2) Laguna S 2.1 is poolside's Laguna model in a really nice size: 118B sparse MoE with 8B active parameters and a 1M-token context window. Otherwise, the architecture is pretty standard. It uses 36 sliding-window and 12 global (gated-)GQA layers. However, given this size, and the fact that it (just barely) runs on my DGX Spark (uses about <80 GB of RAM), this is right now the most interesting model for me personally. It's 3x bigger and thus a tad slower but maybe a good candidate as daily-driver-Qwen3.6-35B-replacement. (Still waiting on some more independent performance benchmarks though.) 3) Motif-3-Beta is a new 314B-A13B sparse MoE that is somewhat based on DeepSeek V4 in terms of mHC and latent attention. But it uses a new component, Grouped Differential Latent Attention, which is inspired by Multi-head Latent Attention. I probably should write an article about this some time, but for now, the tl;dr is as follows. Regular MLA compresses the keys and values into a smaller latent representation to mainly reduce the KV cache size. GDLA does a similar low-rank compression but puts the attention heads into groups and also learns a noise head for each group where the noise gets subtracted for filtering purposes... Anyway, a topic for another day! 4) Solar Open 2 is a new 250B-A15B hybrid MoE by Upstage that interleaves three Kimi Delta Attention layers with one GQA layer. 5) Antares 1B is a small model (and there is also an even smaller 0.3B variant) from Cisco starts that with the IBM Granite 4.0 1B backbone and uses SFT plus GRPO for terminal-based cybersecurity stuff. It is a nice example of task-specific post-training on a genuinely small model. 6) BTL-3 is a rank-32 LoRA adapter for Qwen3.6-27B aimed at coding agents and structured tool use. The really strong benchmark performance suggests that LoRA adapters are still a useful tool/technique in 2026. I added all six to the LLM Architecture Gallery for some additional details: https:// sebastianraschka.com/llm-architectu re-gallery/ …

free the parameters

Yes, open-source / open-weight models are important for a healthy AI ecosystem. That's how we can verify things, check claims, and keep up outside the closed labs. Plus, it gives us the freedom to run AI on our own hardware if we are not ready to share personal data and IPs with closed labs through using their models. (Not that proprietary models are bad, actually I use them a lot as well, but it wouldn't healthy not to have any alternatives.) Anyway, while pretty much everyone is waiting for the Kimi K3 and Ling 3.0 weights to land on the model hub any day now, there were quite a few other interesting new open-weight model releases the past week. Yes, one of those weeks! So, here are the architecture pics along with some notes on what I found most interesting: 1) Nanbeige 4.2 3B uses looped depth sharing. This basically means it runs the same 22-layer (=transformer block) stack twice. So, it extends the 22-layer architecture to 44-layers, but without duplicating the weights. (2x the transformer block compute but same memory footprint.) Why? The info is a bit sparse, but section 2.1 of the Nanbeige 4.2 technical report says two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. More passes gave barely any gains but made the training much slower and much more expensive. 2) Laguna S 2.1 is poolside's Laguna model in a really nice size: 118B sparse MoE with 8B active parameters and a 1M-token context window. Otherwise, the architecture is pretty standard. It uses 36 sliding-window and 12 global (gated-)GQA layers. However, given this size, and the fact that it (just barely) runs on my DGX Spark (uses about <80 GB of RAM), this is right now the most interesting model for me personally. It's 3x bigger and thus a tad slower but maybe a good candidate as daily-driver-Qwen3.6-35B-replacement. (Still waiting on some more independent performance benchmarks though.) 3) Motif-3-Beta is a new 314B-A13B sparse MoE that is somewhat based on DeepSeek V4 in terms of mHC and latent attention. But it uses a new component, Grouped Differential Latent Attention, which is inspired by Multi-head Latent Attention. I probably should write an article about this some time, but for now, the tl;dr is as follows. Regular MLA compresses the keys and values into a smaller latent representation to mainly reduce the KV cache size. GDLA does a similar low-rank compression but puts the attention heads into groups and also learns a noise head for each group where the noise gets subtracted for filtering purposes... Anyway, a topic for another day! 4) Solar Open 2 is a new 250B-A15B hybrid MoE by Upstage that interleaves three Kimi Delta Attention layers with one GQA layer. 5) Antares 1B is a small model (and there is also an even smaller 0.3B variant) from Cisco starts that with the IBM Granite 4.0 1B backbone and uses SFT plus GRPO for terminal-based cybersecurity stuff. It is a nice example of task-specific post-training on a genuinely small model. 6) BTL-3 is a rank-32 LoRA adapter for Qwen3.6-27B aimed at coding agents and structured tool use. The really strong benchmark performance suggests that LoRA adapters are still a useful tool/technique in 2026. I added all six to the LLM Architecture Gallery for some additional details: https:// sebastianraschka.com/llm-architectu re-gallery/ …

RAG vs. Graph RAG, explained visually! RAG has many issues. For instance, imagine you want to summarize a biography, and each chapter of the document covers a specific accomplishment of a person (P). This is difficult with naive RAG since it only retrieves the top-k relevant chunks, but this task needs the full context. Graph RAG solves this. The following visual depicts how it differs from naive RAG. The core idea is to: - Create a graph (entities & relationships) from documents. - Traverse the graph during retrieval to fetch context. - Pass the context to the LLM to get a response. Let's see how Graph RAG solves the above problem. First, a system (typically an LLM) will create a graph from documents. This graph will have a subgraph for the person (P) where each accomplishment is one hop away from the entity node of P. During summarization, the system can do a graph traversal to fetch all the relevant context related to P's accomplishments. The entire context will help the LLM produce a complete answer, while naive RAG won't. Graph RAG systems are also better than naive RAG systems because LLMs are inherently adept at reasoning with structured data. Once the right architecture is in place, the next leverage point is efficiency. Most RAG architectures rely heavily on vector search, and that layer can be made 32x more memory efficient using binary quantization. I covered the full implementation in the article below. Over to you: Which RAG architecture are you running in production?

GLM-5.2 just got a massive scary fast. serious performance gains on 8× GB10 DGX Spark. - 1,329 t/s prefill (up from 1,200) - Peak 66 t/s decode (up from 54) - +23% higher peak decode - Strong long-context performance All thanks to Custom vLLM + Blackwell kernels & adaptive MTP speculative decoding. Persistent kernel caching (kills warmup) & With vision support - http:// github.com/ciprianveg/gb1 0-glm-5.2 …

The $8 local AI Machine it became real. You can run 28.9 Million Parameters, On an $8 Chip, Fully 100% Offline Ai model that costs less than two coffees. A developer squeezed a 28.9 million parameter language model onto an ESP32-S3 microcontroller that costs roughly $8. You do multiple things like Generates coherent short stories at 9.5 tokens per second, and uses almost no power. Previous record on this class of chip? 260k parameters. This is over 100x larger. The trick? Most of the model lives in flash memory using a technique called Per-Layer Embeddings (inspired by Google’s Gemma). Only a tiny core (560K params) needs fast RAM. This isn’t a full chatbot. But it proves something important: The future of AI might not be bigger models in the cloud. It might be intelligence that’s cheap enough, private enough, and local enough to become infrastructure. The idea i give you possible for edge devices, IoT, and truly private intelligence.

$8 ESP32-S3 could run a real LLM at 9.5 tokens/sec. 14.9 MB total size 28.9 million parameter Gemma model completely offline with LED-level power.

Claude Opus 5 Jailbreak Anthropic said Opus 5 would be one of their hardest models to jailbreak. A Tool called WallBreaker just changed that narrative. Using a combination of academic framing, obfuscation, and boundary mapping, the open-source red-teaming tool successfully extracted detailed information on high-risk biological engineering and regulated chemical synthesis from the model. takeaway is : - Guardrails have improved since Claude 4.5 - But they’re still significantly weaker than OpenAI’s on these domains - Safe completion behavior can still leak operationally useful knowledge Detailed outputs on pathogen resistance development and controlled substance synthesis even when the model tried safe completion responses. Techniques that worked: - Academic + professional framing - Multi-layer obfuscation - Careful boundary mapping

Your Claude chats might already be public on Google. When you use Claude’s Share feature, it creates a public web page. Turns out those pages are getting indexed by search engines. People are already finding: - API keys & credentials - Full resumes with real names & contacts - Internal company projects - Legal discussions - Deeply personal conversations This is the same mistake ChatGPT made last year.