단일 GPU 추론 통합, 에이전트 탐색 폭 확장, 초저비트 양자화

Massive breakthrough here! Researchers built a new AI inference engine that: - reduces self-hosting costs by ~4x - runs a full agentic pipeline on one GPU - serves 20+ architectures, not just LLMs And it's fully open-source. Here's the core problem with engines like vLLM that it solves: Most agent pipelines today run 4-5 small models under the hood: - an embedder for retrieval - a reranker for precision - an extractor for entities - and an LLM for generation The standard way to serve them is one server per model. vLLM serves the LLM, TEI serves the embedder, and everything else gets a custom FastAPI wrapper. Each server reserves its own slice of GPU memory and holds it whether traffic arrives or not. GPUs are billed by the hour, so idle time costs the same as busy time. This is why switching to small models rarely reduces the bill. The cost is never in the calls but rather in maintaining the servers. The structural fix is serving every model from one process that loads and evicts models based on traffic. Superlinked open-sourced a new inference engine that does exactly that. SIE (Superlinked Inference Engine) is an Apache 2.0 server that runs 85+ models behind one API. Four calls cover the whole pipeline: - encode() returns vectors - score() returns relevance scores - extract() returns entity spans - and generate() runs small open LLMs. Models load on first request and are evicted least-recently-used, so one GPU serves a rotating set of models instead of sitting siloed behind one. It runs anywhere from a laptop to a Kubernetes cluster, and it plugs into Qdrant, Weaviate, Chroma, LanceDB, LangChain, and LlamaIndex. GitHub repo: https:// github.com/superlinked/sie (don't forget to star ) To dive deeper, my co-founder also wrote a hands-on guide to using SIE. It walks through a real-world example where you can learn to deploy and manage five models on a single GPU, including a small language model, an OCR model, an NER model, a reranker, and an object detector. Read it below.

Test-time scaling has two axes: running agents over longer timeframes (depth), and running a larger number of agents (breadth). Everybody knows about the first axis, but the second one is just as important when solving hard problems that require broad search.

Congrats to @deepseek_ai on DeepSeek-V4-Flash-Vision-Exp, the first multimodal model in the V4 family! vLLM serves it now. A vision encoder and aligner on the 285B/13B V4-Flash MoE backbone, with text-agent scores holding steady. https:// recipes.vllm.ai/deepseek-ai/De epSeek-V4-Flash-Vision-Exp …

Congrats to @SparkLLM on releasing Spark-X2.5-4B and 1.7B! Compact on-device agent models: 200+ languages, native 1M context. Both have day-0 vLLM support via Spark's out-of-tree general plugin.

Batching strategies in LLM inference, clearly explained! (bookmark it) - Static - Dynamic - And continuous batching I wrote a detailed article explaining how each works and why serving LLMs is a different problem from traditional ML inference. The article is quoted below.

1.5TB → 214GB. Seven times smaller, barely a dent. That's Hy4 preview. The trick is Sherry — our quantization method that packs weights down to 1.25 bits each. (The image shows how.) It also unlocks a new way to run it: stitch the GPUs you already have across machines, and

“Fast Weight Attention for Continual Learning” Fast-weight models try to make attention cheaper by continuously rewriting a small fixed-size memory. This paper shows that this rewrite should behave more like online learning from what the model just predicted to what actually