LLM 추론 경로 분리, 필터 검색 그래프, Agent 중심 실행 환경

7 parameters that control every LLM response. An LLM produces one token at a time by sampling from a probability distribution over the whole vocabulary. These seven parameters are what decide how that sampling happens. 1) Max tokens A hard cap on how many tokens come back. It cuts generation off at the limit rather than asking the model to wrap up, so a low value gives you a truncated sentence, not a shorter answer. 2) Temperature Scales the scores before they become probabilities. Below 1 sharpens the distribution toward the most likely token, above 1 flattens it so unlikely tokens get a real chance, and 0 makes the output effectively deterministic. 3) Top_p Keeps the smallest group of tokens whose probabilities add up to p, then samples only from that group. It adapts to the model's confidence, staying narrow when one token dominates and widening when the model is unsure. 4) Top_k Keeps the k most likely tokens and discards the rest, no matter how the probability is spread. It is a fixed size cut, so it can still admit weak tokens even when the model is confident about one answer. 5) Frequency penalty Lowers a token's score in proportion to how many times it has already appeared. This is the one that stops the same phrase from coming back every second line. 6) Presence penalty Lowers a token's score once it has appeared at all, with no scaling by count. It pushes the model toward vocabulary and topics it has not touched yet, which reads as broader coverage rather than less repetition. 7) Stop sequences A list of strings that halt generation the moment they are produced. This is how you keep a model from writing the next turn of a dialogue it was only supposed to answer. Three different jobs are hiding in that list. Temperature, top_p and top_k reshape the distribution before a token is picked. The two penalties edit scores based on what has already been written. Max tokens and stop sequences decide when to quit. Temperature and top_p both widen the same pool, so tuning both at once makes the effect hard to attribute. Move one, hold the other. Value ranges for each are in the visual. The prompt decides what the model is trying to say. These decide how it picks each word while saying it. I wrote a full first-principles breakdown of what happens inside an inference call, prefill, decode, KV cache and all. The article is quoted below.

A good technical LLM interview question: Your LLM chatbot takes 12s before it generates the first token, and the users are complaining. So you move the model onto a GPU with 3x the computing power. The time to first token barely improves. Why did this happen? (answer below) Latency in an LLM app is a placement problem disguised as a model problem. If you profile the 12 seconds, the model's prefill itself may only account for around 1.5 seconds of it. So halving the prefill step saves just 750ms out of 12000, which is under 7%. The rest is spread across stages that never touch the GPU. The request first travels to whatever region the app runs in, and a cross-continent round trip could cost over a second before any code executes. Then the request handler starts. On a container-based serverless platform under load, this adds several seconds of cold start, paid before auth, rate limiting, or prompt assembly even begins. Retrieval adds its own hop, and the response streams back across the same distance. Optimizing a stage that was already fast cannot alter the latency that's majorly affected by other stages. Those other stages are slow for a structural reason. An LLM app runs two workloads that want opposite machines. - The request path is short, spiky, and needs to sit close to users - Inference is long-running, GPU-bound, and billed hourly, whether requests arrive or not. So the actual decision is not which model to run, but where each of these two workloads runs. There are three options, each with its own tradeoffs: > A dedicated GPU box removes inference cold starts, but it bills around the clock and lives in one location, so distant users wait out the round trip on every request > Container-based serverless scales to zero, but the request path pays a cold start, and most of these platforms have no GPU behind them. > Edge runtimes start in under a millisecond, because a WebAssembly module carries no OS or container image to boot. They handle the request path well and cannot hold a model. So the answer is not to pick one, but to split the app across two of them. The request path runs close to users, and inference runs on a dedicated GPU it calls into. That also explains the failed upgrade. More compute made a stage that was already fast faster, and left the 10.5 seconds around it untouched. To actually learn how it's done in practice, Akamai's GitHub has a reference implementation for each half. - vllm-on-lke serves Qwen2.5-7B-Instruct behind an OpenAI-compatible endpoint on one RTX 4000 Ada GPU in Linode Kubernetes Engine, with Terraform creating the cluster, both firewalls, and the GPU operator in one apply. - akamai-functions-llm-chatbot covers the front, where a WebAssembly API checks a KV cache and only calls the GPU-backed instance on a miss. Both are available on Akamai’s new Developer Hub, alongside their tutorials and code samples. It also links to Edge Case, their Discord, where four developer advocates architect and deploy a production app live every other Wednesday. If you create a new Akamai Cloud account, you can also get $300 in credits for joining. Join here: https:// fandf.co/4w7id0z That said, this post treats generation as a single 1.5s block, but that block has its own structure, and knowing it well tells you whether a model is slow to start or slow to stream. I wrote a first-principles walkthrough of it, covering the prefill and decode split, KV caching, and where the time actually goes inside each one. Read it below. Thanks to Akamai Cloud for partnering today!

ACORN fixes filtered search at query time. Qdrant's filterable HNSW fixes it in the index. Filters can break an HNSW graph into disconnected islands, leaving search stranded and recall behind. ACORN works around that at query time by stepping through filtered-out neighbors. Filterable HNSW takes a different approach: it adds edges between points that share an indexed payload value, so the filtered graph remains connected. On a 1% filter over 1M vectors, filterable HNSW reached 99.8% recall at 1.0ms. ACORN reached 67.7% at 4.7ms. ACORN still earns its place for gaps like broad values and AND filters. But it's most effective when it has a graph built for filters underneath it. Full benchmark: https:// qdrant.tech/articles/filte red-vector-search-acorn/ …

ANOLISA (Alibaba Cloud Linux 4 Agentic Edition) is an agent-first operating system designed for AI Agents. Tokenless is its context-efficiency layer — stripping redundant tool outputs at the OS level before they hit the model. No framework changes. Tested with Claude Code — same workflow, fewer tokens. How it works: OS-level proxy: Removes null fields, debug noise, oversized outputs. Compact rewriting: Verbose JSON → tabular notation, same data. Reversible: Dropped content retrievable via markers when needed. Deploy on Alibaba Cloud ECS — full setup guide in the docs: https:// click.alibabacloud.com/m/20000002445/ Or run it yourself: open source at https:// github.com/alibaba/anolisa — give it a Star if you like it.

Kimi Work for Financial Analysts - Tutorial #2 Use Kimi Work for 3 common investment research tasks: - Build a live investor dashboard - Update financial models in spreadsheet - Process and generate reports in batch Stay tuned for more Kimi Work workflows!

Hyra keeps pushing the frontier. Since our last math update, four more advances have landed: 3D Blaschke–Lebesgue: raised the universal lower bound for constant-width bodies from 0.380799w³ → 0.411040w³ (now 97.9% of the conjectured Meissner optimum). ∫ Beurling–Ahlfors: improved the best uniform (L^p) coefficient from 1.575 → 1.523958, moving closer to Iwaniec’s conjectured sharp bound. ± Partial Hadamard matrices: extended asymptotic counting from the cubic scale down to the near-quadratic regime. [,] Operator commutators: improved the cost of approximating the identity from Tao’s O(log⁵(1/ε)) to O(log³), narrowing the gap toward the known logarithmic lower bound. We are making open-source models better at solving open problems! Check the results: