웹 연결 에이전트 확산과 하네스 중심 평가, 실행 효율 경쟁

Greg Brockman

@gdb

6시간 전

you can now have chatgpt work securely sign into accounts:

Greg Brockman

@gdb

2시간 전

triggered tasks in chatgpt:

Vercel Developers

@vercel_dev

5시간 전

The WebMCP Challenge launches today. The first 1,000 builders get $20 in AI Gateway credits, and the top 10 projects get $3,600 in Vercel credits & $600 in AI Gateway credits. Let's ship ↓

OpenAI Developers

@OpenAIDevs

6시간 전

Ready to hack? Join us at the WebMCP Challenge kickoff livestream. We’ll talk through the WebMCP open standard, show off some builds, and cover prizes and how to hack with us.

Great paper on why agent leaderboard comparisons are hard to trust. It's on the hot topic of how much of an agent benchmark score actually belongs to the harness. The harness is the layer between the model and the task. It builds the context the model sees, mediates tool calls, validates outputs, and decides when to retry or stop. Every score comes out of a model and a harness together, but only the model gets reported. The authors ran a controlled grid to measure this. Three frontier models, three harness configurations, 100 tasks from SWE-bench Verified, with task order, execution environment, step budget, and evaluation script all held fixed. Swapping the harness moved GLM-5.1 by 13.0 points. Swapping the model inside a fixed harness moved scores by 3.0, 2.5, and 5.0 points. Harness-induced variance came out 7.8x larger than model-induced variance, and 6 of 9 model-pair comparisons flipped their ranking depending on which harness ran. Public leaderboards show the same thing. On SWE-bench Verified Mini, HAL reports a 34 point swing for Claude Sonnet 4.5 across scaffolds and nearly 48 points for o4-mini. They propose a Harness Card, a structured disclosure across seven layers, so you can tell whether a score gap came from the model, the harness, or the interaction. Paper: https:// arxiv.org/abs/2605.23950 Track more trending AI papers in our academy: https:// academy.dair.ai

// There Is No Neutral Harness // Great work discussing some of the issues in harness evaluation. Twelve open-weight models answer the same 3,679 items from ARC, HellaSwag, MMLU, and TruthfulQA under 26 equally defensible harness configurations. Items, weights, and greedy decoding stay fixed. Only option order, prompt wording, and whether the answer is read from generated text or per-option likelihoods change. gemma4-31b lands anywhere from 31% to 89% depending on the harness alone. On the items that two adjacent models both answer stably, the pair is tied. Config-fragile items carry 95.7% of the gap between them, and four of the twelve models reach rank one under some configuration. Item discrimination, the property benchmark-compression methods maximize when picking a representative subset, correlates with fragility. Compressed benchmarks are selecting for the items most sensitive to configuration. Paper: https:// arxiv.org/abs/2608.21382 Track more trending AI papers in our academy: https:// academy.dair.ai

BREAKING: I'm partnering with @SpaceXAI to give all Lenny's Newsletter annual subscribers a free month of Grok Bot (included in Cursor Pro+) This is the first time @SpaceXAI has offered a deal like this to anyone, and I'm thrilled to make this amazing product accessible to more people. If you're already a paid subscriber, grab your deal here (search for "Grok"): https:// lennysproductpass.com If not, subscribe here and look for the Product Pass link in your welcome email: https:// lennysnewsletter.com/subscribe I've been hooked on Grok @Bot since before it came out, and my usage has only gone up. Seriously, it's really really good. Some of my favorite use cases right now: + After I record a podcast, taking a first pass at key takeaways and promotion ideas + Automatically adding school events to the calendar + Triaging support emails (saves me hours!) + Suggesting things I can do to be happier by analyzing my emails, calendar, and Slack + Landing me great IMAX Odyssey tickets Grab your free month of Grok Bot here (search for Grok): https:// lennysproductpass.com

Elon Musk

@elonmusk

4시간 전

Simple & useful Grok @Bot

Claude Code 2.1.246 has been released. 61 CLI changes Highlights: • Added a tool to launch dedicated agents for complex multi-step tasks, enabling isolated subagent workflows • Auto mode tab added to /permissions to view and edit auto-mode classifier rules, enabling finer control • Increased safety-check timeout for auto-mode tool calls to prevent temporary denials for large sessions Complete details in thread ↓

Claude Code CLI 2.1.246 changelog: New features: • Added a startup warning for Bash allow rules with a wildcard before the subcommand (e.g. Bash(git * main)), since they also match options inserted before the subcommand • Added an Auto mode tab to /permissions for viewing and editing auto mode classifier rules • Added the turn's completion time to the end-of-turn duration line, e.g. ✻ Sautéed for 23s · done 6:05 PM Fixes: • Fixed fullscreen mode showing a blank transcript after resizing the terminal and jumping to the bottom until the next keypress • Fixed a severe transcript slowdown when a diff contained a very long single line (e.g. a base64 string); such lines now render truncated with a marker • Fixed erratic fullscreen scrolling when positioned at an earlier message, including jump-to-bottom getting stuck mid-transcript • Fixed background sessions failing to open after 45 seconds when Claude Code's starting directory had been deleted, the machine had slept, or the host is slow to start processes • Fixed background sessions failing to open with "Couldn't start the background service … EACCES" when another Claude Code process was re-installing the npm package at that moment • Fixed markdown rendering being disabled for a whole message when its first 500 characters contained no markdown, and for +/N) lists and setext headings • Fixed MCP tool calls interrupted by an incoming message in headless/remote sessions being reported to the model as "completed with no output" instead of an explicit interrupted error • Fixed MCP tool arguments being sent as JSON strings when the parameter's schema is empty ({}), instead of their real type • Fixed a command interrupted mid-run showing as "Ran 1 shell command" with no sign it was cut • Fixed pressing ← or running /background during a dynamic workflow restarting its finished subagents; it now asks first and says how many subagents would restart • Fixed opening a just-started session in claude agents while its worker was still booting (common on Windows) stopping it with "was stopped while the respawn was in flight" • Fixed claude agents listing a backgrounded named session twice; backgrounding the same conversation again now numbers the new row (e.g. my-session (2)) • Fixed the background retention sweep removing git worktrees under .claude/worktrees/ that you created yourself when an old background-session record pointed at them • Fixed auto mode tool calls being denied as "temporarily unavailable" on very large sessions by scaling the safety-check deadline with prompt size • Fixed the plugin cache creating duplicate SHA-named directories for the same plugin • Fixed plugin skills whose frontmatter name already includes the <plugin>: prefix showing it doubled in the slash menu (e.g. /plugin:plugin:skill) • Fixed claude plugin update failing for an installed plugin given its bare name (only the fully-qualified name worked) • Fixed plugin installation failing when plugin.json was saved with a UTF-8 byte-order mark (BOM) • Fixed /reload-plugins reporting 0 skills for plugins that define skills under skills/*/SKILL.md • Fixed hook error messages showing a literal ＄{CLAUDE_PLUGIN_ROOT} instead of the resolved plugin path • Fixed /rename replacing the theme's prompt border color (including a custom theme's promptBorder) with the default cyan; the border now keeps your theme's color unless you pick one with /color • Fixed custom theme diff colors (diffAdded/diffRemoved and their dimmed variants) being ignored in diffs and the /theme preview • Fixed a keybindings.json binding with an unknown action name silently deadening that key; it is now skipped so the default binding keeps working, and a warning is logged under --debug • Fixed /stats activity heatmap showing each day's activity one cell off (Sunday's count under Monday) in timezones east of UTC • Fixed /fork from an already-forked or backgrounded session starting the new session with an empty conversation • Fixed prompts beginning with /-- (e.g. Lean doc comments) being rejected as an unknown slash command instead of being sent to Claude • Fixed the @ file picker staying open after the typed text stopped matching a real path • Fixed the status line's cost and duration resetting to zero after navigating to the agents view and back • Fixed fullscreen mode moving keyboard focus onto the control under the pointer when you clicked the terminal window only to bring it back into focus • Fixed path completion failing when the completion token or working directory contained a null byte • Fixed the UI stopping with a render error on the first tool call when a third-party Anthropic-compatible endpoint (ANTHROPIC_BASE_URL) streams a tool_use block without an id • Fixed the Write tool reporting "Out of memory" or freezing for a long time after overwriting a very large existing file, even though the file had been written • Fixed claude plugin install <name> exiting silently (or hanging in a terminal) instead of reporting an error when ~/.claude/plugins/known_marketplaces.json is empty or corrupted • Fixed resumed sessions failing every turn with a 400 when the saved history contains tool blocks the Anthropic API does not accept (typically written by a third-party API proxy) • Fixed curl -fsSL https:// claude.ai/install.sh | bash failing with "Raw mode is not supported" for some Team/Enterprise users with server-managed settings • Fixed sessions that ended in plan mode resuming outside plan mode in the VS Code extension, and in claude -p --continue/--resume with a permission prompt tool, when no permission mode was set • Fixed the Notification hook not firing while the sandbox "Network request outside of sandbox" permission prompt is waiting • Fixed Bash permission checks to always require approval for malformed commands with a dangling && or || operator • Fixed --strict-mcp-config sessions prompting to approve .mcp.json servers they would never load, which left background sessions waiting at startup • Fixed telemetry and metrics requests to Anthropic carrying the API key configured for a third-party gateway (ANTHROPIC_BASE_URL); a credential is now only sent to its own host • Fixed a visible API error on the first prompt after idle when apiKeyHelper returns short-lived JWTs: an expired cached token is now refreshed before sending, and 401/403 auth errors retry quietly • Fixed memory growing with session length in the fullscreen and Ctrl+O transcript views: each rendered message row no longer retains a full copy of the transcript-wide tool lookups • Fixed /ultrareview runs and cloud sessions launched at the same time from one repository (e.g. from several worktrees) sometimes starting with another launch's uncommitted changes • Fixed the task progress count (e.g. 3/5) shown for background cloud sessions such as /autofix-pr occasionally missing a task • Fixed Remote Control sessions keeping their placeholder name in http:// claude.ai and the Claude app until the second prompt; the auto-generated title now appears after the first prompt • Fixed MCP tools marked requiresUserInteraction still offering "Yes, and don't ask again" in their permission prompt; the option wrote an allow rule the tool then ignored • Fixed the self-hosted runner ending its live sessions or exiting when a work-poll response is malformed (e.g. an intercepting proxy's HTML page); it now retries the poll • Fixed the command sandbox's filesystem configuration not respecting --setting-sources Improvements: • Improved /cd: the new directory's project settings, hooks, .mcp.json servers (behind the usual approval prompt), skills, and agents now take effect right after the move instead of on --resume • Improved Bash tool latency on bash shells by replaying snapshot functions without a base64 subshell per function • Improved subagent results: a subagent that stops at its maxTurns limit now returns its output marked as partial, with a hint to continue it via SendMessage, instead of appearing finished • Improved non-interactive sessions (-p, SDK, cloud sessions) to automatically continue a response cut off mid-stream by a server error, connection loss, or stall instead of ending with an error • Improved attribution of usage telemetry to your organization for workload identity federation sessions, events sent while apiKeyHelper runs at startup, and after a login token expired while idle Other changes: • Windows/macOS: Fixed headless sessions not cleaning up stale entries in ~/.claude/sessions left by sessions that exited uncleanly • Changed /code-review so Claude can also start it on its own on Bedrock, Vertex AI, and Foundry, through the Claude apps gateway, and when telemetry or non-essential traffic is disabled • /goal: Changed idle sessions to start at most three check-ins on long-running background work per goal; your next message allows three more • Changed claude install and claude update to defer a pending managed-settings consent prompt to the next interactive session instead of prompting mid-command • Changed OpenTelemetry plugin events for plugins synced from http:// claude.ai: plugin_id_hash now reflects the plugin's real marketplace, and enabled_via is admin-install for admin-installed plugins Source: https:// github.com/anthropics/cla ude-code/blob/main/CHANGELOG.md#21246 …

“Speculative Programmatic Tool Calling” Code agents waste time waiting for the LLM to finish generating code before tool calls can start. So this paper speculatively executes predictable tool calls from partial generations, while a shadow REPL tracks dependencies safely. This provides ~1.2x lower wall-clock time without changing the model or agent logic. https:// alphaxiv.org/abs/2608.spec- ptc …

NVIDIA AI

@NVIDIAAI

6시간 전

When an LLM engine crashes, a cold restart can mean minutes of lost capacity. Shadow engine recovery, a new preview feature in NVIDIA Dynamo, keeps a standby engine warmed up and ready to take over. In our GLM-5.2 test, it restored capacity in 7.3 seconds, nearly 39x faster than a cold restart. Read all about it: https:// nvda.ws/45OhcA0

Granite 4.2 8B is live on CoreWeave Serverless Inference on day 0. Reasons natively, calls tools on its own and you do not have to worry about infra to manage. @IBMResearch built it for enterprise agents. We keep it served. https:// utm.io/usqJY

Love seeing real-world agent benchmarks like this MiniMax-M3: $0.018 to spin up an inbox, write and send a business email end-to-end. The cheapest model on the board - and it finished the job. This is what efficient intelligence looks like. (Yes! We’re a frontier lab - we don't just build video and music models, we also train and open-source our frontier LLMs for you to use.)