실시간 음성·에이전트형 영상·3D 월드 모델, 로컬 실행과 코딩 평가의 확장

Mark Zuckerberg

@finkd

10시간 전

Muse Voice Transcribe is MSL's first real-time audio perception model -- rolling out today. SOTA in streaming speech-to-text, it handles speaker diarization, and endpointing natively in a single model.

AI at Meta

@AIatMeta

10시간 전

Introducing Muse Voice Transcribe, the first real-time audio perception model from Meta Superintelligence Labs. Muse Voice Transcribe delivers real-time streaming ASR, diarization with 20+ speakers, and endpointing. It’s multilingual with seamless code-switching and improves accuracy with language, keyword, and context biasing. The model ranks first on @ArtificialAnlys streaming speech-to-text and on public diarization benchmarks.

Tired of high token costs for video workflows? Agentic video understanding gives Gemini the tools to dynamically search, scan, and inspect target video segments to achieve its goal. Building on agentic vision, Gemini now actively scans visual frames, audio, and transcripts instead of ingesting video at a static 1 FPS. This reduces token consumption by up to 88%, improves accuracy, and unlocks new ways to process long-form video. Tackle complex video workflows like: ▸ Sub-second retrieval: Catch split-second cuts missed at 1 FPS ▸ Long-form search: Query multi-hour videos without overflowing context ▸ Anomaly detection: Resample interesting windows at higher FPS ▸ Counting: Accurately track repeated actions and objects over time See how agentic video understanding transforms token efficiency in Gemini 3.7 Flash when compared with static video processing:

Agentic video understanding is supported by Gemini 3.7 Flash, 3.6 Flash and 3.5 Flash-Lite and is available today for video uploads and @YouTube videos via the Gemini API on @GoogleAIStudio and Gemini Enterprise Agent Platform. More details in the blog:

Justin Johnson

@jcjohnss

10시간 전

Today we sharing Atlas, our new multimodal world model. One reason this model is so special to me is that it combines two core visual intelligence tasks I have worked on for over a decade: generation and reconstruction

martin_casado

@martin_casado

9시간 전

An incredible accomplishment. Think of it as a video model with full camera control. And the scene remains (nearly) 3D consistent. Built on a fully internal base model. There are many use cases, from video editing, to 3D reconstruction to robotics. Great technical blog too.

Aravind Srinivas

@AravSrinivas

12시간 전

We're introducing hybrid compute for all users of the Perplexity Mac app. This will allow Computer to orchestrate local models that can run locally on Mac, particularly for agent steps involving sensitive and private files (eg your bloodwork, tax returns, litigation, etc).

Elon Musk

@elonmusk

11시간 전

Grok @Bot runs on its own computer in the cloud 24/7, so it doesn’t matter if you turn off your laptop

A research agent is only as good as the search layer behind it! I built a Deep Research Agent using LangChain Deep Agents with a search-first architecture designed around grounded, source-backed research. The agent plans the research in real time, breaks the work into smaller steps, searches across the web, reasons over the retrieved sources, and produces a fully cited answer. For this project, I used Liner as the search and grounding layer. Liner returns raw, structured search results with titles, URLs, descriptions, and dates. The agent then handles the reasoning and synthesis itself instead of relying on the search API to generate the final answer. That separation matters. Search is responsible for finding the right information. The agent is responsible for deciding what matters, comparing sources, connecting the evidence, and producing the final response. The workflow looks like this: Research → Plan → Search → Collect sources → Synthesize → Cite → Save report Every claim in the final answer is backed by a source from the retrieved results, and the full report is saved as a downloadable Markdown artifact. You can also inspect the entire run in the Deep Agents UI, including the research plan, search calls, retrieved sources, and final synthesis. Liner handles retrieval, while the agent keeps control over reasoning, synthesis, and how the final answer is constructed. Github Repo: https:// github.com/Sumanth077/Han ds-On-AI-Engineering/tree/main/ai_agents/deep_research_assistant …

Releasing SWE-bench Multimodal v2.0 today 480 tasks where a coding agent must interpret visual assets like screenshots, diagrams, recordings to diagnose and fix a bug in a repository.

We made SWE-bench Multimodal more than a year ago, but the instances were never public- now they are! 480 new tasks, JavaScript/HTML/CSS. Anthropic has used it in Mythos and Fable, and now you can too.

CLIP by hand ~ 13 steps walkthrough below CLIP, Contrastive Language-Image Pre-training, is OpenAI's answer to a question that sounds impossible: how do you put a sentence and a picture in the same space? CLIP shipped when OpenAI was still open, and those embeddings were shared far and wide. Almost every multimodal model you use today descends from them. How does it work? Goal: learn one shared embedding space for text and images. = 1. Given = A mini batch of three text-image pairs. OpenAI trained the original on 400 million. = 2. Text to vectors = Let us look up each word with word2vec. = 3. Image to vectors = We cut each image into two patches and flatten them. Now text and pixels are both just numbers. = 4. The other pairs = Repeat steps 2 and 3 for the rest of the batch. = 5. Encode = Let us push both sides through their encoders, a linear layer and a ReLU. In practice these are transformers, but the shape of the operation is the same. = 6. Mean pooling = We average across the columns, so each image and each sentence collapses to a single vector. = 7. Projection = The text vectors are 3D and the image vectors are 4D, so they cannot be compared at all. A linear layer projects both to 2D. That 2D space is the shared embedding space, and getting here is the whole point of the model. = 8. Prepare for matmul = Let us copy the text vectors down and the transposed image vectors across. = 9. MatMul = We multiply, which takes the dot product of every text vector with every image vector. Each cell is one estimate of how well a sentence matches a picture. = 10. Softmax, e to the power = Raise e to each cell. To keep it hand sized we approximate e with 3. = 11. Softmax, sum = Sum each row for image to text, each column for text to image. = 12. Softmax, normalize = Divide, and out come two similarity matrices, one per direction. = 13. Loss gradients = The targets are identity matrices: a pair that belongs together should score 1, every other cell 0. Subtract the target from the similarity and you have the gradients, in both directions. The takeaway: pairing a picture with a sentence comes down to a single dot product. Everything before step 9 is the work of getting them into one shared space, so that the dot product finally means something. Save this post!

Three ways you get in trouble when building an app: 1. Always calling the best model 2. Always calling the cheapest model 3. Always calling a mid model The first strategy will be very expensive. The second strategy is just dumb. The third strategy is just mediocre. The proper solution is to use a router and decide which model is best for a particular request. Check out these guys. Their platform lets you select a model and a specific reasoning effort for each request. This keeps the quality of the best models while costing much, much less.