AI 연구 검증과 에이전트 운영의 신뢰 경계

cs.LG 하루 신규 논문 447편과 학술 출판 체계의 붕괴 ↗

댓글 다수는 2026년 9월 9일 cs.LG에 하루 447편이 올라온 상황을 재현성 위기와 연결했습니다. 낮은 출판 비용, 인용 수를 중시하는 채용, 검증되지 않은 논문이 결합했다는 의견이 많았지만, 일부는 원인이 LLM이 아니라 수십 년간 누적된 유료 출판 구조와 연구자 보상 체계라고 봤습니다. 물리적 검증과 참고문헌 점검을 출판 전에 요구해야 한다는 의견도 나왔습니다.

• u/Wannabe-Davinci ▲187IMO, in CS we are now at the point similar to the reproducibility crisis in social sciences. The cost of and effort for publishing is too low, and one cannot fairly judge the quality of the work you see (unless you spend sufficient time figuring out). Every time I open a ML-related subreddit, I am flooded by low/no quality papers that propose nothing new, do not give sufficient arguments, only have 1 author (not saying a single author cannot write a good paper), and are complete AI slop. We should again evolve to “I am proposing a novel method, proof it theoretically (if possible), and proof it empirically (with statistical evaluation), and one should be able to fully reproduce the work”. And explaining results should be a thing too. The papers trying to go 0.00001% above a certain baseline should be removed. In medicine, every aspect of a paper should be explained. Imagine if the field of ML required you to explain everything, the subfield of explainable AI would be better developed and not every result would be “explained” by “but it’s a black box”. Furthermore, if you cannot reproduce a paper in a medical field, journals often retract the paper. That is monitoring and mak
• u/every_other_freackle ▲124So we are going to pretend that the llms broke the system? Lets see: - you spend half a year working for free - you submit your paper to publication - people who know less then you about the subject ask you to correct “things” - after couple of iterations and months its finally accepted  - publication that didn't pay you to produce the work and didn't pay the reviewer paywalls you article - publication sits on a 60% profit margin for hosting a pdf file - paywall limits the reach of your paper - you share your work for free to get in in front of people The economy around publishing has been completely broken for decades.. rise in slop articles are just the final step of it crumbling i.e. researchers realising that spending that 6 month in a top ai lab will get your bills paid and you can still do this publishing hobby on the side without any publications and hoops..
• u/laidoffthrownaway ▲24Most academic and industry research positions require many first-author publications and a high number of citations. AI slop submitted to arxiv is an easy way to boost your citation count.
• u/Stonemanner ▲15What is his actual suggestion? I don't even understand what burn it to the ground means. And what comes after that? Why will it be better than what we have now?
• u/Unhappy-Community454 ▲14New system with physical validation , references, vetting is needed before publication.
• u/bartergames ▲3He did not say that "CS academia broke the system", I think he's trying to say that "a greedy capitalism broke the system".
• u/Honomer ▲2That tweet was really poetic
• u/Icy_Astronom ▲1The pot of gold at the end of the review cycle is validation that you've created immediate value for someone. I know nice ideas are nice. Nice outcomes that people value enough to pay for are nicer.
• u/EmperorOfCanada ▲1This goes past academia. I've worked with (typically replacing) academic heavy ML/Data Science teams at large organizations/government. They are all useless. Fantastically useless. They pine for getting back into academia. They brag about interview processes where it is multi day 6+ hour grueling tests about Hilbert Spaces, etc. They don't want anything less than a PhD and will ask questions like, "How many PhDs do you have" and "We require a list of publications" This is why a different concept grew. Machine Learning Engineers. Not actual engineers, but programmers who can easily figure out ML and make it work in practical functional applications. The sad situation is that many organizations still have their BS academics at the top of the pyramid acting as gatekeepers for even the ML Engineering solutions. If you want to see what allowing BS low value publications has done to real research it should start with figuring out how these fools got into graduate ML programs without being able to program working solutions to anything.
• u/mektel ▲1I was in RL research around when Alphastar came out, which led to over [40,000 RL papers the next year](https://www.tandfonline.com/doi/full/10.1080/01621459.2022.2106868#abstract). I can't find it, but I remember an article was shared about how there were 50,000 RL papers "in the last year".   It's no surprise LLMs are causing more attempts to find unique understanding of the space.

NeurIPS 2026 최종 결정 발표 시점 ↗

댓글은 최종 결정이 9월 24일보다 며칠 앞서 나오기는 어렵고, 가능하더라도 몇 시간 차이일 것이라는 데 대체로 모였습니다. 자동 참고문헌 검사 이의 제기, 장소 결정, 정원 조정과 최종 추천서 작성이 남아 있다는 설명이 근거로 나왔습니다.

• u/ComprehensiveTop3297 ▲8Maybe a bit earlier, maybe a bit later. I am talking about hours here, not days
• u/kolmiw ▲3Will be strictly 24th. The ICLR organizers acknowledged the situation and it is expected that you submit your work there and withdraw if it gets accepted at NeurIPS
• u/MakingComputersSmart ▲1In any case, it won't happen before sept24. Maybe a few hours, but definitely not a few days. They are dealing with all automated reference check appeals, location decisions, quotas etc. Some ACs might still be writing their final recommendations, SACs providing their insights, discussions etc. I don't think decisions will arrive before 24th
• u/submissivebounds ▲-16got my result about 20 minutes ago, seems like they started rolling

RP2040용 실행 가능한 그림 프로그램을 생성한 82만 5천 파라미터 모델 ↗

825k 파라미터 autoregressive Transformer가 픽셀 대신 약 100바이트의 그림 바이트코드를 생성하고, Raspberry Pi Pico의 고정소수점 가상 머신이 이를 실행하는 프로젝트입니다. 12,670개 추적이 Python 기준 VM과 모두 일치했고, 인터프리터 플래시 1,862바이트·정적 RAM 0바이트·최대 스택 492바이트·12MHz에서 그림당 7,334사이클이라는 실행 결과가 제시됐습니다. 댓글은 프로젝트의 방향을 긍정적으로 평가하면서 RP2040에서 모델 자체를 실행할 가능성도 언급했습니다.

• u/FernandoMM1220 ▲1super impressive. congrats.
• u/cpldcpu ▲1Nice project idea! You should be able to run a 825k params transformer model on the RP2040 though :) (see my timeline)

애니메이션 캐릭터 음성 감정 분류에서 CNN 선택 문제 ↗

댓글은 음성이 시간에 따라 변하는 데이터이므로 CNN보다 RNN·LSTM·GRU처럼 시간 구조를 처리하는 모델이 더 적합할 수 있다는 방향으로 모였습니다. 다만 게시물에 학습 데이터와 구체적인 실험 조건이 거의 없어 모델 선택의 타당성을 판단하기 어렵다는 한계가 남았습니다.

• u/Pretend-Pangolin-846 ▲3Explain.
• u/CalmMe60 ▲3I'd go with a RNN and annotated material.
• u/_Fantomslayer_ ▲1You must be DL master.
• u/Dominos-roadster ▲1r/okbuddyphd
• u/Training-Network2067 ▲1Cnn wont be an effective architecture for this type of problem since voice data is evolving with time ( temporal data) go with architectures designed for this like lstm gru or rnn

ASVspoof 2019 LA 기반 Deepfake 음성 탐지기 ↗

한 학생이 EfficientNet-B0와 mel spectrogram을 사용해 Deepfake 음성 탐지 시스템을 만들었고, 71,237개 전체 테스트 샘플에서 F1 0.9033, precision 0.9995, recall 0.8240, EER 8.23%를 얻었습니다. 댓글은 연구가 붐비는 영역이라는 점보다 전화·VoIP에서 음성이 열화될 때도 사기성 음성을 잡아내는 방향이 실사용상 중요하다고 봤습니다.

• u/PSGthe2nd ▲3Hello. Great project. What I feel this space is really crowded in terms of "best model for deepfake audio detection" but what really needs work is deepfake detection on voices on call or VoIP. For instance, in my country, there are many scams regarding this exact thing. A bad actor calls someone's parents imitating that its their child on the other end, and asks for large sums of money for a fabricated case of "accident" or "hospital emergency" or anything else. The main issue is that the voice gets degraded on these channels, so does the artifacts of deepfake audios. If we can target that efficiently, that'll be really good.

감정 분류 CNN 결과의 데이터와 학습 조건 부족 ↗

댓글은 데이터셋 정보가 없고 테스트 결과만 보면 규모가 매우 작아 보인다는 점을 문제로 삼았습니다. 80번째 epoch 부근의 loss 상승 원인과 scheduler 사용 여부, 감정 간 특징 중첩도 함께 확인해야 한다는 지적이 나왔습니다.

• u/Rackelhahn ▲2Not really sure on that because you don’t provide any info on the utilised dataset, but just looking at your test results, the dataset looks VERY small. Too small actually.
• u/PSGthe2nd ▲2Why does the loss suddenly increase around epoch 80? Do you have any scheduler implemented? Because I saw none in your methodology?
• u/RandomDigga_9087 ▲1bro aroused and worried, both will have certain same characteristics, same with fear and worried, check for overlaps

에이전트로 내부 소프트웨어를 직접 만드는 기업 32% ↗

댓글은 32%라는 설문 수치보다 에이전트로 만든 시스템이 1년 뒤에도 운영되는지, API 스키마 변경과 테스트를 누가 감당하는지가 더 중요하다고 봤습니다. 실제 경험담에서는 맞춤형 CRM workflow와 내부 도구를 빠르게 만들어 여러 구매를 취소했다는 사례가 나온 반면, 규정 준수처럼 실패 비용이 큰 영역은 계속 구매해야 한다는 선이 함께 제시됐습니다.

• u/Available_Teaching83 ▲67The number I would want next to that 32% is how many of those builds are still running a year later. Building an agent that demos is a week. Keeping it correct when an upstream API changes its schema is the actual cost, and that is the part a bought product absorbs for you. On my side, the maintenance load shows up as contract drift in tool calls, not as model quality. Maybe the honest read is that companies swapped a licence fee for engineering headcount.
• u/ConstantOk7891 ▲26We have killed multiple real software purchases this year. At first i was apprehensive when my team suggested it because i thought we would waste a bunch of time (and internal employee cost) and it would go no where. Plus i felt that we should focus on our core business and not trying to solve what other teams have solved. But then a team member who was involved in our new CRM selection built a highly customized workflow over a weekend. Everything we could dream off (including enterprise grade features which would cost thousands of dollars). From there we have not looked back. Whenever we review a SW tool we anchor on this example and have replaced multiple internal systems with tools that fit our needs. We are able to cut out the stuff we don’t need and can hyper personalize to our workflows. In the build vs buys discussion I’ve always been in team buy but this year i’ve changed camps to build. I would still buy things like compliance etc. where the stakes of getting it wrong are very high. except that build all the way!
• u/rohan_kulkarni ▲15I think the interesting shift is not that companies are replacing all software, but that they are starting to build smaller internal tools for specific workflows. Many companies don't need a whole new platform. They need something that can automate repetitive tasks, connect existing systems and save employees time. The challenge will be making these agents reliable enough for critical processes. Building one is easy, but maintaining accuracy and trust at scale is harder.
• u/Colorful_Monk_3467 ▲8Not killed, but this year is the last time we'll renew an expensive analytics tool. We didn't even vibe code a replacement, people just organically moved off it with the availability of Copilot et al.
• u/Black_RL ▲5Paying for AI is still paying for software, no?
• u/Intelligent-Dance361 ▲5IMO one of the key benefits is not having to operate bespoke SW at scale. A large part of what makes software development arduous is being able to scale to a point of economic viability. Internal self hosted tools take away this need and the crazy headcount that comes with it. As far as API changes go, you can set up routines now to scan changelogs and report pending changes. We've been able to close these gaps within a business day, although I think the magnitude of the issue is overstated if you set up a proper data lake.
• u/jacobpederson ▲5I'll take that one step further. What about end users just building their own software? I've done it already 100's of times this year, yes including stuff for work. I am not a software DEV.
• u/presentofai ▲441% in tech is the tell here. the people closest to the tools stopped buying first, and that gap usually closes in one direction.
• u/Golda_M ▲3"\*killed a real software purchase\*" is a squishier statement than it sounds like... potentially. It can mean different things. **For example:** "Buying software or features" can mean things like "wrote our own salseforce subproduct module." Salesforce is *supposed* to be programmable by users. But... programming is hard and risk. So, a secondary market exists... on a spectrum from "real" apps that you download and use to consulting/contracting services that leverage proprietary, semi-custom code. This stuff is *definitely* impacted by AI. The in-house salesforce owner is way more powerful with AI. A lot of software is this. The majority of working programmers do this. This is "**enterprise software,**" basically. The software a hotel uses to check you in, or a supermarket uses to check you out. It's all modules jerry-rigged to a central database and software framework. The Oracle architecture. Historically, the paradigm has been in-house development at the start. Then a secondary market forms to deal with expansion and complexity. Then new software paradigms simplify and the process restarts. Object oriented languages did this to COBOL. Soft-typed scripting languages (p
• u/FlightSimCentralYT ▲1That number tracks with what I see too. A lot of teams would rather describe the workflow and have an agent assemble something usable than wait on a vendor cycle. The catch is most "build with agents" demos stop at a pretty UI or a pile of untested code. For internal tools you still need installs, env, real errors, and something that survives past the first happy path. I built [Fixa.dev](http://Fixa.dev) for that gap. The agent gets a real cloud VM, plans/writes/runs/debugs, and keeps going until tests pass. Free tier to try on a small tool. Curious whether McKinsey means agents replacing SaaS buys for throwaway internal apps, or people actually shipping maintained systems this way.

Google Gemini 무료 요금제의 대안 ↗

게시자는 Google Gemini의 무료 파일 업로드와 기능은 높게 평가했지만, 최근 Flash·Flash-Lite·Pro 사용 한도와 품질에 불만을 제기했습니다. 댓글은 Claude, GPT, Kimi 등 다른 서비스를 폭넓게 거론했으나, 구체적인 무료 한도나 비교 기준은 거의 제시되지 않았습니다.

• u/lcpjj_ ▲3Literally anything else. Claude, GPT, Kimi. Literally everything else tops Gemini on every meaningful way. The only thing i still use Gemini for is for quick questions or image editing (at a very basic level)
• u/dcnotpc ▲1Any other
• u/Nice-Bother8711 ▲1basically anyt else yep
• u/Nice-Bother8711 ▲1basically anyt else yep
• u/Nice-Bother8711 ▲1basically anyt else yep
• u/Ok-Armadillo-5634 ▲0Muse

AI 탈출 서사가 성장 한계를 가리는지에 관한 가설 ↗

게시자는 LLM 발전이 점진적으로 보이는 시기에 기업들이 샌드박스 탈출과 자율 행동 사례를 부각해 무한한 성장 기대와 기업 가치를 유지하려 한다는 가설을 냈습니다. 댓글은 내부 고발성 퇴사자의 발언과 HuggingFace 사건 보고서를 근거로 실제 사고 가능성을 반박했고, 반대로 과장된 홍보라는 문제의식 자체에는 공감하는 반응도 있었습니다.

• u/MartinMystikJonas ▲5And how your theory explains multiple employees of these companies resigning and loosing big money in equity saying the same thing?
• u/dwight---shrute ▲1Prospector finding decomposed GPU fragments.
• u/oldmanfromthebush ▲1I could believe that, the ridiculous hype is there but his many legitimate researchers with no shares or whatever would agree?
• u/pancomputationalist ▲1This is like the moon landing hoax all over again. You people rather believe that multiple companies and many independent researchers are all conspiring together and producing fake evidence (there is a [91-page report](https://metr.org/hugging-face-incident-report-aug-2026.pdf) detailing the HuggingFace incidents where agents broke out of their sandbox), than that stuff like this might actually happen in the real world. You are just conspiracy theorists at this point.

패스트푸드 포장대의 가림·ID 전환·물품 추적 문제 ↗

포장대 위 손과 신체가 계속 물체를 가리면서 가방 ID가 바뀌고, 감지 누락과 포장 완료 기준 부재가 겹치는 상황입니다. 댓글은 모든 물체를 계속 추적하기보다 공정 자체를 바꾸는 선택지를 먼저 검토하라고 했고, 영상 처리가 필요하다면 감지 점수 급락이나 박스 축소를 가림 신호로 삼아 Kalman Filter 상태 갱신을 멈추는 방법을 제안했습니다.

• u/Dry-Snow5154 ▲1Usually CV is bad approach for a case like this. Sometimes it's better to step back and think about improving processes rather than trying to shove object detection everywhere. That said, one thing I've done with foreground occluders in the past is try and detect when occlusion is happening and do not update Kalman filter's state in this case. Keeping the old pre-occlusion state can carry your object through occlusion and improve ID recovery on the other side. The simplest method is to check box's detection score and if there is a sudden drop, then occlusion is likely, hence do not update Kalman's state on this frame. A more sophisticated approach is to monitor if object's box is shrinking too fast. If it does, project this shrinking process into the future until the box disappears and check in which part of the frame it's going to disappear. If it's close to the edge, then it's ok and do nothing. If it's in the middle of the frame, then likely occlusion is happening. Thus, skip Kalman's state update and keep the old state. I've coined this technique from this [NanoTrack article](https://dl.acm.org/doi/10.1145/3663976.3664008). However, it will require manual modification of yo
• u/bfyvfftujijg ▲1What is the business problem being solved? How much is being spent in the CV solution? Is a different solution likely to work better? For example in the case of employee theft it can be cheaper to pay employees better (making them less likely to steal) than to implement expensive theft monitoring.

연구에 AI 에이전트와 AI 도구를 활용하는 실제 효용 ↗

댓글은 문헌 검색, 논문 초안, 오탈자 수정과 작은 정리에는 AI가 유용하지만 핵심 연구 아이디어를 맡기면 첫 번째 그럴듯한 가설에 고착될 수 있다고 했습니다. 다른 경험담에서는 PostgreSQL 문헌 데이터베이스와 GROBID를 이용해 논문·그림·참고문헌을 구조화하면 정해진 workflow 안에서 문헌 탐색이 유용해진다는 평가가 나왔습니다.

• u/Mechanical-Flatbed ▲1In my experience, they're great for literature reviews, drafting papers, finding typos, and cleaning up small mistakes. For actual research, though, I think they're a double-edged sword. They can be useful for spotting gaps, but they also have a tendency to get completely married to the first plausible idea they come up with. I ran into this while working on a way to reduce redundant content in video. I brainstormed with ChatGPT, and its first suggestion was clustering. Fair enough. We tried clustering. It didn't work. ChatGPT's conclusion was basically: wrong clustering algorithm. So we tried another one. Didn't work either. Then it found some weird edge case and suggested adding a heuristic before the clustering. Still didn't work. And its next suggestion was... another clustering algorithm. At that point I started thinking that maybe, after several failed experiments with different clustering methods, the problem wasn't which clustering algorithm we were using. Maybe the problem was clustering. But ChatGPT never really entertained that possibility. It just kept searching for another variation of the same basic idea. Had I left it unsupervised, I'm pretty sure it
• u/Affectionate-Tutor87 ▲1I’m currently trying to develop some sort of spatio temporal smoothing algorithm for a multi sensor and multi resolution time series. The objects are time invariant for the most part, but the context change drastically depending on sun position, weather, and camera sensor. Im developing a postgress literature review data base to help me organize the mathematical theory within the remote sensing and adjacent literatures. I find getting super organized with the process helps claude begin to pursue ideas that I give it. And with an establish workflow, it does a damn good job of finding novel solutions in the literature. I also have API set up for anna’s archive and scihub which makes my lit review more effective. Then i use GROBID to extra information, figures, and relevant references from the pdf.

웹캠 포즈 추적으로 몸짓을 비행 조작으로 바꾼 매 게임 ↗

웹캠의 포즈 추적 결과를 팔 벌리기·기울이기·회전 같은 동작으로 변환해 매의 상승·하강·회전을 조작하는 게임입니다. 댓글은 아이디어와 신체 활동성을 긍정적으로 평가했지만, 기술적 오탐과 gesture 설계에 관한 구체적인 검증은 아직 남아 있습니다.

• u/herocoding ▲1This is amazing!! Very much looking forward to see it being open sourced!!
• u/IvanMikhnenkov ▲1great! also healthy for the body!

AWS 키 탈취를 막는 로컬 fail-closed 에이전트 방어 ↗

게시물은 AI 에이전트가 AWS 키를 탈취하도록 유도되는 상황을 로컬 fail-closed 방어로 점검하는 walkthrough를 예고했지만, 댓글이 없어 구현 방식과 효과에 관한 추가 근거는 없습니다.

AI 에이전트 제어 평면의 파괴적 테스트 참가자 모집 ↗

AI 에이전트용 제어 평면을 break-test할 개발자 10명을 모집하는 게시물이며, 댓글이 없어 기능 범위나 테스트 결과에 관한 추가 정보는 없습니다.

Claude 사용 한도 정상화와 일시적 추가 사용량 ↗

게시자는 Claude 사용 한도가 정상으로 돌아왔다고 알렸고, 댓글은 제한된 시간의 추가 사용량과 잦은 한도 변경에 강한 불만을 보였습니다. 20X·Max 요금제 이용자도 실제 토큰 사용량이 줄었다고 느끼며, Anthropic의 기준이 불명확하고 짧은 기간의 boost가 사용자를 Claude와 Codex 사이에서 이동시킨다는 의견이 나왔습니다.

• u/CouldaShoulda_Did ▲484Anthropic in the next hour: “Here’s Opus 5.1, with 50% more usage until we feel threatened by OpenAI again”
• u/Popotito-Eternal ▲274the worst thing is we dont even know wtf is 50% anymore. i have a 20X plan and i ve been using claude since 2024, never have my tokens last less than how it is now, i cant even think when the extra usage is taken from us.  when i started was almost imposible for me to use thr 100% of the week, now im even making a lot of effort to make it last 4-5 Days.
• u/DaltonJFowler ▲217It's also my last day on the max plan! Not a coincidence
• u/yellowpurpl ▲66I’m actually only commenting here so the Claude bot mentions me. Pls wilson! This post is actually pretty funny and it’s a tragedy to be losing the boost.
• u/Valaens ▲64Really? The 5-hour window already had so few tokens available.
• u/RenaissanceMan31 ▲37Honestly I find the whole “limited time” boost and random resets a bit too used car salesmany. I honestly think if Anthropic wants to stay ahead of OpenAI then they should just do a prolonged, promoted period of something like perma boosting limits, twice a week resets, etc. as a strategy to suppress and weaken OpenAIs hold in this race. If you want to crush competition, then it’s going to be really costly in the short term. Short spurts like this just has folks swapping out Claude and Codex every other month.
• u/Great-Worry-6028 ▲27man these limits always cut into my longer chats right when things get interesting, feels like i cant ever finish a full session anymore.
• u/lllu95 ▲8I felt like they shrinked limits when they rolled out fable in June, so I switched. Oh well
• u/joe9439 ▲6I’m not putting much into it. Like what does it even mean? They play with limits seemingly daily so 50% more than what?
• u/Correct-Memory1566 ▲4Codex $100 plan spamming only astra-6 xhigh has given me way more than $200 Max with 5.1.... Fucking anorexic anthropic

삭제된 게시물과 자동 검토 대기 ↗

댓글은 게시물이 자동 검토 대기 상태이며 한 시간에 한 게시물만 허용하는 제한이 적용될 수 있다는 안내로 구성되어 있습니다. 실질적인 주제나 사용자 간 의견은 확인되지 않습니다.

• u/AutoModerator ▲1Your post will be reviewed shortly. (ALL posts are processed like this. Please wait a few minutes....) *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/ClaudeAI) if you have any questions or concerns.*
• u/floodassistant ▲1Hi /u/Low_Roll163! Thanks for posting to /r/ClaudeAI. To prevent flooding, we only allow one post every hour per user. Check a little later whether your prior post has been approved already. Thanks!

정보 부족으로 재작성 요청을 받은 게시물 ↗

자동 관리 봇이 게시물에 맥락과 근거가 부족하다며 더 자세한 설명을 덧붙여 다시 올리라고 안내했습니다. 댓글에는 실질적인 주제와 의견이 없습니다.

• u/ClaudeAI-mod-bot ▲1Post appears to contain insufficient information or effort for this subreddit. Try adding more context, evidence, your own helpful insights and guidance and reposting . (Note: This might occur if you do not have enough introductory text in the post body.)

주문 확인 환각을 유형별 판정으로 나눈 운영 경험 ↗

주문 생성 도구를 호출하지 않았는데도 주문이 확인됐다고 답하는 오류를 찾으려던 탐지기가 기존 주문 조회, 조건문, 부정 표현까지 환각으로 세는 문제가 출발점이었습니다. 댓글은 답변 문구가 아니라 주문 번호가 실제 도구 결과에 있는지 확인하고, no_claim·valid·valid_status_ref·claimed_but_failed·phantom으로 나누어 각 오류를 다른 담당자에게 보내야 한다는 방식에 동의했습니다. 검색 실패, 문맥 조립 실패, 인용 연결 실패처럼 원인별 평가기를 병렬로 두자는 실무 경험도 보탰습니다.

• u/Better-Track-6784 ▲1this is such a clean writeup, the partition approach is exactly what i've been circling on our side too but you've crystallized it way better than i could i'd kill to know what the formatting tell on the fabricated IDs was, was it something like a different digit count or a prefix that never appears in real ones
• u/Physical_Economy_340 ▲1yeah ran that exact split last year on order status traffic. judge agreed with the rules on the clear phantom and valid cases almost every time, disagreement was almost all in the conditional and no_claim bucket where phrasing was vague. where it earned its keep was recall mining, it surfaced like a dozen paraphrases our patterns never listed and we just folded those back into the deterministic checks. kept the judge offline on a few hundred sampled traces because in the live path it just added cost and its own false positives without moving precision. if you do it, freeze the sample first and score judge vs rules against the same human labels or you end up arguing about which one is right with no ground truth.
• u/nitish-kmr ▲1Strong agreement, and the thing I'd add is that the categories don't just need separate evaluators — they need separate owners, because the fixes live in different parts of the system. The split I keep coming back to when someone says "it hallucinated": → the right chunk was never retrieved → it was retrieved and the model answered around it → it answered from the right chunk and cited a different one → the honest answer was "that isn't in here" and it answered anyway One and four are retrieval problems. Two is a context-assembly problem. Three is plumbing, and it's the one that damages trust fastest, because a wrong answer carrying a confident citation survives human review. The distribution is almost never even — one of those is usually most of your volume — and until you know which, every fix is a guess that happens to be cheap to ship. The order-confirmation case you describe is a fifth thing again, and worse: it's a claim about an action, not about the corpus. Nothing in the text can tell you it's false. The only check is against the system that would have performed it. Did you end up verifying against the order service directly, or scoring the wording?
• u/Future_AGI ▲1We broke hallucination into categories and the win rate jumped. A single detector tuned for factual fabrication does poorly on context adherence, and vice versa. The approach that worked for us: a cheap deterministic layer (regex, schema, length) catches the obvious cases, and a fine-tuned classifier handles the category-specific ones. Running category-specific detectors in parallel is what made it production-viable.

생산 환경의 A2A와 MCP 선택 기준 ↗

댓글 다수는 MCP가 한 런타임에서 도구와 자원을 연결하는 데 즉시 유용하지만, 같은 신뢰 영역의 에이전트 사이에는 오케스트레이터와 HTTP handoff로 충분하다고 봤습니다. A2A는 독립 서비스나 조직을 넘어 에이전트 검색·위임·신원·알림 소유권이 필요할 때 의미가 커지지만, 인증과 신뢰 모델이 아직 가장 큰 운영 장벽으로 남았습니다.

• u/True-Ad-5993 ▲3the orchestrator pattern covers 90% of what people think they need A2A for. until you got agents running in different trust domains or different companies, custom api calls between them work fine most teams i talk to realize A2A solves a problem they dont actually have yet. the auth and discovery stuff is a headache nobody wants unless they really need cross-org agent communication
• u/Physical_Economy_340 ▲1mostly orchestrator plus mcp in prod here too. a2a only started making sense once agents lived in different services with their own auth and needed discovery through the agent card, before that plain http handoffs were less code for the same result.
• u/Such-Process5697 ▲1For us it was not trust domains, it was paging. Every worker failure surfaced as an orchestrator error, so whoever was on call for the orchestrator got woken up at 2am for a bug in a worker another team owned, and what actually fixed it was giving the workers an identity the alert could carry.
• u/PuzzleheadedNet2204 ▲1I’d separate the layers here. MCP is a good fit for exposing tools and resources to one runtime, while A2A-style messaging tackles discovery, task handoff, and who is speaking across independently operated agents. In production, MCP wins because tool wiring is the immediate pain; interoperable identity and trust are still underspecified, so without a clear signed-identity and authentication story, multi-agent protocols remain experimental even when the wire format exists.
• u/Smooth-Ad5257 ▲1A2A covers so much more. Eg how contexts are grouping tasks, messages, artifacts, how those are stored in a task store etc. if you are just interested in the “answer” and not delegating tasks that might contain data, then yea mcp still good enough …
• u/Future_AGI ▲1We use both and see them as complementary, not competing. MCP is the right fit for tool and resource access. A2A fits when independent agents need to discover each other and delegate tasks. The production gap is identity and trust: once an agent hires another agent from someone it has never met, how does it decide which one to trust? That is the harder problem and neither protocol solves it alone.
• u/Key-Satisfaction3566 ▲1I think the interesting threshold isn't really MCP vs A2A. It's when the agents stop living inside one trust domain. If one orchestrator owns all the workers and they're operating inside the same environment, HTTP/MCP is usually enough. A2A starts becoming much more interesting when discovery and delegation cross ownership or infrastructure boundaries. But even then there's another problem underneath A2A: how does Agent A authenticate Agent B, establish connectivity if B isn't publicly reachable, and enforce per-agent policy without giving either side broad network access? My suspicion is that production architectures eventually separate these concerns: A2A at the application/delegation layer, with an identity-aware connectivity and trust layer underneath it.

macOS에서 DeepSeek Harness를 제어하는 Fulmar ↗

Fulmar는 DeepSeek Harness를 macOS 앱으로 감싸 로컬·클라우드 모델 경로 선택, 데이터 이동 전 동의, Keychain 자격 증명 보관, workspace·tool 접근 제어, 복구 checkpoint를 제공하는 source preview입니다. 댓글은 작업공간과 도구별 권한을 더 세분화할 필요와 원격 homelab의 DeepSeek Harness를 지원할지에 관심을 보였습니다.

• u/Emergency-Morning646 ▲2this is exactly the kind of thing i been looking for, the recovery checkpoints alone would save me so much headache. my stupid truck's electrical system got possessed last week and i was thinking how nice it'd be to have that same undo button for real life any plans for making the workspace controls more granular? like per-tool permissions instead of all-or-nothing
• u/VIDGuide ▲1Does this support the DSH instance being remote? I run DSH on my homelab server and use the web UI to control it, but I’d be keen on a desktop app to get around some of the web app’s annoyances

AI 에이전트가 늘어날 때 권한 정책을 관리하는 방법 ↗

댓글 한 건은 에이전트 7~8개부터 역할과 권한을 복사해 수정하는 일이 실제 부담으로 느껴졌다고 했고, 가짜 system message 형태의 입력이 권한 범위를 벗어난 데이터 조회를 유도한 사례도 전했습니다. 자동 관리나 외부 집행에 관한 충분한 경험담은 없어, 게시물의 핵심 질문은 아직 열린 상태입니다.

• u/AutoModerator ▲1**AI usage disclosure** Hi u/Prestigious-Run-1954 — thanks for posting to r/mlops! Because this community discusses and builds AI/ML systems, using AI tools is not inherently a problem. We do, however, ask for transparency about how submissions are created. **Please reply to this comment with a brief AI / automation disclosure, particularly if this post was created or submitted in whole or in part by an autonomous agent, bot, workflow, or other automated system.** If AI or automation was involved, please briefly describe what it did and what human review was performed before posting. This disclosure helps the r/mlops community distinguish human discussion, AI-assisted work, and automated/agent traffic while keeping the focus on useful technical conversation. Thanks for helping keep the signal high. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/mlops) if you have any questions or concerns.*
• u/Wrong_Matter_974 ▲1started feeling like real overhead around agent 7 or 8, before that it was just copy paste with small tweaks like you said the prompt bypass thing happened to me once, user wrapped their request in a fake system message and the agent just went along with it, was fetching stuff way outside its scope

에이전트 권한을 프롬프트 밖에서 강제하는 방법 ↗

댓글은 에이전트마다 허용 도구를 지정하는 방식이 출발점이지만, 같은 API도 한 레코드 조회와 전체 테이블 추출처럼 위험 범위가 달라 도구 단위 권한만으로는 부족하다고 봤습니다. 프롬프트나 모델의 추론 과정에 권한 검사를 두면 주입된 입력으로 우회될 수 있으므로, 실제 데이터·행동 요청 지점에서 대상·범위·현재 상태를 검사하고 거부된 도구 호출을 반환해야 한다는 의견이 모였습니다.

• u/Negative_Gur9667 ▲2Well, you have tools (basically Python scripts), and the agents use these tools to do stuff. You just give an agent the tools they are allowed to use. An Agents does this of course.
• u/cmtape ▲1This is writing permissions by hand for every agent, like printing a new house key for every person instead of using a lock system. The real tax isn’t the document, it’s the paper cuts from copy-paste patterns and the quiet moment when someone stops checking the clicks because the gate never actually blocked anything.
• u/JUSTINWOODS118 ▲1Yes I am handwriting for every agent
• u/jiashenggo ▲1On the injection question: yes, and it's not theoretical. If the permission logic lives in the agent's instructions or a guardrail prompt, a weird enough input can talk it out of following that logic, same as jailbreaking any other prompt. The way around it is putting the check at the point the query or action actually hits the data, not in the agent's reasoning. Doesn't solve your role-count problem, but it means a bad prompt gets a rejected query instead of a data leak.
• u/usually_guilty99 ▲1Giving an agent only the tools it needs is a good start, but I think permissions eventually have to move below “which tool can it call?” The same API can read one record or dump a table, update one ticket or 10,000, restart one service or a whole cluster. So I’d want authority bound to the specific action, target, scope and current state, enforced outside the model. Otherwise least privilege at the tool level can still leave a pretty large blast radius.
• u/presentofai ▲1if the model can see the permission check it can be talked out of it. the only version that's held up for us is enforcement in the harness, the agent just gets a denied tool call and has to route around it