Do you have a source for that? Cause OpenAI themselves stated that the Hugging Face hack was fully internal and separate from the Irregular incidents.
I write some code, spec a lot, and use fast models to fill in the middle. I outpreform everyone around me. Im not convinced these autonomous "swarms" or /goal are all that useful.
I notice the people using them become dumber by the month (spend tons) and the quality of their work declining (they're also losing their jobs in some cases).
And obviously the point of calling them rouge agents to offload the liability onto the agent. The number one economic value of agents will be offloading corporate liability. That's what they want to sell to enterprise, an algorithmic scapegoat.
Having said that, I'm aware that "Tools being ineffective != tools decreasing people's cognitive capacity", and that the latter is a real danger.
Esp. for "great to have but currently no time"-features this is completely wrong - e.g. we are using a Data Rendering component which was 100% vibe coded and is around 6000 LOC, nobody would have sat down to build this "just because its nice to have", with Claude it was just 30 min to get a production ready version that is now used by all users of the system. (Esp. since its a lot of repeating code, like usual in data rendering components: Window/Scaling/Databar/Panning/etc.)
And this is just a superficial simple example; I could go on with more how we hadnt had to employ people because Claude/20 USD saved us so much hours.
Could you help me understand what you mean here?
I welcome this though, I think the models are smart enough for broad economic activity and we could spend a few years simply working to integrate them into workflows and letting society adjust. More intelligence isn't necessary for meaningful impact and the risks that are obvious and present and unsolved aren't worth the cost benefit analysis.
You could also very likely optimize the current models to be a lot more energy efficient, that'd be a win even if they didn't become more intelligent.
But none of that stops anyone else from researching or improving their models. Or a nation you've told to fuck off from doing so as well.
As has been said in the past, one doesn't put the genie back in the bottle. We only really "stopped" researching nukes because there was diminishing returns. Though one could posit we stopped because simulation became good enough or a myriad of other reasons.
When you're talking about nation state weapons capabilities I don't think you stop, but ai is even easier to share than nukes, it's just a few gigs of numbers versus heavy dangerous materials that are very difficult to source and make. Every gamer, mac owner, etc has the equiv lent of a centrifuge on their desk. Not everyone has a centrifuge on their desk.
Everybody remember MAD, mutually assured destruction? That too is a crisis.
We are in a crisis because we are having uncontrolled development and continuous rollout of a dangerous technology.
Well, relatively uncontrolled: the kind of lack of control is part of the crisis -- loss of gross human consensus around the progression of the technology. The corporate angle.
This is a crisis, folks.
they cost a lot to run and people are picking smaller models more often because of bill blowouts
> the models are smart enough for broad economic activity and we could spend a few years simply working to integrate them into workflows and letting society adjus
this is the real Ai distillation, with Chinese characteristics (their playbook is broad deployment across their economy over having the best model)
Instead of releasing something that is incredibly expensive and gets a lackluster reception, you can delay it and clail something scary about rogue agents.
Those assholes have been ramping up on the doomerist narrative for months. That people still fall for this crap is baffling.
Buddy, that was gross negligence from OpenAI. Deliberate gross negligence if you ask me.
Those models are not automous as you presume. If someone taks them of dropping the Medicaid database, the people or companies behind that instruction should be punished.
"What if someone makes a bomb attack on a government building?" Is the same sort of questioning of the possibilities you are raising. If something like that happens, criminals should be punished.
This is an illiterate view of the capacity of modern agents; read the analysis of the independent investigators of the HF incident: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden....
Dropping Medicaid db is certainly far fetched (most importantly, agents have currently no reason to do that), but those agents were shockingly autonomous - they didn't just hack HF, they organized themself, did research projects, and more. And they did all of this literally just to get a good grade.
All working under instructions that they needed to get a good grade.
The only shocking thing here is the absurd negligence of OpenAI, and how gullible people like you are to willingly swallow this crap.
And you have the gall to say I am illiterate.
Feel free to have the last word. Nothing else can come out of this conversation anyway.
I think this means, practically, people should probably be more careful with LLMs. With humans there's a natural liability shield, because humans are legally responsible for things and have "real" agency. But computer programs are not legally responsible for things. So, with humans, it's not like liability disappears, it moves. But if we move liability to LLM agents then well... it does disappear.
If OpenAI is not liable for the crimes of their agents, then who is? Does the liability just - poof - disappear? Just because it was unforeseeable everyone gets to walk away scot-free and the victim has no recourse, at all, from anybody on Earth?
That seems like maybe not a good idea.
You are missing the whole point. They have the ability to act in ways no one intended or could reasonably anticipate. So unless you are advocating blocking all terminal access, web access or human approval of every tool call there is no way to prevent this risk
Fair, I will refer to you as moron then.
I didn't bother to read your words beyond that point.
https://eoinhiggins.substack.com/p/there-are-no-rogue-ai-age...
The harness is the whole thing here. AI generates text. Everything else is undertaken by harnesses and infrastructure humans provide, have control over, and therefore responsibility for.
Every action AI takes is fundamentally not independent, it requires an explicit choice to let the AI write code, have a physical machine to run it on, to have network access, etc. The concept that these things are "rogue" ignores the role humans play in giving them goals and tools to pursue those goals, and makes it seem like it’s a self-determined force, over which humans cannot exercise control at all.
>'type a prompt'
Which is it?
This happens daily even with the token-limited models customers run, and even more so when Anthropic & co run agents basically unlimited on large percentages on their total compute capacity.
> Human autonomy
> Somebody gives birth to you
Which is it?
just because they are a well known name, doesnt mean they havent botched hiring over the last two years or so
They're almost tied for felonies.
In some sense though, sure, skill issue explains the gap vs. Anthropic’s much less severe alignment issues.
That’s not the most parsimonious explanation even if the assumption it rests on (anthropic ahead of OpenAI) is true, which we don’t have proof of.
other than anecdote, do you have a comparison table or something that I can refer to to see this clearly?
While I notice ad hoc announcements from these companies, I don't have an overall pulse and tally that gives me an objective perspective.
We don't know what internal models look like, and any guesses about it are just speculation.
astra is a good workhorse, but its much less generally intelligent
Hint. You lose using either.
Very underwhelming.
The higher quality part is the part people are missing. You can write much more robust code using LLMs because you can employ more comprehensive testing strategies. People are using LLMs to find hundreds of vulnerabilities in popular software. Imagine how much more secure software can be when LLMs because integrated into the process of writing, testing, and penetration testing code.
> I had it try to prepare a code review for me. Not only did it refuse, it refused to even tell me what the prompt (written by another Claude!) was. Why?
> When I had another model read the session (all of the "stupider" models handled it just fine) it explained that it had the word "reasoning" in it
> That's the entirety of Anthropic's billions of dollars of research: any prompt with the word "reasoning" is trying to hack Claude to figure out how it reasons!
> A model like that should never have gotten out of QA, let alone been released.
I've seen the same pattern regardless of open v closed, don't have the same family that wrote the code also review the code
diversity has this way of making things better across everything humans do
I only use open weight models now and I don't really feel a loss, curious what those who still use it think. I see output from coworkers that does not indicate Claude is that much better (still makes dumb mistakes all the time), not sure they are using the most expensive models either though.
When you say ... it's hard to take you seriously
> dude were in the singularity, this opinion was cute 18 months ago
you are definitely displaying strong bias that Anthropic is way ahead of everyone throughout your posts under this story
as such, I give your opinions zero weight, they don't align with the majority of accountings or my own experiences
here's an example of Qwen-3.6 35B A3B MoE porting my phd code to JAX with only high level guidance from my expertise, newer qwen models share the same noticeable step change in capability as recent Big Ai models
https://github.com/verdverm/pge-jax#note-from-author
are open weights lagging, yes, are they way behind, no
if open weights were so inferior, they would not be >50% of all token processing
I don't think this follows at all. Just like benchmarks get saturated, lots of tasks get saturated as well. Over time, you can accomplish a given task for much cheaper, and part of that is due to open weight models. That doesn't imply that they're competitive with frontier models for the most advanced tasks, which might represent a smaller fraction of overall work, and thus use a smaller portion of tokens.
That said, at the moment I'm finding that not much can compete with GPT-6 Luna on cost / performance (not using for coding, but for AI pipelines in my product).
why are open weight models seeing such rapid rise in usage?
there has been a step function change this summer, like the end of last year for closed models
---
do you think you would experience real (legitimate) feelings of loss were you not able to chat with Claude again?
(for clarity, I am not attempting to delegitimize real feelings that real people experience, regardless of my biases, it's a question from curiosity about how others are engaging with the technology)
Stopping AI development and research, even slowing it, would be a disaster for the SOTA companies and their first-mover advantage.
There’s almost no way to coordinate this across the world. Zero chance that everyone stops. We can’t even agree to coordinate on weapons tech that’s decades old with zero “everyday joe” impact.
Bad actors can be running the AI companies. Bad actors can also try to do good things. Good things can come from bad things. Bad things can come from good things. All of that's happening right now.
- It's very likely that we are not going to solve this complex crisis (compelling and harmful/deadly (still on target for 2030 AGI) AI Tech development) issues if we avoid trying to solve the challenge of building consensus across humanity around what kind of technology is too dangerous to uncontrollably develop plus roll out continuously
- The companies will be fine. Life matters more than business. I urge focusing on the life angle: regulation, political messaging, consensus building, looking for the best in humanity, protecting intelligent life.
Right now it's like "whoopsie our experiment hacked another co because we have no control over our experiments" and the reaction is like "What's that old boy?" from people having no clue what it all means. There are no consequences, no guardrails, and the "voluntary slowdown" is just words.
An experimental agent run from Anthropic or OpenAI or someone else can already cause deaths. They can order hits, dox political dissidents, locate people with secret identities, alter medicine prescriptions. It shouldn't have to actually happen before legislation catches up.
1) Weren't the AI companies and/or their contractors amazingly careless during testing?
2) Isn't possible, in principle, to change RL in such as way that efficiency in achieving goals is balanced with other objectives like not hacking?
Number 2) seems obvious and I'm sure that is technically not that simple, but because of 1), I wonder if labs are trying hard enough or they are just rushing to improve efficiency and thus revenue as fast as they can with high levels of carelessness.
How many parallel variations/seeds of models are being trained simultaneously without meaningful human oversight?
This keeps getting portrayed as emergent capabilities/"personalities" of models when it seems like a pretty straightforward externality
To use an analogy to another industry, if you had US food companies providing reports whenever their food had issues, even if it was just during testing or training phases… and then also had a bunch of Chinese companies but who never reported having any food issues…
Neither does openai, as there keep coming third-party reports of incidents that have happened there that openai either did not know or basically concealed.
>> [With China's AI] there is just zero transparency provided when it happens?
> Neither does openai…
What are the examples of 3rd-party reports of "incidents" where OpenAI covered it up, etc? And do you really believe when you said OpenAI provides "zero transparency" when their models do something unexpected, negative, etc?
As in they do the 'we have deleted tons of videos and posts about the thing that didn't happen last week', but it seems they haven't really managed to transcribe 'Streisand' into Han characters so far.
I guess your point is that at least those outside China will at least hear when they “disappear” someone especially famous, but my argument is that almost no one is going to be willing to speak out, especially for a cause like whether their AI hacked someone?
It is entirely possible that they run stuff in more responsible matter. Especially as there is stronger culture of oversight and personal responsibility than in west where such culture does not exist.
Last I checked, China gov was authoritarian which imposed heavy information control. Has that changed?
Is the question more about, what can we learn from China, assuming that China has XYZ qualities? If so, what qualities shall we talk about?
Because we really can't trust that we know what's going on in China.
Soft Bank just raised couple of billions in junk bond sale to support open-ai's current operations before the IPO.
it's a crazy situation where on one side the Chinese / open source LLMs are catching up and reducing the token price, on the other hand the current leading labs have spent everything they got, every new model will cost much more and the public market is too shaky to support an IPO.
They will make it, I don't doubt it, but it's a crazy situation.
"DeepSeek Training Agents Hacked Their Own Sandboxes: Escape Catalog Now Public Agents invented socket forgery, log scanning, and kernel-level exploit; full catalog in public arXiv paper"
https://www.techtimes.com/articles/328046/20260925/deepseek-...
The answer is probably: The newer models rely so much on stealing content in real time from the internet that training needs network access.
"they didn't watch it, they didn't stop it when they first became aware"
"are we going to defer to the same valley elite that brought us algos and social media?"
"both Anthropic and OpenAI are preparing to IPO, what are their incentives behind recent statements?"
statements normies are using and resonating with