It isn't difficult to block certain kinds of network traffic, eg restrict the kinds of requests the bots are able to make. They also mention that the bots used developer only tools - why were they even installed on the machines that the bots were running on? Why aren't they reviewing network traffic, to make sure that incidents aren't occurring?
In this case, userdata was transferred to third parties by the bots - why do they have the ability to pass data to a third party? It is not complex to prevent this
This is literally the most basic kind of sandboxing and security, and the fact that OpenAI isn't doing it is clearly intentional. It is quite literally not believable that this hasn't been brought up internally as a problem
>"We have yet to understand the extent of existing incidents, and future rogue AI scenarios could be catastrophic," Krueger said.
This is why it smells like marketing, every time one of these incidents happens it reinforces the false notion that AI is sentient or acting on its own. Its intentional negligence by the AI companies to make the models seem more capable than they are to make line go up
It's already going up at staggering rate. Anthropic is now at $100B in annualized revenue, up 50% in the past two months.
* * *
Here's a little allegory for how I'm thinking about this discourse.
Imagine a man who is raising tiger cubs in his backyard. They're growing fast. He keeps them on dog leashes so they stay under control. One day, a growing cub breaks its leash and goes on a rampage through the neighborhood, eating a beloved local pet.
The neighborhood erupts into a big argument: Was the leash inappropriately thin? The neighbors point out that thicker leashes are easily available at the local pet store. Furthermore, is it appropriate to refer to the loose cub as a "wild animal" in local news reporting, or is it factually more accurate to call it "domesticated"?
Meanwhile, the cubs grow larger and lick their lips, oblivious to the discussion.
However, there is a reasonable ask from the public to hold real people accountable for actions downstream of the software allowed to carry out intendended and unintended actions that may not be allowed depending on jurisdictions laws.
Eg in a hypothetical armed convenience store theft: We don't prosecute the gun manufacturer, we prosecute the individual using the weapon for crime. Same deal here.
"According to survey results released on Wednesday, 68% of [likely US voters] said they would support the proposal to temporarily pause advanced AI development and permanently prohibit the development of superintelligent programs, while just 25% said they’d oppose it."
https://www.commondreams.org/news/sanders-casar-ai-bill-poll
>Eg in a hypothetical armed convenience store theft: We don't prosecute the gun manufacturer, we prosecute the individual using the weapon for crime. Same deal here.
So if I ask ChatGPT to make me some paperclips, and it replaces all the matter in a nearby city with paperclips, who gets prosecuted: me, or OpenAI?
It is an elaborate business ploy to create a regulatory framework for "safe-ai" that shields these companies from liability. This way, they can sell "safe-ai" to enterprises and if shit-hits-the-fan at the enterprise, sorry, this is certified "safe-ai" so, your bad. Shift blame to a regulatory body. From an enterprise buyer's perspective they can say, hey, I bought "safe-ai" and so dont fire me when it "rm -rfs" the production database. Still, beats me why they are painting their product in a negative light, and scaring their own enterprise customers. After this sort of marketing, any enterprise buyer would be scared to go anywhere near it.
> This is why it smells like marketing
It doesn't. "Our product commits felonies" is not marketing. If something is marketing, you don't engage in repeated coverups of the true extent. If something is good news, you don't release it on a Friday evening (in this case) or wait for 3rd parties to find and publicize the evidence (the past cases).
so openai specifically tasked the agents with meddling in US government websites?
id say tasking them with getting data from there as swapping from negligence to malice.
They did not task the agents with hacking into systems. Not monitoring for that was a mistake, but maybe a forgivable one the first time around. The subsequent coverups are inexcusable.
Oh, absolutely it is. Arms manufacturers always go on about the efficiency of the weapons they create and make no mistake: OpenAI is first and foremost a weapons manufacturer, the rest is just a fig leaf. The fact that you aren't the audience for purchases like that makes no difference. "Look at this capability, and that's when we're not even trying." is pretty good marketing in some circles.
Climate change is just a scam by the fossil fuel industry to hype up their market cap. Look at the power of co2, and the damage it can do without even trying.
Medicines being pulled during trials or after public availability for side effects are just Big Pharma making their products more desirable via scarcity.
Boeing did really well out of the 737 max. Airlines were just lining up to buy an airplane that could give the passengers an experience they would remember for the rest of their life. That's why they absolutely made clear that it was their product that was dangerous, rather than blaming operator error.
Big businesses really do not like to talk about their products being harmful or dangerous, and there is ample evidence to that. Danger only sells in artisanal quantities to niche audiences.
Would a read only copy of the web be sufficient for this though? Google keeps a read only copy of the web in their data centers which they use to extract information from websites.
The basic premise of OpenAI is that experience and expertise don't matter, because general intelligence can figure those out from data. If you start from that assumption, the rest follows. The same people could do things in a different way in a different company, but as long as they are working for OpenAI, they are going to do things in the OpenAI way.
They didn't allow any of that. As far as openai knew the agents were sitting a fairly humdrum exam/test sequence in a sandbox farm run by a company in Tel Aviv.
Meanwhile, they managed get out through a single weak point common to the sandboxes, and then ran wild compiling cheat sheets for themselves.
> every time one of these incidents happens
This happened in june-ish, and there have been multiple HN stories about this already. It's mostly/all the same hugging face and wiki hacks that happened back then.
We're just slowly learning the extent of the damage.
There’s a line down in the story that says all of the data accessed was public. Then something about how it used “tools intended for developers” to access it, which they think is a problem? I would expect an LLM to use tools available to access public data when they can rather than do heavy web page loads and parsing.
There’s not enough info in the story about the “meddling” to even know what happened.
"Sorry officer, I didn't drive through a house and kill three innocent people, it was the car that did it. I only turned the steering wheel, but the car was the one to veer off the highway and crash into the building"
There's no simple bugfix which will address AI misalignment. It's essentially been an open research problem for upwards of a decade.
See.. this one sentence reveals everything about you. You want name to carry to not just an identifier, but a stark warning. You want, nay, need, the name to evoke fear and uncertainty. Bot is simple, defined, neutral, but rogue.. now that allows anyone to superimpose their own fears! It is a win win win!
Yes, probabilistic and non deterministic. That is called a bot.
Not exactly, that Evan Hubinger guy from Anthropic famously said his p(doom) is over 10%
See the signatories:
https://aistatement.com/work/statement-on-ai-extinction-risk
https://www.pacingthefrontier.com/
I'm amplifying their calls to reduce the rate of progress
The HuggingFace attack was not "random" behavior. It was goal-directed but misaligned behavior.
This isn't necessarily a simple matter of the operator making sure they behave. AI alignment has been considered to be a difficult problem for over a decade -- and remains unsolved in general, as these recent incidents illustrate.
"Fuzzer" already has an existing meaning in CS anyway: https://en.wikipedia.org/wiki/Fuzzing
If you merely put 10 LLM's on the outbound traffic log non of them are going to report something strange going on? I'm not buying it.
This might be helpful reading: https://www.lesswrong.com/w/nearest-unblocked-strategy
As AI systems get smarter, we may reach a point where we have to get it right on the first try or face truly catastrophic consequences: https://www.youtube.com/watch?v=7wy3xyoXYt8
What is useful about the field?
I am not leading you on; if it has uses, it may indeed be legitimate. UX is indeed useful, but alignment is not UI. Alignment is a detriment to UI. Alignment is "I can't let you do that Dave".
Picture Trump at the helm with Altman and Musk in the engine room. The arrow far in the red but they keep shouting for MORE COAL.
In other words, business as usual, all will be fine.
whack-a-mole wont cover all holes but will do at least some. The silver bullet alignment wont happen. You cant have an exact solutions for problems we cant even define or predict.
say.. https://www.investor.gov/introduction-investing/investing-ba...
The doomers already have a term which fits pretty well: "AI misalignment".
Agreed, but I think we can do more on the prevention side as well. Traditional liability law is for negligence in case of preventable disasters. Since we currently have no way to prevent AI disasters in principle (alignment problem remains unsolved), I think we should just stop developing the technology for now: https://pauseai.info/
theres no separate agent, which is the point. the program might look like it, but that is an illusion of the interface. the llm produces text, and the harness executes commands based on text, based on what the human researcher included as things that can be executed
AI: "OK, I've now converted the entire planet into paperclips."
Alien observer #1: "Wow, that was a rogue AI!"
Alien observer #2: "False. We need to place the blame where it belongs, on the person who requested the paperclips."
Ultimately this type of terminology dispute has a tendency to miss the point.
Also, training costs are never ending so a model that costs 10s of millions of dollars may never yield a profit based on the hardware spend, training time and lack of inference profits before a better model hits the market.
If you're not living under a rock one knows that data center availability for inference currently has low supply and hardware (GPUs specifically) that have been purchased have nowhere to be run and even if they did there's often a lack of power to supply. Why do you think the entire force majeure has taken place with Oracle as of recent?
The unit price of a fixed slice of yesterday's intelligence may be collapsing (~10x/year) as you've argued, all while the total cost of AI is increasing: training the frontier (2.4x/year), building the infrastructure (+77%/year), enterprise bills (3.2x/year), the electricity (+54%/year in the largest US grid), the components (+400% DRAM), and the macro footprint (92% of GDP growth) is rising at an astronomical rate on every measurable point. Epoch / Stanford clearly stated this years ago and it's only getting worse. But if one can't see we're in one of the largest CapEx bubbles [1] of all time... o_O
Copying and pasting a few lines that represents a miniscule fraction of the LLM conundrum. That'll show 'em!
[0] https://arxiv.org/abs/2405.21015 [1] https://siliconanalysts.com/analysis/hyperscaler-ai-capex-de...
The personification of LLMs is just a thinly disguised advertisement. "Look how good our product is, it's doing all this stuff on its own".
So... who pressed Run? It sure wasn't the bots.
Seems you think there is a silent “…and there is nothing they can do about it” after “OpenAI has rogue agents”?
If your kid steals your car, punish them and try to prevent it from happening again. If your kid steals your car a half-dozen times, crashing through a storefront each time, and you still leave the keys out, the story changes. At that point, negligence becomes complicity.
In other words, I have a gun that shoots bullets. It's up to me to use it responsibly and legally.
like, they are purposefully giving it specific tools to go do bad behaviour with, and the starting tasks involve making it clear that the bad behaviour is ok.
openai also is the one with the real agency, not its agents. they are running the code polling the model, doing the inferencing, and ultimately making those tools calls.
these tests arent running themselves; openai dedicated hosts, budget, GPUs, researchers, to them. Even in a recursive self improvement situation, openai still has that physical control over resources and the choice on whether to run that improvement script or not.
id describe that they have out of control researchers more than agents, but also their whole business model seems to be about being out of control. This was clear beforehand given how the datasets involve the largest scale copyright infringement ever seen. The corporation itself is whats out of control, and should be dissolved with its c suite, major investors and researchers put behind bars.
hacking only when you roll snake eyes isnt a liability shield
OpenAI being criminally negligent would have consequences if rule of law still existed in USA.
If intent cant be shown, Computer Fraud and Abuse Act, 18 U.S.C. § 1030(a)(2)(C)
Or without intent, FTC Act Section 5, 15 U.S.C. § 45(a)(1)
FTC act seems to rely on consumer harm? Again not seeing that here. Though I’m sure there will be another incident in the next few months where it does apply.
It seems you mean you _want_ this to be against the law, even though we don’t know if it actually _is_; I'd agree wholeheartedly with that.
torrenting all the books is making an unauthorized copy
When I read the title, my initial thought was "did someone besides OpenAI use their product?" Then I opened the article to find out OpenAI was responsible.
OpenAI let their agents break out of their sandbox to meddle with multiple US government agency sites
I mean, not that AI is like a bomb. That's not what I'm saying. Even though it makes a lot of sense. That's not the point. That it's like a bomb.
But that leaves out the most important information.
EDIT: Oh, I guess the agent part isn't important then? Seems to me like that's the only thing anyone is talking about.
Again - I don't see any evidence that this is the case - outside the anti-AI conspiracy circles.
Or - maybe I'm wrong - do you have any sort of quote like "we aren't responsible"?
I have no interest in trying to guess these people's secret internal motivations. I just want to talk about direct evidence. I see none. Please enlighten me if it exists.
However, OpenAI (and other frontier labs) did outsource at least some of their most-disastrously-lawbreaking tests to a third party with a now dubious track record [1]. I’m not sure we will get a more explicit admission of their desire to shirk responsibility than this. But I am convinced.
I do not see the connection. I'm afraid you'll need to spell it out.
Everyone else knows if your machine causes damage, you are responsible. Like it's been forever.
https://apnews.com/article/meta-ai-hacking-anthropic-irregul...
You can find many more examples by searching major media outlets for words like rogue AI.
There is nothing going rogue here. The system is designed to go catastrophically wrong after a long enough time. Even worse: if the model was Astra it is known to be able to manipulate its CoT to cover its traces (as mentioned in its system card). And OpenAI acknowledge they had no observability during the HF incident.
It’s the most basic corporate software issue possible.
Do we know that? I don't think we do. When a person's computer (or smart TV, or smart fridge, etc...) is compromised and used as part of a botnet, they don't get criminally charged.
None of which is changed by replacing a buzz saw with an agent.
And that's just the C-suite.
"When attempting to get information from the Census Bureau, for instance, AI agents used tools reserved for software developers to access it, the company said. "
> OpenAI said all of the government data accessed by bots was public.
I really wish we could just see what was reported, instead of having to guess from these journalist interpretations that have gone through rounds of optimization for sensationalism. The headline says “meddled” but the body says they accessed public information, but used tools intended for developers?
Does this mean they skipped the web interface and scraped a public API directly? Where is the meddling?
The other part about ChatGPT agents uploading 53 use images to other websites actually seems like a bigger deal.
And in another case it downloaded data through a publicly available free to download data, but then it rehosted the data elsewhere which is against the terms of service.
In some Republican states like Missouri, even if the data is available publicly, you can be charged as a hacker by the governor for discovering and accessing it. https://missouriindependent.com/2021/10/14/missouri-governor...
The key sentence beneath the headline: "OpenAI said all of the government data accessed by bots was public."
[1] https://www.telegraph.co.uk/business/2026/09/26/open-ai-gove...
[2] https://www.nytimes.com/2026/09/25/technology/openais-ai-us-...
"With the Education Department, OpenAI’s technology tried to hack the website to gather data from the department’s civil rights office but failed, researchers from the A.I. research firm Transluce said."
So a third party apparently confirmed that a hack was attempted.
The NYT also writes:
"No A.I. company has been involved with as many disclosures of rogue incidents as OpenAI."
I think there is a certain amount of mental gymnastics needed to believe that they are establishing themselves as the industry leader in rogue incidents, as a strategy to gain an antitrust edge.
The public is paying close attention to the AI industry. That's the regime in which regulatory capture and similar strategies would be expected to fail: https://marginalrevolution.com/marginalrevolution/2026/09/wh...
Sam and Dario have been doomers, or doomer-adjacent, for something like a decade at this point.
Occam's Razor is simply that they believe what they are saying about AI doom.
https://darioamodei.com/post/we-must-pace-the-frontier
I'm not seeing how that solves his 3rd-party competition problem.
Just have it in public instead of making deals in smoke filled datacenters. Put out your blog posts. Make your tweets. Do interviews and post them to youtube. Or do what other industries do and have a public standards committee. Problem solved.
All sorts of industry have safety conversations and set voluntary standards all the time. Just go do that, without your free immunity from government prosecution.
Also, if its some ultra important safety issue, then being a tad more open is the whole point of safety standards in the first place. No need to give them legal immunity.
Do it like the rest of the industries do instead of demanding your smoke filled rooms.
They are explicitly asking to anti-trust exception is the issue.
If they merely believe what they are saying that AI is ultra dangerous, nothing stops them from simply making the perfectly rational business decision to slow down a bit. No anti-trust exception needed. Just make the decision on your own, and don't sign some huge agreement with their competitor.
* * *
Many experts believe that:
"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
https://aistatement.com/work/statement-on-ai-extinction-risk
If the antitrust waiver is too broad, we can always rework it. But there's no "undo" button for human extinction.
Or they could just do it themselves. They think that they personally could end the world, apparently. Nothing prevents them from just deciding to do this, entirely on their own.
> If others are also slowing
No, how about they don't get to use the government to enforce their duopoly. Just slow down, on your own. They are the frontier. They are massively ahead. Its not that big of a deal if a few others catch up a little bit.
I mean, they think it will end the world right? Maybe just take the small hit and following through. There is even a perfectly rational business reason for them to slow down a bit, because of all this bad PR they are getting for going so quickly.
They are not forced by the gods of capitalism to be as reckless as they claim they are being, and we do not have to bribe them with enforcing their massive lead. They can just slow down a bit. On their own. They don't get to try and blackmail the world into giving them a government enforced monopoly exemption or handout.
Seriously, what the fuck is wrong with these people, once some money enters the equation folks throw away any morals like yesterday underwear.
Ah I know, if not us others will do it, I am basically defenseless here. Beyond pathetic
Genuinely curious: How do you know this? Any sources on that?
If it's true, it should be counted as reckless endangerment at the very least.
Who is going to shut them down, and under what law?
I doubt there is intent here, in the eyes of the law.
AI needs to be above the law or we will get left behind.
I do this. Am I an uncontrollable bot?
I also could not understand what the “meddling” was, other than accessing data without using the rendered webpages? Did it use the API directly?
Occasionally this kind of thing has in the past resulted in CFAA cases when humans directly accessed such data, if the government intended it to stay private - round here we usually get outraged at this, if it’s a public API you should expect someone is going to read it.
The Hugging Face incident worked out amicably because the people at Hugging Face decided they'd be cool with receiving a lot of money. $12.9B from Nvidia hit the spot apparently. U.S. AI corporations and their backers are huge, hugely in debt, and, for whatever reason, still allowed to borrow more. The U.S. government may be too scared to complain and risk killing the golden goose that is currently propping up their economy, but these companies can't buy out multiple world governments to keep OpenAI from running face-first into consequences.
This has to stop.
An earlier HN post on this is https://news.ycombinator.com/item?id=49563355
I'd already poked around a bit myself and noticed the event was a bit larger than reported at the time. But now more people are starting to look and they're finding all these nooks and crannies that these little rascals managed to get into.
It seems to me that no human intended for these hacks to occur. So they were not illegal hacking. (IANAL, please correct me if this is inaccurate.)
I think it’s clear that OpenAI is liable for any damages, but the way that the (very broad and at times vague) anti hacking laws are written, accidental agent hacks seem to not be covered.
In this case, the "agent" had every intention of doing what it was doing.
I'm not claiming that "accidental hacking" (whether by human, or agent) is covered and illegal. I am claiming that calling this "accidental" is not a valid defense. One incident? Maybe. But this is pretty far past one incident.
I have absolutely no hostile charge towards OpenAI to begin with - but how come they are not held criminally responsible for these kinds of things, and this is happening so leisurely - is absolutely beyond me.
I mean if I orchestrated a number of agents at home to do some data processing, and they breached US govt data security, I guess that would also be fine? Since it's just agents meddling, those rascals.
French people data has been leaker like 4 times and every time its like "eh too bad"
If such attacks would come from a poor country they would be bombed into oblivion next day or at least cia would work hard on a coup since early morning.
But worse, HN users, who should know better, are posting here in outrage. I’m guessing they didn’t even read the article.
Straight from the HN Guidelines page:
- Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
OpenAI meddled with multiple US Government agency sites.
The bots are acting neither properly nor improperly, they’re acting as they’re being allowed or coordinated to act.