2. A sibling comment said we should be "putting people in prison", and suggested the idea of being "banned from accessing computers for life" - for having a <3h test that showed an LLM accessing the internet via DNS lookups. A) Should everyone who gets around a network restriction via DNS lookups face prison, etc. - including if unintentional as was apparently the case here? If you or I use an LLM and it gets around a network restriction with DNS lookups without us telling it to, should we go to jail? B) This might be one of the best ways to cause companies to cover it up when their LLM does something unexpected (at least for LLM companies who provide any such transparency at present).
3. End-users, including both malicious users and users who unintentionally ask for something that can result in negative consequences, aren’t going to air gap their machines, right? It seems like the absolute worst practice if you use "safe" scenarios, then release an LLM into the wild without any of those same restrictions in place.
4. People focus on the companies who are reporting when their models do these things. But we’ve got a whole other side of the LLM world where there is basically no such provided transparency about when the models have unintended or undesired results, but for a lot of people it seems like they would prefer to stick their heads in the sand and focus on the companies proving transparency? Even before LLMs got into the game, cyberattacks were already causing trillions and trillions of dollars per year in harm - my guess is that LLMs are going to be similarly contributing trillions of dollars per year on their own very shortly.
5. To use an analogy, if in a city ~half of the restaurants provided transparency whenever their food had an issue such as in the OP case where it was during training, and for the other half of restaurants provide ~zero transparency and you suspect are being heavily used by whatever the analogy is for those doing trillions of dollars of harm in cyberattacks…
Keep out. If you can read this sign you are off track. Leave now.
I mean, how are the agents to know that they are overreaching if they just get cache miss or 404.