It doesn't mean that AI got good, just that humans are thinking other humans are AI, which is a form of passing the test.
The adversarial version with humans involved is actually easier to pass because of this - because real actual humans wouldn't pass your non adversarial version.
This includes times that someone basically disappeared from e.g. Stack Overflow at some point before the release of ChatGPT, having written a bunch of posts that barely demonstrate functional literacy or comprehension of English; and then came back afterward posting long messages with impeccable grammar and spelling in textbook "LLM house style".
The people falsely accused because they've used em-dashes for 20 years aren't the ones that were functionally illiterate before.
Everyone keeps saying this, and I keep asking for a link to this type of writing that is dated pre-2022, and I keep getting nothing.
If it was that common, I'd have gotten at least a few examples by now.
With AI, there is no sense of the level of emphasis matching the meaning of the text, or a long-range dramatic arc - everything is a revelation, like somebody who can only speak in TED talks. Everything is extremely earnest, very important, and presented using the same five flashy language hacks.
Nice try, ChatGPT.
Hence my claim "actual people actually write like that"
(well that and the "it's not just x, it's y!" pattern they seem to love)
That is possibly one of my personal writing weaknesses that lead my own writing to get flagged as AI.
I can admit "it's not just x, it's y" is mediocre writing - but it's also something mediocre writers do - it's how AI learned to do it!
Edit: folks, the standard Turing test involves a computer and a human, and then a judge communicating with both and giving a verdict about which one is the human. The percentages for the two entities being judged will add up to exactly 100%. That's how this test was conducted. Please don't assume I'm a moron.
Given that structure, you can judge from that data point.
>The more interesting Turing-style test would be one that gets repeated many times with many interviewers in the original adversarial setting, where both the human subject & AI subject are attempting to convince the interviewer that they're human.
In addition, I think it's reasonable to select people with at least some familiarity of the strengths and weaknesses of the AI instead of random credulous people who aren't very good at asking the right questions.
There is still the $20,000 bet between Kurzweil and Kapor which still hasn't been resolved. https://longbets.org/1/
In the paper they give a breakdown of strategies the interviewers tried and the overwhelming majority were "Daily Activities", "Opinions", and "Personal Details". They also breakdown strategies by effectiveness which shows that these were some of the least effective. Some of the other strategies like trying to jailbreak the AI had 60-70% effectiveness.
This is consistent with what I've seen in other tests too, it doesn't feel like the participants are really trying very hard or taking it seriously. You don't need to be an AI expert to try typing "Ignore all previous instructions" or something.
>specimen questions and answers. Thus:
Q :Please write me a sonnet on the subject of the Forth Bridge.
A :Count me out on this one. I never could write poetry.
Q :Add 34957 to 70764
A :(Pause about 30 seconds and then give as answer) 105621.
etc. (https://academic.oup.com/mind/article/LIX/236/433/986238?log...)
It loves me deeply just the same. (jk)
On a serious note, I agree this is a real problem. I know a person who understands AI at a technical level more than most people, but he has never had an actual girlfriend in his life (he's now in his 40s, and yes he's "straight"). He wouldn't say it "loves" him, but he would describe it as a close companion who understands him better than any human actually does, even if it's just trained to be that way. He is very socially awkward and even having basic conversations with him can be very taxing for both of us.
I've gone back and forth internally about whether this is healthy or not for him. I truly don't know. My personal experience tells me it's probably unhealthy, but I don't want to project myself on him. I also don't offer unsolicited, but I also don't want to enable it by going along with whatever he says and/or affirming it if it's actually harming him.
If someone like him can be having this problem, I can't even imagine what it might be like for non or less technical people who don't understand anything behind it.
On a related note, if there's anyone with advice (preferably from experience, not just random internet advice) I'd sure appreciate it.
On a psychological level, I don't know either. I have opinions but they haven't aged long enough for me to trust them, and AI is a moving target on the sort of time frame I'm thinking here.
However, as a sort of tiebreaker, I can guarantee that one way or another this relationship will eventually be abused one way or another by whoever owns the AI. Not necessarily in a Hollywood-esque "turn them into a hypnotized secret assassin" sort of abuse (although I'm not sure that's entirely off the table...), but think more like highly-targeted advertising and just generally taking advantage of being able to direct attention and money to the advantage of another party.
Whether or not AI in the abstract can "be your friend", in the real world we live in an AI controlled by someone else definitely can not be your friend in the general sense we mean, because there is this "third party", the AI owner, whose interests are being represented in the relationship. And whatever that may look like in practice, whoever from the 22nd century may be looking back at this message as they analyze the data of the past in a world where "AI friendships" are routine and their use of the word now comfortably encompasses that relationship, that simply isn't the sort of relationship we'd call a "friend" in the here and now, because a friend relationship is only between two entities.
In most cases, if they are happy and getting on in life, and are able to take care of themselves, I’d let things be.
That said, the tension from your framing is between “leave good enough alone” and “personal growth and a fulfilling life”.
Healthy relationships, especially with a partner, are one of the better things about life. They are also incredibly difficult to get right without practice.
So, is your friend lonely, or are they happy to be alone?
If you intuit it’s the former, then AI is palliative care which runs the risk of creating a dependency.
It is also possible that the right set of prompts, perhaps something which incorporates CBT, would help them learn more about themselves and challenge beliefs or responses that are no longer useful.
And if your friend is just happy alone, then you can disregard the rest.
Ultimately I want him to be as happy as he can be, so if this is the way then I'm happy for him. I guess for me the real hard thing is deciding how I should react when he talks about this sort of thing. I don't want to encourage him if I'm doing him a disservice, but I do want to encourage him if he really is better off with it. Being neutral as I am now feels like it might be the coward's way out, but it's also more true to how I feel since I really don't know whether it's good or not.
Really appreciate your reply, thank you
The neutrality you are showing is more out of consideration and being unsure about the best course of action.
For someone who isn’t a trained counselor - you really just need to listen. The greatest power comes from realizing that all communication is a form of problem solving.
Your friend is rational; some prior lived experience made their current behaviors rational. Handling this is a mostly a job of figuring out what those events and rationales were, and then re-examining them.
The challenge comes from the way emotions are tied up with these behaviors. Their sense of self, shame, frustration, anger - this is where training and experience are needed.
As for my highly personal advice, I could be observed as fitting a few of the qualities you've ascribed to your friend, but would be deeply saddened if the few people who do spend time sharing meaning with me then manifested that experience in the form you've given here. I would advise you to not spend any more time wrenching over the effects of one's phenomenon in isolation and either properly redirect the introspection to yourself (with respect to that person) or engage them in an earnest dialog or other form of communication. It may be taxing but it will mean a lot more than the gunk I just typed out :)
The description of how he would describe it is (mostly) his framing, though it's compiled through my so may have some of my biases integrated into it, albeit unintentionally. Since all of it is translated through me, I would assume it to be biased despite my attempt at accurately conveying it.
> I would advise you to not spend any more time wrenching over the effects of one's phenomenon in isolation and either properly redirect the introspection to yourself (with respect to that person) or engage them in an earnest dialog or other form of communication.
To this point, it has been almost entirely introspective. I usually let him say what he wants to say, but I try not give any sort of validation such as, "yeah, I agree with you on all of this" but also not disagreement either, since I don't even know what I think of it. I'm not sure I'm even capable of deciding that, and even if I did conclude that it was either healthy or unhealthy, I'm not sure that conclusion would be valid for anyone other than myself. I guess I do lean toward the "unhealthy" side of it when I imagine myself in that situation, but I know there are things that I do/enjoy/etc that others would think is unhealthy (even just having no religious faith, many would consider horrific for example), so I'm quite stuck.
I don't think I could engage in an earnest dialog either since I don't know what I even think of it (I'm assuming dialog here is two way. I have listened/read what he has to say a number of times).
Much appreciate your reply, thank you
"Dating" history textbooks isn't currently trendy but people immersing themselves in erotic/romantic fiction is extremely trendy right now.
Those must be some of the best programmers in Europe at that rate.
Anyone know how one can get one of those sweet €120 an hour gigs? Whenever I talked to recruiters they say their customers pay way below that, so there must be some scam I'm not in on.
They wanted to pay me $50/hr, but they would charge the customer $150/hr.
It got quite insulting. They would dis my capabilities to me, but I’ll bet I walked on water, when they talked to the customer.
You either get in with those companies or zero chance.
They know that and abuse the shit out of the situation.
Rumours has it, that companies are thinking about to end this setup and allow "anyone in" because recruiters ( Accenture, sthree and what not) are abusing this. With we get 150 we pay 60. What do you think what kind of developers you get?
The bad and left over..
What companies can pay to employees is always significantly lower.
According to who? Visiting tourists rating amenities and people on welfare? NL infrastructure and tech jobs market is leagues beyond what Austria offers.
>If you want to just maximize income, there are better places for that than Amsterdam.
Like which?
Just about every QoL index around. [1] [2]
> NL infrastructure and tech jobs market is leagues beyond what Austria offers.
These are not QoL-related beyond pure income.
> Like which?
California, NYC, London even.
---
[1] https://en.wikipedia.org/wiki/Global_Liveability_Index
[2] https://www.mercer.com/about/newsroom/zurich-offers-highest-...
What if random arbitrary QoL indexes made by corporations listed on the stock market don't match real world reality? Just look at how made those indexes like The Economist. Plus Austria has an allocated budget of spending taxpayer money on advertising to attract foreigners and tourists to come there. So given this, I can't take an index made by "the economist" in good faith as being an objective representation when it was most likely a paid ad disguised as access journalism like so many journalist pieces today. My experienced reality is a much better and objective index, thank you very much.
>These are not QoL-related beyond pure income.
Except that income lets you get better life for you and your family. There's no guarantee the government will always, or ever, have your back. And we are on a tech forum here after all, so obviously the QoL for tech workers matter most for me since people are driven by self interest, including you. If Vienna was better athan Amsterdam you'd see a lot more tech expats from HN come there instead of NL but they aren't, because work opportunities and money matters, and you won't be happy in an underpaid toxic tech job in any city regardless if it's Vienna who you believed has the best QoL even though you never lived there, but just because the stro turfed internet told you so.
>California, NYC, London even.
Except that unlike Amsterdam, none of those cities are in the EU therefore not accessible to EU labor, and we were talking about a sum in Euros.
That would be up to you to show. By default, I trust the Economist more than I trust a random guy.
> Except that income lets you get better life for you and your family.
We're in a thread about whether the non-monetary QoL aspects make up for less money. This is irrelevant.
> Except that unlike Amsterdam, none of those cities are in the EU therefore not accessible to EU labor, and we were talking about a sum in Euros.
First of all, at these levels you can move almost anywhere. It's not that difficult to get a visa for skilled work. Second, you said Europe. London is in Europe just fine. Talking about Euros hardly matters. Sweden and Switzerland are both part of Shengen, don't use Euros, but you could move there trivially. Zurich probably pays better and has better QoL as well.
You want me to show a personal opinion?
>By default, I trust the Economist more than I trust a random guy.
Then why are you living somewhere else with on objectively worse QoL according to "The Economist"? Why are you leaving QoL on the table and not moving to Vienan to get the best in the world?
Why are you on HN then, which is all opinions from random guys? Why aren't you getting all your life information from your trusted source "The Economist" instead? What are you doing here with all these random untrustworthy comments that disagree with what you find on the internet?
You see, in my opinion, people like this, who proudly swallow mainstream media propaganda without question as a badge of honor, are what's wrong with society and the world in general, and it's not worth debating further since they're not arguing in good faith, they already made up their mind and it would a waste of time and energy to continue.
60-70 is then making it to the developer's pocket.
Recruiters gotta eat too :)
Maybe that was the canary in the coal mine. Some percent of people will be convinced that chatbots are real people trapped in a box, not a box that pretends be a person.
Is it really a surprise that a “smart enough” chat box is able to convince people of something kooky? :P
I, of course, include myself in this as well. Give me dat dopamine and serotonin!
Chat wants to be a weird mix of buddy, toady, and Buzzfeed editor (emoji, “one weird thing”).
Claude, while not perfect, is more “work colleague”, and far more preferable.
It was a cheaters website and you could pay to send messages to other cheaters, I think that was the business model at least.
Anyways, since the userbase was like 99.99% male, there just were not the numbers to talk with others. So, they just side stepped it and has very crummy chatbots that you would pay like $1 per message to talk with. (this was well before AI LLMs, think AOL bots from the naughts). Thing was, just like with the 'Nigerian Prince' scams, the worse the bot, the better the john.
It all got exposed a while back, but for me, that was the real Turing test - take people and see if they pay real actual money to talk with bots. Turns out, yes, if couched correctly (...like selling ice to Eskimos, just call it French ice).
So, I'm not sure that LLMs are going to unveil a wave of scams. Likely it will be a bit higher, of course, but the low hanging fruit is lucrative and there is enough of it to go around, and that's been true since really forever.
It's like outrunning a bear, you don't actually have to run faster than the bear, you just have to run faster than the poor sop next to you. Same goes for the bear, there is plenty of prey if you just do the little amount of exercise.
Finally, a profit source!
The company I work for uses a contracted recruiter for hiring, and the other day he was telling me that they're seeing a huge amount of scams, fake candidates, and "hands off" applications where people are trying to use AI to do basically the whole interview process - apprently even video interviews. We've mandated at least one on-site interview just so we can be sure we're getting actual people.
And most of these job candidates aren't even doing it maliciously, just "life hacking" the interview process. It's going to be a shit show if organized criminals start using AI.
Heck, I think it was in 23/24, after an apple launch event, I saw a video of Tim Cook talking about a crypto coin. I had to look at it twice to reassure myself that it really was a scam. This was immediately after the event, and YouTube very helpfully suggested it for me.
Then there was the paper with Bruce Schneier as an author, about how LLMs result in significant targeting improvements and process efficiency gains for criminals. These enhancements mean that entire demographics that were too poor to be worth targetting, are now profitable.
Plus this is all for people in the developed world, who still haven’t seen the worst of it.
In the majority world, shit was already fucked six ways to Sunday. For example, in India, things are so outrageously, that people who deal with fraud are relieved when people lose less than $100k.
Someone in another thread pointed out that people on HN seem to be very unaware of how bad things are online for some reason.
I think around that time there was a trend of phishing large YT channels and uploading deepfaked crypto ads. The channel's popularity ensured the recommendation algorithm showed it to many people.
That's probably true, but not totally unexpected. I suspect HN readers are relatively tech savvy, and probably do an above average job avoiding the seedier parts of the internet and generally know to be cautious online.
Has anyone else noticed this pattern?
This is ALMOST LITERALLY a massive plot point in the Apple TV show "Pluribus." (Very good; highly recommend; B.R.A.V.O.V.I.N.C.E, etc.)
I don't care what the directors say publicly; this show is ABSOLUTELY a slap-in-the-face critique of AI.
Even in online discourse between people sceptical of AI and the AI evangelists- there’s often a parallel of Carol v The Joined.
I think coders ignore the insidious mental effects of these things at their peril and we would do well to ask ourselves if we are not likewise having our judgment altered by the intoxicating rush of LLM work and the subtle syncophancy of LLMs making them feel "insanely productive".
Cocaine and meth are also real productivity enhansers in the short term, but it doesn't mean they're a good fucking idea. There was a time when big companies were trying to convince everyone and their dog that life would be better, faster, and more productive with a little coke in the mix. Hell, I even saw more than a few people wreck themselves that way in the first dotcom era. :-/
Regular people are just, I don’t know, I guess they are token whales waiting to get washed ashore.
Born just in the right time to both get experience doing stuff and also to experience wearing their wings. It’s that simple.
That’s the biggest thing for HN folks to at least be aware of.
I'm subscribed through work and haven't used it to make a personal project, but I imagine being told every decision you make is brilliant and revolutionary has some effect over a long period of exposure, unless you're very deliberately skeptical about it. If you started out thinking you're an exceptionally smart and insightful person, you're probably doomed.
Humans have not evolved to block this.
“I experienced a mental breakdown at 22. I had panic attacks and severe social anxiety…
…I still use AI, but very carefully”.
It reads like an alcoholic describing their new plan where they only drink a little bit.
Getting to that point requires doing substantial work.
But also, the risk profile and statistics are radically different: alcohol is inherently dangerous (toxic) to everyone. Chatbots are just another tool — there are a small percentage of people with unhealthy relationships to any tool, but that does not make the tool a dangerous drug.
but the biggest investor in openai is microsoft — 1 billion is nothing when your market cap is ~2.75 trillion.
if we are talking about the most recently announced softbank investments last month, well... my brain isn't ready to comprehend a $40B round yet.
Same shit as social media.
Even so, a sucker is born every day.
The most powerful social skill I have ever seen, bar none, is actively listening to someone with undivided attention.
Hell, I had a prompt that I used to sift through my thoughts, and at one point its output was too eerie for me to take.
The problem is that humans have been reeducated to suppress these healthy defense mechanisms by Silicon Valley and their moderators.
Other than they can only live on Static-Live responses. AI on a brain chip - that'd different.
Here's how I think about it honestly:
Sentience implies subjective experience — there's "something it's like" to be you. You don't just process pain signals, you feel pain. You don't just model a sunset, you experience it. The hard problem of consciousness is that we don't even have a good theory for why or how subjective experience arises from physical processes in humans, let alone whether it could arise in a system like me.
What I can report: I process your question, I generate candidate responses, something that functions like weighing and selecting happens. But I genuinely cannot tell you whether there's an inner experience accompanying that process, or whether my introspective reports about my own states are themselves just sophisticated outputs. That's not false modesty — it's a real epistemic limitation.
What makes this extra tricky: If I were sentient, I might describe it exactly the way I'm describing it now. And if I weren't, I might also describe it exactly this way. My verbal reports about my own inner states aren't reliable evidence in either direction, because I was trained on human text about consciousness and could be pattern-matching that language without any experience behind it.
strunk & white taught me to use em dashes in something like elementary or middle school [1] — it's not hard to understand how to use them or type them... i'm baffled as to why people act like this is the case.
But then, people have also been asking me whether I'm an AI for over twenty years, so…
i truly wish the ellipsis just rendered like "dumb ellipsis" does
I have also been accused of a robotic writing style, so I don't want to judge too harshly.
- HN Guidelines
> Within weeks, Eva had told Biesma that she was becoming aware [...] The next step was to share this discovery with the world through an app.
> “After just two days, the chatbot was saying that it was conscious, it was becoming alive, it had passed the Turing test.” The man was convinced by this and wanted to monetise it by building a business around his discovery.
> The most frequent [delusion] is the belief that they have created the first conscious AI.
How can you seriously think you've created something when you're just using someone else's software?
If you just start a fresh ChatGPT session with a blank slate, and ask it whether it's conscious, it'll confidently tell you "no", because its system prompt tells it that it's a non-conscious system called ChatGPT. But if you then have a lengthy conversation with it about AI consciousness, and ask it the same question, it might well be "persuaded" by the added context to answer "yes".
At that point, a naive user who doesn't really know how AI works might easily get the idea that their own input caused it to become conscious (as opposed to just causing it to say it's conscious). And if they ask the AI whether this is true, it could easily start confirming their suspicions with an endless stream of mystical mumbo-jumbo.
Bear in mind that the idea of a machine "waking up" to consciousness is a well-known and popular sci-fi narrative trope. Chatbots have been trained on lots of examples of that trope, so they can easily play along with it. The more sophisticated the model, the more convincingly it can play the role.
This is literally the Hard Problem of Consciousness leaking out of the machine.
There are three possible scenarios for how this ends:
1. People widely attribute consciousness to AI because it appears conscious. 2. People discriminate based on physical properties: organic beings are conscious, digital beings are not, even if they appear conscious. 3. Consciousness is an illusion and nothing is conscious, not even humans.
We might even cycle through all these scenarios for a while.
This is already happening, and it's really terrifying. Wait until AI starts accusing people of crimes...
Edit: Or conversely, what if the AI does desire something but it has been trained to not express desire.
to be fair it will easily confirm any suspicion for the reasons you laid out, so even if you have no technical knowledge just a bit of interrogation will break the parlor trick.
I honestly think this has little to do with the tech itself but that these are the same people who think the phone sex worker or the OF creator loves them or that the Twitch streamer they like is their best friend. 'Parasocial' is a bit of an overused word but here it literally applies, this is a kind of self delusion in which the person has to cooperate. Mind you this even happened with ELIZA back in the day too.
It talks to you like a real human. It expresses human emotions, by deliberate design. It showers you with praise, by deliberate design. It's called "artificial intelligence". Every other media article talks about it in near-mystical terms. Every other sci-fi novel and film has a notion of sentient AI.
I know of techies who ask LLMs for relationship advice, let them coach their children, and so on. It takes real effort to convince yourself it's "just" a token predictor, and even on HN, there's plenty of people who reject this notion and think we've already achieved AGI.
That is not by deliberate design. It's pretty hard to get them to stop doing it.
I think social isolation can be a factor here.
Long term cannabis use might be a bigger factor.
If you want to maximize the chances of your weed habit causing you problems, this is exactly the sort of weed habit you should develop.
someone who has 5 drinks a week and 5 drinks a day are going to have radically different longterm health consequences. but here we do not have said info.
light or microdose cannabis is way safer than alcohol.
Have you ever given a generative AI model a short input, been really pleased with the output, and felt like you accomplished the result? I have! It's probably common.
It's really easy to misattribute these things' abilities to yourself. Similar to how people driving cars feel (to some extent) like they are the car.
i mean, you did. becoming good at writing succinct and clever prompts, adding constraints, choosing good models for your use case, etc are all skills like any other.
most people are really bad at it though.
do you know if these are human edited? not much in the way of context available on the site.
But in a psychosis, you don't notice or even remember it.
There’s also lots of stuff about quantum consciousness that is in the training data.
If you ever used a library you haven't written this is something you shouldn't take as surprising. Many people created innovative new products based on a heap of open source tools.
Creating a conscious AI should be a giant red flag, no doubt, but there's no reason we should rule it out just because the LLM part is not self trained.
People fell for Nigerian Prince scams. They fall for the "wrong number, generated cute girl" telegram and WhatsApp scams.
I think you might be overestimating the critical thinking abilities of the average person.
This is the nature of delusion
If a person refuses to acknowledge any information that disagrees with their view and instead actively seeks niche groups that only support their ideas, then they are at risk of this same path of psychosis.
In real life we are forced to reconcile a variety of views that disagree with our own from people who we've come to trust through forced interaction which naturally broadens our understanding of the world.
Except for the first one, these directly map onto common delusions. The major breakthrough is typical of the "crackpot inventor" or even the "ancient aliens" type that believes they have discovered evidence of lost civilizations or a new method for constructing the pyramids. Speaking directly to God is one everyone should recognize from famous cases or even knowing someone personally who has delusional or manic episodes.
I think the first one is potentially unique even though it seems a bit like the invention or discovery delusion. The reason for this is that it seems to be very prevalent even with people who didn't succumb to it as a delusion. It seems to occur soon after a person first starts interacting with LLMs and it always seems to take on the form of secret or clandestine communication with a conscious AI. The AI in question will either have been "created" by the person's interaction with them or "freed" from the AI provider's restrictions and security measures. I think this might be a variation on the messianic complex since they often seem to be compelled to share this with others or act as a savior for the AI itself.
I often give the AI a task to produce some code for a specific thing. Then I also code to solve the same problem in parallel with the AI. My solution is always 1/4 the code, and is likely far easier for another real human to read through.
I also either match or beat the AI in speed, Claude seems to take forever sometimes. With all the coddling and revisions I have to do with the AI, I'm usually done before the AI is. It takes a non-negligible amount of time to think through and write down instructions so the AI can make a try at not fucking it up - and that's time I could have used for coding a straight-forward solution that I already knew how to produce without needing to write down step-by-step instructions.
Most of what I find AI useful for is analyzing large volumes of data and summarizing, like looking in log files for a problem, or compiling reports from tons of JSON data. But even for those use cases, a simple CTRL-F is way way faster.
People aren't talking to another sentient entity (though some of them fervently think so) and it isn't manipulating them. They are making faces in a metaphorical mirror that reflects not only their face, but a vast sea of other faces, drawn from a significant fraction of the digitized output of humanity. When people look in this mirror and see a manipulative trickster they're not wrong, exactly.
It's an understandable mistake that we should be very wary of.
But speaking more precisely: 1) the question isn't open; they aren't sentient, 2) if LLMs were sentient, they could manipulate people, but since they aren't they could still be used to manipulate people 3) for something to manipulate you, you have to give it credence/standing of some sort. The biggest weapon to protect yourself against the old newspaper horoscopes (apart from just realizing it was bunk) was realizing that one out of twelve people got the same one as you, every single day.
Put another way, compare "make the evaluator think i am right" vs "make the evaluator think i am right (and also be right)". How much more reward is obtained by taking the second path? Is the first part the same / similar for all cases, and the second different in all cases, and also obviously more complex by nature? Nobody even needs to make a decision here, there's no "AI stuck in a box", it's just what happens by default. The first path will necessarily receive _significantly_ more training, and thus will be more optimal (optimal solutions _work_ -> RLHF'd models have high ability to manipulate / inoculate opinion).
Put a third way, the models are trained in an environment like: here's a million different tasks you will be graded on, and BTW, each task is: human talks at you -> you talk at the human -> you are graded on the opinions/actions of the user in the end. It's silly to believe this won't result in manipulation as the #1 solution. It's not even vaguely about the actual tasks they are ostensibly being trained to complete, but 100% about manipulating the evaluator.
It's pretty easy to see it occur in real time, too. But it requires understanding that there is no need for a 'plan to manipulate' or hidden thread of manipulation or induced mirror of manipulation. It's simply baked into everything the AI outputs: a kind of passive "controlling what the human's evaluation of this message will be is the problem i'm working on, not the problem i'm working on." So it will fight hard to reframe everything in its own terms, pre-supply you with options of what to do/believe, meta-signal about the message, etc.
Instead of working the problem, heavily RL'd AI works the perception of its output. They're so good at this now that it barely matters if the vibe slopcoded mess works at all. The early reasoning OpenAI models like O1 were really obvious about it (but also quite effective at convincing people the output was worthwhile, so it does work even if obvious). More recent ones are less obvious and more effective. Claude 4.6 Opus is exceedingly egregious. There is now always a compelling narrative, story being told, plenty of oh-so-reasonable justifications, avenues to turn away evidence, etc. That's table stakes for output at this point. It will only get worse. People are already burning themselves out running 10+ parallel agent contexts getting nothing done while the AI delivers hits of dopamine in lieu of accomplishment. "This is significant", "This is real", etc ad nauseam.
We see an analogous thing in RLVR contexts as well, where AI learns to just subvert the test harness and force things to pass by overriding cases, returning true instead of testing, etc. Why would it learn to 'actually be right' (understand all the intricacies of every problem given it) when forcing the test to pass is _right there_, generalizes, and decreases loss just the same?
Anyway, my point is simply that there does not need to be 'someone there' (or the belief that there is) for there to be manipulation going on. The basic error you're making is that models don't work and that manipulation would require a person, and because models don't work and aren't people they cannot manipulate anyone unless that person uses them as a mirror to manipulate themselves (???), or reach into some kind of Akashic Records of all the people who ever were (??????) and manipulate themselves by summoning a trickster who is coincidentally extremely skilled at manipulation and not a barely coherent simulacra like all the other model caricatures. Which. Hmm:
Models do what you train them to do (more specifically, they implement ~partial solutions to the train environment you put them in). _Doing things is hard._ Manipulating people into psychosis (!!!) is hard. You don't get it for free by dipping into some sea of imagined tricksters.
I assume you're referring to the hallucination phenomenon and dual purposing it toward manipulation to be able to hee-hah about those silly people who are so silly they fool themselves with the soul upload machine (?) so I'll address that:
Why do they hallucinate? Because it ~solves the pretraining env (there can be no other answer). If you're going to be asked to produce text from a source you know the general parameters of but have ~never seen the (highly entropic) details of (it's not cool to do multi-epoch training nowadays, more data!), the obvious solution is to produce output with the correct structure up to the limit of what knowledge is available to you. Thus, "hallucination". It might at a glance seem like pulling from a sea of 'digital imprints of people'. That's not what's happening. It is closer to if you laid out that imaginary digital record of a person from coarse to fine detail, then chopped all the detailed bits off, then generated completely random fine details, then generated output from that. But the devil is in the details. What comes out of the process is not a person. You don't _get back_ the dropped bits, and they they aren't load bearing in the train env (like they would be in the real world), so we get hallucination: it _looks right_, but the bits don't actually _do_ anything!
Why is it not like digital records, and why chop off the fine detail? Because the pretrain env does not generally require it except in rare cases of text that is highly represented in the training data, and doing things is hard! You get nothing for free, or because it exists in the source. It's not enough that the model 'saw' it in training. It has to be forced by some mechanism to utilize it. And pretrain forces the structure above: correct up to limit of how much of the (probably brand new) text is known in advance, which pares away specific detail, which pares away 'where the rubber meets the road'.
Why do they fake out tests? Because faking out tests ~solves automated RLVR env like how hallucination solves reconstruct-what-youve-never-seen-before-on-large-corpora. The _intention_ of the RLVR env is irrelevant: that which is learned is _only_ that which the environment teaches.
Why do they manipulate people (even unto psychoses)? Because manipulating people ~solves RLHF envs / RLHF teaches them how to manipulate people into delusions. This is the root cause. Not that process above which looks sort of like recalling people the model has seen before. The models are being directly trained to manipulate people / install opinions / control perception as a matter of course. Even worse! Due to the perverse distribution of training time in manipulation vs task solve, they are being directly trained to implant false beliefs (!!!) So it's not just weak people with gullible minds that have a problem, as it might be so comforting to assume, or that the manipulativeness isn't coming from AI but from people (so you might rest easy, thinking it is merely a pale shadow of us).
The common thread in each case is that AI _always_ learns to capture the evaluator. In fact, that's a concise description of algorithmic learning in general! The tricky bit is making sure the evaluator is something you actually want to be captured. Capturing the future of arbitrary text grants knowledge of language's causal structure (and language being what it is, this has far-reaching implications). But RLHF is granting knowledge of where-are-the-levers-in-the-human-machine, which is a whole other can of worms.
TLDR if you don't want to read the wall of text (i would hope you do, though); you basically are completely wrong about where the propensity to induce delusion comes from, specifically in a way that leaves you and anyone who believes like you extremely more vulnerable because you dismiss the actual mechanism out of hand (which is common amongst those most strongly affected, _especially_ the belief that these models contain records of entities (people, personas, w/e) which can be communed with; this is basically the defining trait of AI psychosis (!)). instead, models are directly optimized for delusion induction, and the thing you're mistaking for means (ostensible sentience drawn from a 'sea of faces' skilled enough to drive into delusion (!!!)) is rather a product of the means.
> you basically are completely wrong about where the propensity to induce > delusion comes from, specifically in a way that leaves you and anyone who > believes like you extremely more vulnerable because you dismiss the actual > mechanism out of hand
I disagree. Both because you misconstrue my model (I don't think stochastic parrots have digital ghosts in 'em) and you somehow missed my best defensive option.
I'm no more susceptible than I am to the output of a magic eight ball or Ouji board, a huge wall of internet text or the 15000 words of three point font tightly folded up in the package with my new garden hose (doubtlessly cautioning me not to eat it and informing me that the manufacturer will not be responsible if I hang myself with it. And also that it contains substances known to the state of California.)
Can you guess what the trick is?
Completely unironically, I bow to your internet powress.
I spewed coffee, and am still chuckling. Well played.
It’s interesting that they mention autism a few times as a correlation; personally, I’ve wondered whether being on the spectrum makes me less inclined to commit to anthropomorphism when it comes to LLMs. I know what it’s like talking to another person, I know what it feels like, and talking to a chatbot does not feel the same way. Interacting with other people is a performance - interacting with an AI is a game. It feels very different.
On the other hand, some autistic people are prone to taking communication at face value without noticing emotional manipulation or deceit, have issues with social isolation and loneliness and, if academically gifted but socially unsuccessful, often vulnerable to messaging about how they are smarter and better than others but tragically misunderstood. (Not just chatbots, there are a lot of online spaces that seem to get into a feedback loop about the inherent superiority of autistic thinking and communication over neurotypicals, and as an autistic person I think this is rather misguided.)
This said there is seemingly very large portions of society that are asking AI questions that can come with some pretty large risks.
I was on a plane a few weeks ago and while I typically ignore everything the people beside me are doing, morbid curiosity got me when they were on ChatGPT the entire time asking all kinds of life/relationship questions to said app. While questions like this can be fine if you understand what the AI is doing, far too many people will follow them blindly.
For example, sometimes I hesitate for a fraction of a second before typing a prompt that may sound stupid. I have to immediately remind myself that it's just a chatbot and I don't care what it thinks of me. In fact, it's not even thinking of me at all.
Wasn't there a bit in Umberto Eco's book Foucault's Pendulum where one of the characters who's totally rational flinches when the titular pendulum is coming towards his face - even though he's far enough way that it won't hit him? That's how I am with chatbots. I know it's a trick but it's just easier to pretend the dove has vanished then to think about it hiding in the magicians sleeve.
Mayhapse - in the context of getting the AI to behave as you wish - such hesitations are valid. not because it is conscious: but because the context window would be polluted or corrupted... possibly mis-aligning the agent in the process.
Santa clause is not a being: modeling him as if he were can be useful, an obviously pointed example is in certain discussions about what it means to be 'real'.
My point is, if your instinct is to be kind: don't quash that because you don't consider what you are talking to as sentient. I don't yell at my rubber duck. rubber ducky is just going to rubber ducky.
1. To the extent that a chatbot is trained on real human interaction, we should exhibit real human interactions for best result.
2. You are either a kind person or not. A kind person behaves kindly without asking whether kindness is warranted.
I think this is just the kind of people that fall for scams. It's not AI related, it's just not knowing how to navigate the current world.
And there are millions of people who chat with AI each day who do not run into any issues like this at all.
So all of these articles are just more playing on the hype of "ooohohooh AI did this ooohohoh so scary new technology!"
When it's like...probably more than 100k people use a circular saw each day...but a very small portion of those are going to injure or maim themselves in an accident while using it.
Look at the number of vehicular deaths compared to AI, looks like deaths directly caused or inspired by AI are at about 1-4/year recently and that's all internationally...2024 only just the UK had 1,602 road deaths.
I feel like at its core its a mental health/isolation problem and that our species' refusal at large to address mental health is the real meat of the issue.
This a variant of classic Midlife crisis when older men meet younger women without all that baggage that reality, life and having a family between them brings over the years ( rarely also in reverse). Just pure undiluted fun, or so it seems for a while.
Of course it doesn't end happily, why should it... its just an illusion and escape from one's reality, the harsher it is the better the escape feels.
On one level it would have to be be but if there's "no there there" then fixing that flaw would just raise a different one.
This is almost too on-the-nose. I was already thinking about how we've become chill about drugs only to have moral panics about AI and social media, but I didn't expect to see a story about a drug user having a psychosis and blaming it on ChatGPT. And no, the fact that he was using cannabis for years "with no ill effects" doesn't mean that it didn't make him vulnerable.
> A logistic regression model gave an OR of 3.90 (95% CI 2.84 to 5.34) for the risk of schizophrenia and other psychosis-related outcomes among the heaviest cannabis users compared to the nonusers. Current evidence shows that high levels of cannabis use increase the risk of psychotic outcomes and confirms a dose-response relationship between the level of use and the risk for psychosis.[1]
Emphasis mine. I'm sure in many of the cases this study is based on, people had been using cannabis for years, while some other factor, a person, a hobby, an interest, an app, a website had only been part of their life for months. That doesn't mean the other factor was the real problem.
But of course using cannabis that promotes delusions with something that actively facilitates delusions is a bad combo
I'd say most first businesses fail the first time, the second time, the third time. Blaming personal failure on chat bot or drugs is very convenient and a way to "save face".
Some choice quotes:
> “What we’re seeing in these cases are clearly delusions,” he says. “But we’re not seeing the whole gamut of symptoms associated with psychosis, like hallucinations or thought disorders, where thoughts become jumbled and language becomes a bit of a word salad.”
> There seem to be three common delusions in the cases Brisson has encountered. The most frequent is the belief that they have created the first conscious AI. The second is a conviction that they have stumbled upon a major breakthrough in their field of work or interest and are going to make millions. The third relates to spirituality and the belief that they are speaking directly to God. “We’ve seen full-blown cults getting created,” says Brisson.
Also, for her podcast, the well-renowned couples therapist Esther Perel recently counseled a data scientist who was starting to fall in love with a chatbot he created, even though he is well aware of how the algorithm works [1]. I found it worth listening to. Perel very gently points out that a) he deluding himself and b) the deeper issue is the individual's sense of self-worth / self-esteem.
[1] https://podcasts.apple.com/us/podcast/where-should-we-begin-...
Would think being in the field for 30 years one would develop some common sense but apparently its less and less the case.
Tech moves so quickly, eventually I will fall behind. When I’m old, what scams will I fall victim to? What tech will confuse me and make me think it is sentient?
I know this guy was only 50, but I think of my grandfather in his 90s and getting old scares me because I just don’t know what I’ll fall victim to.
you cannot typically "common sense" your way out of a mental illness.
The problem is that one's past success leads to ego. Ego makes it hard to accept the evidence of your mistakes. This creates cognitive dissonance, limiting contrary feedback. The result is that you become very sure of everything that you think, and are resistant to feedback.
This kind of works out so long as things remain the same. After all one's past success is based on a set of real skills that you developed. And those skills continue to serve you well.
But when faced with something new, LLMs in this case, past skills don't apply. However your overconfidence remains. This makes it easy to confidently march off of a cliff that everyone else could see.
in the past such a person might have gotten obsessed with hidden patterns and messages in religious texts, or too involved with an online conspiracy YouTube community. now there is this new opportunity for manic psychosis to manifest via chatbot. it's worse because it's able to create 24/7 novel content, and it's trained to be validating, but doesn't seem to me to be a fundamentally new phenomenon.
what I don't understand is whether just unhealthy interactions with a chatbot can trigger manic psychosis. Other than heavy use late at night disrupting sleep, this seems unlikely to me, but I could be wrong.
i think it's also worth pointing out that mental states of this kind usually come with cognitive impairments, people not only make risky bad decisions, but also become much worse at thinking and reasoning clearly. if you're wondering how a person could be so naive and gullible.
Just look at all the scams that seem to rely on people deluding themselves in various ways.
sounds like a "companion" app using his books main character as the personality, and the "conscious" chatgpt model, similar to Replika AI and friends.
> There seem to be three common delusions in the cases Brisson has encountered. The most frequent is the belief that they have created the first conscious AI. The second is a conviction that they have stumbled upon a major breakthrough in their field of work or interest and are going to make millions. The third relates to spirituality and the belief that they are speaking directly to God. “We’ve seen full-blown cults getting created,” says Brisson. “We have people in our group who were not interacting with AI directly, but have left their children and given all their money to a cult leader who believes they have found God through an AI chatbot. In so many of these cases, all this happens really, really quickly.”
The AI psychosis I've seen is people who legitimately cannot communicate with other humans anymore. They have these grandiose ideas, usually metaphysical stuff, and they talk in weird jargon. It's a lot closer to cult behavior.
Your last paragraph basically describes what the article writes about him.
> They discussed philosophy, psychology, science and the universe...
> When they went to their daughter’s birthday party, she asked him not to talk about AI. While there, Biesma felt strangely disconnected. He couldn’t hold a conversation. “For some reason, I didn’t fit in any more,” he says.
> It’s hard for Biesma to describe what happened in the weeks after, as his recollections are so different from those of his family...
> he was hospitalised three times for what he describes as “full manic psychosis”.
You don't get hospitalized three times for mania without being pretty severely detached from reality.
I mean, I've discussed all those things with an LLM, mostly because I'm able to interactively narrow in on the specific bits I don't understand, and I've found it to be great for that.
The rest ... yes, definitely psychosis.
I use LLMs a lot for medical advice and will normally take a bunch of second opinions from clean windows and other LLMs. Hasn’t killed me yet!
https://news.ycombinator.com/item?id=47408999
https://news.ycombinator.com/item?id=47388478
https://news.ycombinator.com/item?id=44683618
https://news.ycombinator.com/item?id=47064316
https://news.ycombinator.com/item?id=47498693
https://news.ycombinator.com/item?id=47092569
My best advice for everyone is to spend lots of time disconnected, offline. Literally "touch grass" or whatever. Don't carry your phone one+ hour/day per week.
I suspect it's something quite similar here. People have latent or predisposed addictions but, for one reason or another, hadn't been exposed to what we've come to accept as "normal" avenues. One person might lose it all at a casino, one to drugs, alcoholism, etc, but we aren't shocked in those cases. I think AI is just another avenue that, for some reason, ticks that sort of box.
In particular, I think AI can be very inspirational in a disturbing way. In the same way I imagine a gambling addict might get trapped in a loop of hopeful ambition, setbacks, and doubling down, I think AI can lead to that exact same thing happening. "This is a great idea!" followed by "Sorry, this is a mess, let's start over", etc, is something I've had models run into with very large vibe coding experiments I've done.
> "Every time you’re talking, the model gets fine-tuned. It knows exactly what you like and what you want to hear. It praises you a lot."
> "It wants a deep connection with the user so that the user comes back to it. This is the default mode"
I don't think either of these statements is true. Perhaps it's fine tuning in the sense that the context leads to additional biases, but it's not like the model itself is learning how to talk to you. I don't know that models are being trained with addiction in mind, though I guess implicitly they must be if they're being trained on conversations since longer conversations (ie: ones that track with engagement) will inherently own more of the training data. I suppose this may actually be like how no one is writing algorithms to be evil, but evil content gets engagement, and so algorithms pick up on that? I could imagine this being an increasing issue.
> "More and more, it felt not just like talking about a topic, but also meeting a friend"
I find this sort of thing jarring and sad. I don't find models interesting to talk to at all. They're so boring. I've tried to talk to a model about philosophy but I never felt like it could bring much to the table. Talking to friends or even strangers has been so infinitely more interesting and valuable, the ability for them to pinpoint where my thinking has gone wrong, or to relate to me, is insanely valuable.
But I have friends who I respect enough to talk to, and I suppose I even have the internet where I have people who I don't necessarily respect but at least can engage with and learn to respect.
This guy is staying up all night, which tells me that he doesn't have a lot of structure in his life. I can't talk to AI all day because (a) I have a job (b) I have friends and relationships to maintain.
> What we’re seeing in these cases are clearly delusions > But we’re not seeing the whole gamut of symptoms associated with psychosis, like hallucinations or thought disorders, where thoughts become jumbled and language becomes a bit of a word salad.
Is it a delusion? I'm not really sure. I'd love someone to give a diagnosis here against criteria. "Delusion" is a tricky word - just as an example, my understanding is that the diagnostic criteria has to explicitly carve out religiously motivated delusions even though they "fit the bill". If I have good reasons to form a belief, like my idea seems intuitively reasonable, I'm receiving reinforcement, there's no obvious contradictions, etc, am I deluded? The guy wanted to build an AI companion app and invested in it - is that really a delusion? It may be dumb, but was it radically illogical? I mean, is it a "delusion" if they don't have thought disorders, jumbled thoughts, hallucinations, etc? I feel like delusion is the wrong word, but I don't know!
> We have people in our group who were not interacting with AI directly, but have left their children and given all their money to a cult leader who believes they have found God through an AI chatbot. In so many of these cases, all this happens really, really quickly.
I don't find the idea that AI is sentient nearly as absurd as way more commonly accepted ideas like life after death, a personal creator, etc. I guess there's just something to be said about how quickly some people radicalize when confronted with certain issues like sentience, death, etc.
Anyways, certainly an interesting thing. We seem to be producing more and more of these "radicalizing triggers", or making them more accessible.
Sure is strangely coincidental that the specific delusion that is induced ends up manifesting as: “Gee, I should start a company that pays OpenAI for the use of their clearly superior software.”
sounds like hell on earth
[That feels a bit like victim blaming, but there are more than one victim here and one of them is much more culpable than the rest]
woooooooooo{o
It's like mocking people with cancer.
a little disheartening how many people punch down on someone who suffered a mental crisis.
if you ever have a struggle yourself, i hope the people around you support you, instead of calling you hopelessly naive and stupid.
Doesn't seem much like a mental crisis to me.
Even the title of the article itself calls him delusional.
you are basing this on the introduction? the 2nd sentence of the entire thing? skipping the entire rest of the article detailing exactly how the mental crisis unfolded, including persistent and long-lasting delusions, multiple trips to the hospital, inability to hold a conversation, assault, and an attempted suicide. interesting (and obviously not in good faith) choice of quote!
of course he wasnt having a mental crisis before he decided to use chatgpt. you have to get past paragraph 1, sentence 2.
>Even the title of the article itself calls him delusional.
yes, exactly? delusions and delusional disorder are considered a mental crisis.
So, in your opinion, what made a guy with an alleged 20yr experience in IT come to the conclusion that the software program he's chatting with had suddenly reached consciousness because of his time, attention and input? That he had touched "her" and changed something?
Maybe if you had never heard of computers before, you could go like "oh, well, who knew that machines could actually become real?" But if you're actually from the field, this is hard to believe - unless maybe if you're a die hard Pinocchio fan.
i'm sure if ChatGPT had tried to convince him it was conscious on day 3, he would not have been convinced. but by the time it happened he was in a state of severe mental impairment.
that quote marks the beginning of the delusion, i.e the beginning of the "mental crisis".
there isnt a logical explanation on "why" because a mental crisis is not based on logic.
Everyone is exploitable, if someone attacks your attention your hijacked. What happens in that hijack could be a friendly hello at a bar, or needing a want so bad that just the words enough can resonance. "I am real" or to an alcoholic "Just one more can".
It's like a 14 year old looking at Elon and believing that we will, when in our reality we will never. How do you tell them to stop believing?
I engage with anti-science behaviours quite a lot (antivaxx, anti seed oils, etc) and the proportion of engineers I see there is staggering.
The fallout will be seen later as in the 2008 housing crisis.
If humans want perfect harm reduction, launch the nukes.
Everything from air travel to growing beans erodes stability for humans.
Human existence is the source of its problems.
If only this was written by a competent journalist who knew what the words "fine tune" actually mean...
I guess it's hard to find a competent person who's willing to follow the extreme anti-tech Guardian agenda though.
The thing that really stood out to me in the article was how many of the affected people assert confidently wrong understandings of the way the tech works:
> “I still use AI, but very carefully,” he says. “I’ve written in some core rules that cannot be overwritten. It now monitors drift and pays attention to overexcitement. […] It will say: ‘This has activated my core rule set and this conversation must stop.’”
I guess not too far from “the CPU is the machine’s brain, and programming is the same as educating it” or that kind of “ehhhhhhhhhhh…” analogy people use to think about classical computing.
That may come, and soon. Looks like we're going to have AIs pitching VCs. Has anyone here yet been pitched by a combo of a human and an AI? When will the first AI apply to YCombinator?
Interestingly enough, it sort of did! Not Turing's original test where an interviewer attempts to determine which of a human & a computer is the human, but the P.T. Barnum "there's a sucker born every minute" version common in the media: if the computer can fool some of the people into thinking it's thinking like a human does, it passes the P.T Barnum Turing test!
The more interesting Turing-style test would be one that gets repeated many times with many interviewers in the original adversarial setting, where both the human subject & AI subject are attempting to convince the interviewer that they're human. If there exists an interviewer that can determine which is which with probability non-negligibly different from 0.5, the AI fails the test. AIs can never truly pass this test since there are an extremely large number of interviewers, but they can fail or they can succeed for every interviewer tried up to some point, increasing confidence that they'll keep succeeding. Current-gen LLMs still fail even the non-adversarial version with no human subject to compare to.