Character AI has filed a motion to dismiss a case brought against it by the parent of a teen who committed suicide allegedly after becoming hooked on the company's technology.
Of course they have logical flaws. Everyone should be made aware of that before using AI. A table saw will cut your finger off. Matches will burn down your house. It’s the nature of the thing. That doesn’t make them unuseful. I use them to help with coding all the time. It’s wrong frequently, but it’s still useful and saves me a lot of time. But absolutely no one should ever rely on any output as if it were gospel. Ever. That is a user flaw, not a tool flaw. Though, possibly a communication flaw as you can’t rely on every random person to understand that.
What single point of failure? In fact, what was even the failure here? The AI was roleplaying and has no capacity to understand the person it’s talking with is taking it seriously or is mentally unstable.
The failure is reasonable scenarios where the fantasy needs to end. AFAIK the only other way this could’ve ended, without harm, would be if the kid just decided to stop chatting (highly unlikely) or if someone looked over his shoulder at what was being typed (almost as unlikely). As others have said, it’s hard to know what is the AI thought process or predict how it would react to a situation without testing it. So for all they know the bot could have said, in the first place, “Let’s die together.”
The AI tried to talk him out of killing himself and responded as though he would instead come home to her. I’m not sure what’s unreasonable about that. Hell, I’d justify far less reasonable responses because an AI is incapable of reason.
There is no thought process. The AI looks at the existing conversation and then responds using words a human would be statistically likely to. It doesn’t understand anything it’s saying. It doesn’t understand human life, nor the fragility or preciousness of it. It doesn’t know what life and death are. It doesn’t know about depression or suicide. It doesn’t know the difference between real and make believe. It just spits out stochastic tokens. And it does so in a way that it’s impossible to understand why it outputs what it does on the scale of a human lifetime because every single token depends on billions of parameters, each informed by every single bit of training data.
For as smart as AI appears to be, it’s just a completely dumb computation black box. Exactly in the way power tools and fire are dumb.
Of course they have logical flaws. Everyone should be made aware of that before using AI. A table saw will cut your finger off. Matches will burn down your house. It’s the nature of the thing. That doesn’t make them unuseful. I use them to help with coding all the time. It’s wrong frequently, but it’s still useful and saves me a lot of time. But absolutely no one should ever rely on any output as if it were gospel. Ever. That is a user flaw, not a tool flaw. Though, possibly a communication flaw as you can’t rely on every random person to understand that.
Not saying flaws make them useless, I’m saying the flaws mean they shouldn’t be a single point of failure.
What single point of failure? In fact, what was even the failure here? The AI was roleplaying and has no capacity to understand the person it’s talking with is taking it seriously or is mentally unstable.
The failure is reasonable scenarios where the fantasy needs to end. AFAIK the only other way this could’ve ended, without harm, would be if the kid just decided to stop chatting (highly unlikely) or if someone looked over his shoulder at what was being typed (almost as unlikely). As others have said, it’s hard to know what is the AI thought process or predict how it would react to a situation without testing it. So for all they know the bot could have said, in the first place, “Let’s die together.”
The AI tried to talk him out of killing himself and responded as though he would instead come home to her. I’m not sure what’s unreasonable about that. Hell, I’d justify far less reasonable responses because an AI is incapable of reason.
There is no thought process. The AI looks at the existing conversation and then responds using words a human would be statistically likely to. It doesn’t understand anything it’s saying. It doesn’t understand human life, nor the fragility or preciousness of it. It doesn’t know what life and death are. It doesn’t know about depression or suicide. It doesn’t know the difference between real and make believe. It just spits out stochastic tokens. And it does so in a way that it’s impossible to understand why it outputs what it does on the scale of a human lifetime because every single token depends on billions of parameters, each informed by every single bit of training data.
For as smart as AI appears to be, it’s just a completely dumb computation black box. Exactly in the way power tools and fire are dumb.