The AI tried to talk him out of killing himself and responded as though he would instead come home to her. I’m not sure what’s unreasonable about that. Hell, I’d justify far less reasonable responses because an AI is incapable of reason.

There is no thought process. The AI looks at the existing conversation and then responds using words a human would be statistically likely to. It doesn’t understand anything it’s saying. It doesn’t understand human life, nor the fragility or preciousness of it. It doesn’t know what life and death are. It doesn’t know about depression or suicide. It doesn’t know the difference between real and make believe. It just spits out stochastic tokens. And it does so in a way that it’s impossible to understand why it outputs what it does on the scale of a human lifetime because every single token depends on billions of parameters, each informed by every single bit of training data.

For as smart as AI appears to be, it’s just a completely dumb computation black box. Exactly in the way power tools and fire are dumb.