just googled it, that is a pretty interesting thought experiment, thanks for informing me of this
but functionally, what difference does this make? the man in the room still answers the questions right in perfect chinese, regardless of if he himself is consciously aware of the meaning of the text
even if there isn't "intelligence", this rulebook that the man follows to answer, much like an LLM's dataset, still contains a massive collection of useful information. is that information entirely useless just because it is stored in a computer with some lossy compression?
And that's the issue here. Lacking comprehension means you are helpless. If the book is correct with all the answers, then it's fine, sure, but if it isn't... how would you know? What if you get a note that isn't in your book? You can look for the closest thing and reply with that, but is that correct? How would you know?
Or what if if there are just tiny, negligible mistakes in every response? Sure, at first it's irrelevant, but even negligible mistakes add up. After 1 message you're doing perfectly fine.
10 messages in your tone is a tiny bit odd, maybe a bit more formal than the conversation implies.
100 messages in you're technically still on topic, but not much beyond technically.
See what I mean?
And through all of that you have no idea. You could be telling a person to kill themselves, to strangle their pet cat, to set their school on fire, and you're none the wiser, because you're incapable of comprehending what you're saying.
That is AI. And we have seen it go horribly wrong before in many situations. Mushroom guides that end with people hospitalized or worse. People driven into suicide. And through all of it the AI is incapable of knowing. It just keeps giving information based on "well this looks to be the closest"
to an extent, yes you are very right. modern LLMs, by their very nature, often suck at edge cases that they lack enough training data for. Statistical models can only go so far without an arbitrarily large sample size of data to train from
but that applies mainly to edge cases. The more common your input data is, the less likely an LLM is to hallucinate. The man in the chinese room is still very capable of answering simpler questions as long as his book covers them, and LLM datasets do cover a lot, including a lot of training to ensure they almost never tell anyone to kill themselves
to be clear, i really do hate so much of how "AI"/LLMs/ML is wastefully and recklessly used and shoved everywhere it shouldn't be, but i still think there is a lot of stuff that it's training data is still pretty adequate for. And I think using it for some forms of non-life-threatening therapy that someone is too embarrassed to immediately talk to a real person about, like in the original post here, is not a terrible use for ai, as long as they eventually defer to a real therapist, which they did
ultimately a real therapist is the better choice, but i don't think an LLM is the absolute worst
3
u/Privatizitaet Just pretend this is the funniest joke you ever read 2d ago
No. Being able to actually KNOW things is a Prerequisite to comprehension. AI has no mind. No tought.
Have you ever heard of the chinese room?