LLMs: The actual “Chinese Room Experiment”

(NOT a “thought” experiment!)

By now, everyone should be familiar with the “Chinese Room” argument by John Searle. It argues that even if a machine behaves as if it can understand Chinese, the machine does not “really” understand it, and basically goes like this:

Imagine a person who speaks only English sitting inside a room. They have a massive rule book that tells them how to manipulate Chinese symbols. When a slip of paper with Chinese characters is slid under the door, the person looks up the symbols in the book, follows the instructions to produce a new set of symbols, and slides the response back out. To the person outside, it appears as though the person inside understands Chinese. However, the person inside understands nothing of the language; they are merely blindly manipulating symbols they do not understand.

The supposed conclusion of the thought experiment was that pure symbolic manipulation is insufficient for semantic understanding. Searle argued that because the person inside the room follows rules without grasping meaning, a computer running a program, no matter how sophisticated, cannot possess true consciousness or intent. It simulates understanding rather than embodying it.

Personally, I think the flaws of Searle’s argument are clear, which is that he performs a sleight of hand by putting a presumed intelligent being into a mundane position of mindlessly shuffling symbols around, and then immediately proclaims the system has no real understanding of Chinese because the only apparent candidate with intelligence does not possess understanding of Chinese. Obviously, the (IMHO correct) counter-argument is that the overall system understands Chinese, even though the system contains a conscious human being who does not.

I do not think anyone in 2026 would make such a mistake. These days, the “Chinese Room” is not a thought experiment at all. It is a real thing, and we call them Large Language Models. The “rule book” corresponds to the weights of the LLM, and the “person” mindlessly shuffling symbols around is the architecture. As Andrej Karpathy showed, it can be implemented in a couple hundred lines of code. Nobody for a second thinks that the 300 lines of Python is intelligent. The “magic” has always been in the billions of parameters of weights.

Interestingly, there was a less famous variant of the Chinese Room idea that makes a more compelling argument against functionalism. It was (as I learned when writing this article) from Ned Block, who supposed that the “rule book” was simply a huge lookup table and contained an exhaustive list of all sensible conversations and their responses. The machine would simply have to perform relatively simple searches to find the appropriate responses, thus could not have had any real understanding; But the rule book also had no intelligence, since it was just a simple (though huge) list of conversations. Neither of the components could be reasonably considered as having any intelligence, and thus it can be concluded that the whole system did not have any intelligence either.

I personally think the last reasoning step may not be logically airtight, since it is generally possible that a system is greater than the sum of its individual parts. But that does not matter, because as we have discovered, LLMs’ weights are simply highly compressed data. With the success of LLMs, people have recognized that compression is intelligence. LLMs’ weights are essentially compressed data generated by human civilization: books, papers, articles, conversations… On an abstract level it truly is a lookup table as imagined by Ned Block, except that it is compressed. The lookup table feels lifeless and mechanical, but after heavy compression something that feels qualitatively different emerges.

Of course, people today still have heated debates about the nature of LLMs. Nobody really doubts they can write code, complete tasks, etc. to a useful extent in some contexts. But are they truly intelligent? Or are they merely “stochastic parrots”? Can they ever be as creative as humans?

I have no strong opinions on these issues (except perhaps regarding “stochastic parrots” – I’m strongly convinced the term can apply to humans as well), this essay is intended to simply highlight the fact that the “Chinese Room” is no longer a “thought experiment” as described in old philosophy textbooks. It is now a real thing, and we call them LLMs.