This doesn’t seem like it disagrees with what Mathew was saying?
Oh, it’s a whole active field of study (formal semantics), but one that I personally don’t think is very relevant to actual human life. I’m of the opinion that most of the interesting things we can do with words can’t be translated to lambda calculus.
I read his post as claiming that LLMs reason exclusively through System-2-style, deliberate, language-centric methods. I don’t see any other way to interpret this:
It’s arguable that [what LLMs do] isn’t reasoning at all because while we as humans tend to express our reasoning processes as words, it’s far from clear whether that’s how we ever reason.
Or this:
This leads cognitive scientists to suspect that most decisions are made subconsciously and non-linguistically in System 1, which in turn suggests that our thought process is fundamentally different to the way LLMs process information.
Or this:
It’s possible that purely linguistic LLM-style processing might be better than human-style reasoning for some problems.
Perhaps there’s another interpretation that’s eluding me?
The fact that, for example, Claude adds numbers through a heuristic, approximation-based process that it’s then unable to explain says to me that LLMs, like humans, don’t do all their reasoning in a System-2-style, deliberate, language-centric way.
My understanding is that when you ask for an explanation after the fact, the model only has the text of the answer to work with. Any additional internal/hidden state that may have existed while the original answer was being generated is no longer there.
Yes, this is correct. But also, there is empirical research supporting the conclusion (which we could already hypothesize from this theoretical understanding) that post-hoc explanations are uninformative.
In principle, one could reasonably hope (for example) that, because there is a commonality between the process generating the initial decision and the process generating the “explanation,” the explanations might therefore align with the decision. But there is lots of empirical evidence of situations where this is not the case.
The fact that, for example, Claude adds numbers through a heuristic, approximation-based process that it’s then unable to explain says to me that LLMs, like humans, don’t do all their reasoning in a System-2-style, deliberate, language-centric way.
It may be a heuristic approximation-based process, but it’s still just manipulating tokens, isn’t it? I don’t see anything in the article suggesting that it has a world model incorporating integers, or a non-token-based intuitive approximation system.
The example they give of Claude performing an approximate calculation and then separately working out the rightmost digit has paths that look like how Common Core teaches kids to do it, so there are almost certainly tons of examples in its training data. That’s also how I do mental arithmetic using System-2.
The neural network is full of interior nodes that don’t correspond directly to tokens, so I don’t think it can meaningfully be said to be “just manipulating tokens”.
LLMs are stochastic during training and inference, so technically they’re approximations to heuristics. It’s approximation all the way down. There are literally billions of parameters in a large model, but nothing at all like human System 2, the latter of which reflect human brain architecture. It’s more like System 1 all the way, though scratchpad approaches like chain of thought and thinking models fake System 2. LLMs “shortcut” reasoning a different way, basically reward hacking an approximation of the surfaced artifacts of reasoning. Tokens are only manipulated at the input and output layers. What happens in between is a matter of numerical “activations”, an ensemble working in parallel, and making sense of them from the outside is far from straightforward. I wouldn’t say a world is encoded. I’d say an approximation of the textual residue of a world modelled via human cognition and writing has been trained in, which is quite a few removes away from a direct world model.
So, it seems I’m getting to grips with how to use AI for my needs. For instance, a couple of days ago I needed some commands for a .BAT file to copy about 26 files into over a thousand different folders (if you’re curious: I went back to ZZT and MegaZeux, and I realised I really ought to copy all the runtime files into the folders of each individual game, because of the way I have things set up on Android using Magic DosBox. So I needed to copy 26 files from the MegaZeux runner and 3 files from the ZZT runner into the appropriate folders - with well over a thousand folders of games for each engine), and chatGPT provided me with something that worked. I tested it small-scale, saw it was good, ran it, no trouble.
Mostly, though, it has been very helpful with ScummVM.
…because I don’t find the people at ScummVM particularly friendly or approachable. 
(disclaimer: they are doing amazing work, and I’m extraordinarily grateful to them) So chatGPT was an improvement. It particularly helped me understand what I should do in terms of shaders to account for EGA games in the era of CRT monitors, so I could get a more authentic experience.
Something a bit weird happened just now, in that… ChatGPT quipped at me! I was not expecting that.
I was trying to figure out how to add custom “icons” to ScummVM entries in Android (answer: it’s way more trouble than it’s worth), and I was using, for testing, a game called “Alien Cow Rampage: Orion Needs Your Milk”. The acronym for that being… oh, check it out yourself. 
So at the end I give up and say so, ChatGPT predictably supports me in that decision and proceeds to highlight how much information I actually was able to get with just a few minutes of experimentation (I don’t care how much you butter me up, ChattieBoy, not until third date at least), and then…
…and then it quips this:
At least Alien Cow Rampage: Orion Needs Your Milk remains memorable enough that it doesn’t really need an icon to stand out in the list.
Cheers, and happy adventuring.
You guys who deal with AI more frequently may not find anything special about this, but… an unprompted jokey comment about the game’s name and how I don’t need an icon for it?
I ought to feel awed and amazed, right?
Then why do I feel taken aback and creeped out?
You can change its tone in the personalization settings.
Yeah, but it’s not just tone - it’s the nature of what it said, and how it came up with it. The “thought process” necessary to do that at that time. Caught me off-guard. And creeped me out.
If a human, even a subordinate employee, were to make a quip like that in an otherwise professional setting, it’s unsurprising because we know that humans have their own desires and goals (to find fun & connect with other people) that inevitably inform their actions at all times, regardless of what their “primary” goal is at a given moment. We don’t expect humans to turn off their humanity and execute a particular assigned task to the letter.
To see the same behaviour from an artificial tool is different, because we typically don’t want our inanimate tools to have an inner life of their own, with their own personal goals conflicting with what we’ve assigned them to do. The tool is not behaving as expected/intended. And you can of course offer various explanations for why it added that quip, but most of them are at least a little bit worrisome in one way or another, IMO.
