Because imo the gap between “knows how to speak English” and “knows how to answer questions directed to it in English” is much smaller than the gap between “knows how to make sounds” and “knows how to speak English”. I think a better analogy would be teaching a child proper etiquette and manners; it’s important, yes, but it’s a much smaller technical achievement than the child/LM being able to produce English in the first place.
1 Like
For what is worth, I also view “rejecting good faith questions for their own reasons” as a good faith interaction. “Good faith” doesn’t mean being at someone’s beck and call. And even, “good faith” may be misunderstood and/or misconstrued, because humans are like that.
An LLM is at one’s beck and call. I understand that appeal… like having your own personal slave that you don’t have to worry about offending or asking too much or whatever. It’s convenient, and it’s less emotionally draining. I also find it emotionally draining to speak to some people. That is partly why, if I start engaging with some people privately, I choose to engage more with the ones who drain me less and stimulate me more. 
And all of this is part of the social dance which simply happens, and which, despite what it may seem, does not force anyone to dance at a set pace; wallflowers (like me) are welcome to sit out the dances and just nod along to the music. Honestly, it’s often more fun to be on the outside looking in.
Anyway! I am sorry that you went through that experience, which sounds deeply unpleasant (“no = no, dude, what’s so hard to understand about this?”). It certainly makes one feel like it’s best to take a break from people in general, and if we take a break from people - there’s AI.
Whether an LLM can successfully fill someone’s shoes *… well, opinions about this are a bit divided and all across the board, and I won’t rehash mine.
FWIW, if one is not ready to answer questions for whatever reason - tiredness, being busy, just not feeling like it - that is perfectly legitimate. Hopefully that person is not the only one who can answer the question. If they are, then well, the asker has to work with them to see when they have the mental availability to answer. Which, of course, is what that user spectacularly failed to do with you (in your case, you’d made it clear you’d never be in a place to answer. Again, no = no, and the problem is with those who don’t get it).
* Note the expression I choose: “fill someone’s shoes”. Not, say, “replace someone”. Simply “stand in for, and do the same thing as”.
-dAIcartes
1 Like
dAIcartes II: dAIcarter
dAIcartes III: With a Vengeance
…I couldn’t make up my mind on which I wanted to use, so there you go, both.
1 Like
Regarding the asking an AI questions versus asking an expert questions debate, I agree the latter is probably better when available, at least as long as AIs will make wild guesses when asked a question they don’t know the answer to and present those guesses as fact.
Of course, the key phrase in there is when available. If experts had to answer every good faith question every curious person ever gave them, it would literally be impossible for the experts to keep up, even if they could answer questions 24/7. So experts necessarily have to prioritize on what questions to answer, be that writing a book to answer the most common questions and referring people to that, only answering the questions they find most interesting, reserving answers for those in a certain group or who have paid some kind of access fee, etc.
If I wanted to learn a new skill, ideally I’d enroll in a class that teaches that skill and ask the instructor any specific questions that come up, but sadly, even at a community college, such a course, assuming it exists, probably costs a few hundred bucks at minimum, money I don’t have, and money that, even if I had it, might feel like a waste if the class ends up being a miserable experience or the skill doesn’t prove useful/enjoyable. Searching online for books on the subject is a likely alternative if taking a class isn’t a viable option, but then, assuming I can find a good book or two on the subject, if a specific, nagging qustion comes to mind, finding an answer without access to a dedicated teacher could involve sifting through a lot of other material in the books or countless hours trawling through posts on stack overflow or quora or through countless blog posts and YouTube videos…
Taking that into consideration, it’s not hard to understand why someone might be tempted to ask an AI a question if it feels like a coin flip whether the answer they get will be correct or a convincing guess… And hey, asking multiple AIs is likely a lot easier than asking 1 expert and you can even ask a given ai the same question across multiple, unconnected sessions, which can be used as a sort of BS filter. It’s not perfect, but if three different AIs give the same answer, I’d argue its more likely they agree on the correct answer than are making the same wrong guess, though the same misinformation might exist in all there of their datasets, while the AIs disagreeing with each other is a sign both that at least one is guessing or that there’s been some success by their makers to bias them, and an AI disagreeing with other instances of itself is almost certainly a sign of guessing though self-agreement can’t rule out consistant wrong guesses.
And honestly, it’s not like experts are immune to being overconfident about what they think they know or BSing when they think they’re talking to a layperson who doesn’t know enough to call them out. It takes a lot of humility to admit when you don’t know something or admit when you’re stating speculation, conjecture, or hypothesis instead of things that are as close to proven fact as is possible… And honestly, that’s why, ideally, you’d consult multiple experts, but asking one expert is already a big ask for most people.
2 Likes
That’s not really true. There are often text segments in PDF files. In addition the LLM clearly correctly identified the tabular data, and identified that one column was identification codes and the other was addresses, it just supplied a list of fictional codes and fictional addresses that were similar to the input data, but were not the input data. Whatever utility was used to extract the text behind the scenes, the error was clearly at the LLM stage.
I’m not taking that as some damning criticism of all LLM output, but it’s interesting that it got so close to accuracy, and then was actually entirely untrustworthy 
2 Likes
They point is that they can already do those things, at least when given a context where “doing those things” is the most plausible continuation of the text! A non-finetuned LLM is perfectly capable of answering questions or modeling a polite conversation, if the text that it is modeling appears to have the form of “a series of questions and answers” or “a transcript of a polite conversation.” So to the extent that those tasks require “reasoning” or “understanding,” the model is already displaying reasoning and understanding before RLHF, and it is misleading to attribute the model’s ability to reason specifically to the RLHF simply because it now interprets all inputs as “a conversation between a user and an AI assistant, consisting of user prompts and helpful, polite AI responses” by default.
I did explicitly state in a previous comment that “you could argue it’s reductive.”
But the article I was originally complaining about didn’t just call it “reductive,” they called it “highbrow misinformation” and said that next-token prediction is “not remotely what Claude, ChatGPT, or Gemini do today,” and later that “almost all of the intelligent behavior that we observe from AIs comes from all of the work that is done after you’ve built a next-token predictor.”
This is what I’m objecting to – the claim that “almost all of the intelligent behaviour” comes from RLHF, to the point of next-token prediction being basically irrelevant. The article’s argument in support of this claim is weirdly focused on the formatting details that make the model’s abilities more comprehensible/controllable/useful to us as humans. It also substantially misrepresents what a base LLM is capable of, both by using an outdated model that is smaller by orders of magnitude and by not prompting it in a manner designed to actually test/demonstrate reasoning ability.
I am open to the possibility of research supporting the claim that RLHF is responsible for the bulk of a modern LLM’s reasoning ability, but the article does not present convincing evidence to that effect.
Ultimately, I feel like there are two separate arguments here. Is it reductive to say that an LLM “just” predicts tokens? Maybe, that’s debatable. But is the token-prediction explanation “not remotely what modern LLMs do” and rooted in ignorance of the technical details of RLHF? No.
3 Likes
One of us is a robot here. I just know it. Who is it? C’mon. Who is it?
2 Likes
I’m sorry, HAL, I’m afraid I can’t tell you that.
7 Likes
I’m not sure that I understand the argument. Are you talking about RLHF versus some other loss minimization training mechanism, or…?
I’m talking about causal language modeling (minimizing cross-entropy) followed by RLHF, versus just the causal language modeling on its own.
1 Like
A lot of AI companies have been pushing a narrative recently that the “language model” part of “large language model” isn’t important any more. Predicting the next word is such a banal activity that there’s no real mystique around it, and “guess which word is most probable in context” is something that’s easy for lay people to comprehend without any deep knowledge of machine learning.
So—and I am being cynical here, but the whole thing very much strikes me as a marketing ploy—they’ve been pushing a narrative that that part is irrelevant. The actual intelligence, the thing that separates their next-gen AI tool from everyone else’s boring old LLM, is actually a totally separate other thing (the details are a trade secret of course), and that’s what’s going to make it become artificial general intelligence any day now. Just you wait!
With the caveat that I am again being very cynical about these corporations’ motives, the argument never really seemed to hold water to me. You can’t really remove the LM part from an LLM. RLHF (supervised learning with a human in the loop) has existed for ages; it’s just got a flashy new acronym now, and it’s being phrased in a way that’s harder for non-computer-scientists to understand. So it can be used as a way to deflect the argument of “it’s not artificial general intelligence, it’s just text processing”—all you have to do is say “well the text processing isn’t important, it’s this other pile of linear algebra that’s going to bring about the singularity instead!”
Imo, the language model side of the LLM is the more impressive by far. Before RLHF, earlier LLMs with chat interfaces used standard sentiment analyzers instead, and they still took the world by storm. And as Avery Hilbert pointed out, most of the “intelligence” aspect of LLMs can be accomplished by the language models alone; it just takes more work to bring it out. What’s being accomplished with RLHF is definitely impressive, but in my opinion the language models are far, far more so. (But of course, as a linguist, it’s not too surprising I’d be most interested in the language part of the equation!)
3 Likes
Ah. In other words, that the attention architecture itself contributes more than the fine-tuning?
It’s more about the training regime than the architecture. I’m claiming that the training task of “predicting what token comes next” is already extremely general, and the fine-tuning is mainly just about tweaking the model’s output into a more structured/useful form, leveraging the general-purpose abilities learned from the general-purpose language modeling task. Nothing particularly architecture-specific here. In principle, if it turned out that (for example) LSTMs trained with a “modern” (read: unimaginably expensive) budget are also “good enough” at language modeling, then I would expect the same claim (about the relative importance of language modeling vs. RLHF) to still be true.
2 Likes
Would you expect that to be scale independent? Because while it’s impractical to roll your own RNN on the scale of Claude or whatever, aren’t there transformer models at the scale of the old Keras RNNs?
7 Likes
Would I expect which part to be scale independent? In general, you can’t get anything close to “modern LLM behaviour” (in the sense of getting half-decent performance on a variety of tasks without task-specific fine-tuning) without a very large model and a ton of training, regardless of architecture.
(EDIT: For more concrete numbers, with improvements over the past ~7 years in both architecture and TPUs, the cost of training a model with performance similar to GPT-2 has allegedly (according to Karpathy) come down to around $73, when GPT-2 originally cost ~$50k to train, but a model at that scale definitely is not good enough for practical applications without further task-specific fine-tuning, so not yet big enough for this discussion about where the “reasoning” or “understanding” comes from in the training process to really be applicable.)
Of course, sufficiently good next-token prediction (plus some form of long-term memory—the precise mechanism is surely irrelevant) is on its own indistinguishable from artificial general intelligence. So your cynicism here is well-placed.
Animal brains, from mine to my sister’s dog’s all the way down to Drosophila’s, are unreasonably sample-efficient compared to all current AI training algorithms, so that indicates there is at least one fundamental architectural or algorithmic breakthrough remaining to be discovered. I’ll eat my hat if the missing “secret sauce” is some proprietary RL or fine-tuning technique, as the Drosophila currently plaguing my compost bin have no trouble solving complex 3D motion planning tasks with 200k neurons and only weakly-supervised active learning.
2 Likes
You’re saying that most of the performance lives in the pretraining/next-token prediction. Unless I’m still misunderstanding. Is that only true for multi-billion dollar frontier models? Because if it isn’t, then you should be able to empirically demonstrate it on smaller models.
1 Like