@Giger_Kitty said: (and anyway, if we’re making such a big deal to replace humans with machines that are so human, even in their mistakes, jeez, just use humans. we’ve managed so far, and with the internet as it is we’re closer than ever.
AI doesn’t commit ALL of a human’s mistakes. AI will answer my rudimentary coding questions endlessly with no ego, without talking down to me, or rolling their eyes and saying I just “don’t get it”. It will teach me a concept with language at my current level. It won’t tell me to “just go look it up” in documentation that later turned out to not exist.
With AI, I’ve learned a lot and gotten some personal projects to a level I wouldn’t have solely on my own and learned a lot by seeing how AI did it. Especially if we had to do long back-and-forth sessions on details.
I don’t mind if anyone wants to avoid AI. Me? It bridges a gap that takes me from being stuck to at least having something to iterate on.
You’re talking about instruction tuning, but as the bit I quoted from the article said, instruction tuning isn’t the final step either.
I’m no ML engineer, but from what I can tell, RLHF doesn’t work in the way you’re describing. The model isn’t given part of a known sequence and then “rewarded” based on how close it gets to completing the sequence, as it is in earlier training steps. Instead, the model’s output is rated by a function that estimates its quality along some other axis like helpfulness (having been trained to do so based on human ratings).
Well, I did gloss over this a bit with the vague remark about “modern developments.” But the argument I would make is this: the LLM is still fundamentally a language model, meaning that:
The model estimates a probability distribution for the “next token,” conditioned on a preceding set of tokens.
There is some ideal, vaguely-defined “true distribution” of “all possible uses of natural language” (or “natural language + code” or whatever) that the model is supposed to be approximating during initial training, based on the idea that the training data is an empirical sample from this “true distribution.”
For the final application, we don’t want to just model “natural language text.” We want to restrict the model to text with specific properties (e.g. text in the form of alternating prompts and responses that are helpful/inoffensive/insert other goals here), which should be reasonable to achieve if this is a subset of the broader distribution of “all uses of natural language” that the model has already learned.
So the fine-tuning steps (including RLHF) can be seen as tweaking the initially learned distribution to better match some desired final distribution, but the fact that we’re trying to create a model which samples from some distribution of “text but with desirable properties” has not changed.
I do not claim to be 100% up-to-date on recent scholarship regarding the extent to which behaviours promoted during RLHF are already implicitly present in the base model. But the fact that we still rely on training the base model on an absolutely insane amount of straightforward language modeling still strongly suggests to me that the fine-tuning, including RLHF, is still primarily about getting the model to express a specific subset of what it was already potentially capable of, rather than teaching it fundamentally new abilities unique to that step of training.
In the end, I am open to the argument that calling it “just” next-token-prediction is dismissive (since the entire premise of using LLMs for general AI is that language modeling is, in its full generality, an “AI-hard” problem), but I maintain that it is not wrong to understand the behaviour of modern chatbots as a form of “predicting the next token” with respect to some particular preferred domain of text. And the fact that LLMs accomplish at least some portion of what looks like “reasoning” through questionable statistical heuristics is pretty unambiguously true (see e.g. the paper mavnn mentioned earlier about diagnosing from medical imaging without actually having access to the images, although for that we’d have to get into multi-modal models, which is a different question I guess).
A tool doesn’t need to be perfect to be useful. If you need to chop down a tree, an ax is much more useful than a steak knife, but you’re probably not going to throw out the ax and go buy a new one the first time the blade gets nicked or the first time you spot a spot of rust, and unless you’re a lumberjack chopping down tries by the hundreds, you’re probably not too bothered by your ax taking 11 swings to cut down a given tree compared to another ax only taking 10 swings to cut down the same tree, and even then, the lumberjack is probably cutting down enough trees to make the investment in a chainsaw worth it where it might not be for the home owner wanting to clear a single tree.
As for the AI coding example, maybe the AI produces something that only half works and needs a day’s work of manual debugging to make it work, and maybe that doesn’t sound great… But if you would need a day’s work to get to an equivalent point of half working handwritten code that the AI spit out in 5 minutes, that’s still a significant time savings… Of course, if the AI’s code is so much worse than your own draft code would likely be that any time saved not writing code manually would be lost to more time spent debugging the Ai’s code, it’s a bit of a wash.
Speaking more generally, current AI is probably best used for stuff where the AI’s output is hard or tedious for a human to generate, but the quality of the output is easy to evaluate… something that can also be said for a lot of traditional programs as well… Sadly, a lot of LLM-based AI is being marketed to and used by people who don’t know enough to verify the AI’s output… or don’t know when to use one specialized AI or a non-AI tool over another AI tool, kind of like the guy who makes the mistake of thinking all bladed tools are equally good at cutting wood and tries to use the steak knife to cut down a tree… only the AI might not be as obviously the wrong tool for the job once you get started.
We might also observe that teaching a baby to speak is about getting it to express a specific subset of the sounds it was already potentially capable of, but that doesn’t mean we aren’t teaching it a new ability, right? Because the most straightforward way for it to produce that specific subset is to learn the ability.
Or, as the article I linked earlier put it: “if you encounter a tiger, the best way of predicting the resulting sense-data (the appearance of the tiger pouncing, the sound of the tiger’s roar, the burst of pain at the tiger’s jaws closing around your arm) is to know things about tigers.”
Similarly, we might observe that what the human brain has evolved to do is “sample from a distribution of electrical impulses but with desirable properties”, or indeed that what human DNA has evolved to do is “sample from a distribution of meat but with desirable properties”. But the fact that those desirable properties happen to include everything we care about makes that an awfully reductive way to look at it.
There is a lot of knowledge in the base models, and one way of accessing it is through few-shot prompting. You establish a pattern for the model to follow.
Say you give this text to the model:
Q: What is the capital of France
A: Paris
Q: What is the capital of Germany
A: Berlin
Q: What is the capital of Italy
A: Rome
Q: What is the capital of Spain
A:
The model recognizes the pattern and will draw on its knowledge to give the correct answer.
However, left to its own devices a base model will not just complete the capital of Spain, it will continue making new question and answer pairs, since that was the pattern we gave it.
However (there’s always a however, huh? It’s like butts), resoving the problem of people being jerks by removing people from the equation is… not comforting at all to me.
Still, that’s what we’ve been doing collectively for the past few decades, so, nothing new there. Anyway, asocial and isolated-by-choice as I am, I probably seem hypocritical in this. How should I put it… people can be mean, evil, a pain, selfish, pretentious, and all sorts of thing, but I’ll always defend to the death their inclusion and growth over their exclusion. Again, coming from someone who feels uncomfortable in social situations and will avoid groups of more than 2 when having a chance (but still having to put up with them occasionally). And this strange hipocrisy is also human.
Having said all that…
On a short-term individual basis, you have found something that works for you and you know how to work with it. Like everything, if you find that you can easily use it in a “responsible” way (I hesitated a lot before using this word. I mean “responsible” in the sense that you are well aware of the AI’s shortcoming and the traps of using AI, so that you can work around those things; have a stable framework around AI usage to draw on its best while avoiding its worst * ), it’s not only your choice but your right to do it your way, without being pointed a finger at **.
* doesn’t this sound much like “managing a difficult person”, though? Instead of learning how to work with a difficult person, you’re learning how to deal with an AI who might possible, as seen before, just up and delete all of your e-mails? I still don’t see THAT big of a difference between those.
** Those who are seriously against AI will probably point the finger anyway, for a number of reasons, some of them bad and some of them good; some of them terrible, some of them great. Knowing that is also part of using AI these days. I’m sure I’m not telling you anything you don’t already know, I mostly want to show that I’m aware of it too, even as I say the things I say.
EDIT - To clarify, those examples you posted of human behaviour are things I really dislike to hear from others, same you. Still, not everyone is like that. Usually one looks for a forum, or a messageboard, or a Discord, or whatever it’s called these days, that is either populated by better people or that has a couple of better people in there, people who won’t be jerks. And one stops going to a place where one doesn’t feel well. (There was this one place I went to, great people, great discussions, a welcoming lot, where I got involved in a discussion that went a certain way that I felt extremely uncomfortable about (and it was nothing major, and it’s not like everyone had a mass opinion that was opposed to mine, and it ALSO involved AI! Which is partly why I’m a bit cautious discussing AI now), and I simply felt that, although it was possible to “salvage” the situation socially, I simply felt I could not stay in a place where such an opinion was being expressed (even if it was only one or two people expressing it. That no one else seemed at the time to be countering those opinions also bothered me - but it’s a forum with no bustling activity, where one needs to be patient). So I preferred to leave and not come back; entirely my right, for the sake of my mental wellbeing that others have no say in. A choice to stop interacting. As a result, I came here. That’s part of it: to continue looking for a place where you feel comfortable)
That’s option a). Option b) is removing the human from the equation. Again, still valid, if not everyone’s preferrerd choice. To be fair, it does resolve the immediate problem.
EDIT - I suddenly wonder how many of the people so opposed against AI rooted for Robin Williams’ character in Bicentennial Man, or for their relationship in Chobits! I know I am one of those people! It’s suddenly not quite the same when the fantasy becomes reality, huh?
There’s a lot of very sensible answers to the original question, which is that you don’t trust (or possibly even use) LLMs to do the things that traditional computer programs do without supervision, because they are not actually good at it, and instead should use them (and trust them more) when doing things they are good at, such as text transformation, analysis and summary, but it’s worth noting that they can be terribly bad at that too.
I recently needed to extract a lot of tabular information from a PDF that someone had made out of a word document. Although what was there was very readable and clearly laid out it wouldn’t actually copy/paste clearly because of PDF text-box ordering. This seemed like a good job for an LLM to extract.
ChatGPT happily extracted the text for me in the format I had specified, but all the text it extracted was hallucinated. It was all quite similar to the original document, but none of it was right, and it was the sort of text where some minor mistakes would not have been acceptable.
I found another way to get the text, but it was interesting to me how in that case the LLM was entirely untrustworthy when given a task that was in its general area of expertise (though still not 100%, since again I was hoping for reproducible accuracy).
PDFs are not text at all: they are binary data, and nasty binary data at that. An LLM is ill suited for dealing with them. Either the frontend converted your PDF to images or ran a text extraction utility, and the model never saw the original PDF itself.
So the original problem was caused by the frontend, but then the LLM acted confidently wrong instead of alerting you.
That is to say, there have been plenty of times that LLMs have patiently answered questions that humans would probably consider offensive, annoying, or a total waste of their time, because I’m on the autism spectrum and I tend to ask questions like that at times. I saw this as a net plus for both me and other people.
My preferred solution to this problem is to not give the AI the authority to do that. Of course, sometimes some corporation with ultimate control over the system has decided the opposite, but then the real problem is what the real problem usually is in debates about AI: greedy, short-sighted humans.
I think it’s an issue of perception, because in the example you give me, I don’t see the “asker” as a jerk. I never do. I think the jerks are the ones who get fed up with good faith questions. In my perception, you got questions, they should be answered.
And, I don’t know how this will come across, but if I’m speaking to someone who I know is in that spectrum, I will be more tolerant of questions which I might otherwise find offensive. I dunno if others are the same. For that reason, I find it helpful to know whether I’m speaking to someone on the spectrum. But I understand that type of disclosure isn’t something that one’s immediately comfortable with. Almost a catch-22.
I do know for a fact that I made my share of questions about transgenders and their struggles, and many of them were, on some level, offensive - partly because I was regurgitating the false information that many spew. But because I was doing so in good faith, and I had the good fortune of speaking to people who saw that, I was corrected. It took a number of corrections, and some self-guidance and research, but I arrived at the point where I think I can converse about that subject without being offensive or falling into the traps that many cis people do because of ever-pervasing poor arguments that float around incessantly,
My point: people acted towards me in good faith. Those who don’t? Those are the jerks. Not you. Well, unless you double down on something really jerky. Then you’re being a jerk. But if you don’t, you’ve got a question and want to know the answer.
…maybe my problem is that I’m too detached from how things actually work in the real world, admittedly. But I seem to do real good in aventure games and IF!
I do lose my patience with people who ask but then make no effort to understand, or who ask and are pointed in a way which might help them and then whine about not getting the work all done for them. That’s something else, though. Let those people ask AI and be happy with their work done with no brains involved.
(to clarify, I don’t mean that everyone who uses AI is happy with brainless work. But people who do want brainless work find their holy grail in AI).
Absolutely, and that’s why I loved Draconis’ post:
Says it all. And with not putting it in a place where it can be accountable, it follows that it can’t be put in a place where it can make real damage.
The scary thing aobut it? Every program can make real damage. But guardrails exist to prevent that. But AI seems to bypass the guardrails sometimes. And again we come into the question of “how much do we trust it”. And the answer is usually “How much do you trust the person that’s closest to you not to flip out and do something crazy”. And… we’re circling back.
One of my favorite games to play the past couple of months is to ask Google AI about what people are saying about one of my podcasts. (search term: this week in retro podcast reviews)
The game features include:
On any given day, who will Google think is hosting the show? (the correct answer is clearly documented and hasn’t changed in nearly a year but Google routinely gives an out-of-date answer)
Will Google put a comma after every word in the output? That happens every so often and looks utterly insane.
Will Google get confused and show a preview video from another podcast with a similar name and/or theme? More than one?
It’s not always wrong about something. But it’s frequently wrong about something which isn’t a changing fact, or just spams commas, and yeah. I would not ask it to do my homework at that level of accuracy.
If it were accurate, would you ask it to do your homework?
Loadaed question, having fun, I don’t mean to say that you’d choose not to do your homework at all. But it was just too obvious not to put out there. Because I’m sure some people would. And the harm that causes them individually is a precursor of the harm that causes society.
The difference is that babies don’t start out fluent in every language and then forget most of them. (They do start out capable of a wider range of sounds, but the process of learning a language still involves learning to use those sounds to communicate, which they couldn’t previously do.)
In contrast, I am asserting that the base LLM prior to RLHF already has the actual ability to do things like translate text, write code, answer general-knowledge questions, etc. depending on context, and the RLHF is largely about making it easier for humans to deliberately induce those behaviours through more structured prompting.
That’s basically just the “predicting the next token is AI-hard in general” stance that I’ve already mentioned. It doesn’t explain why I’m supposed to view RLHF/fine-tuning as a super-important bonus step that makes it incorrect to describe LLMs as next-token-predictors.
In the end, it is definitely still the case that the “backbone” of the LLM’s reasoning abilities is a big language model that learns primarily from modeling language. The RLHF step does not work without the prior language modeling, and the reverse is not true (models without any fine-tuning/RLHF can still be evaluated on general-purpose tasks with slightly more involved prompting, like the example Pontus I shows, and achieve reasonable results).
EDIT: Seems like I failed to reply to the comment this was supposed to be a reply to, is there a way to retroactively change that?
It is frustrating that the “reply to a specific comment” button and the “create a top-level comment” button are both labeled “Reply,” with the same icon and everything. Although the wrong button is way at the bottom of the screen, so I have no idea how I clicked it?
How is this any different from “the LLM starts out capable of a wide range of word sequences, but the process of RLHF involves learning to use them to answer questions, perform tasks, and hold a polite conversation, which they couldn’t previously do”?
I don’t think there’s a meaningful difference between “it has the ability to answer questions, somewhere deep down inside, but it won’t do it, because it hasn’t been trained to” and “it doesn’t have the ability to answer questions”.
For the same reason it’s incorrect to describe people as next-sense-datum-predictors, or gamete-seekers, or some other reductive term that conflates the conditions under which we evolved with what we actually do. Perhaps you disagree that it’s incorrect to do that?
In an ideal world, this is true, but I think in the real world there’s a lot more nuance and there are plenty of valid reasons someone might not answer even good faith questions. They might be busy with other things, or not in the right mental place, or just tired. Maybe they got asked some bad faith questions previously and are feeling sensitive. This is all especially relevant if the person belongs to some marginalized group and you are asking them about it. I feel like there are plenty of cases where one party may want to opt out for reasons unrelated to being a jerk, and in a lot of these cases a LLM can helpfully fill in.
I am admittedly a bit sensitive about this due to a bizarre incident from my past. For some time I maintained documentation for a piece of software, but I eventually left the project on somewhat bad terms and the documentation was no longer maintained. A short time later, someone unaware of what had happened contacted me with a bunch of questions about the software and my documentation. I politely told them that I was no longer involved with the project and couldn’t help. They then responded with this long diatribe about how leaving the project doesn’t mean all my expertise immediately evaporates and surely it wouldn’t be that much trouble for me to help them with these few small things and… I don’t know. It felt extremely entitled and I drew a hard boundary, and wrote back with something like, “Do not contact me with questions about this.” The person then decided to go on a multi-thread public tirade on the forums, which continued to the point of them being banned, and I almost quit that forum myself because of how I felt after that. Obviously there was something deeply wrong with this person, but my point is I am usually going to be understanding of people who might reject good faith questions for their own reasons even if I don’t know what those reasons are.
But I seem to do real good in aventure games and IF! 
