ChatGPT: Understanding Its Power Through RLHF and RLAIF

Key takeaways

RLHF turns human rankings into a reward model that then guides language model training.
This stage follows pre-training and supervised fine-tuning to move model responses closer to human preferences.
RLAIF replaces part of the human feedback with judgments from another model guided by a constitution of principles.

Do you know why ChatGPT made so much noise? No, it is not because it is free, even though that helps. It is because of four letters: RLHF, the secret ingredient that makes ChatGPT so effective.

RLHF, or Reinforcement Learning from Human Feedback, marks the difference between earlier language models and ChatGPT, which is so good at understanding what we want and interacting with us the way a friend or colleague would. In a human way.

This is not an abstract topic that only curious people should explore. In fact, many companies already use this method to build incredibly powerful language models.

By the end of this article, you will have an excellent understanding of RLHF and a recent alternative used to improve the performance and safety of large language models, which we call LLMs. I am Louis-François, and we are going to dive into this essential technique for language models that helped make GPT-4 and other recent models so powerful!

Reinforcement Learning from Human Feedback, or RLHF, is a method that combines two parts: human feedback and reinforcement learning. I have already explained reinforcement learning separately if you are curious about this type of learning in artificial intelligence. Basically, we use real people to give the model feedback and improve it step by step. By model, I mainly mean language models such as GPT-4 here. We do this primarily to improve their alignment with desired goals and make the model more human. How?

Humans essentially steer the model during training to favor some formulations and answers over others, encouraging it to produce responses that are more reliable and aligned with our expectations. This is done with a brilliant reinforcement-learning technique that helps the model learn from its mistakes without radically changing its behavior. The model uses feedback from human rankings to improve over time instead of doing everything automatically with a fixed, prebuilt dataset. But let’s look at this training in a little more detail…

It is important to know that RLHF is only one additional stage in the complete training process of a language model. You already need an initial version of the model that is fairly good and capable of generating answers. So we generally begin by training the model itself on data from the Internet. Then we can start the RLHF training process.

The first step is therefore to take an already powerful model and fine-tune it with supervised learning, which I have also covered separately. But what does that mean? It means taking a model such as GPT-3.5, an improved and updated version of GPT-3 that knows roughly the entire Internet, and retraining it specifically on examples of conversations. In this case, we try to narrow its potential strictly to conversation, theoretically making it better at conversing than GPT-3 because a specialist is almost always better than a generalist at a specific task.

The GPT-4 training process. Step 1: Pre-training. Step 2: Training the reward model. Step 3: Retraining the final model using the reference model. Image translated by the author from the original version on the OpenAI blog.

Now, your model is supposed to be better at conversations, but that is not enough. We are still not at GPT-4 level, only at an improvement over GPT-3. It is like learning the rules of a sport such as tennis. Even if you know how to play, you need to practice to improve. It is the same for LLMs: they need practice, and this is where RLHF enters the picture. So that is what we do. We talk with the model and collect many model completions from conversations for every exchange. Then we ask humans to rank the generated completions. For example, we could ask the model to generate 4 answers for each message and ask human evaluators to rank them from 1 to 4 based on how “human” they feel.

But how can we take message rankings and tell the model to follow the best ones? Well, as always, when facing a very complicated problem, we use more AI! We train another model on the dataset we just built. Essentially, we create an automatic classifier based on how humans rated the answers. We give it all the messages, outputs, and rankings generated by the language model and teach it to reproduce the same classification process as the humans in the previous step. This new model is called the reward model.

Now we need to put everything together! We have our pre-trained and fine-tuned LLM, and we ask it to generate several completions, which are then sent to the reward model. We can use the rankings from this new reward model to train the language model for a third time, giving it automatic feedback so it converges toward what humans prefer most, assuming our reward system understood us correctly. This is much more efficient than having humans read and rate millions of examples!

So, a model trained on the Internet, such as GPT-3.5, is specifically fine-tuned for conversation, making it a little better at conversations. However, knowing is not enough. The model needs to practice. This is done by collecting a few thousand model responses and having humans rank those responses according to how much they resemble human answers. A new “reward model” is then trained using the ranked data, hopefully learning to reproduce those human preferences. Finally, the last step introduces RLHF by using the language model to generate responses that the reward model ranks. In return, this automatically guides the language model to produce outputs that more closely align with human preferences. And there you have it! You now have a super-powerful system such as ChatGPT.

This is really cool, but you can see how incredibly expensive it can become if you need to hire hundreds of people to have conversations and then rank the model’s answers. It is also difficult to evaluate an answer precisely because all of this is very subjective. Fortunately, there is a fascinating and innovative alternative to RLHF: Reinforcement Learning from AI Feedback (RLAIF), developed by Anthropic.

In RLAIF, instead of relying on human feedback, an AI Feedback Model provides feedback for training. Yes, an AI now trains another AI. We have reached that point. This new AI Feedback Model is guided by what they call a “constitution” provided by the model’s creators. This constitution is simply a small set of principles or instructions the model must follow, defining the essential principles for its judgments.

Image from the paper introducing RLAIF by Google.

We first create a dataset of ranked preferences, but this time it is generated automatically by the AI Feedback Model, which is essentially a model such as ChatGPT with the constitution in its prompt. This dataset is then used to train our reward model, exactly like the reward model in RLHF. The reward model then serves as the reward signal in the same reinforcement-learning setup for our language model as it does in RLHF.

RLAIF is therefore essentially the same thing as RLHF, but with one more step that further automates the complete training process and requires even less human labor to create a powerful language model. This makes training much more accessible to smaller businesses.

Fun fact: a recent Google paper, which I linked below, ran more experiments with RLAIF and found that humans often prefer answers from the RLAIF-trained model over those from the model trained with human feedback!

Conclusion:

RLHF and its automated counterpart, RLAIF, are revolutionizing how we train and fine-tune language models such as those behind ChatGPT. They also demonstrate the need for human feedback and, as people say, the importance of always having humans in the loop. Even when RLAIF is highly automated, a great deal of human fine-tuning and judgment is still required to improve the results. It remains a machine that predicts the words that come next, not a conscious and intelligent being that understands the world as we do!

That is RLHF and RLAIF, and how these learning techniques make our language models smarter and more effective. I hope you enjoyed this explanation, and I will see you next time with more exciting new AI techniques!

FAQ

What does RLHF mean?

RLHF means reinforcement learning from human feedback. It uses human preferences to better align a model.

At what stage of training is RLHF used?

It is applied after pre-training and supervised fine-tuning, once the model can already generate answers and hold a conversation.

How are human preferences collected?

Evaluators rank several responses to the same prompt according to their quality and fit with the desired behavior.

What is a reward model?

It is a model trained on human rankings so it can reproduce those preferences and automatically score new answers.

What does RLAIF mean?

RLAIF means reinforcement learning from AI feedback. An AI model provides the preferences used for training.

What role does the constitution play in RLAIF?

It provides a small set of principles that guides the model responsible for judging and ranking responses.

Does RLAIF completely remove human work?

No. It further automates preference creation, but humans still define the principles, evaluate results, and adjust the system.