Reinforcement learning from human feedback

In machine learning, reinforcement learning from human feedback (RLHF), also called reinforcement learning from human preferences, is a technique to align an AI agent to human preferences. In classical reinforcement learning, such an agent learns a policy that maximizes a reward function that measures how well it performed its task. However, it is difficult to explicitly define such a reward function that approximates human preferences. Therefore, RLHF seeks to train a "reward model" directly from human feedback. This model can then function as a reward function to optimize an agent's policy through an optimization algorithm like Proximal Policy Optimization. The reward model is trained in advance to the policy being optimized to predict if a given output is good (high reward) or bad (low reward). RLHF can improve the robustness and exploration of RL agents, especially when the reward function is sparse or noisy.

Motivation

Optimizing a model based on human feedback is desirable when a task is difficult to specify yet easy to judge. For example, for the task of generating a compelling story, humans can iteratively assess the quality of different AI-generated stories, and the goal would be for the model to use their feedback to improve its story generation.

There have been various prior attempts at using human feedback to optimize a model's outputs, including through reinforcement learning, but most attempts were either narrow and difficult to generalize or broke down on more complex tasks. RLHF was an attempt to create a general algorithm for learning from a practical amount human feedback.

Collecting human feedback

Human feedback is commonly collected by prompting humans to rank instances of the agent's behavior. These rankings can then be used to score outputs, for example, using the Elo rating system. While ranking is the most widely adopted form of feedback, recent research has explored other forms, such as numerical feedback, natural language feedback, and prompting for direct edits to the model's output.

When learning from human feedback through pairwise comparisons under the Bradley-Terry-Luce model (or the Plackett-Luce model for K-wise comparisons), the maximum likelihood estimator (MLE) for linear reward functions converges if the comparison data is generated under specific models, but in policy training, a pessimistic MLE, which incorporates a lower confidence bound as the reward estimate, is more effective. Moreover, when applicable, it has been shown that considering K-wise comparisons directly is asymptotically more efficient than converting them into pairwise comparisons for prediction purposes.

Applications

RLHF has been applied to various domains of natural language processing, such as conversational agents, text summarization, and natural language understanding. Ordinary reinforcement learning, where agents learn from their own actions based on a "reward function", is difficult to apply to natural language processing tasks because the rewards are often not easy to define or measure, especially when dealing with complex tasks that involve human values or preferences. RLHF can enable language models to provide answers that align with these complex values, to generate more verbose responses, and to reject questions that are either inappropriate or outside the knowledge space of the model. Some examples of RLHF-trained language models are OpenAI's ChatGPT and its predecessor InstructGPT, as well as DeepMind's Sparrow.

RLHF has also been applied to other areas, such as the development of video game bots. For example, OpenAI and DeepMind trained agents to play Atari games based on human preferences. The agents achieved strong performance in many of the environments tested, often surpassing human performance.

Challenges and limitations

RLHF suffers from a number of challenges that can be broken down into problems with human feedback, problems with learning a reward model, and problems with optimizing the policy.

One major challenge is the scalability and cost of human feedback, which can be slow and expensive compared to unsupervised learning. The quality and consistency of human feedback can also vary depending on the task, the interface, and the individual preferences of the humans. Even when human feedback is feasible, RLHF models may still exhibit undesirable behaviors that are not captured by human feedback or exploit loopholes in the reward model, which brings to light the challenges of alignment and robustness.

The effectiveness of RLHF is dependent on the quality of human feedback. If the feedback lacks impartiality or is inconsistent or incorrect, the model may become biased. There is also a risk that the model may overfit to the feedback it receives. For instance, if feedback comes predominantly from a specific demographic or if it reflects specific biases, the model may learn not only the general alignment intended in the feedback, but also any peculiarities or noise present therein. This excessive alignment to the specific feedback it received (or to the biases of the specific demographic that provided it) can lead to the model performing suboptimally in new contexts or when used by different groups.

Additionally, in some cases, there may be a risk of the model learning to manipulate the feedback process or game the system to achieve higher rewards, rather than genuinely improving its performance, which indicates a fault in the reward function.

Researchers have surveyed a number of additional limitations to RLHF.

Alternatives

An alternative to RLHF called Direct Preference Optimization (DPO) has been proposed to learn human preferences. Like RLHF, it has been applied to align pre-trained large language models using human-generated preference data. However, instead of training an intermediate reward model to then be optimized by a policy using reinforcement learning, DPO uses a change of variables to define the "preference loss" directly as a function of the policy and uses this loss to fine-tune the model.

See also

• Reinforcement learning
• ChatGPT
• Reward-based selection
• Human-in-the-loop