RLHF and post-training glossary
18 terms buyers meet when evaluating RLHF vendors.
- RLHF (reinforcement learning from human feedback)
- Post-training that uses human judgments of which outputs are better, usually by training a reward model and then optimizing the model against it.
- Preference pair
- One prompt with two responses and a human choice of the better one (chosen and rejected). The basic unit of most preference data.
- Ranking data
- A human ordering of three or more responses to the same prompt. It can be split into many pairs.
- Reward model
- A model trained on human preference data to predict which output a person would prefer. RLHF uses its score as the training signal.
- DPO (direct preference optimization)
- A method that trains directly on chosen/rejected pairs without a separate reward model or reinforcement learning loop.
- SFT (supervised fine-tuning)
- Training on prompts paired with ideal responses written or approved by people. Usually done before preference tuning.
- Demonstration data
- Example responses written by skilled people to show the model the target behavior. The main input to SFT.
- RLAIF (reinforcement learning from AI feedback)
- Post-training where an AI model, guided by rules or a rubric, supplies some or all of the preference judgments. Human-labeled sets are still used to check it.
- RLVR (reinforcement learning with verifiable rewards)
- Post-training on tasks whose answers a program can check, such as unit tests or exact math answers, instead of human comparisons.
- Rubric
- The written criteria raters use to judge responses, such as accuracy, helpfulness, safety, and style, with examples.
- Rater calibration
- Training and testing raters against known answers and shared examples so that they apply the rubric the same way before production.
- Inter-rater agreement
- How often different raters make the same judgment on the same item. Common measures include percent agreement, Cohen’s kappa, and Krippendorff’s alpha.
- Gold set
- Items with known correct judgments mixed into the work to check rater quality.
- Adjudication
- A process where a senior reviewer resolves items on which raters disagree.
- Reward hacking
- When a model finds outputs that score well with the reward model but are not actually better, such as longer or flattering answers.
- Red teaming
- Deliberate adversarial testing to find unsafe, insecure, or unwanted model behavior. Results often feed safety preference data.
- SOC 2 Type II
- An independent auditor’s report on how a company’s security controls operated over a period.
- ISO/IEC 27001
- An international certification for an information security management system.