How to choose an RLHF or preference data vendor

01

Choose the training method first

• RLHF with a reward model: you need many pairwise or ranked comparisons, often with ties and reasons, to train a reward model.
• DPO and related methods: you need clean chosen/rejected pairs on the same prompt; no separate reward model.
• SFT before preference tuning: you need expert-written demonstrations, usually a smaller and more costly set.
• RL with verifiable rewards: you need tasks with automatic checks, rubrics, or verifiers instead of human comparisons.
• RLAIF: AI feedback replaces part of the human labeling; you still need a human-labeled set to check the AI judge.

Write down the method, the target volume, and how many responses each prompt will have. These drive the task design and the price.

02

Design the preference format

Decide between binary choice, choice with ties, Likert-scale ratings, or full rankings of several responses. Decide whether raters must write a short reason, and whether they rate single turns or whole conversations. Ask each vendor to show the rater interface for your format. Get the rubric in writing before the pilot.

03

Choose the raters

Match the raters to the task. Code, medicine, law, and math preferences need verified domain experts. Helpfulness and tone preferences may need a representative population. Ask how experts are verified, how many qualified raters exist for your exact domain and language, and whether the same raters stay on your project between rounds.

04

Check calibration and inter-rater agreement

Ask how raters are calibrated before production: training tasks, gold pairs with known answers, and calibration sessions. Ask which agreement metric the vendor reports and how often. Ask how disagreements are adjudicated and whether low-agreement items are removed or relabeled. Run a paid pilot with two vendors on the same prompts, then compare agreement and your own spot checks.

05

Choose a buying model

• Managed service: the vendor recruits, calibrates, runs QA, and delivers preference data.
• Expert network: you get vetted specialists; you may own more of the task design.
• Platform you operate: you run preference collection with your own or sourced raters; some platforms publish prices.
• Off-the-shelf datasets: existing post-training data licensed now; check license terms and overlap with public benchmarks.

06

Security and compliance evidence

Prompts and model outputs often reveal unreleased model behavior. Ask for current reports, not badges: SOC 2 Type II report, ISO/IEC 27001 certificate and scope, HIPAA terms for health data, GDPR processing terms, subprocessors, retention and deletion, and whether raters work on managed devices. A certification on a website may cover only part of a company.

07

Commercial terms and conflicts

Clarify the unit of price (per comparison, per hour, per seat, per dataset), minimums, pilot price, and rework policy for low-agreement batches. Confirm IP ownership of outputs and exclusivity. Check ownership and major investors, and ask how the vendor separates your project from competitors’ projects.