Skip to content
Your data, your models, your cloud. See it running on your stack. Book a call
DataObserve
Book Demo

Human preference optimization, explained

Research7 mins read

Introduction

Supervised examples teach a model what a good answer looks like. Preference data teaches it which of two good-looking answers your experts would actually choose.

RLHF and DPO

RLHF trains a reward model on preferences and optimises the policy against it. DPO skips the reward model and learns directly from preferred and rejected pairs, which is simpler and often more stable.

Data quality decides the outcome

Who writes the pairs matters

Pairs written by domain experts capture the trade-offs that matter: caution in clinical answers, precision in financial ones. Generic raters tend to reward fluency over correctness.

Related articles

Research

The silent failure in long RL runs: trainer–inference mismatchA tiny numerical gap between rollout and training log-probs can bend a week of learning. How we detect it early.In on-policy reinforcement learning the trainer assumes the samples came from the policy it's optimising. In practice, rollouts are generated by a separately optimised inference engine, and the two rarely agree to the last decimal.Continue reading

Research

How much domain data is enough?Early results on scaling curves for domain adaptation, and why the first thousand examples matter most.9 mins read

Research

Understanding inference cost per tokenBatching, context length and model size: the three levers that decide your serving bill.6 mins read

Guides

Fine-tuning or prompting? A practical decision guidePrompting gets you started; fine-tuning gets you consistent. A checklist for choosing the right tool for each task.5 mins read

Products

Turn one-off answers into scheduled reports with CoworkerAsk once, then let Coworker deliver the answer every Monday, shared with the people who need it.3 mins read

Model training

What's new in the model catalog: 174 open-weight modelsNew MoE releases, longer contexts and per-model correctness tests: the autumn update to the catalog.4 mins read

Build frontier models and agents that learn from experience.

Model training

Introduction

General-purpose models are remarkable, but they don't know your contracts, your claims rules or the shorthand your engineers use. The gap between a demo and a dependable system is almost always domain knowledge. Learning from experience closes that gap. Instead of prompting harder, you let a model practise on your tasks, score the results against what good looks like, and keep the improvements.

6 mins readContinue Reading

// Ready to see the DataObserve in action?

Handle complex workflows at scale, from live delivery issues to compliance decisions, with every answer and action traceable in production.

Book a Demo

// Designed for enterprises

We safeguard your company data and protect workspace privacy.

Review security, data handling and deployment requirements with our team.

  • Security and privacy

    Keep sensitive information within your organization’s approved data boundaries.

  • Governance and access

    Define who can access connected data and which actions require approval.

  • Enterprise deployment

    Plan deployment around your infrastructure and operational requirements.

We are HiringJoin the // Leap

Work with us