The silent failure in long RL runs: trainer–inference mismatch

In on-policy reinforcement learning the trainer assumes the samples came from the policy it's optimising. In practice, rollouts are generated by a separately optimised inference engine, and the two rarely agree to the last decimal.

Small disagreements in log-probabilities compound across long sequences. The run doesn't crash; rewards still go up for a while. But the policy drifts toward behaviour the trainer never actually measured.

We recompute a sample of rollout log-probs with the training engine and track the KL divergence between the two. A rising mismatch triggers an alert before the damage is baked into a checkpoint.

We pair it with numerical stability checks and load tests on the serving path, so researchers can focus on algorithms rather than plumbing.