What PPO Limits: Probability Ratios, Clipping, and Value Estimates

Clip an incentive, not the policy itself#

The ratio compares the current probability of a sampled action with its probability under the policy that generated the batch. Old log probabilities stay fixed while that batch is reused. A ratio of 1.4 means a relative increase of 40%, not an increase of 40 percentage points.

With epsilon 0.2, PPO-Clip maximizes the minimum of the ordinary ratio-times-advantage term and its clipped counterpart. For advantage 2 and ratio 1.4, the contribution is 2.4. For advantage 2 and ratio 0.6, it is 1.2. For advantage −2, ratios 0.6 and 1.4 give −1.6 and −2.8 respectively.

Favorable movement eventually reaches a flat branch; unfavorable movement is not flattened in the same way. Shared parameters, other samples, and optimizer state can still move ratios beyond the interval. Clipping is not a hard probability or KL constraint. See the official PPO explanation and implementation.

Old policy and reference policy have different roles#

The old policy explains how the current batch was sampled. A reference model provides a separate anchor, often the initial SFT model in language-model training. Small updates relative to successive old policies do not guarantee proximity to the initial model. InstructGPT includes a separate SFT-reference KL penalty in its RL setup.

Tokenwise ratios and a complete response’s product of ratios also differ. Report the direction, reference, and sampling distribution for each KL metric.

Estimate advantage with explicit boundaries#

A value function predicts future return from a prefix; it does not certify a mathematical statement. GAE accumulates discounted TD residuals, introducing an estimation tradeoff described in the GAE paper.

In a constructed three-step terminal trajectory, rewards are (0,0,1), values are (0.2,0.4,0.7), terminal value is zero, and gamma is one. Residuals are (0.2,0.3,0.3). Lambda 0.5 gives advantages (0.425,0.45,0.3); lambda one gives (0.8,0.6,0.3).

True termination and collection cutoffs require different boundary treatment. A generation budget limit may be a real terminal failure if the task defines it that way, or an artificial collection boundary in a continuing trajectory. The field name alone does not decide.

Inspect metrics before interpreting them#

A reported clip fraction may count every out-of-range ratio, even when the sample is not on the flat branch. Check the implementation. Track task success, reward components, old-policy and reference-policy KL, entropy, length, and value error separately.

The Chinese article contains the full numerical table, GAE calculation, failure cases, and proposed controlled training comparison. Only the arithmetic checks have been run.