Practical Notes on Tuning Reinforcement Learning

Why tuning is difficult#

Reinforcement-learning data changes as the policy changes. Rewards may be sparse or delayed, gradient estimates noisy, and small parameter changes can produce noticeably different learning curves. This makes a single run a weak basis for choosing a configuration.

Parameters to reason about#

The source focuses on several commonly adjusted settings:

• Learning rate: large values can destabilize updates; very small values can make progress slow. The article gives example ranges for DQN, Actor–Critic, PPO, SAC, and TD3.
• Discount factor: values closer to one emphasize longer-term rewards; shorter-horizon tasks may call for a different balance.
• Batch size: affects gradient noise, memory use, and update frequency.
• Exploration rate: an epsilon schedule controls the balance between exploratory actions and actions favored by the current value estimate.

The recommended values depend on the algorithm, environment, reward scale, implementation, and compute budget. They should be recorded and evaluated rather than copied as guarantees.

Diagnose before changing everything#

The article recommends beginning with a known baseline, inspecting reward and loss curves, and changing parameters in a way that makes cause and effect interpretable. It also discusses learning-rate schedules and common symptoms such as oscillating reward, slow progress, or unstable values.

A stronger comparison fixes the environment and evaluation procedure, runs multiple seeds when feasible, and reports variability alongside averages. The Chinese original contains additional parameter tables and troubleshooting suggestions.