Model-Based Reinforcement Learning Foundations

Represent the problem as an MDP#

A Markov decision process (MDP) is described by states $\mathcal{S}$, actions $\mathcal{A}$, a transition model $P(s'\mid s,a)$, a reward function $R(s,a,s')$, and a discount factor $\gamma$. The Markov property says that the current state contains the information needed to model the next state and reward, given the chosen action.

The tutorial defines a policy $\pi(a\mid s)$ and the discounted return:

The state-value function $V^\pi(s)$ measures expected return from a state when following a policy; the action-value function $Q^\pi(s,a)$ measures expected return after taking an action and then following that policy. These values give a common language for evaluating and improving behavior.

Bellman equations turn long-term value into recursive updates#

The Bellman expectation equation expresses a policy’s value in terms of immediate reward and the discounted value of possible next states. The Bellman optimality equation replaces the policy’s action average with a maximum over available actions. These recursive relationships are the basis for the dynamic-programming methods in the article.

The known-model assumption is central: the algorithms need transition probabilities and rewards to calculate expectations over next states. When a model is unknown, an agent must estimate it or use model-free learning methods; planning with an inaccurate learned model can compound its prediction errors.

Policy iteration and value iteration#

Policy iteration alternates between evaluating the current policy and improving it by choosing actions with higher expected value. Value iteration applies the Bellman optimality update directly, then extracts a greedy policy from the resulting value function. The source includes pseudocode-style Python implementations for both approaches and compares their update patterns.

The code uses a stopping threshold on the maximum value change and accesses transitions through env.P. That attribute is an environment implementation detail, and the tutorial’s Gym interface may differ across versions. In policy improvement, ties between actions are resolved by argmax, so the resulting deterministic policy can select one of several equally valued actions.

FrozenLake as a small planning example#

The guide uses the 4×4 slippery FrozenLake grid: the agent starts at S, aims for G, and avoids holes H. It builds the environment, passes its known transition table to policy iteration and value iteration, and sketches visualizations of state values and actions. A final helper runs episodes to estimate how often a policy reaches the goal.

This is a useful small example for seeing planning over a model. The article’s success-rate code is a proposed evaluation procedure, not a published measurement: the post provides no run output, package versions, seeds, or comparison table. Results in the stochastic environment should be estimated over repeated episodes and reported with uncertainty. The implementation should also be checked against the installed Gym/Gymnasium API before use.

Limits of the tutorial’s scope#

Dynamic programming is most direct when the state and action spaces are manageable and the transition model is available. For larger or continuous problems, exact tables become impractical; function approximation, sampling, learned models, or model-free approaches are needed. The source introduces those as subsequent topics rather than implementing them here.

The article is a theory-and-code tutorial. It does not document a personal experiment, a trained agent, or a measured improvement. Its discussion of planning should therefore be read as a foundation for reasoning about learning systems, not as evidence that a particular system learned or improved.

Connection to RSI and AGI#

An explicit transition model gives an agent a way to reason about possible consequences before acting. That makes model-based planning relevant to the broader question of how a system could use experience to choose better actions. For RSI, planning alone is not enough: improvements must be measured across iterations, the model’s errors must be tracked, and tests must distinguish real capability gains from changes in environment assumptions or evaluation conditions.

For the full Chinese tutorial—including MDP definitions, Bellman derivations, policy and value iteration code, FrozenLake visualizations, and the proposed policy-testing loop—see the original article.