How experience travels

γ = 1 · Initial values = 0. Changing α or λ restarts the episodes. Shading counts updates, including those with no value change; ×N marks repeated updates. Trajectory symbols brighten when visited. An update is not necessarily a new visit.

All methods start with zero state values and γ = 1. Rewards are received on entering the right edge. Those states are terminal and always have value zero; numbers shown inside them are entry rewards.

MC averages every observed return with 1/N(s). 3- and 5-step TD update when a target becomes available and finish pending shorter targets at termination. TD(λ) uses conventional online accumulating traces, reset each episode. Purple halos indicate eligibility, not state value.

Batch TD begins only after all 31 transitions have been observed. Each sweep averages targets by source state using the previous table throughout, with step size 1. It stops when the largest change is below 10⁻⁹. Green backward arrows show sampled dependencies used by updates; they are not additional transitions. MC’s reverse animation is a calculation after termination.

The counters include coordinate updates whose values do not change. Methods have different computation budgets and step sizes. This is a comparison of information propagation, not a ranking of accuracy or a control experiment. With γ = 1 and no step costs, equally rewarded completed trajectories have the same return regardless of length.

Algorithm conventions: Sutton & Barto, chapters 6, 7, and 12.