TD(0) & Q-Learning, as Robbins-Monro
A derivation-first note connecting TD(0), Q-learning, Bellman residuals, and Robbins-Monro stochastic approximation.
Read post →Reinforcement Learning
Reinforcement Learning
Batch TD(0) as Dynamic Programming on the Empirical MRP
A step-by-step derivation showing that the fixed point of batch TD(0) coincides with the Bellman solution of the empirical Markov reward process defined by transition-count MLEs and empirical conditional mean rewards.
Read post →