Mathematical spine: statements, proofs, and limits

Read each proof in this chapter alongside its corresponding lecture. Before starting, identify the object being estimated, the distribution generating the observations, and the assumption that justifies the main step. Replacing a table with a neural network does not automatically extend a finite-state theorem to that network.

A.0 Objects and standing conventions

Unless a statement says otherwise, and are finite, , rewards satisfy almost surely, and the initial distribution is fixed independently of policy parameters. Policies are stationary and Markov. Terminal states are represented by an absorbing zero-reward state. This makes infinite sums and normalized occupancies unambiguous. Finite-horizon statements instead have time-indexed values and terminal value zero. All expectations include environment and policy randomness.

Symbol	Complete definition
	Conditional next-state probability; each row sums to one



	Expected discounted return, conditional on state or state and first action


	; a probability distribution

For action sets that depend on the state, every maximum and sum uses the admissible set. Norms of distributions below use row vectors where convenient; this does not change their numerical meaning.

A.1 Bellman equations are conditional-expectation identities

Question. Why is it legitimate to replace the rest of a trajectory by a single value?

Proposition. and .

Proof. The return is absolutely bounded by and satisfies . Condition first on . The Markov property and stationary policy imply . Taking conditional expectations, and then averaging over , proves both identities. Absolute boundedness justifies the expectations and infinite sums.

Because , the matrix series converges in operator norm. Multiplication of its partial sum by gives . Consequently

This is an inverse with a probabilistic interpretation: the th term is the expected reward steps into the future.

A.2 Contraction, optimality, and an executable stopping certificate

Theorem. Each of is a -contraction in its appropriate sup norm. Each has a unique fixed point. The fixed point of is the optimal value, attained by a deterministic stationary greedy policy.

Proof. Stochastic averaging gives . For the maximum use the elementary inequality ; to prove it, take an index attaining the first maximum, bound one difference, and reverse the roles. These yield all three contraction inequalities.

For completeness, let for any one of these operators. Then . Summing the geometric tail shows that is Cauchy in a finite-dimensional complete space. Let its limit be . Lipschitz continuity gives . Two fixed points would satisfy , hence are equal. Also .

Let be the fixed point of and choose a maximizer in every state. Then ; uniqueness for gives . For any possibly history-dependent policy, the optimality equation implies

where is the observed history before the action. Iterating and taking expectations gives . The bounded final term vanishes. Thus no history-dependent policy beats . The same argument and identify the fixed point.

Residual certificate. If , then

where is greedy with respect to . The first inequality follows from . For the second write

The first two terms have joint norm at most . Move the last norm to the left and insert the first bound. This is a conservative certificate, not an estimate of the typical observed error.

Pause and predict. Halving the Bellman residual need not halve the number of iterations: the contraction factor determines how quickly the residual decreases. At , neither the geometric proof nor these certificates is available without additional structure.

Computed Bellman error and its proved bound in the running MDP; these coincide here, not in general.

A.3 Policy improvement without a hidden infinite remainder

Theorem. If coordinatewise, then .

Proof. Subtract the two policy-evaluation equations and rearrange:

Every entry of the inverse series from A.1 is nonnegative. It maps a nonnegative vector to a nonnegative vector. This proves the result including ties. Equivalently, repeated one-step improvement has a final remainder , whose norm tends to zero. For finite deterministic policy iteration, retain the old action at ties. A changed policy then strictly improves at least one state, cannot revisit an old value vector, and therefore terminates because there are finitely many deterministic policies. At termination it is greedy with respect to its own value, hence optimal by A.2.

A.4 A running MDP that can be solved three ways

Use states , actions and . The following table is the entire environment, not a sample dataset.

State	Action	Reward	Next state
0	stay	1	0
0	switch	0	1
1	stay	2	1
1	switch	0	0

The always-stay policy has value . A greedy improvement switches in state 0 because , and stays in state 1 because . The improved value is . Its other action values are at state 0 and at state 1, so the policy is greedy with respect to its own value and therefore optimal. Value iteration, matrix policy evaluation, and tabular sampling all refer to these same numbers.

For a randomized policy, let and . Then

This gives an exact differentiable objective for checking sampled gradients. At , . With and independent logits for , ; A.9 gives the gradient . Negative first component is sensible: at this current policy, entering state 1 is not yet as valuable as entering it under the eventual optimal policy. Local policy improvement is not clairvoyance.

A.5 Monte Carlo consistency with repeated visits

Assumptions. Episodes are independent and identically distributed under a fixed policy and reset distribution. For a state , let be its number of visits in episode and the sum of returns following those visits. Assume and . Future dynamics conditional on each visit follow the same Markov policy. Time must be included in the state for a finite-horizon problem whose values depend on time.

Theorem. The every-visit sample average converges almost surely to .

Proof. Visits within an episode are dependent; do not apply an independent-sample law to individual visits. Conditional expectation at each visit instead gives . The absolute-integrability assumption allows summation over , giving . Apply the strong law of large numbers to the iid episode pairs . The ratio tends to , with eventually nonzero denominator. First-visit MC uses one return from each episode that visits and the identical argument with a zero-one visit count.

The strong law for integrable iid variables is a probability prerequisite, not a consequence of independence of visits. This theorem concerns sample averages; arbitrary adaptive step sizes require separate hypotheses.

A.6 Stochastic contraction: a complete tabular convergence proof

Contraction of the mean update is not enough to prove convergence of a random algorithm. The following argument supplies the stochastic conditions and proof.

Theorem. Let be a -contraction in sup norm, with fixed point . At time update coordinates by

Assume: is measurable before is observed; ; almost surely for each ; for deterministic constants ; ; the conditional second moments are uniformly bounded; and the iterates are almost surely bounded. There are finitely many coordinates. Then almost surely. Setting implements asynchronous updates.

Step 1: the weighted martingale noise has a convergent series. Write for a fixed coordinate. Conditional mean zero implies orthogonality of different increments in , and . Here is the required maximal argument. For partial sums and , stop at the first with , if it occurs. Orthogonality of future martingale increments to the stopped sum yields

This follows by expanding the square of the final sum after the stopping time; its expected square is at least the expected squared stopped sum, which is at least times the stopping probability. Let . Choose so the remaining sum of variances is at most , and use . The resulting probabilities sum to at most . By the elementary Borel-Cantelli implication (the probability of any event after index is at most the tail sum), almost surely all sufficiently late tails beginning at stay within . Thus the series is Cauchy and converges. Finitely many coordinates can be intersected without losing probability one.

Step 2: the moving average of this noise tends to zero. Define

For a fixed , expanding the recursion expresses as a decaying initial term plus , where is nondecreasing in and lies in . Put . Summation by parts gives

Its absolute value is at most . The initial term tends to zero because . Convergence of the series in Step 1 makes the tail supremum arbitrarily small as . Hence .

Step 3: contraction removes the remaining error. Let . Then

Boundedness and imply . For any , eventually and , so the absolute value of the input in brackets is at most . A scalar convex-averaging recursion with divergent step sum has limiting upper absolute value at most any eventual uniform bound on its input: expand the recursion, observe that its initial coefficient vanishes and its input coefficients sum to at most one. Therefore for every coordinate. For finitely many coordinates the limsup of the maximum equals the maximum of their limsups. Let to get , so . Together with this proves the theorem.

Application to Q-learning. At an observed the sample target is . Condition on the entire past and the selected pair, before the transition is sampled. Its conditional expectation is . If , then every sample target lies in whenever does. Convex updates preserve that bound by induction, and the target-minus-mean noise is bounded by . A deterministic per-visit schedule with and , together with infinite visits to every pair, supplies all step-size assumptions. A.2 supplies the contraction. Therefore almost surely.

Application to TD(0). For a fixed policy, condition before sampling the action and transition given the current state. The conditional mean is . The identical boundedness and per-state visitation argument proves tabular TD convergence.

SARSA corollary. For on-policy tabular SARSA in the same bounded discounted setting, suppose its behavior policies become greedy in the limit while retaining infinite state-action exploration (GLIE). Conditional on the selected pair, its target mean uses the expected next Q-value under the next action-selection policy. Its difference from is a vanishing perturbation: if the probability of a nongreedy next action is at most uniformly over states, boundedness gives absolute difference at most . The contraction proof above still works with an additional vanishing input term in Step 3; the limiting inequality remains . With the same per-pair step sizes and martingale conditions this proves convergence to . Here "greedy in the limit" must refer to the action distribution actually used for the next sampled action and target. It is an assumption, not a property of arbitrary exploration schedules.

Scope of the proof. Q-learning does not itself guarantee infinite exploration; that is an assumption. Constant step sizes fail the square-summability condition, and an uncorrected replay distribution may lack the required conditional expectation. The proof excludes nonlinear approximation, delayed parameter copies, and arbitrary moving target policies. The GLIE SARSA corollary covers the specific vanishing perturbation proved above. Tsitsiklis (1994), listed in the references, gives the general stochastic-approximation treatment. The discounted argument is proved here in full.

A.7 Linear approximation: projection, error, and sampled convergence

Let admit a stationary distribution with all entries positive, , and have full column rank. Define , , and the orthogonal projection

Projection theorem. is a -contraction in norm; its fixed point is unique in the feature space and satisfies

Proof. Jensen's inequality and stationarity give

Orthogonal projection is nonexpansive: is an orthogonal decomposition, so . This proves the contraction and, by A.2's Cauchy argument, the unique fixed point. The equation is equivalent, after multiplication by , to .

For the error, orthogonality yields

But , whose norm is at most . Rearrangement proves the bound. Finally for any ,

Full column rank makes the right side positive for , so is invertible even though it need not be symmetric.

Sampled theorem, with a precise sampling model. Suppose independent transition tuples are sampled by drawing , and then from the environment. With deterministic nonnegative steps , , linear semi-gradient TD

converges almost surely to . Here is the row of at .

Proof. Put . Conditional mean drift is . Finite features and bounded rewards give for some finite : expand the affine update and bound each finite feature norm. The last inequality in the previous proof gives for . Expanding the squared update therefore yields

Convergence lemma. If nonnegative integrable adapted obeys , where are deterministic summable nonnegative sequences and , let and

Then and . A nonnegative supermartingale converges almost surely: the upcrossing inequality applied to any rational interval bounds its expected number of completed crossings by a finite constant divided by ; infinite oscillation would cross some rational interval infinitely often. Fatou's inequality and uniformly bounded expectations rule out a positive probability of an infinite limit. This is the usual supermartingale convergence theorem from the probability prerequisite. Summing expectations also gives , so that sum is finite almost surely. Because has a finite positive limit, has a finite limit and .

Apply the lemma with , and . If the limit of were positive, would diverge, contradicting the lemma. Thus .

A single Markov trajectory does not provide iid stationary samples. Stationarity preserves the projection algebra, but the sampled convergence proof above does not cover this dependent case. The Markov-noise extension of Tsitsiklis and Van Roy (1997) is an advanced external result, unnecessary for the iid theorem proved here. This proof explicitly uses the strong law, conditional expectation, and the nonnegative-supermartingale convergence theorem as probability prerequisites, rather than RL-specific assumptions.

A complete divergence example. There are two states, zero rewards, and two actions. Action always moves to state 2; action always moves to state 1. The target policy always chooses , while the behavior policy chooses with probability independently in both states. Its stationary distribution is . Use one feature, , discount , and per-action importance correction , equal to 10 for and 0 for . Under iid draws from the stationary behavior transition distribution, the mean TD coefficient is

Thus the expected update is , which expands every nonzero for , although the true target value is exactly represented by . This establishes instability of the mean recursion; no claim that every sampled trajectory diverges is needed. Importance correcting actions does not replace the behavior state weighting by the target stationary weighting.

Exact mean TD recursions from A.7 with step size 0.1 and initial weight 1. These are algebraic expected-update curves, not sampled learning runs.

A.8 The double-sampling problem, derived

Let , , and condition on a state . The mean-squared Bellman error is . Under bounded differentiable features,

Differentiating the squared sample residual instead gives , which differs by the conditional covariance term

The product is unbiased for the desired product of expectations only if the two factors use conditionally independent next-transition samples, or the covariance happens to vanish. Independently draw and given and use . Conditional independence factors its expectation correctly. Ordinary replay usually supplies only one next transition for each sampled state-action event. Therefore "differentiate both sides of the TD target" specifies an objective, but not generally the MSBE gradient. This distinction disappears with deterministic transitions and rewards, not with arbitrary stochastic dynamics.

An exact two-outcome calculation. At a fixed state let the scalar feature be , the reward be zero, and . Suppose the next feature is zero or two, each with probability one half. For , the sampled residual is or . Thus

Indeed the mean residual coefficient is , whereas the mean squared coefficient is . The difference is the conditional covariance contribution. The laboratory computes these exact numbers before asking you to introduce sampling noise.

A.9 Policy gradients and the baseline condition

Assumptions. Policies have common full support in a neighborhood of , are continuously differentiable, and score vectors are uniformly bounded there. Transition and reward laws and do not depend on . Discounted rewards are bounded as in A.0.

Theorem.

Proof. Begin with a horizon . Its trajectory probability is the product of initial, action, and environment probabilities. Only action probabilities differentiate, giving score . Thus

For , is measurable before is sampled, and . Those terms vanish. The remaining expression is . If , the total absolute contribution at reward index is at most , a summable sequence. Dominated convergence (or uniform convergence of the differentiated finite sums) therefore justifies . Conditional expectation of given is by A.1. The occupancy definition gives the second equality.

Baseline theorem. A random baseline is valid when it is measurable with respect to the information available before the current action is sampled, or more generally conditionally independent of that action given that information. Then by the same conditional score identity. A fixed function qualifies. A fitted function evaluated at does not automatically qualify if its fitted parameters depend on the same action and its return. Stop-gradient blocks differentiation through a target; it does not create statistical independence.

For a vector score and scalar return , minimizing over scalar gives

when the denominator is positive. Differentiate the quadratic and solve its unique stationary point; its second derivative is . If the denominator is zero, the score is zero almost surely and any baseline is equivalent. The value function is optimal only under additional conditions, such as action-independent squared score norm.

Implementation consequence. Generate a batch with , compute all scores at that same parameter, sum the gradient estimator, and only then update. Updating inside the return-weighted timestep loop and recomputing scores at new parameters does not implement this theorem.

A.10 Finite-rollout GAE and boundary conditions

Let consecutive transitions belong to one episode segment, be fixed while computing targets, and define . Let the final value be zero only at a true terminal state. For ,

Identity. If , then the difference from the same estimator using is

Proof. Subtract the two residuals to obtain . In the weighted sum the coefficient at an interior index is ; the first and last indices occur once. Collecting terms proves the formula. At , read the original sum directly, or use the convention in this identity.

With an exact critic, every later residual has conditional mean zero before its action is sampled. Hence for a fixed segment length, and also for bounded stopping boundaries determined without using future unsampled noise. At on a complete episode, the terminal critic error is zero and the only error is , a valid fixed baseline. On a rollout cutoff, the final error remains. Smaller is not a theorem of monotone variance reduction; covariance among residuals matters.

Two different masks. Let be the event that transition truly terminates the task, and let be the event that the next stored step is outside the current episode segment. Define and . Then compute backward:

At an external time limit, but if the environment resets. At true termination both vanish. At an ordinary transition both equal one. At the final storage boundary ; its bootstrap still enters . Use the actual final observation, not the next episode's automatic reset observation. A time limit is terminal only when remaining time is part of the task definition and the objective ends there.

A.11 Performance difference and the actual trust-region bound

Performance-difference lemma.

Proof. Along a trajectory, replace the advantage by its conditional form . Its discounted sum telescopes to , with a bounded tail . The occupancy definition proves the displayed form.

Define and

Theorem.

Proof. Normalized occupancies obey . Subtract the equations for the two policies:

For row vectors, multiplication by a stochastic matrix contracts the norm; the inverse series therefore has induced norm at most . Each row of has norm at most . Thus . Also , so . Multiply these two bounds and the prefactor in the performance-difference lemma.

This is a maximum-over-states TV statement. A measured average KL on visited states is not the same constraint. PPO clips contributions to a sampled surrogate; it does not enforce either constraint. For example, if an unsampled state's action probabilities change arbitrarily, the finite-batch PPO objective and its clipping statistics do not detect the change. Parameter sharing can produce such changes even when every sampled ratio is modest.

Natural-gradient subproblem. For a symmetric positive definite Fisher matrix , solve subject to . If and , let . Cauchy-Schwarz gives , with equality for aligned vectors. Therefore

For , the zero step is optimal. If is singular and has a nonzero null-space component, the problem is unbounded. If lies in its range, replace the inverse by a pseudoinverse and restrict to that range; null directions are undetermined. Damping changes the local quadratic model. None of these algebraic solutions guarantees that a nonlinear policy satisfies the exact KL constraint without checking it.

A.12 Model error and planning error are distinct

Consider true and learned models on the same finite spaces, both with mean rewards bounded by . Assume

Theorem. For every policy,

If a planner returns with , then .

Proof. Subtract the Bellman equations, retaining the learned value in the transition-error term:

The inverse norm is at most and . The transition convention gives the stated bound. For planning, insert and between the two true returns. The two model differences are bounded by and the middle optimization gap by .

If denotes total variation instead, multiply its contribution by two. A held-out average prediction error is not the uniform error assumed here. An ensemble disagreement score is not a proved upper bound on either error without calibration assumptions.

A.13 Entropy regularization and the DPO algebra

For a finite action set, positive reference probabilities , rewards , and , define and .

Variational identity. For every distribution ,

Proof. Substitute and expand the two KL sums. Nonnegativity of KL follows from : , with the usual limiting convention for zero . Equality occurs exactly at . Thus the optimizer is unique.

For a countable sequence space, the same identity requires and finite relevant expectations; bounded rewards suffice. Support outside is excluded, not created by optimization.

Apply the identity in every state to . The soft operator is

If , each differs by at most , so the exponential sum is multiplied by a factor between and . Taking logs proves a -contraction. Its fixed point and normalized exponential policy solve the regularized control problem by the same bounded telescoping verification as A.2. Uniform yields entropy regularization plus a per-step constant only when the number of admissible actions is fixed.

DPO's cancellation. For a fixed prompt , suppose the KL-regularized optimum is representable, and preferences follow at the specified reward scale. Rearranging the optimal policy gives

Subtract the expressions for the two responses. The prompt-only partition function cancels. Replacing with a parameterized in this preference model gives the negative log likelihood

The cancellation establishes the algebraic result. It does not establish offline preference coverage, a logistic model of human preferences, or convergence of a neural optimizer. Those need separate statistical and optimization arguments.

A.14 Group-relative estimators: what centering actually does

For one fixed prompt, draw iid responses. Let be the reward and , with finite required moments, , and . Let .

Proposition.

With the leave-one-out mean , the corresponding expectation is .

Proof. Independence gives for . The self term contributes to , so the centered contribution is . Average over . The leave-one-out mean has no self term and therefore zero score correlation.

Dividing by a random within-group standard deviation is not covered by this theorem. That denominator is correlated with the numerator and changes prompt weighting. For a Bernoulli action and reward with success probability and logit score , . Enumerating all groups verifies the factor exactly; the lab does this rather than relying on a noisy simulation.

If rewards are iid Bernoulli(), the probability of an informative mixed-reward group is

Proof. The only constant groups are all successes and all failures, disjoint events with probabilities and . Subtract them from one. Filtering to mixed groups replaces the prompt distribution by one proportional to this probability times its original probability, unless corrected. This is not automatically the original objective.

Exact probability of a mixed binary-reward group, derived in A.14. Curves assume independent responses with the same success probability.

A.15 Importance sampling, doubly robust evaluation, and impossibility

Consider a horizon , target policy , behavior policy , and the same environment and initial law. Require whenever a target-reachable action has positive target probability. Define and .

IS identity. For an integrable function of the history through transition , when the weighted absolute moment is finite.

Proof. Write the joint trajectory mass. Multiplication by the product of action ratios cancels each behavior action probability and replaces it by the target probability; initial and transition factors are unchanged. Sum over histories. For continuous outcomes this is the same Radon-Nikodym change of measure. Missing support prevents the cancellation on omitted histories.

Finite-horizon DR theorem. Let fixed (or independently trained) functions define and . Then

has expectation if the ratios are exact and the required moments exist, regardless of the value-model error.

Proof. Use IS on each term. Under the target policy . All the expected value terms telescope, including cancellation of the initial term; the terminal term is zero. The remaining sum is the expected target return.

The other robustness route. If , each bracket has conditional mean zero given the history through . It remains mean zero after multiplication by any integrable weight measurable at that point, including misspecified action-ratio products, provided their fitting does not use the current or future residual in a way that invalidates that conditioning. Thus the expectation is . This is a precise version of double robustness, not a promise of finite-sample accuracy when both nuisances are imperfect or estimated on the same trajectories. Cross-fitting can enforce the required training/evaluation separation. Self-normalized weights generally introduce finite-sample bias.

Nonidentifiability without support. In a one-step problem there are two actions, . The behavior always takes , whose reward is zero. In environment , gives reward ; in it gives reward . Every possible logged dataset has the identical distribution in these environments. The target that always takes has value or . No statistic of those logs can identify which is correct: a statistic has the same distribution under identical data laws and cannot be almost surely equal to two distinct values. Structural assumptions or new observations can narrow the possibilities; estimator complexity alone cannot.

A.16 Retrace and V-trace: fixed points, not just formulas

All operators in this section are exact tabular expected operators for fixed target and behavior policies. They do not by themselves prove convergence of a neural implementation. See Munos et al. (2016) and Espeholt et al. (2018) for the algorithms and broader results.

Retrace. Index vectors by state-action pairs. Let be the stochastic kernel that takes an environment transition and then chooses the next action from . For put , on behavior support, and

Then entrywise and . The expected infinite Retrace operator is

The inverse is the convergent nonnegative series ; expansion is exactly the expected product-of-traces target. Because ,

This matrix is nonnegative. Its row-sum vector is

since the inverse series begins with . Nonnegativity and this bound prove a sup-norm contraction with factor at most . Its fixed point satisfies , hence is . This proves why truncating the trace in this specific construction preserves the target fixed point.

V-trace. For states, take , and . Assume sufficient overlap that

Define , and kernels and analogously. Both are substochastic, because sums to at most one. The expected operator is

Its difference matrix is , entrywise nonnegative since and . Its row-sum vector is . Thus it contracts. A fixed point obeys , the Bellman equation for

It is the value of this clipped target policy, not generally . The cap changes the fixed point; the cap changes trace propagation. This is a fundamental distinction from the Retrace construction above.

A.17 Three useful counterexamples to overgeneralization

Monotone mixing is a sufficient argmax property. If and is coordinatewise nondecreasing, choosing a local maximizer of every gives a global maximizer: increase one coordinate at a time and the value cannot decrease. This proves , not equality of the sets. If , every joint action is globally optimal even if each has a unique maximizer.

Hindsight relabeling can condition on transition noise. Consider one action, one transition, and an independent fair coin determining the achieved state . For a fixed goal , reward is , so expected reward is . Relabel every transition with the achieved goal . Every relabeled reward is now one. Dynamics are goal-independent and rewards are perfectly recomputable, yet the relabeled conditional transition law for a given goal is not the original fair coin. Therefore these two properties alone do not make naive hindsight targets unbiased in stochastic dynamics.

A trajectory Gibbs density need not be a causal policy. In an uncontrolled one-step system, a fair coin gives or 1, with reward . Reweighting complete trajectories by changes the probability of to . No policy can change the coin: there are no actions. Thus a Gibbs distribution over entire trajectories is not automatically a realizable controlled distribution. In stochastic environments, causal maximum-entropy control regularizes action choices and retains the transition law; its backward recursion is A.13's soft Bellman equation. Maximum-entropy inverse modeling must specify whether it models unconstrained trajectories or causal decisions.

A.18 Shaping, options, and return-distribution contraction

Potential shaping. For fixed bounded , replace reward by . Summing to gives

For discounted continuing problems the final term vanishes, so every policy's value is shifted by and its action ordering is unchanged. For finite episodes the terminal term need not vanish. Require terminal potential zero, or a common discounted boundary term independent of policy; variable terminal times and a nonzero constant potential generally violate the latter. Finite-horizon time-dependent shaping uses with . This is the complete boundary argument.

Options. An option has a policy, initiation set, and termination rule; let its duration be almost surely finite and at least one. Its discounted backup is . The difference of two such backups is bounded by . Taking a maximum over available options preserves this bound by A.2's maximum inequality. Thus the option Bellman operator contracts. Zero-duration cycles or undefined termination require additional treatment.

Distributional policy evaluation. Let be return laws with finite th moments, , and define , where is the infimum of over couplings of the two laws. The fixed-policy distributional Bellman operator draws the same in both copies, then couples their next return laws within of their infimal th transport costs. The rewards cancel and the next returns are multiplied by , so

Formally the conditional th cost is at most ; average and let . This proves the inequality without assuming a common global optimal coupling. It is a fixed-policy result. Greedy action selection can switch discontinuously when means tie, so the argument does not establish the same contraction for distributional control or for a categorical projection in an arbitrary metric.

A.19 What a mathematical result licenses

The course assumes finite-dimensional linear algebra, elementary probability including conditional expectation, strong laws and martingale convergence, multivariable calculus, and elementary convex optimization. It states these prerequisites explicitly rather than developing all of probability within an RL course. The RL results above specify their sampling laws and boundary conditions. Many modern neural algorithms have exact specifications but no general global-convergence theorem for the implementations discussed here. In those cases, distinguish the exact objective and update from approximate optimization and empirical evidence.

For each research claim, write the theorem's domain alongside its practical counterpart: finite versus continuous state, known versus learned dynamics, fixed versus changing policy, exact versus fitted critic, independent versus reused data. A rigorous explanation ends at the boundary of its assumptions and makes that boundary useful.