PhiBE: A PDE-Based Bellman Equation for Continuous-Time Policy Evaluation

Yuhua Zhu

In this paper, we study policy evaluation in continuous-time reinforcement learning (RL), where the state follows an unknown stochastic differential equation (SDE) but only discrete-time data are available. We first highlight that the discrete-time Bellman equation (BE) is not always a reliable approximation to the true value function because it ignores the underlying continuous-time structure. We then introduce a new Bellman equation, PhiBE, which integrates the discrete-time information into a continuous-time PDE formulation. By leveraging the smooth structure of the underlying dynamics, PhiBE provides a provably more accurate approximation to the true value function, especially in scenarios where the underlying dynamics change slowly or the reward oscillates. Moreover, we extend PhiBE to higher orders, providing increasingly accurate approximations. We further develop a model-free algorithm for PhiBE under linear function approximation and establish its convergence under model misspecification, together with finite-sample guarantees. Directly applying classical discrete-time LSTD bounds to the continuous-time RL setting yields a model-misspecification bound with an $O({\Delta t}^{-1/2})$ stability factor and a sample-complexity bound of order $O({\Delta t}^{-2})$. By exploiting the smooth structure of the underlying dynamics, our analysis instead gives a model-misspecification bound with a stability factor independent of ${\Delta t}$ and a sample-complexity bound of order $O({\Delta t}^{-1})$. Moreover, we identify a fundamental trade-off between discretization error and sample error that is intrinsic to continuous-time policy evaluation: finer time discretization reduces bias but amplifies variance, so excessively frequent sampling does not necessarily improve performance. This is an insight that does not arise in classical discrete-time RL analyses. Numerical experiments are provided to validate the theoretical guarantees we propose.

PhiBE: A PDE-Based Bellman Equation for Continuous-Time Policy Evaluation

Abstract