Talk:Hamilton–Jacobi–Bellman equation

Physics and Optimal Control

Um, I don't think the Hamilton-Jacobi-Bellman equation is the Hamilton-Jacobi equation anymore than let's say Shannon information is the thermodynamic entropy. Phys 02:57, 15 Aug 2004 (UTC)

Phys is right. There is some mixing together here of Hamilton-Jacobi-Bellman and Hamilton-Jacobi, of Optimal Control and Physics. The result is confusing. I will rewrite from the point of view of O.C. only. Someone else can add the relation to physics and to the pre-Bellman work. Encyclops July 2005

The historical reason for the name is that Bellman got the idea from the mathematics book of Carathéodory on the Calculus of Variations which used the Hamilton-Jacobi theory. JFB80 (talk) 21:23, 13 November 2010 (UTC)

Bracket notation

Is there a reason for using the notation $\langle a,b\rangle$ to denote inner product here? I'd prefer ordinary matrix notation: $a^{T}b$. The latter is less confusing, since it can't be confused with other variations of scalar product, such as $\int a^{\prime }b\,\mathrm {d} x$--PeR 12:08, 14 June 2006 (UTC)

$a^{T}b$ is very clear. But when a and b are somewhat messy expressions it becomes less readable. In our case what would we have $\left({\frac {\partial }{\partial x}}V(x,t)\right)^{T}F(x,u)$ ? I don't know if I like it or not. Encyclops 00:23, 15 June 2006 (UTC)

I think really the notation used should be $\nabla$ and the $\cdot$. The notation for ${\frac {\delta }{\delta x}}$ where x is a vector is not particularly intuitive.- (User) Wolfkeeper (Talk) 18:28, 20 May 2009 (UTC)

Yes, $\nabla$ or $\nabla _{x}$ seems good to me, FWIW. And a dot for the inner product, I like also. Encyclops (talk) 19:08, 20 May 2009 (UTC)

Sufficient condition?

The current article claims that the HJB is a sufficient condition. That sounds wrong to me, because first of all the equation itself is not a sufficient condition: I assume what is meant is that "if V solves HJB, this suffices to conclude that it optimizes the objective". But is this true in general? I know that in discrete-time, infinite-horizon cases, a solution of the Bellman equation only serves to identify a candidate solution for the original sequence problem, that is, solving the Bellman equation is necessary but not sufficient for optimality. (See Stokey-Lucas-Prescott, Recursive Methods in Economic Dynamics. Theorem 4.3 shows that for an infinite horizon problem, satisfying the Bellman equation and an appropriate 'transversality condition' suffices for optimality; but the Bellman equation alone is not sufficient.)

Is the sufficiency claim in this article based on the fact that the example given has a finite horizon T? If so, this should be clarified, and it would be helpful to add more general cases too. --Rinconsoleao (talk) 08:07, 30 May 2008 (UTC)

The continuous time/continuous state case we are looking at here is more complex than the discrete time case you mention; there are some delicate technical issues that do not arise in d.t. control. A number of "verification theorems" have been proven, using various assumptions. The simplest theorems, one of which goes back to Bellman, says that if a control satisfies HJB and the terminal condition, then that control is optimal. HJB => optimality. In this sense HJB is a sufficient condition. However, there could exist optimal solutions that are not smooth (not continuous or not differentiable), do not satisfy HJB, but are nevertheless optimal. There are also other verification theorems that establish HJB as necessary and sufficient, but that requires additional assumptions, so they are more restrictive. We also have to ask what kind of "solutions" of HJB we are talking about, the "classical" PDE solutions that Bellman used or the modern viscosity solutions. Frankly, my knowledge of this area is not sufficient ;-) to give an overview of all these theorems. Encyclops (talk) 23:49, 30 May 2008 (UTC)

Sufficient condition? A confirmation

I do agree with this remark. For me, HJB is a necessary condition i.e. an optimal control should necessarily satisfied HJB. I refer to Oksendal (Stochastic differential equations, theorem 11.2.1).

The important question is to know if the finding of a solution of a HJB equation is sufficient. No. Once found, the solution of the HJB PDE has to satisfies some criteria (a verification theorem, Oksendal 11.2.2).

HJB is necessary but not sufficient.

130.104.59.97 (talk) 09:31, 3 December 2009 (UTC)Devil may cry.

I am quite confused by this article. In my references the HJB equation is a sufficient not necessary condition. In the book of Bertsekas: Dynamic programming and optimal control, Athena Scientific at pag. 93 it is stated that the theorem about HJB is a sufficient condition. And in all my courses of optimal control, every professor always remarks the difference between the Pontryagin minimum principle (necessary) and the HJB (sufficient). Checking the book of Oksendal I saw that the formulation presented is for stochastic process from a stochastic/mathematical point of view. Now, I do not have the knowledge to understand where is the trick, however, I am quite sure that the HJB equation for optimal control problem, as formulated in this article and proved in Bertsekas, is a sufficient condition not necessary. Could someone make a more deep investigation? Pivs (talk) 20:43, 4 May 2012 (UTC)

Multiply by dt?

I wonder if the two last terms (before the big O) of the last equation should not be multiplied by dt. —Preceding unsigned comment added by 61.26.5.133 (talk) 03:18, 13 March 2009 (UTC)

I think you may be right. Any other opinions ? Encyclops (talk) 01:42, 16 March 2009 (UTC)

Agreed. Done. --Rinconsoleao (talk) 09:44, 8 June 2009 (UTC)

Terminal condition

In section The partial differential equation I see a terminal condition which does not quite look like a meaningful terminal condition. Is it actually meant to be

$V(x(T),T)=D(x(T))$

and if so, may I correct that? Thank you --Andylong (talk) 11:32, 3 July 2012 (UTC)

Terminal Constraint

Here we have a control problem with x(0) = x0.

Okay. But often we solve control problems where in addition there is a terminal constraint x(T) = xE.

I ran through the literature and I must say that I fail at finding the HJB in that case. In particular, right now we use V(t,X) := min_(x,u){ int_t^T C(x(s),u(s)) ds + D(x(T)) s.t. x(t)=X }. If we were to attempt V(t,X) := min_(x,u){ int_t^T C(x(s),u(s)) ds + D(x(T)) s.t. x(t)=X, x(T)=xE } then V(T,X) for X!=xE would no longer be well-defined.

It would be crazy if that problem was unsolvable via the article's technique. Certainly, because xE can be chosen freely or as parameter of the optimization problem, once the case xE has been presented in this article, the one-sided case can be deleted afterwards in favour of brevity. — Preceding unsigned comment added by 2A02:908:1657:A860:71C7:F597:421E:F6B8 (talk) 21:47, 26 October 2021 (UTC)