@(@\newcommand{\W}[1]{ \; #1 \; }
\newcommand{\R}[1]{ {\rm #1} }
\newcommand{\B}[1]{ {\bf #1} }
\newcommand{\D}[2]{ \frac{\partial #1}{\partial #2} }
\newcommand{\DD}[3]{ \frac{\partial^2 #1}{\partial #2 \partial #3} }
\newcommand{\Dpow}[2]{ \frac{\partial^{#1}}{\partial {#2}^{#1}} }
\newcommand{\dpow}[2]{ \frac{ {\rm d}^{#1}}{{\rm d}\, {#2}^{#1}} }@)@
This is cppad-20221105 documentation. Here is a link to its
current documentation
.
exp_2: Second Order Reverse Mode
Purpose
In general, a second order reverse sweep is given the
first order expansion
for all of the variables in an operation sequence.
Given a choice of a particular variable,
it computes the derivative,
of that variables first order expansion coefficient,
with respect to all of the independent variables.
Mathematical Form
Suppose that we use the algorithm exp_2.hpp
to compute
@[@
f(x) = 1 + x + x^2 / 2
@]@
The corresponding second derivative is
@[@
\Dpow{2}{x} f (x) = 1
@]@
f_5
For our example, we chose to compute the derivative of @(@
v_5^{(1)}
@)@
with respect to all the independent variable.
For the case computed for the
first order sweep
,
@(@
v_5^{(1)}
@)@ is the derivative
of the value returned by exp_2.hpp
.
This the value computed will be the second derivative of
the value returned by exp_2.hpp
.
We begin with the function @(@
f_5
@)@ where @(@
v_5^{(1)}
@)@
is both an argument and the value of the function; i.e.,
@[@
\begin{array}{rcl}
f_5 \left(
v_1^{(0)}, v_1^{(1)} , \ldots , v_5^{(0)} , v_5^{(1)}
\right)
& = & v_5^{(1)}
\\
\D{f_5}{v_5^{(1)}} & = & 1
\end{array}
@]@
All the other partial derivatives of @(@
f_5
@)@ are zero.
Index 5: f_4
Second order reverse mode starts with the last operation in the sequence.
For the case in question, this is the operation with index 5.
The zero and first order sweep representations of this operation are
@[@
\begin{array}{rcl}
v_5^{(0)} & = & v_2^{(0)} + v_4^{(0)}
\\
v_5^{(1)} & = & v_2^{(1)} + v_4^{(1)}
\end{array}
@]@
We define the function
@(@
f_4 \left( v_1^{(0)} , \ldots , v_4^{(1)} \right)
@)@
as equal to @(@
f_5
@)@
except that @(@
v_5^{(0)}
@)@ and @(@
v_5^{(1)}
@)@ are eliminated using
this operation; i.e.
@[@
f_4 =
f_5 \left[ v_1^{(0)} , \ldots , v_4^{(1)} ,
v_5^{(0)} \left( v_2^{(0)} , v_4^{(0)} \right) ,
v_5^{(1)} \left( v_2^{(1)} , v_4^{(1)} \right)
\right]
@]@
It follows that
@[@
\begin{array}{rcll}
\D{f_4}{v_2^{(1)}}
& = & \D{f_5}{v_2^{(1)}} +
\D{f_5}{v_5^{(1)}} * \D{v_5^{(1)}}{v_2^{(1)}}
& = 1
\\
\D{f_4}{v_4^{(1)}}
& = & \D{f_5}{v_4^{(1)}} +
\D{f_5}{v_5^{(1)}} * \D{v_5}{v_4^{(1)}}
& = 1
\end{array}
@]@
All the other partial derivatives of @(@
f_4
@)@ are zero.
Index 4: f_3
The next operation has index 4,
@[@
\begin{array}{rcl}
v_4^{(0)} & = & v_3^{(0)} / 2
\\
v_4^{(1)} & = & v_3^{(1)} / 2
\end{array}
@]@
We define the function
@(@
f_3 \left( v_1^{(0)} , \ldots , v_3^{(1)} \right)
@)@
as equal to @(@
f_4
@)@
except that @(@
v_4^{(0)}
@)@ and @(@
v_4^{(1)}
@)@
are eliminated using this operation; i.e.,
@[@
f_3 =
f_4 \left[ v_1^{(0)} , \ldots , v_3^{(1)} ,
v_4^{(0)} \left( v_3^{(0)} \right) ,
v_4^{(1)} \left( v_3^{(1)} \right)
\right]
@]@
It follows that
@[@
\begin{array}{rcll}
\D{f_3}{v_2^{(1)}}
& = & \D{f_4}{v_2^{(1)}}
& = 1
\\
\D{f_3}{v_3^{(1)}}
& = & \D{f_4}{v_3^{(1)}} +
\D{f_4}{v_4^{(1)}} * \D{v_4^{(1)}}{v_3^{(1)}}
& = 0.5
\end{array}
@]@
All the other partial derivatives of @(@
f_3
@)@ are zero.
Index 3: f_2
The next operation has index 3,
@[@
\begin{array}{rcl}
v_3^{(0)} & = & v_1^{(0)} * v_1^{(0)}
\\
v_3^{(1)} & = & 2 * v_1^{(0)} * v_1^{(1)}
\end{array}
@]@
We define the function
@(@
f_2 \left( v_1^{(0)} , \ldots , v_2^{(1)} \right)
@)@
as equal to @(@
f_3
@)@
except that @(@
v_3^{(0)}
@)@ and @(@
v_3^{(1)}
@)@ are
eliminated using this operation; i.e.,
@[@
f_2 =
f_3 \left[ v_1^{(0)} , \ldots , v_2^{(1)} ,
v_3^{(0)} ( v_1^{(0)} ) ,
v_3^{(1)} ( v_1^{(0)} , v_1^{(1)} )
\right]
@]@
Note that, from the
first order forward sweep
,
the value of @(@
v_1^{(0)}
@)@ is equal to @(@
.5
@)@
and @(@
v_1^{(1)}
@)@ is equal 1.
It follows that
@[@
\begin{array}{rcll}
\D{f_2}{v_1^{(0)}}
& = &
\D{f_3}{v_1^{(0)}} +
\D{f_3}{v_3^{(0)}} * \D{v_3^{(0)}}{v_1^{(0)}} +
\D{f_3}{v_3^{(1)}} * \D{v_3^{(1)}}{v_1^{(0)}}
& = 1
\\
\D{f_2}{v_1^{(1)}}
& = &
\D{f_3}{v_1^{(1)}} +
\D{f_3}{v_3^{(1)}} * \D{v_3^{(1)}}{v_1^{(1)}}
& = 0.5
\\
\D{f_2}{v_2^{(0)}}
& = & \D{f_3}{v_2^{(0)}}
& = 0
\\
\D{f_2}{v_2^{(1)}}
& = & \D{f_3}{v_2^{(1)}}
& = 1
\end{array}
@]@
Index 2: f_1
The next operation has index 2,
@[@
\begin{array}{rcl}
v_2^{(0)} & = & 1 + v_1^{(0)}
\\
v_2^{(1)} & = & v_1^{(1)}
\end{array}
@]@
We define the function
@(@
f_1 ( v_1^{(0)} , v_1^{(1)} )
@)@
as equal to @(@
f_2
@)@
except that @(@
v_2^{(0)}
@)@ and @(@
v_2^{(1)}
@)@
are eliminated using this operation; i.e.,
@[@
f_1 =
f_2 \left[ v_1^{(0)} , v_1^{(1)} ,
v_2^{(0)} ( v_1^{(0)} ) ,
v_2^{(1)} ( v_1^{(1)} )
\right]
@]@
It follows that
@[@
\begin{array}{rcll}
\D{f_1}{v_1^{(0)}}
& = & \D{f_2}{v_1^{(0)}} +
\D{f_2}{v_2^{(0)}} * \D{v_2^{(0)}}{v_1^{(0)}}
& = 1
\\
\D{f_1}{v_1^{(1)}}
& = & \D{f_2}{v_1^{(1)}} +
\D{f_2}{v_2^{(1)}} * \D{v_2^{(1)}}{v_1^{(1)}}
& = 1.5
\end{array}
@]@
Note that @(@
v_1
@)@ is equal to @(@
x
@)@,
so the second derivative of
the function defined by exp_2.hpp
at @(@
x = .5
@)@
is given by
@[@
\Dpow{2}{x} v_5^{(0)}
=
\D{ v_5^{(1)} }{x}
=
\D{ v_5^{(1)} }{v_1^{(0)}}
=
\D{f_1}{v_1^{(0)}} = 1
@]@
There is a theorem about Algorithmic Differentiation that explains why
the other partial of @(@
f_1
@)@ is equal to the
first derivative of
the function defined by exp_2.hpp
at @(@
x = .5
@)@.
Verification
The file exp_2_rev2.cpp
contains a routine
which verifies the values computed above.
It only tests the partial derivatives of
@(@
f_j
@)@ that might not be equal to the corresponding
partials of @(@
f_{j+1}
@)@; i.e., the
other partials of @(@
f_j
@)@ must be equal to the corresponding
partials of @(@
f_{j+1}
@)@.
Which statement in the routine defined by exp_2_rev2.cpp
uses
the values that are calculated by the routine
defined by exp_2_for0.cpp
?
Which statements use values that are calculate by the routine
defined in exp_2_for1.cpp
?
Consider the case where @(@
x = .1
@)@
and we first preform a zero order forward sweep,
then a first order sweep,
for the operation sequence used above.
What are the results of a
second order reverse sweep; i.e.,
what are the corresponding derivatives of
@(@
f_5 , f_4 , \ldots , f_1
@)@.
Create a modified version of
exp_2_rev2.cpp
that verifies the values you obtained for the previous exercise.
Also create and run a main program that reports the result
of calling the modified version of
exp_2_rev2.cpp
.
