An Introduction to Complex Step Differentiation

In an old post we went over the finite difference method, where we approximated the derivative of a function by sampling the function at multiple specific points, and then, as the name implies, taking the difference between these points. Finite differencing is both easy to understand and good enough for a lot of problems leading to widespread use (heck, whole textbooks have been written about them). To improve accuracy you either need to change the locations of the specific points, which is hard to do without knowing the function true derivative (which if you know, why are you using finite differencing), or you can use more samples in a higher-order finite differencing scheme which is computationally more expensive. What if you could get the best of both worlds, improvements in accuracy and improvements in performance from using a single sample?

Complex Step Differentiation

Let’s say we want to take the derivative of some function F(x). In the finite difference approximation we expand the function using a Taylor series with a  finite step of size h, in a real direction. What if instead of taking the step in a real direction, we took it in the imaginary direction? If we were to do that we would obtain

Taking the imaginary portion of both sides and then lopping off all terms above second-order thren gives us

Flip the sides and re-arrange to obtain exactly what we wanted

Note that as h gets small, h^2 gets even smaller, meaning that this approximation has a truncation error of

Using standard finite differences to get this level of truncation error would require two samples, versus complex step’s single sample. But remember that truncation error isn’t the only kind of error, we’ll explore other sources of error and how they affect these approximations in a later post.

But what if we want a higher-order derivative, like the second derivative? Then instead of taking the imaginary component of both sides, we can take the real component of both sides and lop off the fourth-order terms and higher to obtain.

re-arranging we obtain

which has a truncation error of

Caveat Emptor

Do note that this only works on analytic functions, which are those that are both infinitely differentiable and smoothly extendable into the complex plane.

Let’s see how well these work on real functions

Now that we have the formulas for complex step differentiation, lets see how well they work on an example. I’ve arbitrarily chosen the following function

whose derivative is

Plotting them together we get the following

If we then evaluate the derivative at x=1, and compare it to the central difference and complex step approximations with different step sizes, h, we can see how the error changes. Apriori we would expect the error to decrease with decreasing step size, and we see that behavior for both until a step size of approximately 1e-5, when the noise in the central difference begins to grow again. In comparison, the error in the complex step approximation continues to fall until it reaches a step size of approximately 10e-8. It achieves a best error of approximately 3 orders of magnitude lower than that of the central difference. 

What’s more, as the step size continues to decrease the central difference performance degrades, while the complex step simply stops improving.  The exact location of this inflection point for the central difference changes with the function being evaluated, so the step size needs to be tuned for the specific problem. The complex step may stop getting better after a certain step size, but it also doesn’t get worse, allowing you to simply set the step size to the smallest it can go (standard machine precision is about 1e-12, so I set my step size in real problems at that).

Want More Gereshes?

If you want to receive new Gereshes blog post directly to your email when they come out, you can sign up for that here!

Don’t want another email? That’s ok, Gereshes also has a twitter account and subreddit!