Calculus I

When You Can’t Solve It, Chase It

Or: How to find an answer by taking smarter and smarter guesses

Companion notebook. Play with this chapter’s ideas — sliders, animations, and drills — in notebooks/ch9_newton_gradient_descent.ipynb.

By the end of this chapter you will be able to do three things, and the third one is, more or less, how every modern AI system is trained.

You will be able to find where a function crosses zero even when no formula on Earth can do it for you — by starting with a guess and improving it. This is Newton’s method, and you will derive it yourself out of the one idea from Chapter 6 that we said would keep paying off: near any point, a curve looks like its tangent line.

You will be able to find where a function is smallest by repeatedly stepping downhill. This is gradient descent, and you will understand exactly why it sometimes glides smoothly to the bottom, sometimes bounces around, and sometimes flies off the edge of the world — all controlled by a single dial called the learning rate.

And you will see that these two methods, which look like they are doing different jobs, are secretly the same idea pointed at two different targets. Understanding how they differ is the difference between knowing what a neural network does when it learns and just believing that it does.

Here is the mindset for the whole chapter. Up to now, “solving” a problem has meant finding a formula for the exact answer. But most equations worth caring about have no such formula. The honest move, the move that runs the real world, is to stop demanding the exact answer in one shot and instead build a guess that gets better every time you repeat a step. That repeated step is called iteration, and it is one of the most powerful ideas you will learn in this book.

The wall: equations you can’t solve

In Chapter 7 we learned that to minimize a function $f$, we look for where its slope is zero — we solve $f'(x) = 0$. That is the right idea. The trouble is that “solve” usually fails.

Try to find where $f(x) = x - \cos x$ equals zero. Set $x = \cos x$ and… there is no algebra that isolates $x$. The answer exists — it is about $0.739$ — but no rearranging, factoring, or formula will hand it to you. Equations mixing polynomials with $\cos$, or with $e^x$, or with $\ln$, are the normal case, not the exotic one. The clean solvable equations you met in algebra are the rare, lucky exceptions.

So we need a method that finds the answer without a formula for it. Watch how the tangent line — the star of Chapter 6 — turns out to be exactly the tool.

All 12 chapters of Calculus I · Foundations · All courses