Calculus I · When You Can’t Solve It, Chase It

Where this is heading

Everything in this chapter happened on a number line — $x$ was a single number, $f'$ a single slope, “downhill” a choice between left and right. In Book 3, $x$ becomes a vector — a point in a space of many dimensions — and the derivative $f'$ becomes the gradient, a vector that points in the steepest uphill direction. “Step against the slope” becomes “step against the gradient,” and the update rule keeps the exact same shape: $x_{n+1} = x_n - \eta\,(\text{gradient at } x_n)$. The picture you built here — a ball rolling to the bottom of a valley, a learning rate that must be neither too timid nor too bold — is the whole picture. Book 3 just gives the ball more directions it is allowed to roll.

When you eventually watch a neural network train and see its “loss” tick downward step by step, you will not be watching magic. You will be watching this chapter, run a few million times.

Problems

This chapter is about iteration, so its “doing” is mostly computational — you learn gradient descent by running it and watching it converge or blow up. So the tiers are shaped differently here. The Computational Lab is the heart of the practice: structured experiments in the companion notebook. By hand is the short list genuinely worth doing on paper. Concept tests judgment. ◆ Diamond is where this chapter’s real paper depth lives — the proofs are the practice. Teach it and Looking ahead close it out.

Computational Lab — the real practice; run these in the notebook

1. Learning-rate sweep. On $f(x)=x^2$, run gradient descent from $x_0=1$ for $\eta=0.1,\,0.3,\,0.5,\,0.7,\,0.9,\,1.0,\,1.1,\,1.5$. Plot $x_n$ versus $n$ for each. Sort them into four behaviors (smooth glide, shrinking spiral, stuck bounce, explosion) and check the boundary against the prediction $|1-2\eta|<1$.
2. Hunt the fastest rate. Still on $x^2$: which $\eta$ reaches within $10^{-6}$ of the minimum in the fewest steps? Compare your experimental winner with the predicted optimum $\eta=\tfrac12$.
3. Newton, fast and broken. Implement $x_{n+1}=\tfrac12(x_n+\tfrac{2}{x_n})$ for $\sqrt2$; print $x_n$ and the error $|x_n-\sqrt2|$ each step and watch the correct digits roughly double. Then point Newton at $f(x)=x^{1/3}$ from $x_0=1$ and watch it fly apart.
4. A landscape with two valleys. Run gradient descent on $f(x)=x^4-3x^2+x$ from several starting points. Show that different starts land in different minima. (You’ve just discovered, by experiment, the central headache of training: gradient descent finds a local minimum, not necessarily the best one — and which one depends on where you begin.)
5. The race. On a single quadratic, count the steps gradient descent (good $\eta$) and Newton’s method each need to reach a fixed tolerance. Watch Newton win by a landslide — then find a setup where Newton fails and gradient descent doesn’t.
6. (Stretch) A first real optimizer. Add “momentum”: $x_{n+1}=x_n-\eta f'(x_n)+\beta\,(x_n-x_{n-1})$ with $\beta=0.9$. On a stretched bowl $f(x)=10x^2$, compare plain gradient descent against the momentum version. (This is a baby version of the optimizers that actually train neural networks.)

By hand — worth doing on paper

1. For $f(x)=x^2-5$: write $f'$, write Newton’s rule, simplify it, then do two steps from $x_0=2$ toward $\sqrt5$.
2. Gradient descent on $f(x)=x^2$ with $\eta=0.1$, $x_0=1$: compute $x_1,x_2,x_3$.
3. For $f(x)=x^2-4x+7$: find $f'$, find the exact minimizer from $f'=0$, then take two gradient-descent steps from $x_0=0$ with $\eta=0.1$ and check you’re heading the right way.
4. Classify the behavior for each factor $1-2\eta$: $\eta=0.2,\ 0.5,\ 0.8,\ 1.2$ (monotone glide / one-step / oscillating glide / explosion).

Concept & counterexample — think; true/false with justification, or build the case

1. 1. A smaller learning rate is always safer, so always better. (b) Gradient descent can overshoot the minimum. (c) Newton’s method always converges faster than gradient descent. (d) If gradient descent’s steps keep shrinking, it must be approaching the minimum.
2. Predict, then check. For $f(x)=x^2$ with $\eta=0.5$, where is $x_1$ — without computing step by step? For $\eta=1$, describe the long-run behavior in one sentence.
3. When it breaks. Give (a) a function/start where Newton diverges, and (b) a function/start where gradient descent diverges, naming the cause of each. Which failure is fixable by changing only one number, and which isn’t?
4. The two-valley trap (on paper). Sketch a function with a shallow valley and a deeper one. Mark a start that gradient descent sends to the shallow (worse) minimum and one that sends it to the deeper one. In one sentence, what does this say about “gradient descent finds the best answer”?
5. An asymmetry worth seeing. For $f(x)=x^2$, argue that too small an $\eta$ only makes you slow (you still converge), while too large an $\eta$ makes you fail outright. Why are the two kinds of “wrong learning rate” not symmetric?

◆ Diamond — the proofs are the practice

1. ◆ Derive Newton’s method from the tangent-line approximation $f(x)\approx f(x_n)+f'(x_n)(x-x_n)$: set the line to zero and solve for $x_{n+1}$.
2. ◆ Show Newton on $f(x)=x^2-A$ gives the Babylonian rule $x_{n+1}=\tfrac12(x_n+\tfrac{A}{x_n})$, and use it to compute $\sqrt7$ to five decimals from $x_0=3$.
3. ◆ The cube-root disaster: for $f(x)=x^{1/3}$, show Newton’s rule simplifies to $x_{n+1}=-2x_n$; conclude divergence from every nonzero start and explain geometrically (what is the tangent near $x=0$ doing?).
4. ◆ The learning-rate range, in general. For $f(x)=\tfrac{a}{2}(x-m)^2$ (a bowl of curvature $a>0$, minimum at $m$), show gradient descent gives $x_{n+1}-m=(1-\eta a)(x_n-m)$, hence $x_n-m=(1-\eta a)^n(x_0-m)$. Find the exact convergence range and the optimal $\eta$. Then state the single most important practical lesson: what must happen to the safe learning rate as the curvature $a$ grows?
5. ◆◆ Why Newton’s error squares. For $x_{n+1}=\tfrac12(x_n+\tfrac{A}{x_n})$, let $e_n=x_n-\sqrt A$. Prove the exact identity $e_{n+1}=\dfrac{e_n^2}{2x_n}$. Conclude that near the root the error roughly squares each step — which is exactly why the correct digits double (quadratic convergence).
6. ◆◆ Newton reaches a quadratic’s minimum in one step. Using $x_{n+1}=x_n-\tfrac{f'(x_n)}{f''(x_n)}$, show that for any $f(x)=ax^2+bx+c$ ($a>0$) the first step lands on the exact minimizer, regardless of $x_0$. Explain via Chapter 6 why this had to happen (how well does a quadratic match its own second-order Taylor model?).
7. ◆◆ The fixed-point view that unifies everything. Both methods have the form $x_{n+1}=g(x_n)$, and a limit $L$ must satisfy $g(L)=L$ — a fixed point, exactly like Chapter 1’s recursive sequences. A general fact (which you may use): the iteration converges to $L$ when $|g'(L)|<1$, and the smaller $|g'(L)|$, the faster. (a) For gradient descent, $g(x)=x-\eta f'(x)$; compute $g'(L)$ at a minimum $L$ and recover the convergence range from problem 19. (b) For Newton, $g(x)=x-\tfrac{f(x)}{f'(x)}$; show $g'(L)=0$ at a simple root. Explain how this one criterion accounts for both the learning-rate range and Newton’s blazing speed.

Teach it — explain it, write it up, or fix it

1. In three or four sentences with the ball-in-a-valley picture, teach a friend what the learning rate is and what goes wrong at both extremes — too big and too small.
2. Debug your classmate. They say: “Newton’s method is faster, so real AI should obviously use it instead of gradient descent.” Give the one true sentence that explains why neural networks are trained with gradient descent anyway.
3. Write a worked solution for “use Newton’s method to find $\sqrt{10}$ to four decimals,” narrating the setup, the rule, and each iteration so a peer could reuse it for any square root.
4. Design a problem. Cook up a specific quadratic $f$ and a learning rate $\eta$ for which gradient descent reaches the minimum in exactly one step. Show why your $\eta$ works (relate it to $f''$), and write the solution key.
5. In a few sentences, teach the connection between “finding where $f=0$” (Newton’s first job) and “finding where $f$ is smallest” (gradient descent’s job) — how does the same machine do both?

Looking ahead — Book 3, where $x$ becomes a vector

1. In Book 3, $x$ becomes a vector and $f'$ becomes the gradient — a vector pointing in the steepest uphill direction. Predict what the rule $x_{n+1}=x_n-\eta f'(x_n)$ becomes, and what stays exactly the same.
2. Your safe range was $0<\eta<\tfrac{2}{f''}$. In many dimensions there isn’t one curvature $f''$ but many — different in different directions. Predict why a single number $\eta$ might be too big in one direction and too small in another at the same time. (This is why advanced optimizers adapt the rate per direction.)
3. Newton needs $f''$: one number here, an $N\times N$ table in $N$ dimensions. For a model with one million parameters, how many entries is that table? Say in one sentence why that number ends Newton’s method at real scale — and reconnect it to this chapter’s punchline.

All 12 chapters of Calculus I · Foundations · All courses