8.1: Minimizing Error using Derivatives
 Page ID
 7239
In calculus, the derivative is a measure of the slope of any function of x, or f(x)f(x), at each given value of xx. For the function f(x)f(x), the derivative is denoted as f′(x)f′(x) or, pronounced as “f prime x”. Because the formula for ∑ϵ2∑ϵ2 is known and can be treated as a function, the derivative of that function permits the calculation of the change in the sum of the squared error over each possible value of ^αα^ and ^ββ^. For that reason, we need to find the derivative for ∑ϵ2∑ϵ2 with respect to changes in ^αα^ and ^ββ^. That, in turn, will permit us to “derive” the values of ^αα^ and ^ββ^ that result in the lowest possible ∑ϵ2∑ϵ2.
Look – we understand that this all sounds complicated. But it’s not all that complicated. In this chapter, we will walk through all the steps so you’ll see that it's really rather simple and, well, elegant. You will see that differential calculus (the kind of calculus that is concerned with rates of change) is built on a set of clearly defined rules for finding the derivative for any function f(x)f(x). It’s like solving a puzzle. The next section outlines these rules, so we can start solving puzzles.
8.1.1 Rules of Derivation
Derivative Rules
 Power Rule
 Constant Rule
 A Constant Times a Function
 Differentiating a Sum
 Product Rule
 Quotient Rule
 Chain Rule
The following sections provide examples of the application of each rule.
Rule 1: The Power Rule
Example:f(x)=x6f′(x)=6∗x6−1=6x5f(x)=x6f′(x)=6∗x6−1=6x5
A second example can be plotted in R
. The function is f(x)=x2f(x)=x2 and therefore, using the power rule, the derivative is: f′(x)=2xf′(x)=2x.
x < c(5:5)
x
## [1] 5 4 3 2 1 0 1 2 3 4 5
y < x^2
y
## [1] 25 16 9 4 1 0 1 4 9 16 25
plot(x,y, type="o", pch=19)
Rule 2: The Constant Rule
Example:f(x)=346f′(x)=0=10xf(x)=346f′(x)=0=10x
Rule 3: A Constant Times a Function
Example:f(x)=5x2f′(x)=5∗2x2−1=10xf(x)=5x2f′(x)=5∗2x2−1=10x
Rule 4: Differentiating a Sum
Example:
f(x)=4x2+32xf′(x)=(4x2)′+(32x)′=4∗2x2−1+32=8x+32f(x)=4x2+32xf′(x)=(4x2)′+(32x)′=4∗2x2−1+32=8x+32
Rule 5: The Product Rule
Example:f(x)=x3(x−5)f′(x)=(x3)′(x−5)+(x3)(x−5)′=3x2(x−5)+(x3)∗1=3x3−15x2+x3=4x3−15x2f(x)=x3(x−5)f′(x)=(x3)′(x−5)+(x3)(x−5)′=3x2(x−5)+(x3)∗1=3x3−15x2+x3=4x3−15x2
In a second example, the product rule is applied to the function y=f(x)=x2−6x+5y=f(x)=x2−6x+5. The derivative of this function is f′(x)=2x−6f′(x)=2x−6. This function can be plotted in R
.
x < c(1:7)
x
## [1] 1 0 1 2 3 4 5 6 7
y < x^26*x+5
y
## [1] 12 5 0 3 4 3 0 5 12
plot(x,y, type="o", pch=19)
abline(h=0,v=0)
We can also use the derivative and R
to calculate the slope for each value of XX.
b < 2*x6
b
## [1] 8 6 4 2 0 2 4 6 8
The values for XX, which are shown in Figure \(\PageIndex{2}\), range from 8 to +8 and return derivatives (slopes at a point) ranging from 25 to +25.
Rule 6: the Quotient Rule
Example:f(x)=xx2+5f′(x)=(x2+5)(x)′−(x2+5)′(x)(x2+5)2=(x2+5)−(2x)(x)(x2+5)2=−x2+5(x2+5)2f(x)=xx2+5f′(x)=(x2+5)(x)′−(x2+5)′(x)(x2+5)2=(x2+5)−(2x)(x)(x2+5)2=−x2+5(x2+5)2
Rule 7: The Chain Rule
Example:f(x)=(7x2−2x+13)5f′(x)=5(7x2−2x+13)4∗(7x2−2x+13)′=5(7x2−2x+13)4∗(14x−2)f(x)=(7x2−2x+13)5f′(x)=5(7x2−2x+13)4∗(7x2−2x+13)′=5(7x2−2x+13)4∗(14x−2)
8.1.2 Critical Points
Our goal is to use derivatives to find the values of ^αα^ and ^ββ^ that minimize the sum of the squared error. To do this we need to find the minima of a function. The minima is the smallest value that a function takes, whereas the maxima is the largest value. To find the minima and maxima, the critical points are key. The critical point is where the derivative of the function is equal to 00, or f′(x)=0f′(x)=0. Note that this is equivalent to the slope is equal to 00.
Example: Finding the Critical Points
To find the critical point for the function
y=f(x)=(x2−4x+5)y=f(x)=(x2−4x+5);

First find the derivative; f′(x)=2x−4f′(x)=2x−4

Set the derivative equal to 00; f′(x)=2x−4=0f′(x)=2x−4=0

Solve for xx; x=2x=2

Substitute 22 for xx into the function and solve for yy

Thus, the critical point (there’s only one in this case) of the function is (2,1)(2,1)
Once a critical point is identified, the next step is to determine whether that point is a minima or a maxima. The most straightforward way to do this is to identify the x,y coordinates and plot. This can be done in R
, as we will show using the function y=f(x)=(x2−4x+5)y=f(x)=(x2−4x+5). The plot is shown in Figure \(\PageIndex{3}\).
x < c(5:5)
x
## [1] 5 4 3 2 1 0 1 2 3 4 5
y < x^24*x+5
y
## [1] 50 37 26 17 10 5 2 1 2 5 10
plot(x,y, type="o", pch=19)
As can be seen, the critical point (2,1)(2,1) is a minima.
8.1.3 Partial Derivation
When an equation includes two variables, one can take a partial derivative with respect to only one variable, while the other variable is simply treated as a constant. This is particularly useful in our case because the function ∑ϵ2∑ϵ2 has two variables – ^αα^ and ^ββ^.
Let’s take an example. For the function y=f(x,z)=x3+4xz−5z2y=f(x,z)=x3+4xz−5z2, we first take the derivative of xx holding zz constant.
∂y∂x=∂f(x,z)∂x=3x2+4z∂y∂x=∂f(x,z)∂x=3x2+4z
Next we take the derivative of zz holding xx constant.
∂y∂z=∂f(x,z)∂z=4x−10z