Gradient Descent#

In the previous lesson, we built our first model capable of learning its weights from data.

We were left with an important question:

How do we know in which direction to modify a weight to make the loss decrease?

To answer that question, let’s deliberately simplify the problem. Let’s start with a single data point.

A deliberately simple problem#

Our model must learn a very simple relationship:

    x → y

Let’s use four examples:

At first glance, the relationship is obvious:

y = 2 × x

But this time, we will not give the value 2 directly to the model.

Let’s start with a weight w equal to 0.

The model therefore calculates its prediction with:

At the beginning:

w = 0

So, for x = 1:

prediction = 0 × 1 = 0

The predicted value is therefore far from the expected value y = 2.

Our model still has some work to do.

Measuring the error#

We need a way to measure how far the prediction prediction is from the expected value y.

Let’s use a very simple loss function:

Why square error?

Because the result is always positive, and large errors contribute more to the loss.

Let’s take our first situation, with w = 0.

For x = 1 and y = 2:

Let’s perform the same calculation with the four data points.

The average loss obtained is 30.

The current value of w therefore gives us a model that is still far from the expected values.

We now need to find out how to modify the weight w.

Finding the direction#

We know that the loss is too high and that the current value of w is 0.

We also know, just by looking at the data, that the final value of w should be close to 2.

But our program does not know that value.

It has to find it from the data.

One idea would be to try different values of w and keep the one that gives the smallest loss.

That would work for our small problem.

But as soon as our model has more parameters, this approach quickly becomes impractical.

So we need more useful information:

In which direction should we move the weight w to make the loss decrease?

This information is given by the gradient.

The gradient#

The gradient tells us how the loss changes when we modify the weight w.

In our model:

The derivative of the loss function with respect to the weight w gives us:

Let’s calculate gradient for each data point, then its average.

We therefore obtain an indication of the direction in which to modify w.

Gradually modifying the weight w#

Once gradient has been calculated, we can modify w:

The variable learning_rate controls the size of the change applied to w.

In our experiment:

learning_rate = 0.05

The mechanism is then:

We have just built a real learning loop.

Our Python program#

Here is the complete program to run.

data = [
    (1, 2),
    (2, 4),
    (3, 6),
    (4, 8),
]

w = 0.0
learning_rate = 0.05
number_of_epochs = 20

for epoch in range(number_of_epochs):
    gradient = 0.0
    loss = 0.0

    for x, y in data:
        prediction = w * x
        error = prediction - y
        loss += error ** 2
        gradient += 2 * x * error

    gradient /= len(data)
    loss /= len(data)

    w -= learning_rate * gradient

    print(
        "Epoch :", epoch + 1,
        "| loss :", round(loss, 6),
        "| gradient :", round(gradient, 6),
        "| w :", round(w, 6)
    )

print()
print("=== Final model ===")
print("Weight w :", w)

print()
print("=== Test ===")

for x, y in data:
    prediction = w * x

    print(
        "x =", x,
        "| expected =", y,
        "| predicted =", round(prediction, 4)
    )

Let’s look at what actually happened#

The program displays the evolution of loss, gradient, and w at each epoch.

Here is the result we obtained:

Epoch : 1 | perte : 30.0 | gradient : -30.0 | w : 1.5
Epoch : 2 | perte : 1.875 | gradient : -7.5 | w : 1.875
Epoch : 3 | perte : 0.117188 | gradient : -1.875 | w : 1.96875
Epoch : 4 | perte : 0.007324 | gradient : -0.46875 | w : 1.992188
Epoch : 5 | perte : 0.000458 | gradient : -0.117188 | w : 1.998047
Epoch : 6 | perte : 2.9e-05 | gradient : -0.029297 | w : 1.999512
Epoch : 7 | perte : 2e-06 | gradient : -0.007324 | w : 1.999878
Epoch : 8 | perte : 0.0 | gradient : -0.001831 | w : 1.999969
Epoch : 9 | perte : 0.0 | gradient : -0.000458 | w : 1.999992
Epoch : 10 | perte : 0.0 | gradient : -0.000114 | w : 1.999998
Epoch : 11 | perte : 0.0 | gradient : -2.9e-05 | w : 2.0
Epoch : 12 | perte : 0.0 | gradient : -7e-06 | w : 2.0
Epoch : 13 | perte : 0.0 | gradient : -2e-06 | w : 2.0
Epoch : 14 | perte : 0.0 | gradient : -0.0 | w : 2.0
Epoch : 15 | perte : 0.0 | gradient : -0.0 | w : 2.0
Epoch : 16 | perte : 0.0 | gradient : -0.0 | w : 2.0
Epoch : 17 | perte : 0.0 | gradient : -0.0 | w : 2.0
Epoch : 18 | perte : 0.0 | gradient : -0.0 | w : 2.0
Epoch : 19 | perte : 0.0 | gradient : -0.0 | w : 2.0
Epoch : 20 | perte : 0.0 | gradient : -0.0 | w : 2.0

=== Final model ===
Weight w : 1.999999999998181

=== Test ===
x = 1 | expected = 2 | predicted = 2.0
x = 2 | expected = 4 | predicted = 4.0
x = 3 | expected = 6 | predicted = 6.0
x = 4 | expected = 8 | predicted = 8.0

The result is quite telling.

From the very first epoch, the weight w moves from 0 to 1.5.

Then w gradually evolves:

At the same time, the value of loss decreases very quickly.

The program therefore never knows the final value of w directly. It discovers it by using gradient to guide successive corrections of w.

And the predictions?#

Once the learning process is complete, we test the model on the same data.

The result is:

x = 1 | expected = 2 | predicted = 2.0
x = 2 | expected = 4 | predicted = 4.0
x = 3 | expected = 6 | predicted = 6.0
x = 4 | expected = 8 | predicted = 8.0

The model has therefore learned a weight w extremely close to 2:

w = 1.999999999998181

With this value of w, the four values of prediction correspond to the expected values y.

What actually happened?#

We never wrote in the program:

w = 2

The program started with:

w = 0

It then used the values of x and y stored in data to gradually correct w.

At each epoch:

It is this repetition that allows the model to improve.

We have gone from a model with w = 0 to a model whose w is practically equal to 2.

Why do we call it “descent”?#

We are trying to reduce the value of loss.

We can imagine the loss as a surface with different heights.

Each possible value of w then corresponds to a position on that surface.

The gradient gradient tells us the slope at that point.

The update:

w = w - learning_rate × gradient

moves w in the direction that reduces loss.

We are therefore gradually looking for an area where the value of loss is minimal.

Hence the name:

gradient descent.

In our experiment, this descent is especially simple: the model has a single weight w.

A first real learning loop#

With our first watering model, we had already encountered the idea of correction:

error = label - result

We then modified the weights when the prediction was incorrect.

Here, we go one step further.

The model no longer simply notices that prediction differs from the expected value y.

It measures a loss loss and uses the gradient gradient to determine how to gradually modify the weight w.

We now have the essential elements of a learning loop:

This loop is one of the fundamental mechanisms of Machine Learning.

What’s next#

Our experiment was deliberately simple.

A single weight w, one relationship between x and y, and four data points in data were enough to make the mechanism appear.

But our watering model already had two measurements:

• temperature;
• humidity.

It therefore had two weights to learn.

And soon, we will want to go much further.

What happens when our model has several parameters and several layers of computation?

This is where we will start to see neural networks appear.