Back to our watering model#

In the first two lessons, we gradually built a model able to learn from data.

We then studied gradient descent with a single weight w.

Let’s return to our original problem: deciding whether a plant should be watered from temperature and humidity.

This time, our model has several parameters.

It must learn:

• one weight for temperature;
• one weight for humidity;
• a bias.

We still have a few simple observations:

The value 0 means “do not water”.

The value 1 means “water”.

Our goal is now to let gradient descent learn the three model parameters.

Our model#

The model first calculates a score:

score = temperature × weight_temperature
        + humidity × weight_humidity
        + bias

Then we turn this score into a decision:

if score > 0
    water
else
    do not water

We are deliberately keeping things simple.

We are not using any other function yet. The score can therefore take any value.

The question is whether this approach is enough.

The complete program#

Before looking at Python, let’s summarize what the program must do.

We find exactly the mechanism studied in the previous lesson, but now with several parameters.

Here is the complete Python program:

data = [
    (22, 70, 0),
    (24, 65, 0),
    (27, 42, 1),
    (29, 35, 1),
    (25, 55, 0),
]

weight_temperature = 0.0
weight_humidity = 0.0
bias = 0.0

learning_rate = 0.001
epochs = 100


def prediction(temperature, humidity):
    return (
        temperature * weight_temperature
        + humidity * weight_humidity
        + bias
    )


for epoch in range(epochs):
    gradient_temperature = 0.0
    gradient_humidity = 0.0
    gradient_bias = 0.0
    loss = 0.0

    for temperature, humidity, label in data:
        result = prediction(temperature, humidity)
        error = result - label

        loss += error ** 2

        gradient_temperature += 2 * temperature * error
        gradient_humidity += 2 * humidity * error
        gradient_bias += 2 * error

    gradient_temperature /= len(data)
    gradient_humidity /= len(data)
    gradient_bias /= len(data)
    loss /= len(data)

    weight_temperature -= learning_rate * gradient_temperature
    weight_humidity -= learning_rate * gradient_humidity
    bias -= learning_rate * gradient_bias

    print(
        "Epoch:", epoch + 1,
        "| loss:", round(loss, 6),
        "| grad_T:", round(gradient_temperature, 6),
        "| grad_H:", round(gradient_humidity, 6),
        "| grad_bias:", round(gradient_bias, 6)
    )

print("\n=== Final model ===")
print("Temperature weight:", weight_temperature)
print("Humidity weight:", weight_humidity)
print("Bias:", bias)

print("\n=== Test ===")

for temperature, humidity, label in data:
    score = prediction(temperature, humidity)

    if score > 0:
        decision = "water"
    else:
        decision = "do not water"

    print(
        "temperature =", temperature,
        "| humidity =", humidity,
        "| score =", round(score, 6),
        "| expected =", label,
        "| decision =", decision
    )

This time, we no longer have a single weight w.

We now have three parameters that gradient descent must adjust.

A first attempt#

Let’s start with:

learning_rate = 0.001
epochs = 100

The result is not very encouraging.

From the first epochs, the loss increases sharply:

Epoch: 1 | loss: 0.4 | grad_T: -22.4 | grad_H: -30.8 | grad_bias: -0.8
Epoch: 2 | loss: 3.979453 | grad_T: 88.41312 | grad_H: 215.14896 | grad_bias: 3.62896
Epoch: 3 | loss: 145.603543 | ...
Epoch: 4 | loss: 5662.132883 | ...

The values then keep growing.

The model diverges.

The learning rate follows the same principle as in the previous lesson.

But our data is no longer on the same scale.

Temperature is around 20 or 30.

Humidity is around 40 to 70.

The humidity weight therefore receives much larger corrections than when we worked with a simple value x between 1 and 4.

Our first reflex is therefore to reduce the learning rate.

Let’s try a smaller value#

Set:

learning_rate = 0.00001

With 100 epochs, training becomes stable.

The loss decreases:

Epoch: 1 | loss: 0.4
Epoch: 10 | loss: 0.323331
Epoch: 100 | loss: 0.249006

This time, the model no longer diverges.

But the final result is still not satisfactory.

The learned parameters are:

Temperature weight: 0.008899723076964385
Humidity weight: 0.0016567612636759176
Bias: 0.00026465299479816375

The scores remain close to one another.

Our model therefore decides:

water
water
water
water
water

We have stabilized learning, but we have not really learned our decision rule.

Finding a compromise#

Let’s increase the learning rate slightly:

learning_rate = 0.0001

After 100 epochs:

Epoch: 1 | loss: 0.4
Epoch: 10 | loss: 0.253249
Epoch: 100 | loss: 0.066838

This time, the result is much better.

The model learns:

Temperature weight: 0.04433169880296688
Humidity weight: -0.01417226800761924
Bias: 0.0012139153511863618

The decisions become:

temperature = 22 | humidity = 70 | score = -0.0155 | expected = 0 | decision = do not water
temperature = 24 | humidity = 65 | score = 0.144... | expected = 0 | decision = water
temperature = 27 | humidity = 42 | score = 0.602... | expected = 1 | decision = water
temperature = 29 | humidity = 35 | score = 0.790... | expected = 1 | decision = water
temperature = 25 | humidity = 55 | score = 0.33... | expected = 0 | decision = water

We therefore get 3 correct decisions out of 5.

That is not catastrophic.

But it is not the model we are looking for either.

The temptation to wait#

Maybe the problem is simply the number of epochs.

After all, we only used 100 epochs.

Let’s move to:

learning_rate = 0.0001
epochs = 10000

But we do not want to print 10,000 lines in the terminal.

We only need to modify the program output.

We replace:

print(
    "Epoch:", epoch + 1,
    "| loss:", round(loss, 6),
    "| grad_T:", round(gradient_temperature, 6),
    "| grad_H:", round(gradient_humidity, 6),
    "| grad_bias:", round(gradient_bias, 6)
)

with:

if epoch < 10 or (epoch + 1) % 1000 == 0:
    print(
        "Epoch:", epoch + 1,
        "| loss:", round(loss, 6),
        "| grad_T:", round(gradient_temperature, 6),
        "| grad_H:", round(gradient_humidity, 6),
        "| grad_bias:", round(gradient_bias, 6)
    )

We do not change the learning.

We only change what is displayed.

epoch starts at 0.

We therefore use epoch + 1 to display a counter starting at 1.

The modulo operator % gives the remainder of a division.

Thus:

2000 % 1000

is 0.

We can therefore display the first ten epochs, then only the epochs that are multiples of 1,000.

The result becomes much easier to read:

Epoch: 1 | loss: 0.4
Epoch: 2 | loss: ...
...
Epoch: 10 | loss: ...
Epoch: 1000 | loss: 0.042614
Epoch: 2000 | loss: 0.042614
...
Epoch: 10000 | loss: 0.04261

We can now let the 10,000 epochs run without turning the terminal into a wall of text.

And the result surprises us.

After 10,000 epochs, the loss remains practically the same:

0.04261

The final parameters are:

Temperature weight: 0.06243206569256493
Humidity weight: -0.02229944125735614
Bias: 0.003899875682740519

And the decisions remain identical.

The model therefore does not move toward a perfect solution simply because we let it work longer.

We could let this program run ten times longer; that would not solve the problem.

We need to look elsewhere.

Let’s look at the scale of our data#

So far we have used the values directly:

temperature: 22 to 29
humidity:    35 to 70

Let’s see what this means for the gradients.

In our formula:

gradient_temperature += 2 × temperature × error
gradient_humidity += 2 × humidity × error

temperature and humidity are used directly in the calculation.

The gradient therefore depends strongly on their scale.

We can bring the data to a smaller range.

Let’s divide both measurements by 100.

Let’s normalize the data#

Normalization means transforming input values so that they are brought to a comparable and smaller scale.

In our case, we simply divide temperature and humidity by 100 to bring them closer to the 0 to 1 interval.

The data becomes:

data = [
    (22 / 100, 70 / 100, 0),
    (24 / 100, 65 / 100, 0),
    (27 / 100, 42 / 100, 1),
    (29 / 100, 35 / 100, 1),
    (25 / 100, 55 / 100, 0),
]

Temperature and humidity are now approximately between 0 and 1.

The rest of the program does not change.

We can keep:

learning_rate = 0.0001
epochs = 10000

This time, the behavior is different.

The loss decreases progressively:

Epoch: 1 | loss: 0.4
Epoch: 10 | loss: 0.399295
Epoch: 100 | loss: 0.392428
Epoch: 1000 | loss: 0.339117
Epoch: 2000 | loss: 0.303096
...
Epoch: 10000 | loss: 0.245249

Normalization therefore changes the scale of the calculations.

But with the same learning rate, training is now much slower.

We changed the scale of the inputs.

We therefore need to adapt the learning rate to this new scale.

Adapt the learning rate to this new scale#

Now use:

learning_rate = 0.001
epochs = 10000

The loss decreases further:

Epoch: 1 | loss: 0.4
Epoch: 10 | loss: 0.393092
Epoch: 100 | loss: 0.33948
Epoch: 1000 | loss: 0.245259
Epoch: 2000 | loss: 0.232627
...
Epoch: 10000 | loss: 0.158705

The final parameters are:

Temperature weight: 0.3567913070729241
Humidity weight: -0.710991373229769
Bias: 0.6721755384720498

The loss is now lower.

Yet our decision rule is still not ideal.

The model still produces:

water
water
water
water
water

We have therefore discovered something important.

Gradient descent works well.

It adjusts the parameters.

The loss decreases.

But this is not enough to guarantee that our model output can be directly interpreted as a decision.

What the model calculates is a score#

Our model currently produces:

score = temperature × weight_temperature
        + humidity × weight_humidity
        + bias

This score is perfectly useful for comparing two observations.

An observation with a higher score is more oriented toward class 1.

An observation with a lower score is more oriented toward class 0.

But we used:

if score > 0:
    decision = "water"
else:
    decision = "do not water"

We therefore added an artificial boundary at 0 ourselves.

The model produces a continuous number.

We then turn it abruptly into two choices.

We are missing a step between the two.

From learning to prediction#

We now have:

temperature
      +
humidity
      ↓
 parameters
      ↓
   score
      ↓
 decision

The problem lies between the score and the decision.

We would like to interpret the model output as a value between 0 and 1.

A value close to 0 could represent a low probability of watering.

A value close to 1 could represent a high probability of watering.

This would preserve progressive information instead of moving directly from an arbitrary score to a binary decision.

How can we transform our score into an output between 0 and 1?

That is what we will look at next.

The answer will lead us to a function that is very simple in appearance, but particularly important in machine learning: the sigmoid.