A first model that learns its weights#

In the previous lesson, we started with a few Python variables and gradually arrived at data associated with an expected result.

We had:

• temperature
• humidity
• data
• yes / no

We then calculated a threshold from the temperatures. Let’s now use both features.

1. Our training data#

Let’s reuse the same observations:

data = [
    ([22, 70], 0),
    ([24, 65], 0),
    ([27, 42], 1),
    ([29, 35], 1),
    ([25, 55], 0),
]

We have simply replaced no with 0 and yes with 1.

Our dataset therefore means:

• [22, 70] → 0
• [24, 65] → 0
• [27, 42] → 1
• [29, 35] → 1
• [25, 55] → 0

We can now talk about classification:

• 0 → do not water
• 1 → water

2. The model#

The model has two weights and a bias:

temperature_weight = 0.0
humidity_weight = 0.0
bias = 0.0

The calculation is:

Then:

• score >= 0 → 1
• score < 0 → 0

def prediction(temperature, humidity):
    score = (
        temperature * temperature_weight
        + humidity * humidity_weight
        + bias
    )

    if score >= 0:
        return 1
    else:
        return 0

3. At first, the weights are zero#

We start with:

• temperature weight = 0
• humidity weight = 0
• bias = 0

The score therefore starts at 0 for every observation.

We now give the model an opportunity to learn.

4. How do we learn?#

Let’s go through the data several times. Each pass is called an epoch.

For each observation, the model:

• calculates a prediction;
• compares the prediction with the expected answer;
• calculates an error;
• modifies its parameters when necessary.

In our experiment:

error = label - result

A correct prediction gives 0. An error gives a different value and triggers an adjustment.

5. Updating the weights#

temperature_weight += (
    learning_rate
    * error
    * temperature
)

humidity_weight += (
    learning_rate
    * error
    * humidity
)

bias += learning_rate * error

Our learning rate is:

learning_rate = 0.01

The model gradually modifies its parameters according to the examples it encounters.

6. The complete program#

Then its direct translation into Python:

# ==========================================
# Lesson 2.2
# First model that learns its weights
# ==========================================

data = [
    ([22, 70], 0),
    ([24, 65], 0),
    ([27, 42], 1),
    ([29, 35], 1),
    ([25, 55], 0),
]

temperature_weight = 0.0
humidity_weight = 0.0
bias = 0.0
learning_rate = 0.01


def prediction(temperature, humidity):
    score = (
        temperature * temperature_weight
        + humidity * humidity_weight
        + bias
    )

    if score >= 0:
        return 1
    else:
        return 0


for epoch in range(100):
    errors = 0

    for features, label in data:
        temperature = features[0]
        humidity = features[1]

        result = prediction(temperature, humidity)
        error = label - result

        if error != 0:
            errors += 1

            temperature_weight += (
                learning_rate
                * error
                * temperature
            )

            humidity_weight += (
                learning_rate
                * error
                * humidity
            )

            bias += learning_rate * error

    print(
        "Epoch:", epoch + 1,
        "| errors:", errors
    )

    if errors == 0:
        break


print()
print("=== Learned model ===")
print("Temperature weight:", temperature_weight)
print("Humidity weight   :", humidity_weight)
print("Bias              :", bias)

temperature = 28
humidity = 40
result = prediction(temperature, humidity)

print()
print("=== New observation ===")
print("Temperature:", temperature, "°C")
print("Humidity   :", humidity, "%")

if result == 1:
    print("Prediction: water")
else:
    print("Prediction: do not water")

7. The model learns something#

The observed experiment gives:

• Epoch 1 → 4 errors
• Epoch 2 → 2 errors
• Epoch 3 → 1 error
• Epoch 4 → 0 errors

The parameters obtained are:

• temperature weight: 0.38
• humidity weight: -0.19
• bias: 0.01

Our model therefore calculates:

8. What do the weights mean?#

• The temperature weight is positive: +0.38. A higher temperature increases the score.
• The humidity weight is negative: -0.19. A higher humidity decreases the score.
• These relationships come from the examples provided to the model.

The model modified its parameters to reduce its errors. This is where we really begin to see the difference between a programmed rule and a learned model.

9. A new observation#

Let’s test:

• temperature = 28 °C
• humidity = 40 %

The model calculates:

Since 3.05 >= 0, the model predicts water.

With 22 °C and 70 %:

The model predicts do not water.

10. We now have a decision boundary#

Our model separates the two categories according to:

• score >= 0 → water
• score < 0 → do not water

The boundary is:

It depends on three learned parameters:

• temperature weight;
• humidity weight;
• bias.

11. But how does the model know what to change?#

We used a simple error rule:

Then we directly modified the weights.

This is an excellent way to discover the principle. Let’s now look for a more general way to measure error: a loss function.

The question then becomes:

How do we know in which direction to change a weight to make this loss decrease?

This is where gradient descent appears.

To understand this mechanism without immediately getting lost in the two weights of our watering model, let’s return to a simpler problem:

The next step is to understand how a machine can gradually find:

from a few examples.