Several Neurons, Why?#
In the previous lesson, we built our first artificial neuron.
Temperature and humidity enter our model. A few weights, a bias and a sigmoid later, we get an output between 0 and 1.
We even taught it how to correct its parameters.
And it works.
Across our five observations, every result ends up on the correct side of 0.5.
What if we pushed our neuron to its limits?#
Can we give it a watering rule that still makes perfect sense to us, but sits right at the edge of what it can learn?
A simple extra constraint: would you water when it is extremely hot in full sun, with the ground cracked and dry?
In our experimental scenario, it is better to wait until the temperature drops.
We would therefore have something like:
| Temperature | Humidity | Watering |
|---|---|---|
| 22 °C | 40 % | no |
| 28 °C | 40 % | yes |
| 36 °C | 40 % | no |
Same humidity.
Temperature rises.
And yet we ask our neuron for:
no → yes → no
Can it learn this rule?
Let’s not pull our snake out of our pocket just yet…
Let’s try.
We reuse the neuron from the previous lesson and change only its training data:
training_data = [
([22, 40], 0),
([28, 40], 1),
([36, 40], 0),
]
After ten iterations:
Initial loss: 1.4647299641116458
Iteration: 10 Loss: 1.0883803995070394
=== Predictions after training ===
[22, 40] target = 0 output = 0.224012789428278
[28, 40] target = 1 output = 0.7134254811096002
[36, 40] target = 0 output = 0.9777902202427355
The first two cases move in the right direction.
The third one, much less so.
At 36 °C, while we ask for 0, our neuron answers almost 1.
Maybe it simply needs more time.
Let’s move to 1000 iterations.
Initial loss: 1.4647299641116458
Iteration: 100 Loss: 1.087317171813268
Iteration: 500 Loss: 1.0764500144047928
Iteration: 1000 Loss: 1.0558732819682475
=== Predictions after training ===
[22, 40] target = 0 output = 0.1732503426654294
[28, 40] target = 1 output = 0.8173583777871752
[36, 40] target = 0 output = 0.9962427608620434
No.
It insists even more.
The higher the temperature, the higher its output.
Why?
With humidity always equal to 40, its calculation eventually reduces to something like:
score = temperature × weight + constant
If the weight is positive, the score rises with temperature.
If it is negative, it falls.
But our neuron cannot do:
go down → go up → go down again
We have just hit a limitation of our single neuron.
What if we shared the work?#
Let’s try another approach.
One neuron could answer:
“Is it warm enough?”
And another:
“Is it not too hot?”
At 22 °C, the first one should answer no.
At 28 °C, both should answer yes.
At 36 °C, the second one should answer no.
To test the idea, let’s start by giving them their roles ourselves:
def neuron_min_temperature(temperature):
score = temperature - 25
return sigmoid(score)
def neuron_max_temperature(temperature):
score = 32 - temperature
return sigmoid(score)
Result:
22 min = 0.04742587317756678 max = 0.9999546021312976
28 min = 0.9525741268224334 max = 0.9820137900379085
36 min = 0.999983298578152 max = 0.01798620996209156
At 22 °C, the first neuron blocks.
At 28 °C, both are strongly activated.
At 36 °C, the second one blocks.
Let’s combine their answers:
def watering_decision(minimum, maximum):
return minimum * maximum
For each temperature, we compute both neuron outputs and then their combined decision:
minimum = neuron_min_temperature(temperature)
maximum = neuron_max_temperature(temperature)
decision = watering_decision(
minimum,
maximum
)
Result:
22 min = 0.04742587317756678 max = 0.9999546021312976 watering = 0.04742372014400317
28 min = 0.9525741268224334 max = 0.9820137900379085 watering = 0.935440928572949
36 min = 0.999983298578152 max = 0.01798620996209156 watering = 0.017985909566811537
With 0.5 as the threshold:
22 °C → no
28 °C → yes
36 °C → no
It works.
Except that we cheated a little.
We wrote ourselves:
score = temperature - 25
and:
score = 32 - temperature
So our two neurons learned nothing.
We whispered the answer to them.
Let them learn#
This brings us back to what we saw previously: let the neuron learn its parameters instead of choosing them ourselves.
Let’s start with the neuron that must recognize “warm enough”.
We give it:
training_data = [
(22, 0),
(28, 1),
(36, 1),
]
And most importantly:
weight = 0.0
bias = 0.0
This time, there is no hidden 25 in our formula.
After 1000 iterations:
22 target = 0 output = 0.5224750647105219
28 target = 1 output = 0.583933496214407
36 target = 1 output = 0.6617084149920676
It is moving in the right direction, but 22 °C is still just above 0.5.
Let’s continue.
After 10000 iterations:
Weight: 0.2444448774407787
Bias : -5.9646538878523465
22 target = 0 output = 0.357354127926033
28 target = 1 output = 0.7067813299744308
36 target = 1 output = 0.9445570592542989
This time:
no → yes → yes
Our first neuron has learned its role.
Its switching point sits around 24.4 °C.
We did not give it that temperature: it comes from the parameters it learned.
Now the second neuron:
training_data = [
(22, 1),
(28, 1),
(36, 0),
]
We now want:
yes → yes → no
After 10000 iterations, something bothers us:
Iteration: 3000 Loss: 0.49476354205827655
Iteration: 5000 Loss: 0.8418379912683794
Iteration: 7000 Loss: 0.31766551719763914
Iteration: 8000 Loss: 0.6487793462315034
Iteration: 10000 Loss: 0.43142388969196016
It goes down.
It goes back up.
Then down again.
Our learning process is doing a bit of a yo-yo.
We are using:
learning_rate = 0.01
Let’s reduce it:
learning_rate = 0.001
The oscillations disappear.
But after 10000 iterations:
22 target = 1 output = 0.6295212385672573
28 target = 1 output = 0.585818945176154
36 target = 0 output = 0.5255027722192142
It is more stable, but slower.
At 36 °C, we are still on the wrong side.
Let’s push it to 100000 iterations.
Iteration: 10000 Loss: 0.5849536224578121
Iteration: 50000 Loss: 0.30110131056896317
Iteration: 100000 Loss: 0.1871744257344013
Weight: -0.21687684001056218
Bias : 6.880062347894262
22 target = 1 output = 0.891752839001062
28 target = 1 output = 0.6915788232571953
36 target = 0 output = 0.28343139488534946
This time:
yes → yes → no
The second neuron has learned its role too.
Its switching point sits around 31.7 °C.
Let’s remember our trio:
- the weight determines how an input influences the neuron: in which direction and how strongly;
- the bias shifts the point from which the neuron starts to activate;
- the
learning_ratedetermines the size of the corrections made during learning.
In short: the weight tilts, the bias shifts, the learning_rate controls the step size.
If the steps are too large, learning can oscillate. If they are too small, it moves forward… but takes its time.
We have just seen exactly that.
Bring our two neurons together#
We now have:
neuron 1: "warm enough?"
22 → no
28 → yes
36 → yes
neuron 2: "not too hot?"
22 → yes
28 → yes
36 → no
Let’s reuse our multiplication:
watering = minimum * maximum
Result:
22 min = 0.3608567127375629 max = 0.891752839001062 watering = 0.3217949980563124
28 min = 0.7049013197602678 max = 0.6915788232571953 watering = 0.48749482523224996
36 min = 0.9423475981684872 max = 0.28343139488534946 watering = 0.2670908942157531
Ah.
At 28 °C, each neuron answers yes:
0.705 > 0.5
0.692 > 0.5
But:
0.705 × 0.692 ≈ 0.487
We drop below 0.5 again.
no → no → no
Having several neurons is therefore not enough.
We still need to know what to do with their answers.
What if we let another neuron learn that too?
A neuron that listens to the others#
Let’s add a third neuron.
This time, its inputs will be neither temperature nor humidity.
They will be the outputs of our first two neurons:
neuron 1 ──┐
temperature ──── ├──> output neuron ──> watering
neuron 2 ──┘
Its training data becomes:
[0.3608567127375629, 0.891752839001062] → 0
[0.7049013197602678, 0.6915788232571953] → 1
[0.9423475981684872, 0.28343139488534946] → 0
Again, we initialize its weights and bias to zero.
Before learning:
Loss: 0.75
22 → 0.5
28 → 0.5
36 → 0.5
Normal.
With zero weights and a zero bias, its score is zero.
And sigmoid(0) = 0.5.
Let’s let it learn.
After 100000 iterations:
22 → 0.3627140055495805
28 → 0.382212386991098
36 → 0.32137259020941683
Still:
no → no → no
But something has changed.
The highest value is now the one for 28 °C.
So our neuron is beginning to distinguish the case we care about.
Let’s continue.
After 500000 iterations:
Weight min: 5.740437152115647
Weight max: 5.7586645184847685
Bias : -7.962631163158695
And more importantly:
22 min = 0.3608567127375629 max = 0.891752839001062 target = 0 output = 0.3195478557091965
28 min = 0.7049013197602678 max = 0.6915788232571953 target = 1 output = 0.5165891570076458
36 min = 0.9423475981684872 max = 0.28343139488534946 target = 0 output = 0.28476279992382514
There we are.
22 °C → no
28 °C → yes
36 °C → no
What one neuron could not learn, several neurons have just learned together.
Well.
It still took 500000 iterations for our output neuron to finally cross 0.5 in the right place.
That may sound huge for our small Python program. At the scale of modern AI model training, it is tiny.
Our Arduino UNO Q is not training ChatGPT. :)
But those 500000 small steps already give us a very concrete idea of what it means to learn by repeating small corrections again and again.
The program that got us there#
Since this small network eventually did what we asked, let’s keep the complete program.
# ==========================================
# Lesson 06
# Two neurons learn their roles
# then an output neuron learns to combine them
# ==========================================
import math
training_data_min = [
(22, 0),
(28, 1),
(36, 1),
]
training_data_max = [
(22, 1),
(28, 1),
(36, 0),
]
min_weight = 0.0
min_bias = 0.0
max_weight = 0.0
max_bias = 0.0
step = 0.001
learning_rate = 0.001
def sigmoid(x):
return 1 / (1 + math.exp(-x))
def neuron(temperature, weight, bias):
score = temperature * weight + bias
return sigmoid(score)
def compute_total_loss(training_data, weight, bias):
total = 0
for temperature, target in training_data:
output = neuron(
temperature,
weight,
bias
)
error = target - output
total += error ** 2
return total
def train_neuron(training_data, weight, bias, iterations):
for iteration in range(iterations):
original_weight = weight
loss_before = compute_total_loss(
training_data,
original_weight - step,
bias
)
loss_after = compute_total_loss(
training_data,
original_weight + step,
bias
)
gradient_weight = (
loss_after - loss_before
) / (2 * step)
original_bias = bias
loss_before = compute_total_loss(
training_data,
weight,
original_bias - step
)
loss_after = compute_total_loss(
training_data,
weight,
original_bias + step
)
gradient_bias = (
loss_after - loss_before
) / (2 * step)
weight -= learning_rate * gradient_weight
bias -= learning_rate * gradient_bias
return weight, bias
print("=== Training minimum temperature neuron ===")
min_weight, min_bias = train_neuron(
training_data_min,
min_weight,
min_bias,
100000
)
print("Weight:", min_weight)
print("Bias :", min_bias)
print()
print("=== Training maximum temperature neuron ===")
max_weight, max_bias = train_neuron(
training_data_max,
max_weight,
max_bias,
100000
)
print("Weight:", max_weight)
print("Bias :", max_bias)
# ==========================================
# Build training data for the output neuron
# ==========================================
output_training_data = []
for temperature, target in [
(22, 0),
(28, 1),
(36, 0),
]:
minimum = neuron(
temperature,
min_weight,
min_bias
)
maximum = neuron(
temperature,
max_weight,
max_bias
)
output_training_data.append(
([minimum, maximum], target)
)
# ==========================================
# Output neuron
# ==========================================
output_weight_min = 0.0
output_weight_max = 0.0
output_bias = 0.0
def output_neuron(minimum, maximum):
score = (
minimum * output_weight_min
+ maximum * output_weight_max
+ output_bias
)
return sigmoid(score)
def compute_output_loss():
total = 0
for features, target in output_training_data:
minimum = features[0]
maximum = features[1]
output = output_neuron(
minimum,
maximum
)
error = target - output
total += error ** 2
return total
def train_output_neuron():
global output_weight_min
global output_weight_max
global output_bias
original_weight_min = output_weight_min
output_weight_min = original_weight_min - step
loss_before = compute_output_loss()
output_weight_min = original_weight_min + step
loss_after = compute_output_loss()
gradient_weight_min = (
loss_after - loss_before
) / (2 * step)
output_weight_min = original_weight_min
original_weight_max = output_weight_max
output_weight_max = original_weight_max - step
loss_before = compute_output_loss()
output_weight_max = original_weight_max + step
loss_after = compute_output_loss()
gradient_weight_max = (
loss_after - loss_before
) / (2 * step)
output_weight_max = original_weight_max
original_bias = output_bias
output_bias = original_bias - step
loss_before = compute_output_loss()
output_bias = original_bias + step
loss_after = compute_output_loss()
gradient_bias = (
loss_after - loss_before
) / (2 * step)
output_bias = original_bias
output_weight_min -= (
learning_rate * gradient_weight_min
)
output_weight_max -= (
learning_rate * gradient_weight_max
)
output_bias -= (
learning_rate * gradient_bias
)
print()
print("=== Output neuron training data ===")
for features, target in output_training_data:
print(
features,
"target =", target
)
print()
print("=== Training output neuron ===")
print("Initial loss:", compute_output_loss())
for iteration in range(500000):
train_output_neuron()
if (iteration + 1) % 50000 == 0:
print(
"Iteration:",
iteration + 1,
"Loss:",
compute_output_loss()
)
print()
print("=== Output neuron parameters ===")
print("Weight min:", output_weight_min)
print("Weight max:", output_weight_max)
print("Bias :", output_bias)
print()
print("=== Final predictions ===")
for temperature, target in [
(22, 0),
(28, 1),
(36, 0),
]:
minimum = neuron(
temperature,
min_weight,
min_bias
)
maximum = neuron(
temperature,
max_weight,
max_bias
)
output = output_neuron(
minimum,
maximum
)
print(
temperature,
"min =", minimum,
"max =", maximum,
"target =", target,
"output =", output
)
With our experiment:
22 min = 0.3608567127375629 max = 0.891752839001062 target = 0 output = 0.3195478557091965
28 min = 0.7049013197602678 max = 0.6915788232571953 target = 1 output = 0.5165891570076458
36 min = 0.9423475981684872 max = 0.28343139488534946 target = 0 output = 0.28476279992382514
That is our first network.
But before going further, let’s look at the path we took.
We started with a very simple question:
Why would we need several neurons?
This time, we have an answer.
So, why several neurons?#
We started with a single neuron.
As long as the decision moved in one direction, it handled things fairly well.
But then we asked for:
no → yes → no
And that is where it got stuck.
We split the problem between two neurons: one to recognize “warm enough”, the other “not too hot”.
It works when we assign their roles ourselves.
So we let them learn.
The first one learned its boundary.
The second one did too — not without reminding us that a learning_rate that is too large can make learning swing around.
Then we still had to combine their answers.
Our first idea, multiplying them, gave us:
0.705 × 0.692 ≈ 0.487
Just missed.
So instead of deciding ourselves how to combine them, we gave that job to a third neuron.
And after learning:
22 °C → no
28 °C → yes
36 °C → no
We have our answer.
One neuron can learn a simple decision.
Several neurons can split the problem, then learn how to combine what they have understood.
Almost without noticing it, we have just built our first small neural network.
There is still something we did for it, though.
We decided the role of the first two neurons and trained them separately.
Could a network learn all of this together, without us telling it the role of each neuron?