Programming a Perceptron in Python

Background Information

A perceptron classifier is a simple model of a neuron. It has different inputs ($x_1$...$x_n$) with different weights ($w_1$...$w_n$).

The weighted sum $s$ of these inputs is then passed through a step function $f$ (usually a Heaviside step function).

To make things cleaner, here's a little diagram:

Python!

Here's a simple version of such a perceptron using Python and NumPy. It will take two inputs and learn to act like the logical OR function. First, let's import some libraries we need:

from random import choice
from numpy import array, dot, random

Then let's create the step function. In reference to Mathematica, I'll call this function unit_step.

unit_step = lambda x: 0 if x < 0 else 1

Next we need to map the possible input to the expected output. The first two entries of the NumPy array in each tuple are the two input values. The second element of the tuple is the expected result. And the third entry of the array is a "dummy" input (also called the bias) which is needed to move the threshold (also known as the decision boundary) up or down as needed by the step function. Its value is always 1, so that its influence on the result can be controlled by its weight.

training_data = [
    (array([0,0,1]), 0),
    (array([0,1,1]), 1),
    (array([1,0,1]), 1),
    (array([1,1,1]), 1),
]

As you can see, this training sequence maps exactly to the definition of the OR function:

Next we'll choose three random numbers between 0 and 1 as the initial weights:

w = random.rand(3)

Now on to some variable initializations. The errors list is only used to store the error values so that they can be plotted later on. If you don't want to do any plotting, you can just leave it away. The eta variable controls the learning rate. And n specifies the number of learning iterations.

errors = []
eta = 0.2
n = 100

In order to find the ideal values for the weights w, we try to reduce the error magnitude to zero. In this simple case n = 100 iterations are enough; for a bigger and possibly "noisier" set of input data much larger numbers should be used.

First we get a random input set from the training data. Then we calculate the dot product (sometimes also called scalar product or inner product) of the input and weight vectors. This is our (scalar) result, which we can compare to the expected value. If the expected value is bigger, we need to increase the weights, if it's smaller, we need to decrease them. This correction factor is calculated in the last line, where the error is multiplied with the learning rate (eta) and the input vector (x). It is then added to the weights vector, in order to improve the results in the next iteration.

for i in xrange(n):
    x, expected = choice(training_data)
    result = dot(w, x)
    error = expected - unit_step(result)
    errors.append(error)
    w += eta * error * x

And that's already everything we need in order to train the perceptron! It has now "learned" to act like a logical OR function:

for x, _ in training_data:
    result = dot(x, w)
    print("{}: {} -> {}".format(x[:2], result, unit_step(result)))

[0 0]: -0.0714566687173 -> 0
[0 1]: 0.829739696273 -> 1
[1 0]: 0.345454042997 -> 1
[1 1]: 1.24665040799 -> 1

If you're interested, you can also plot the errors, which is a great way to visualize the learning process:

from pylab import plot, ylim
ylim([-1,1])
plot(errors)

It's easy to see that the errors stabilize around the 60th iteration. If you doubt that the errors are definitely eliminated, you can re-run the training with an iteration count of 500 or more and plot the errors:

You could also try to change the training sequence in order to model an AND, NOR or NOT function. Note that it's not possible to model an XOR function using a single perceptron like this, because the two classes (0 and 1) of an XOR function are not linearly separable. In that case you would have to use multiple layers of perceptrons (which is basically a small neural network).

Wrap Up

Here's the entire code:

from random import choice
from numpy import array, dot, random

unit_step = lambda x: 0 if x < 0 else 1

training_data = [
    (array([0,0,1]), 0),
    (array([0,1,1]), 1),
    (array([1,0,1]), 1),
    (array([1,1,1]), 1),
]

w = random.rand(3)
errors = []
eta = 0.2
n = 100

for i in xrange(n):
    x, expected = choice(training_data)
    result = dot(w, x)
    error = expected - unit_step(result)
    errors.append(error)
    w += eta * error * x

for x, _ in training_data:
    result = dot(x, w)
    print("{}: {} -> {}".format(x[:2], result, unit_step(result)))

If you have any questions, or if you've discovered an error (which is easily possible as I've just learned about this stuff), feel free to leave a comment below.