Overview

A simple, illustrative implementation of a single-layered perceptron in Java. When a pattern is impressed on the perceptron the activation of the network is adjusted according to an activation formula and a given bias value. To adjust the way the perceptron reacts to a given input, a learning algorithm (the delta rule) is implemented to adjust the weights connecting the neurons of the perceptron, which are initially set to 0. This rule keeps adjusting the weights until the resulting output for a given input corresponds to a supplied correct output, resulting in a perceptron trained to react to a certain perception in a certain way. The input patterns and the teaching output are hard coded as integer matrices. [edit]

Implementation

The patterns to be impressed on the input layer of the perceptron:

<<patterns>>= int[][] patterns { 0, 0, 0, 0 { 0, 0, 0, 1 { 0, 0, 1, 0 { 0, 0, 1, 1 { 0, 1, 0, 0 { 0, 1, 0, 1 { 0, 1, 1, 0 { 0, 1, 1, 1 { 1, 0, 0, 0 { 1, 0, 0, 1 = { }, }, }, }, }, }, }, }, }, } };

The teaching output for the perceptron, the desired reaction or learning goal:

<<teaching>>= int[][] teachingOutput = { { 0, 0, 0, 0, 0, 0, 0, { 1, 0, 0, 0, 0, 0, 0, { 1, 1, 0, 0, 0, 0, 0, { 1, 1, 1, 0, 0, 0, 0, { 1, 1, 1, 1, 0, 0, 0, { 1, 1, 1, 1, 1, 0, 0, { 1, 1, 1, 1, 1, 1, 0, { 1, 1, 1, 1, 1, 1, 1, { 1, 1, 1, 1, 1, 1, 1, { 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0 0 0 0 0 0 0 0 0 0 }, }, }, }, }, }, }, }, }, } };

The number of input and output neurons and the number of patterns depend on these matrices. The weights of the connections between neurons in the perceptron are

represented in a matrix of doubles. The constructor initializes the weights with the given numbers of input and output neurons.
<<attributes>>= int numberOfInputNeurons = patterns[0].length; int numberOfOutputNeurons = teachingOutput[0].length; int numberOfPatterns = patterns.length; double[][] weights; public Perceptron() { weights = new double[numberOfInputNeurons][numberOfOutputNeurons]; }

The delta rule is the learning algorithm for adjusting the weights of the perceptron given an input value, a desired output and an untrained perceptron. The output of the perceptron is compared to the desired output and weights are adjusted if an error is found, where an error is a difference between the actual output and the expected output (the teaching output).
<<delta>>= int[] output = setOutputValues(i); for (int j = 0; j < numberOfOutputNeurons; j++) { if (teachingOutput[i][j] != output[j]) { for (int k = 0; k < numberOfInputNeurons; k++) { weights[k][j] = weights[k][j] + learningFactor * patterns[i][k] * (teachingOutput[i][j] - output[j]); } } } for (int z = 0; z < output.length; z++) { if (output[z] != teachingOutput[i][z]) error = true; }

Impress a pattern on the input layer and set the output layer for the applied pattern. The method takes one parameter: the index of the pattern to apply and returns an array of integers: the resulting output, the reaction. Depending on the result of the activation formula and the bias a neuron is activated or deactivated by setting the value of the corresponding position to 1 or 0.
<<impress>>= for (int j = 0; j < result.length; j++) { double net = weights[0][j] * toImpress[0] + weights[1][j] * toImpress[1] + weights[2][j] * toImpress[2] + weights[3][j] * toImpress[3]; if (net > bias) result[j] = 1; else result[j] = 0;

A util method for printing a matrix of doubles formatting the values to one decimal digit (using DecimalFormat) to get a readable output of weights, representing the connections of neurons in the perceptron.
<<print_matrix>>= for (int i = 0; i < matrix.length; i++) { for (int j = 0; j < matrix[i].length; j++) { NumberFormat f = NumberFormat.getInstance(); if (f instanceof DecimalFormat) { DecimalFormat decimalFormat = ((DecimalFormat) f); decimalFormat.setMaximumFractionDigits(1); decimalFormat.setMinimumFractionDigits(1); System.out.print("(" + f.format(matrix[i][j]) + ")"); } } System.out.println(); }

[edit]

Usage

Demonstration of the usage in a JUnit 4 unit test (requires Java 5).

<<usage>>= Perceptron p = new Perceptron(); System.out.println("Weights before training: "); p.printMatrix(p.weights); p.deltaRule(); System.out.println("Weights after training: "); p.printMatrix(p.weights);

The complete program:

<<Perceptron.java>>= import java.text.DecimalFormat; import java.text.NumberFormat; import org.junit.Test; public class Perceptron { patterns teaching attributes public void deltaRule() { boolean allCorrect = false; boolean error = false;

double learningFactor = 0.2; while (!allCorrect) { error = false; for (int i = 0; i < numberOfPatterns; i++) { delta } if (!error) { allCorrect = true; } }

int[] setOutputValues(int patternNo) { double bias = 0.7; int[] result = new int[numberOfOutputNeurons]; int[] toImpress = patterns[patternNo]; for (int i = 0; i < toImpress.length; i++) { impress } return result; } public void printMatrix(double[][] matrix) { print_matrix } @Test public void testPerceptron() { usage } }

1 /* Copyright (c) 2012 the authors listed at the following URL, and/or 2 the authors of referenced articles or incorporated external code: 3 http://en.literateprograms.org/Perceptron_(Java)? action=history&offset=20080801150718 4 5 Permission is hereby granted, free of charge, to any person obtaining 6 a copy of this software and associated documentation files (the 7 "Software"), to deal in the Software without restriction, including 8 without limitation the rights to use, copy, modify, merge, publish, 9 distribute, sublicense, and/or sell copies of the Software, and to 10 permit persons to whom the Software is furnished to do so, subject to 11 the following conditions: 12 13 The above copyright notice and this permission notice shall be 14 included in all copies or substantial portions of the Software. 15 16 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,

17 EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 18 MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 19 IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY 20 CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 21 TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 22 SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 23 24 Retrieved from: http://en.literateprograms.org/Perceptron_(Java)?oldid=14163 25 */ 26 27 28 import java.text.DecimalFormat; 29 import java.text.NumberFormat; 30 31 import org.junit.Test; 32 33 public class Perceptron { 34 35 int[][] patterns = { 36 { 0, 0, 0, 0 }, 37 { 0, 0, 0, 1 }, 38 { 0, 0, 1, 0 }, 39 { 0, 0, 1, 1 }, 40 { 0, 1, 0, 0 }, 41 { 0, 1, 0, 1 }, 42 { 0, 1, 1, 0 }, 43 { 0, 1, 1, 1 }, 44 { 1, 0, 0, 0 }, 45 { 1, 0, 0, 1 } }; 46 47 48 int[][] teachingOutput = { 49 { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, 50 { 1, 0, 0, 0, 0, 0, 0, 0, 0, 0 }, 51 { 1, 1, 0, 0, 0, 0, 0, 0, 0, 0 }, 52 { 1, 1, 1, 0, 0, 0, 0, 0, 0, 0 }, 53 { 1, 1, 1, 1, 0, 0, 0, 0, 0, 0 }, 54 { 1, 1, 1, 1, 1, 0, 0, 0, 0, 0 },

55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100

{ 1, 1, 1, 1, 1, 1, 0, 0, 0, 0 }, { 1, 1, 1, 1, 1, 1, 1, 0, 0, 0 }, { 1, 1, 1, 1, 1, 1, 1, 1, 0, 0 }, { 1, 1, 1, 1, 1, 1, 1, 1, 1, 0 } }; int numberOfInputNeurons = patterns[0].length; int numberOfOutputNeurons = teachingOutput[0].length; int numberOfPatterns = patterns.length; double[][] weights; public Perceptron() { weights = new double[numberOfInputNeurons][numberOfOutputNeurons]; } public void deltaRule() { boolean allCorrect = false; boolean error = false; double learningFactor = 0.2; while (!allCorrect) { error = false; for (int i = 0; i < numberOfPatterns; i++) { int[] output = setOutputValues(i); for (int j = 0; j < numberOfOutputNeurons; j++) { if (teachingOutput[i][j] != output[j]) { for (int k = 0; k < numberOfInputNeurons; k++) { weights[k][j] = weights[k][j] + learningFactor * patterns[i][k] * (teachingOutput[i][j] - output[j]); } } } for (int z = 0; z < output.length; z++) { if (output[z] != teachingOutput[i][z]) error = true; } } if (!error) { allCorrect = true; } } } int[] setOutputValues(int patternNo) {

101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146

double bias = 0.7; int[] result = new int[numberOfOutputNeurons]; int[] toImpress = patterns[patternNo]; for (int i = 0; i < toImpress.length; i++) { for (int j = 0; j < result.length; j++) { double net = weights[0][j] * toImpress[0] + weights[1][j] * toImpress[1] + weights[2][j] * toImpress[2] + weights[3][j] * toImpress[3]; if (net > bias) result[j] = 1; else result[j] = 0; } } return result; } public void printMatrix(double[][] matrix) { for (int i = 0; i < matrix.length; i++) { for (int j = 0; j < matrix[i].length; j++) { NumberFormat f = NumberFormat.getInstance(); if (f instanceof DecimalFormat) { DecimalFormat decimalFormat = ((DecimalFormat) f); decimalFormat.setMaximumFractionDigits(1); decimalFormat.setMinimumFractionDigits(1); System.out.print("(" + f.format(matrix[i][j]) + ")"); } } System.out.println(); } } @Test public void testPerceptron() { Perceptron p = new Perceptron(); System.out.println("Weights before training: "); p.printMatrix(p.weights); p.deltaRule(); System.out.println("Weights after training: "); p.printMatrix(p.weights);

147 } 148 } 149

package neuralNetwork; /** * A basic abstract perceptron that stores its inputs, weights, and output. It * handles back propagation training. Subclasses must define the activation * function and its derivative and the min and max output and weights. * * @author Amos Yuen * @version 1.05 - 16 August 2008 */ public abstract class Perceptron implements Neuron { protected protected protected protected protected float error; float output; final Neuron[] inputs; final float[] weights; int numErrors;

public Perceptron(Neuron... inputs) { this.inputs = inputs; weights = new float[inputs.length]; float range = getMaxWeight() - getMinWeight(); for (int i = 0; i < weights.length; i++) weights[i] = (float) Math.random() * range + getMinWeight(); } public abstract float activationDerivative(float input); public abstract float activationFunction(float input); @Override public void addError(float error) { this.error += error; numErrors++; } public void applyBackPropagation(float learningFactor) { float input = getInput(); // Compute the difference of the input from // the expected input using linear approximation. float delta = getError() / activationDerivative(input); for (byte i = 0; i < weights.length; i++) { // Recalculate the weight for this input using the percentage this // input contributes to the overall input. float inputError = delta * weights[i] / input;

learningFactor); Multiply by recalculated weight learningFactor)); } }

setWeight(i, weights[i] + inputError * // Add this input's percentage of the error. // (1f - learningFactor) to compensate for the inputs[i].addError(inputError * (1f -

@Override public float fireNeuron() { output = activationFunction(getInput()); return output; } @Override public float getError() { return error / numErrors; } public float getInput() { float input = 0; for (int i = 0; i < inputs.length; i++) input += inputs[i].getOutput() * weights[i]; return input; } public Neuron getInput(int index) { return inputs[index]; } public abstract float getMaxOutput(); public float getMaxWeight() { return 1f; } public abstract float getMinOutput(); public float getMinWeight() { return 0.001f; } public int getNumInputs() { return inputs.length; } @Override public float getOutput() { return output; } public float getWeight(int index) { return weights[index];

} @Override public void resetError() { error = 0; numErrors = 0; } public void setWeight(int index, float weight) { // Check to prevent setting NaN values if (weight > 0) weights[index] = Math.max(getMinWeight(), Math.min(getMaxWeight(), weight)); } }

Introduction
Perceptron is the simplest type of feed forward neural network. It was designed by Frank Rosenblatt as dichotomic classifier of two classes which are linearly separable. This means that the type of problems the network can solve must be linearly separable. Basic perceptron consists of 3 layers:

Sensor layer Associative layer Output neuron

There are a number of inputs (xn) in sensor layer, weights (wn) and an output. Sometimes w0 is called bias and x0 = +1/-1 (In this case is x0=-1).

For every input on the perceptron (including bias), there is a corresponding weight. To calculate the output of the perceptron, every input is multiplied by its corresponding weight. Then weighted sum is computed of all inputs and fed through a limiter function that evaluates the final output of the perceptron. The output of neuron is formed by activation of the output neuron, which is function of input:

(1)

The activation function F can be linear so that we have a linear network, or nonlinear. In this example, I decided to use threshold (signum) function:

(2)

Output of network in this case is either +1 or -1 depending on the input. If the total input (weighted sum of all inputs) is positive, then the pattern belongs to class +1, otherwise to class -1. Because of this behavior, we can use perceptron for classification tasks. Let's consider we have a perceptron with 2 inputs and we want to separate input patterns into 2 classes. In this case, the separation between the classes is straight line, given by equation: (3) When we set x0=-1 and mark w0=?, then we can rewrite equation (3) into form: (4) Here I will describe the learning method for perceptron. Learning method of perceptron is an iterative procedure that adjust the weights. A learning sample is presented to the network. For each weight, the new value is computed by adding a correction to the old value. The threshold is updated in the same way:

(5)

where y is output of perceptron, d is desired output and ? is the learning parameter.

Using the Program

When you run the program, you see area where you can input samples. Clicking by left button on this area, you will add first class sample (blue cross). Clicking by right button on this area, you will add first class sample (red cross). Samples are added to the samples list. You can also set learning rate and number of iterations. When you have set all these values, you can click on Learn button to start learning.

Using the Code

All samples are stored in generic list samples which holds only Sample class objects. Collapse | Copy Code
public class Sample { double x1; double x2; double cls; public Sample(double x1, double x2, int cls) { this.x1 = x1; this.x2 = x2; this.cls = cls; } public double X1 { get { return x1; } set { this.x1 = value; } } public double X2 { get { return x2; } set { this.x2 = value; } } public double Class { get { return cls; } set { this.cls = value; } } }

Before running a learning of perceptron is important to set learning rate and number of iterations. Perceptron has one great property. If solution exists, perceptron always find it but problem occurs, when solution does not exist. In this case, perceptron will try to find the solution in infinity loop and to avoid this, it is better to set maximum number of iterations.

The next step is to assign random values for weights (w0, w1 and w2). Collapse | Copy Code
Random rnd = new Random(); w0 = rnd.NextDouble(); w1 = rnd.NextDouble(); w2 = rnd.NextDouble();

When random values are assigned to weights, we can loop through samples and compute output for every sample and compare it with desired output.
double x1 = samples[i].X1; double x2 = samples[i].X2; int y; if (((w1 * x1) + (w2 * x2) - w0) < 0) { y = -1; } else { y = 1; }

Collapse | Copy Code

I decided to set x0=-1 and for this reason, the output of perceptron is given by equation: y=w1*w1+w2*w2-w0. When perceptron output and desired output doesnt match, we must compute new weights:
if (y != samples[i].Class) { error = true; w0 = w0 + alpha * (samples[i].Class - y) * x0 / 2; w1 = w1 + alpha * (samples[i].Class - y) * x1 / 2; w2 = w2 + alpha * (samples[i].Class - y) * x2 / 2; }

Collapse | Copy Code

Y is output of perceptron and samples[i].Class is desired output. The last 2 steps (looping through samples and computing new weights), we must repeat while the error variable is <> 0 and current number of iterations (iterations) is less than maxIterations.
int i; int iterations = 0; bool error = true; maxIterations = int.Parse(txtIterations.Text);

Collapse | Copy Code

Random rnd = new Random(); w0 = rnd.NextDouble(); w1 = rnd.NextDouble(); w2 = rnd.NextDouble(); alpha = (double)trackLearningRate.Value / 1000; while (error && iterations < maxIterations) { error = false; for (i = 0; i <= samples.Count - 1; i++) { double x1 = samples[i].X1; double x2 = samples[i].X2; int y; if (((w1 * x1) + (w2 * x2) - w0) < 0) { y = -1; } else { y = 1; } if (y != samples[i].Class) { error = true; w0 = w0 + alpha * (samples[i].Class - y) * x0 / 2; w1 = w1 + alpha * (samples[i].Class - y) * x1 / 2; w2 = w2 + alpha * (samples[i].Class - y) * x2 / 2; } } objGraphics.Clear(Color.White); DrawSeparationLine(); iterations++;

Function DrawSeparationLine draws separation line of 2 classes.