Neural Network Part 3: Perceptrons

I went back to my code for building a Perceptron and I made some changes.  I realized that although McCaffrey combines the code together, there are actually two actions for the perceptron: training and predicting. I created a diagram to help me keep the functions that I need for each in mind:

image

I also skeletoned out some data structures that I think I need:

image

With the base diagrams out of the way, I created different data structures that were tailored to each action.   These are a bit different than the diagrams –> I didn’t go back and update the diagrams because the code is where you would look to see how the system works:

type observation = {xValues:float List} type weightedObservation = {xws:(float*float) List} type confirmedObservation = {observation:observation;yExpected:float} type weightedConfirmedObservation = {weightedObservation:weightedObservation;yExpected:float} type neuronInput = {weightedObservation:weightedObservation;bias:float} type cycleTrainingInput = {weightedConfirmedObservation:weightedConfirmedObservation;bias:float;alpha:float} type adjustmentInput = {weightedConfirmedObservation:weightedConfirmedObservation;bias:float;alpha:float;yActual:float} type adjustmentOutput = {weights:float List; bias:float} type rotationTrainingInput = {confirmedObservations:confirmedObservation List;weights:float List;bias:float;alpha:float} type trainInput = {confirmedObservations:confirmedObservation List; weightSeedValue:float;biasSeedValue:float;alpha:float; maxEpoches:int} type cyclePredictionInput = {weightedObservation:weightedObservation;bias:float} type rotationPredictionInput = {observations:observation List;weights:float List;bias:float} type predictInput = {observations:observation List;weights:float List;bias:float}

Note that I am composing data structures with the base being an observation.  The observation is a list of different xValues for a given, well, observation.  The weighted observation is the XValue paired with the perceptron weights.  The confirmedObservation is for training –> given an observation, what was the actual output? 

With the data structures out of the way, I went to the Perceptron and added in the basic functions for creating seed values:

member this.initializeWeights(xValues, randomSeedValue) = let lo = -0.01 let hi = 0.01 let xWeight = (hi-lo) * randomSeedValue + lo xValues |> Seq.map(fun w -> xWeight) member this.initializeBias(randomSeedValue) = let lo = -0.01 let hi = 0.01 (hi-lo) * randomSeedValue + lo

Since I was doing TDD, here are the unit tests I used for these functions:

[TestMethod] public void initializeWeightsUsingHalfSeedValue_ReturnsExpected() { var weights = _perceptron.initializeWeights(_observation.xValues, .5); var weightsList = new List<double>(weights); var expected = 0.0; var actual = weightsList[0]; Assert.AreEqual(expected, actual); } [TestMethod] public void initializeWeightsUsingLessThanHalfSeedValue_ReturnsExpected() { var weights = _perceptron.initializeWeights(_observation.xValues, .4699021627); var weightsList = new List<double>(weights); var expected = -0.00060; var actual = Math.Round(weightsList[0],5); Assert.AreEqual(expected, actual); } [TestMethod] public void initializeBiasesUsingHalfSeedValue_ReturnsExpected() { var expected = 0.0; var actual = _perceptron.initializeBias(.5); Assert.AreEqual(expected, actual); } [TestMethod] public void initializeBiasesUsingLessThanHalfSeedValue_ReturnsExpected() { var expected = -0.00060; var bias = _perceptron.initializeBias(.4699021627); var actual = Math.Round(bias, 5); Assert.AreEqual(expected, actual); } [TestMethod] public void initializeBiasesUsingGreaterThanHalfSeedValue_ReturnsExpected() { var expected = 0.00364; var bias = _perceptron.initializeBias(.6820621978); var actual = Math.Round(bias,5); Assert.AreEqual(expected, actual); }

I then created a base neuron and activation function that would work for both training and predicting:

member this.runNeuron(input:neuronInput) = let xws = input.weightedObservation.xws let output = xws |> Seq.map(fun (xValue,xWeight) -> xValue*xWeight) |> Seq.sumBy(fun x -> x) output + input.bias member this.runActivation(input) = if input < 0.0 then -1.0 else 1.0

[TestMethod] public void runNeuronUsingNormalInput_ReturnsExpected() { var expected = -0.0219; var perceptronOutput = _perceptron.runNeuron(_neuronInput); var actual = Math.Round(perceptronOutput, 4); Assert.AreEqual(expected, actual); } [TestMethod] public void runActivationUsingNormalInput_ReturnsExpected() { var expected = -1; var actual = _perceptron.runActivation(-0.0219); Assert.AreEqual(expected, actual); }

I then created the functions for training –> specifically to return adjusted weights and biases based on the result of the activation  function

member this.calculateWeightAdjustment(xValue, xWeight, alpha, delta) = match delta > 0.0, xValue >= 0.0 with | true,true -> xWeight - (alpha * abs(delta) * xValue) | false,true -> xWeight + (alpha * abs(delta) * xValue) | true,false -> xWeight - (alpha * abs(delta) * xValue) | false,false -> xWeight + (alpha * abs(delta) * xValue) member this.calculateBiasAdjustment(bias, alpha, delta) = match delta > 0.0 with | true -> bias - (alpha * abs(delta)) | false -> bias + (alpha * abs(delta)) member this.runAdjustment (input:adjustmentInput) = match input.weightedConfirmedObservation.yExpected = input.yActual with | true -> let weights = input.weightedConfirmedObservation.weightedObservation.xws |> Seq.map(fun (x,w) -> w) let weights' = new List<float>(weights) {adjustmentOutput.weights=weights';adjustmentOutput.bias=input.bias} | false -> let delta = input.yActual - input.weightedConfirmedObservation.yExpected let weights' = input.weightedConfirmedObservation.weightedObservation.xws |> Seq.map(fun (xValue, xWeight) -> this.calculateWeightAdjustment(xValue,xWeight,input.alpha,delta)) |> Seq.toList let weights'' = new List<float>(weights') let bias' = this.calculateBiasAdjustment(input.bias,input.alpha,delta) {adjustmentOutput.weights=weights'';adjustmentOutput.bias=bias'}

[TestMethod] public void calculateWeightAdjustmentUsingPositiveDelta_ReturnsExpected() { var xValue = 1.5; var xWeight = .00060; var delta = 2; var weightAdjustment = _perceptron.calculateWeightAdjustment(xValue, xWeight, _alpha, delta); var actual = Math.Round(weightAdjustment, 4); var expected = -.0024; Assert.AreEqual(expected, actual); } [TestMethod] public void calculateWeightAdjustmentUsingNegativeDelta_ReturnsExpected() { var xValue = 1.5; var xWeight = .00060; var delta = -2; var weightAdjustment = _perceptron.calculateWeightAdjustment(xValue, xWeight, _alpha, delta); var actual = Math.Round(weightAdjustment, 5); var expected = .0036; Assert.AreEqual(expected, actual); } [TestMethod] public void calculateBiasAdjustmentUsingPositiveDelta_ReturnsExpected() { var bias = 0.00364; var delta = 2; var expected = .00164; var actual = _perceptron.calculateBiasAdjustment(bias, _alpha, delta); Assert.AreEqual(expected, actual); } [TestMethod] public void calculateBiasAdjustmentUsingNegativeDelta_ReturnsExpected() { var bias = 0.00364; var delta = -2; var expected = .00564; var actual = _perceptron.calculateBiasAdjustment(bias, _alpha, delta); Assert.AreEqual(expected, actual); } [TestMethod] public void runAdjustmentUsingMatchingData_ReturnsExpected() { var adjustmentInput = new adjustmentInput(_weightedConfirmedObservation, _bias, _alpha, -1.0); var adjustedWeights = _perceptron.runAdjustment(adjustmentInput); var expected = .0065; var actual = Math.Round(adjustedWeights.weights[0],4); Assert.AreEqual(expected, actual); } [TestMethod] public void runAdjustmentUsingNegativeData_ReturnsExpected() { weightedConfirmedObservation weightedConfirmedObservation = new NeuralNetworks.weightedConfirmedObservation(_weightedObservation, 1.0); var adjustmentInput = new adjustmentInput(weightedConfirmedObservation, _bias, _alpha, -1.0); var adjustedWeights = _perceptron.runAdjustment(adjustmentInput); var expected = .0125; var actual = Math.Round(adjustedWeights.weights[0], 4); Assert.AreEqual(expected, actual); } [TestMethod] public void runAdjustmentUsingPositiveData_ReturnsExpected() { var adjustmentInput = new adjustmentInput(_weightedConfirmedObservation, _bias, _alpha, 1.0); var adjustedWeights = _perceptron.runAdjustment(adjustmentInput); var expected = .0005; var actual = Math.Round(adjustedWeights.weights[0], 4); Assert.AreEqual(expected, actual); }

With these functions ready, I could run a training cycle for a given observation

member this.runTrainingCycle (cycleTrainingInput:cycleTrainingInput) = let neuronTrainingInput = {neuronInput.weightedObservation=cycleTrainingInput.weightedConfirmedObservation.weightedObservation; neuronInput.bias=cycleTrainingInput.bias} let neuronResult = this.runNeuron(neuronTrainingInput) let activationResult = this.runActivation(neuronResult) let adjustmentInput = {weightedConfirmedObservation=cycleTrainingInput.weightedConfirmedObservation; bias=cycleTrainingInput.bias;alpha=cycleTrainingInput.alpha; yActual=activationResult} this.runAdjustment(adjustmentInput)

[TestMethod] public void runTrainingCycleUsingNegativeData_ReturnsExpected() { var cycleTrainingInput = new cycleTrainingInput(_weightedConfirmedObservation, _bias, _alpha); var adjustmentOutput = _perceptron.runTrainingCycle(cycleTrainingInput); var expected = .0125; var actual = Math.Round(adjustmentOutput.weights[0], 4); Assert.AreEqual(expected, actual); } [TestMethod] public void runTrainingCycleUsingPositiveData_ReturnsExpected() { var cycleTrainingInput = new cycleTrainingInput(_weightedConfirmedObservation, _bias, _alpha); var adjustmentOutput = _perceptron.runTrainingCycle(cycleTrainingInput); var expected = .0065; var actual = Math.Round(adjustmentOutput.weights[0], 4); Assert.AreEqual(expected, actual); }

And then I could run a cycle for each of the observations in the training set, a rotation.  I am not happy that I am mutating the weights and biases here, though I am not sure how to fix that.  I looked for a Seq.Scan function where the results of a function applied to the 1st element of a Seq is used in the input of the next –> all I could see were examples of threading a collector of int (like Seq.mapi).  This will be something I will ask the functional ninjas when I see them again.

member this.runTrainingRotation(rotationTrainingInput: rotationTrainingInput)= let mutable weights = rotationTrainingInput.weights let mutable bias = rotationTrainingInput.bias let alpha = rotationTrainingInput.alpha for i=0 to rotationTrainingInput.confirmedObservations.Count-1 do let currentConfirmedObservation = rotationTrainingInput.confirmedObservations.[i] let xws = Seq.zip currentConfirmedObservation.observation.xValues weights let xws' = new List<(float*float)>(xws) let weightedObservation = {xws=xws'} let weightedTrainingObservation = {weightedObservation=weightedObservation;yExpected=currentConfirmedObservation.yExpected} let cycleTrainingInput = { cycleTrainingInput.weightedConfirmedObservation=weightedTrainingObservation; cycleTrainingInput.bias=bias; cycleTrainingInput.alpha=alpha} let cycleOutput = this.runTrainingCycle(cycleTrainingInput) weights <- cycleOutput.weights bias <- cycleOutput.bias {adjustmentOutput.weights=weights; adjustmentOutput.bias=bias}

[TestMethod] public void runTrainingRotationUsingNegativeData_ReturnsExpected() { var xValues = new List<double>(); xValues.Add(3.0); xValues.Add(4.0); var observation = new observation(xValues); var yExpected = -1.0; var confirmedObservation0 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(1.5); xValues.Add(2.0); yExpected = -1.0; var confirmedObservation1 = new confirmedObservation(observation, yExpected); var trainingObservations = new List<confirmedObservation>(); trainingObservations.Add(confirmedObservation0); trainingObservations.Add(confirmedObservation1); var weights = new List<double>(); weights.Add(.0065); weights.Add(.0123); var rotationTrainingInput = new rotationTrainingInput(trainingObservations, weights, _bias, _alpha); var trainingRotationOutput = _perceptron.runTrainingRotation(rotationTrainingInput); var expected = -0.09606; var actual = Math.Round(trainingRotationOutput.bias, 5); Assert.AreEqual(expected, actual); } [TestMethod] public void runTrainingRotationUsingPositiveData_ReturnsExpected() { var xValues = new List<double>(); xValues.Add(3.0); xValues.Add(4.0); var observation = new observation(xValues); var yExpected = 1.0; var confirmedObservation0 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(1.5); xValues.Add(2.0); yExpected = 1.0; var confirmedObservation1 = new confirmedObservation(observation, yExpected); var trainingObservations = new List<confirmedObservation>(); trainingObservations.Add(confirmedObservation0); trainingObservations.Add(confirmedObservation1); var weights = new List<double>(); weights.Add(.0065); weights.Add(.0123); var rotationTrainingInput = new rotationTrainingInput(trainingObservations, weights, _bias, _alpha); var trainingRotationOutput = _perceptron.runTrainingRotation(rotationTrainingInput); var expected = -.09206; var actual = Math.Round(trainingRotationOutput.bias, 5); Assert.AreEqual(expected, actual); }

With the rotation done, I could write the train function which runs rotations for N number of times to tune the weights and biases:

member this.train(trainInput:trainInput) = let currentObservation = trainInput.confirmedObservations.[0].observation let weights = this.initializeWeights(currentObservation.xValues,trainInput.weightSeedValue) let weights' = new List<float>(weights) let mutable bias = this.initializeBias(trainInput.biasSeedValue) let alpha = trainInput.alpha for i=0 to trainInput.maxEpoches do let rotationTrainingInput={rotationTrainingInput.confirmedObservations=trainInput.confirmedObservations; rotationTrainingInput.weights = weights'; rotationTrainingInput.bias=bias; rotationTrainingInput.alpha=trainInput.alpha} this.runTrainingRotation(rotationTrainingInput) |> ignore {adjustmentOutput.weights=weights'; adjustmentOutput.bias=bias}

[TestMethod] public void trainUsingTestData_RetunsExpected() { var xValues = new List<double>(); xValues.Add(1.5); xValues.Add(2.0); var observation = new observation(xValues); var yExpected = -1.0; var confirmedObservation0 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(2.0); xValues.Add(3.5); observation = new observation(xValues); yExpected = -1.0; var confirmedObservation1 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(3.0); xValues.Add(5.0); observation = new observation(xValues); yExpected = -1.0; var confirmedObservation2 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(3.5); xValues.Add(2.5); observation = new observation(xValues); yExpected = -1.0; var confirmedObservation3 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(4.5); xValues.Add(5.0); observation = new observation(xValues); yExpected = 1.0; var confirmedObservation4 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(5.0); xValues.Add(7.5); observation = new observation(xValues); yExpected = 1.0; var confirmedObservation5 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(5.5); xValues.Add(8.0); observation = new observation(xValues); yExpected = 1.0; var confirmedObservation6 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(6.0); xValues.Add(6.0); observation = new observation(xValues); yExpected = 1.0; var confirmedObservation7 = new confirmedObservation(observation, yExpected); var trainingObservations = new List<confirmedObservation>(); trainingObservations.Add(confirmedObservation0); trainingObservations.Add(confirmedObservation1); trainingObservations.Add(confirmedObservation2); trainingObservations.Add(confirmedObservation3); trainingObservations.Add(confirmedObservation4); trainingObservations.Add(confirmedObservation5); trainingObservations.Add(confirmedObservation6); trainingObservations.Add(confirmedObservation7); var random = new Random(); var weightSeedValue = random.NextDouble(); var biasSeedValue = random.NextDouble(); var alpha = .001; var maxEpoches = 100; var trainInput = new trainInput(trainingObservations, weightSeedValue, biasSeedValue, alpha, maxEpoches); var trainOutput = _perceptron.train(trainInput); Assert.IsNotNull(trainOutput); }

With the training out of the way, I could concentrate on the prediction.  The prediction was much easier because there are no adjustments and the rotation is run once.  The data structures are also simpler because I don’t have to pass in the knownY values.  I also only have 1 covering (all be it long) unit test that looks that the results of the prediction.

member this.runPredictionCycle (cyclePredictionInput:cyclePredictionInput) = let neuronInput = {neuronInput.weightedObservation=cyclePredictionInput.weightedObservation; neuronInput.bias=cyclePredictionInput.bias} let neuronResult = this.runNeuron(neuronInput) this.runActivation(neuronResult) member this.runPredictionRotation (rotationPredictionInput:rotationPredictionInput) = let output = new List<List<float>*float>(); let weights = rotationPredictionInput.weights for i=0 to rotationPredictionInput.observations.Count-1 do let currentObservation = rotationPredictionInput.observations.[i]; let xws = Seq.zip currentObservation.xValues weights let xws' = new List<(float*float)>(xws) let weightedObservation = {xws=xws'} let cyclePredictionInput = { cyclePredictionInput.weightedObservation = weightedObservation; cyclePredictionInput.bias = rotationPredictionInput.bias} let cycleOutput = this.runPredictionCycle(cyclePredictionInput) output.Add(currentObservation.xValues, cycleOutput) output member this.predict(predictInput:predictInput) = let rotationPredictionInput = { rotationPredictionInput.observations = predictInput.observations; rotationPredictionInput.weights = predictInput.weights; rotationPredictionInput.bias = predictInput.bias } this.runPredictionRotation(rotationPredictionInput)

[TestMethod] public void predictUsingTestData_ReturnsExpected() { var xValues = new List<double>(); xValues.Add(3.0); xValues.Add(4.0); var observation0 = new observation(xValues); xValues = new List<double>(); xValues.Add(0.0); xValues.Add(1.0); var observation1 = new observation(xValues); xValues = new List<double>(); xValues.Add(2.0); xValues.Add(5.0); var observation2 = new observation(xValues); xValues = new List<double>(); xValues.Add(5.0); xValues.Add(6.0); var observation3 = new observation(xValues); xValues = new List<double>(); xValues.Add(9.0); xValues.Add(9.0); var observation4 = new observation(xValues); xValues = new List<double>(); xValues.Add(4.0); xValues.Add(6.0); var observation5 = new observation(xValues); var observations = new List<observation>(); observations.Add(observation0); observations.Add(observation1); observations.Add(observation2); observations.Add(observation3); observations.Add(observation4); observations.Add(observation5); var weights = new List<double>(); weights.Add(.0065); weights.Add(.0123); var bias = -0.0906; var predictInput = new predictInput(observations, weights, bias); var predictOutput = _perceptron.predict(predictInput); Assert.IsNotNull(predictOutput); }

When I run all of the unit tests the all run green:

image

With the Perceptron created, I can now go back and change the code and figure out:

1) Why my weights across the XValues are the same (wrong!)

2) How to implement a more idomatic/recursive way of running rotations so I can remove the mutation

With my unit tests running green, I know I am covered in case I make a mistake

Advertisements

One Response to Neural Network Part 3: Perceptrons

  1. Pingback: F# Weekly #32, 2014 | Sergey Tihon's Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: