Consuming and Analyzing Census Data Using F#

As part of my Nerd Dinner refactoring, I wanted to add the ability to guess a person’s age and gender based on their name.  I did a quick search on the internet and the only place that I found that has an API is here and it doesn’t have everything I am looking for.  Fortunately, the US Census website has some flat files with the kind of data I am looking for here.

I grabbed the data and  pumped it into Azure Blob Storage here.  You can swap out the state code to get each dataset.  I then loaded in a list of State Codes found here that match to the file names.

I then fired up Visual Studio and created a new FSharp project.  I added FSharp.Data to use a Type Provider to access the data.  I don’t need to install the Azure Storage .dlls b/c the blobs are public and I just have to read the file

image

Once Nuget was done with its magic, I opened up the script file, pointed to the newly-installed FSharp.Data, and added a reference to the datasets on blob storage:

#r "../packages/FSharp.Data.2.0.9/lib/portable-net40+sl5+wp8+win8/FSharp.Data.dll" open FSharp.Data type censusDataContext = CsvProvider<"https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/AK.TXT"> type stateCodeContext = CsvProvider<"https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/states.csv">

(Note that I am going add FSharp as a language to my Live Writer code snippet add-in at a later date)

In any event, I then printed out all of the codes to see what it looks like:

let stateCodes = stateCodeContext.Load("https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/states.csv"); stateCodes.Rows |> Seq.iter(fun r -> printfn "%A" r)

image

And by changing the lambda slightly like so,

stateCodes.Rows |> Seq.iter(fun r -> printfn "%A" r.Abbreviation)

I get all of the state codes

image

I then tested the census data with code and results are expected

let arkansasData = censusDataContext.Load("https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/AK.TXT"); arkansasData.Rows |> Seq.iter(fun r -> printfn "%A" r)

image

So then I created a method to load all of the state census data and giving me the length of the total:

let stateCodes = stateCodeContext.Load("https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/states.csv"); let usaData = stateCodes.Rows |> Seq.collect(fun r -> censusDataContext.Load(System.String.Format("https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/{0}.TXT",r.Abbreviation)).Rows) |> Seq.length

image

Since this is a I/O bound operation, it made sense to load the data asynchronously, which speeded things up considerably.  You can see my question over on Stack Overflow here and the resulting code takes about 50% of the time on a my dual-processor machine:

stopwatch.Start() let fetchStateDataAsync(stateCode:string)= async{ let uri = System.String.Format("https://portalvhdspgzl51prtcpfj.blob.core.windows.net/censuschicken/{0}.TXT",stateCode) let! stateData = censusDataContext.AsyncLoad(uri) return stateData.Rows } let usaData' = stateCodes.Rows |> Seq.map(fun r -> fetchStateDataAsync(r.Abbreviation)) |> Async.Parallel |> Async.RunSynchronously |> Seq.collect id |> Seq.length stopwatch.Stop() printfn "Parallel: %A" stopwatch.Elapsed.Seconds

image

With the data in hand, it was time to analyze the data to see if there is anything we can do.   Since 23 seconds is a bit too long to wait for a page load (Smile), I will need to put the 5.5 million records into a format that can be easily searched.  Thinking what we want is:

Given a name, what is the gender?

Given a name, what is the age?

Given a name, what is their state of birth?

Also, since we have their current location, we can also input the name and location and answer those questions.  If we make the assumption that their location is the same as their birth state, we can narrow down the list even further.

In any event, I first added a GroupBy to the name:

let nameSum = usaData' |> Seq.groupBy(fun r -> r.Mary) |> Seq.toArray

image

And then I summed up the counts of the names

let nameSum = usaData' |> Seq.groupBy(fun r -> r.Mary) |> Seq.map(fun (n,a) -> n,a |> Seq.sumBy(fun (r) -> r.``14``)) |> Seq.toArray

image

And then the total in the set:

let totalNames = nameSum |> Seq.sumBy(fun (n,c) -> c)

image

And then applied a simple average and sorted it descending

let nameAverage = nameSum |> Seq.map(fun (n,c) -> n,c,float c/ float totalNames) |> Seq.sortBy(fun (n,c,a) -> -a - 1.) |> Seq.toArray

image

So I feel really special that my parents gave me the most popular name in the US ever…

And focusing back to the task on hand, I want to determine the probability that a person is male or female based on their name:

let nameSearch = usaData' |> Seq.filter(fun r -> r.Mary = "James") |> Seq.groupBy(fun r -> r.F) |> Seq.map(fun (n,a) -> n,a |> Seq.sumBy(fun (r) -> r.``14``)) |> Seq.toArray

image

So 18196 parents thought is would be a good idea to name their daughter ‘James’.  I created a quick function like so:

let nameSearch' name = let nameFilter = usaData' |> Seq.filter(fun r -> r.Mary = name) |> Seq.groupBy(fun r -> r.F) |> Seq.map(fun (n,a) -> n,a |> Seq.sumBy(fun (r) -> r.``14``)) let nameSum = nameFilter |> Seq.sumBy(fun (n,c) -> c) nameFilter |> Seq.map(fun (n,c) -> n, c, float c/float nameSum) |> Seq.toArray nameSearch' "James"

image

So if I see the name “James”, there is a 99% chance it is a male.  This can lead to a whole host of questions like variance of names, names that are closest to gender neutral, etc….  Leaving those questions to another day, I now have something I can put into Nerd Dinner.  Now, if there was only a way to handle nicknames and friendly names….

You can see the full code here.

 

 

 

 

 

 

Controlling Servos Using Netdunio and Phidgets

As part of the Terminator program I am creating, I need a way of controlling servos to point the laser (and then gun) and different targets.  I decided to create a POC project and evaluate two different ways of controlling the servos.  As step one, I purchased a pan and tilt chassis from here

image

After playing with the servos from the kit, I decided to use my old stand-by servos that had a much higher quality and whose PWM signals I already know how to use.  With the chassis done, I needed a laser pointer so I figured why not get a shark with fricken laser?

I found one here.

image

So with the servos and laser ready to go, it was time to code.  I started with Netduninos:

public class Program { private const uint TILT_SERVO_STRAIGHT = 1500; private const uint TILT_SERVO_MAX_UP = 2000; private const uint TILT_SERVO_MAX_DOWN = 1000; private const uint PAN_SERVO_STRAIGHT = 1500; private const uint PAN_SERVO_MAX_LEFT = 1000; private const uint PAN_SERVO_MAX_RIGHT = 2000; private static PWM _tiltServo = null; private static PWM _panServo = null; private static uint _tiltServoCurrentPosition = 0; private static uint _panServoCurrentPosition = 0; public static void Main() { SetUpServos(); InputPort button = new InputPort(Pins.ONBOARD_BTN, false, Port.ResistorMode.Disabled); while (true) { if (button.Read()) { MoveServo(); } } } private static void SetUpServos() { uint period = 20000; _tiltServoCurrentPosition = TILT_SERVO_STRAIGHT; _panServoCurrentPosition = PAN_SERVO_STRAIGHT; _tiltServo = new PWM(PWMChannels.PWM_PIN_D3, period, _tiltServoCurrentPosition, PWM.ScaleFactor.Microseconds, false); _tiltServo.Start(); _panServo = new PWM(PWMChannels.PWM_PIN_D5, period, _panServoCurrentPosition, PWM.ScaleFactor.Microseconds, false); _panServo.Start(); } private static void MoveServo() { _panServo.Duration = PAN_SERVO_MAX_LEFT; Thread.Sleep(2000); _panServo.Duration = PAN_SERVO_MAX_RIGHT; Thread.Sleep(2000); _panServo.Duration = PAN_SERVO_STRAIGHT; Thread.Sleep(2000); _tiltServo.Duration = TILT_SERVO_MAX_UP; Thread.Sleep(2000); _tiltServo.Duration = TILT_SERVO_MAX_DOWN; Thread.Sleep(2000); _tiltServo.Duration = TILT_SERVO_STRAIGHT; } }

And sure enough the servos are behaving as expected

I then implemented a similar app using Phidgets.  Because the code is being executed on the PC, I could use F# to code (It does not look like the Netdunino/Microframework supports F#?)

open System open Phidgets let _servoController = new AdvancedServo() let mutable _isServoControllerReady = false let servoController_Attached(args:Events.AttachEventArgs) = let servoController = args.Device :?> AdvancedServo servoController.servos.[0].Engaged <- true servoController.servos.[7].Engaged <- true _isServoControllerReady <- true [<EntryPoint>] let main argv = _servoController.Attach.Add(servoController_Attached) _servoController.``open``() while true do if _isServoControllerReady = true then _servoController.servos.[0].Position<- 100. _servoController.servos.[7].Position<- 100. Console.ReadKey() |> ignore printfn "%A" argv 0

 

The choice then becomes using the Netduino or the Phidgets with my Kinect program.  I decided to defer the decision and use an interface for now.

type IWeaponsSystem = abstract member Activate: unit -> unit abstract member AquireTarget : float*float -> bool abstract member Fire: int -> bool

My decision about using Phidgets or Netduino is a series of trade-offs.  I can code Phidgets in C# or F# but I have to code Netduino in C#.  I would prefer to do this in F# so that makes me learn towards Phidgets.  I can put the Netduino anywhere and have it communicate via an Ethernet signal but I have to have the Phidgets wired to the PC.  Since the targeting system needs to be near the Kinect and the Kinect has to be tethered to the PC also, there is no real advantage of using the mobile Netduino.  Finally, the Phidgets API handles all communication to the servo control board for me, with the Netduino I would have to hook up a router to the Netduino and write the Ethernet communication code.  So I am leaning towards Phidgets, but since I am not sure, the interface allows me to swap in the Netduino at a later point without changing any code.  Love me some O in SOLID…

Up next, integrating the targeting system into the Terminator program.

 

 

Neural Network Part 3: Perceptrons

I went back to my code for building a Perceptron and I made some changes.  I realized that although McCaffrey combines the code together, there are actually two actions for the perceptron: training and predicting. I created a diagram to help me keep the functions that I need for each in mind:

image

I also skeletoned out some data structures that I think I need:

image

With the base diagrams out of the way, I created different data structures that were tailored to each action.   These are a bit different than the diagrams –> I didn’t go back and update the diagrams because the code is where you would look to see how the system works:

type observation = {xValues:float List} type weightedObservation = {xws:(float*float) List} type confirmedObservation = {observation:observation;yExpected:float} type weightedConfirmedObservation = {weightedObservation:weightedObservation;yExpected:float} type neuronInput = {weightedObservation:weightedObservation;bias:float} type cycleTrainingInput = {weightedConfirmedObservation:weightedConfirmedObservation;bias:float;alpha:float} type adjustmentInput = {weightedConfirmedObservation:weightedConfirmedObservation;bias:float;alpha:float;yActual:float} type adjustmentOutput = {weights:float List; bias:float} type rotationTrainingInput = {confirmedObservations:confirmedObservation List;weights:float List;bias:float;alpha:float} type trainInput = {confirmedObservations:confirmedObservation List; weightSeedValue:float;biasSeedValue:float;alpha:float; maxEpoches:int} type cyclePredictionInput = {weightedObservation:weightedObservation;bias:float} type rotationPredictionInput = {observations:observation List;weights:float List;bias:float} type predictInput = {observations:observation List;weights:float List;bias:float}

Note that I am composing data structures with the base being an observation.  The observation is a list of different xValues for a given, well, observation.  The weighted observation is the XValue paired with the perceptron weights.  The confirmedObservation is for training –> given an observation, what was the actual output? 

With the data structures out of the way, I went to the Perceptron and added in the basic functions for creating seed values:

member this.initializeWeights(xValues, randomSeedValue) = let lo = -0.01 let hi = 0.01 let xWeight = (hi-lo) * randomSeedValue + lo xValues |> Seq.map(fun w -> xWeight) member this.initializeBias(randomSeedValue) = let lo = -0.01 let hi = 0.01 (hi-lo) * randomSeedValue + lo

Since I was doing TDD, here are the unit tests I used for these functions:

[TestMethod] public void initializeWeightsUsingHalfSeedValue_ReturnsExpected() { var weights = _perceptron.initializeWeights(_observation.xValues, .5); var weightsList = new List<double>(weights); var expected = 0.0; var actual = weightsList[0]; Assert.AreEqual(expected, actual); } [TestMethod] public void initializeWeightsUsingLessThanHalfSeedValue_ReturnsExpected() { var weights = _perceptron.initializeWeights(_observation.xValues, .4699021627); var weightsList = new List<double>(weights); var expected = -0.00060; var actual = Math.Round(weightsList[0],5); Assert.AreEqual(expected, actual); } [TestMethod] public void initializeBiasesUsingHalfSeedValue_ReturnsExpected() { var expected = 0.0; var actual = _perceptron.initializeBias(.5); Assert.AreEqual(expected, actual); } [TestMethod] public void initializeBiasesUsingLessThanHalfSeedValue_ReturnsExpected() { var expected = -0.00060; var bias = _perceptron.initializeBias(.4699021627); var actual = Math.Round(bias, 5); Assert.AreEqual(expected, actual); } [TestMethod] public void initializeBiasesUsingGreaterThanHalfSeedValue_ReturnsExpected() { var expected = 0.00364; var bias = _perceptron.initializeBias(.6820621978); var actual = Math.Round(bias,5); Assert.AreEqual(expected, actual); }

I then created a base neuron and activation function that would work for both training and predicting:

member this.runNeuron(input:neuronInput) = let xws = input.weightedObservation.xws let output = xws |> Seq.map(fun (xValue,xWeight) -> xValue*xWeight) |> Seq.sumBy(fun x -> x) output + input.bias member this.runActivation(input) = if input < 0.0 then -1.0 else 1.0

[TestMethod] public void runNeuronUsingNormalInput_ReturnsExpected() { var expected = -0.0219; var perceptronOutput = _perceptron.runNeuron(_neuronInput); var actual = Math.Round(perceptronOutput, 4); Assert.AreEqual(expected, actual); } [TestMethod] public void runActivationUsingNormalInput_ReturnsExpected() { var expected = -1; var actual = _perceptron.runActivation(-0.0219); Assert.AreEqual(expected, actual); }

I then created the functions for training –> specifically to return adjusted weights and biases based on the result of the activation  function

member this.calculateWeightAdjustment(xValue, xWeight, alpha, delta) = match delta > 0.0, xValue >= 0.0 with | true,true -> xWeight - (alpha * abs(delta) * xValue) | false,true -> xWeight + (alpha * abs(delta) * xValue) | true,false -> xWeight - (alpha * abs(delta) * xValue) | false,false -> xWeight + (alpha * abs(delta) * xValue) member this.calculateBiasAdjustment(bias, alpha, delta) = match delta > 0.0 with | true -> bias - (alpha * abs(delta)) | false -> bias + (alpha * abs(delta)) member this.runAdjustment (input:adjustmentInput) = match input.weightedConfirmedObservation.yExpected = input.yActual with | true -> let weights = input.weightedConfirmedObservation.weightedObservation.xws |> Seq.map(fun (x,w) -> w) let weights' = new List<float>(weights) {adjustmentOutput.weights=weights';adjustmentOutput.bias=input.bias} | false -> let delta = input.yActual - input.weightedConfirmedObservation.yExpected let weights' = input.weightedConfirmedObservation.weightedObservation.xws |> Seq.map(fun (xValue, xWeight) -> this.calculateWeightAdjustment(xValue,xWeight,input.alpha,delta)) |> Seq.toList let weights'' = new List<float>(weights') let bias' = this.calculateBiasAdjustment(input.bias,input.alpha,delta) {adjustmentOutput.weights=weights'';adjustmentOutput.bias=bias'}

[TestMethod] public void calculateWeightAdjustmentUsingPositiveDelta_ReturnsExpected() { var xValue = 1.5; var xWeight = .00060; var delta = 2; var weightAdjustment = _perceptron.calculateWeightAdjustment(xValue, xWeight, _alpha, delta); var actual = Math.Round(weightAdjustment, 4); var expected = -.0024; Assert.AreEqual(expected, actual); } [TestMethod] public void calculateWeightAdjustmentUsingNegativeDelta_ReturnsExpected() { var xValue = 1.5; var xWeight = .00060; var delta = -2; var weightAdjustment = _perceptron.calculateWeightAdjustment(xValue, xWeight, _alpha, delta); var actual = Math.Round(weightAdjustment, 5); var expected = .0036; Assert.AreEqual(expected, actual); } [TestMethod] public void calculateBiasAdjustmentUsingPositiveDelta_ReturnsExpected() { var bias = 0.00364; var delta = 2; var expected = .00164; var actual = _perceptron.calculateBiasAdjustment(bias, _alpha, delta); Assert.AreEqual(expected, actual); } [TestMethod] public void calculateBiasAdjustmentUsingNegativeDelta_ReturnsExpected() { var bias = 0.00364; var delta = -2; var expected = .00564; var actual = _perceptron.calculateBiasAdjustment(bias, _alpha, delta); Assert.AreEqual(expected, actual); } [TestMethod] public void runAdjustmentUsingMatchingData_ReturnsExpected() { var adjustmentInput = new adjustmentInput(_weightedConfirmedObservation, _bias, _alpha, -1.0); var adjustedWeights = _perceptron.runAdjustment(adjustmentInput); var expected = .0065; var actual = Math.Round(adjustedWeights.weights[0],4); Assert.AreEqual(expected, actual); } [TestMethod] public void runAdjustmentUsingNegativeData_ReturnsExpected() { weightedConfirmedObservation weightedConfirmedObservation = new NeuralNetworks.weightedConfirmedObservation(_weightedObservation, 1.0); var adjustmentInput = new adjustmentInput(weightedConfirmedObservation, _bias, _alpha, -1.0); var adjustedWeights = _perceptron.runAdjustment(adjustmentInput); var expected = .0125; var actual = Math.Round(adjustedWeights.weights[0], 4); Assert.AreEqual(expected, actual); } [TestMethod] public void runAdjustmentUsingPositiveData_ReturnsExpected() { var adjustmentInput = new adjustmentInput(_weightedConfirmedObservation, _bias, _alpha, 1.0); var adjustedWeights = _perceptron.runAdjustment(adjustmentInput); var expected = .0005; var actual = Math.Round(adjustedWeights.weights[0], 4); Assert.AreEqual(expected, actual); }

With these functions ready, I could run a training cycle for a given observation

member this.runTrainingCycle (cycleTrainingInput:cycleTrainingInput) = let neuronTrainingInput = {neuronInput.weightedObservation=cycleTrainingInput.weightedConfirmedObservation.weightedObservation; neuronInput.bias=cycleTrainingInput.bias} let neuronResult = this.runNeuron(neuronTrainingInput) let activationResult = this.runActivation(neuronResult) let adjustmentInput = {weightedConfirmedObservation=cycleTrainingInput.weightedConfirmedObservation; bias=cycleTrainingInput.bias;alpha=cycleTrainingInput.alpha; yActual=activationResult} this.runAdjustment(adjustmentInput)

[TestMethod] public void runTrainingCycleUsingNegativeData_ReturnsExpected() { var cycleTrainingInput = new cycleTrainingInput(_weightedConfirmedObservation, _bias, _alpha); var adjustmentOutput = _perceptron.runTrainingCycle(cycleTrainingInput); var expected = .0125; var actual = Math.Round(adjustmentOutput.weights[0], 4); Assert.AreEqual(expected, actual); } [TestMethod] public void runTrainingCycleUsingPositiveData_ReturnsExpected() { var cycleTrainingInput = new cycleTrainingInput(_weightedConfirmedObservation, _bias, _alpha); var adjustmentOutput = _perceptron.runTrainingCycle(cycleTrainingInput); var expected = .0065; var actual = Math.Round(adjustmentOutput.weights[0], 4); Assert.AreEqual(expected, actual); }

And then I could run a cycle for each of the observations in the training set, a rotation.  I am not happy that I am mutating the weights and biases here, though I am not sure how to fix that.  I looked for a Seq.Scan function where the results of a function applied to the 1st element of a Seq is used in the input of the next –> all I could see were examples of threading a collector of int (like Seq.mapi).  This will be something I will ask the functional ninjas when I see them again.

member this.runTrainingRotation(rotationTrainingInput: rotationTrainingInput)= let mutable weights = rotationTrainingInput.weights let mutable bias = rotationTrainingInput.bias let alpha = rotationTrainingInput.alpha for i=0 to rotationTrainingInput.confirmedObservations.Count-1 do let currentConfirmedObservation = rotationTrainingInput.confirmedObservations.[i] let xws = Seq.zip currentConfirmedObservation.observation.xValues weights let xws' = new List<(float*float)>(xws) let weightedObservation = {xws=xws'} let weightedTrainingObservation = {weightedObservation=weightedObservation;yExpected=currentConfirmedObservation.yExpected} let cycleTrainingInput = { cycleTrainingInput.weightedConfirmedObservation=weightedTrainingObservation; cycleTrainingInput.bias=bias; cycleTrainingInput.alpha=alpha} let cycleOutput = this.runTrainingCycle(cycleTrainingInput) weights <- cycleOutput.weights bias <- cycleOutput.bias {adjustmentOutput.weights=weights; adjustmentOutput.bias=bias}

[TestMethod] public void runTrainingRotationUsingNegativeData_ReturnsExpected() { var xValues = new List<double>(); xValues.Add(3.0); xValues.Add(4.0); var observation = new observation(xValues); var yExpected = -1.0; var confirmedObservation0 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(1.5); xValues.Add(2.0); yExpected = -1.0; var confirmedObservation1 = new confirmedObservation(observation, yExpected); var trainingObservations = new List<confirmedObservation>(); trainingObservations.Add(confirmedObservation0); trainingObservations.Add(confirmedObservation1); var weights = new List<double>(); weights.Add(.0065); weights.Add(.0123); var rotationTrainingInput = new rotationTrainingInput(trainingObservations, weights, _bias, _alpha); var trainingRotationOutput = _perceptron.runTrainingRotation(rotationTrainingInput); var expected = -0.09606; var actual = Math.Round(trainingRotationOutput.bias, 5); Assert.AreEqual(expected, actual); } [TestMethod] public void runTrainingRotationUsingPositiveData_ReturnsExpected() { var xValues = new List<double>(); xValues.Add(3.0); xValues.Add(4.0); var observation = new observation(xValues); var yExpected = 1.0; var confirmedObservation0 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(1.5); xValues.Add(2.0); yExpected = 1.0; var confirmedObservation1 = new confirmedObservation(observation, yExpected); var trainingObservations = new List<confirmedObservation>(); trainingObservations.Add(confirmedObservation0); trainingObservations.Add(confirmedObservation1); var weights = new List<double>(); weights.Add(.0065); weights.Add(.0123); var rotationTrainingInput = new rotationTrainingInput(trainingObservations, weights, _bias, _alpha); var trainingRotationOutput = _perceptron.runTrainingRotation(rotationTrainingInput); var expected = -.09206; var actual = Math.Round(trainingRotationOutput.bias, 5); Assert.AreEqual(expected, actual); }

With the rotation done, I could write the train function which runs rotations for N number of times to tune the weights and biases:

member this.train(trainInput:trainInput) = let currentObservation = trainInput.confirmedObservations.[0].observation let weights = this.initializeWeights(currentObservation.xValues,trainInput.weightSeedValue) let weights' = new List<float>(weights) let mutable bias = this.initializeBias(trainInput.biasSeedValue) let alpha = trainInput.alpha for i=0 to trainInput.maxEpoches do let rotationTrainingInput={rotationTrainingInput.confirmedObservations=trainInput.confirmedObservations; rotationTrainingInput.weights = weights'; rotationTrainingInput.bias=bias; rotationTrainingInput.alpha=trainInput.alpha} this.runTrainingRotation(rotationTrainingInput) |> ignore {adjustmentOutput.weights=weights'; adjustmentOutput.bias=bias}

[TestMethod] public void trainUsingTestData_RetunsExpected() { var xValues = new List<double>(); xValues.Add(1.5); xValues.Add(2.0); var observation = new observation(xValues); var yExpected = -1.0; var confirmedObservation0 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(2.0); xValues.Add(3.5); observation = new observation(xValues); yExpected = -1.0; var confirmedObservation1 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(3.0); xValues.Add(5.0); observation = new observation(xValues); yExpected = -1.0; var confirmedObservation2 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(3.5); xValues.Add(2.5); observation = new observation(xValues); yExpected = -1.0; var confirmedObservation3 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(4.5); xValues.Add(5.0); observation = new observation(xValues); yExpected = 1.0; var confirmedObservation4 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(5.0); xValues.Add(7.5); observation = new observation(xValues); yExpected = 1.0; var confirmedObservation5 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(5.5); xValues.Add(8.0); observation = new observation(xValues); yExpected = 1.0; var confirmedObservation6 = new confirmedObservation(observation, yExpected); xValues = new List<double>(); xValues.Add(6.0); xValues.Add(6.0); observation = new observation(xValues); yExpected = 1.0; var confirmedObservation7 = new confirmedObservation(observation, yExpected); var trainingObservations = new List<confirmedObservation>(); trainingObservations.Add(confirmedObservation0); trainingObservations.Add(confirmedObservation1); trainingObservations.Add(confirmedObservation2); trainingObservations.Add(confirmedObservation3); trainingObservations.Add(confirmedObservation4); trainingObservations.Add(confirmedObservation5); trainingObservations.Add(confirmedObservation6); trainingObservations.Add(confirmedObservation7); var random = new Random(); var weightSeedValue = random.NextDouble(); var biasSeedValue = random.NextDouble(); var alpha = .001; var maxEpoches = 100; var trainInput = new trainInput(trainingObservations, weightSeedValue, biasSeedValue, alpha, maxEpoches); var trainOutput = _perceptron.train(trainInput); Assert.IsNotNull(trainOutput); }

With the training out of the way, I could concentrate on the prediction.  The prediction was much easier because there are no adjustments and the rotation is run once.  The data structures are also simpler because I don’t have to pass in the knownY values.  I also only have 1 covering (all be it long) unit test that looks that the results of the prediction.

member this.runPredictionCycle (cyclePredictionInput:cyclePredictionInput) = let neuronInput = {neuronInput.weightedObservation=cyclePredictionInput.weightedObservation; neuronInput.bias=cyclePredictionInput.bias} let neuronResult = this.runNeuron(neuronInput) this.runActivation(neuronResult) member this.runPredictionRotation (rotationPredictionInput:rotationPredictionInput) = let output = new List<List<float>*float>(); let weights = rotationPredictionInput.weights for i=0 to rotationPredictionInput.observations.Count-1 do let currentObservation = rotationPredictionInput.observations.[i]; let xws = Seq.zip currentObservation.xValues weights let xws' = new List<(float*float)>(xws) let weightedObservation = {xws=xws'} let cyclePredictionInput = { cyclePredictionInput.weightedObservation = weightedObservation; cyclePredictionInput.bias = rotationPredictionInput.bias} let cycleOutput = this.runPredictionCycle(cyclePredictionInput) output.Add(currentObservation.xValues, cycleOutput) output member this.predict(predictInput:predictInput) = let rotationPredictionInput = { rotationPredictionInput.observations = predictInput.observations; rotationPredictionInput.weights = predictInput.weights; rotationPredictionInput.bias = predictInput.bias } this.runPredictionRotation(rotationPredictionInput)

[TestMethod] public void predictUsingTestData_ReturnsExpected() { var xValues = new List<double>(); xValues.Add(3.0); xValues.Add(4.0); var observation0 = new observation(xValues); xValues = new List<double>(); xValues.Add(0.0); xValues.Add(1.0); var observation1 = new observation(xValues); xValues = new List<double>(); xValues.Add(2.0); xValues.Add(5.0); var observation2 = new observation(xValues); xValues = new List<double>(); xValues.Add(5.0); xValues.Add(6.0); var observation3 = new observation(xValues); xValues = new List<double>(); xValues.Add(9.0); xValues.Add(9.0); var observation4 = new observation(xValues); xValues = new List<double>(); xValues.Add(4.0); xValues.Add(6.0); var observation5 = new observation(xValues); var observations = new List<observation>(); observations.Add(observation0); observations.Add(observation1); observations.Add(observation2); observations.Add(observation3); observations.Add(observation4); observations.Add(observation5); var weights = new List<double>(); weights.Add(.0065); weights.Add(.0123); var bias = -0.0906; var predictInput = new predictInput(observations, weights, bias); var predictOutput = _perceptron.predict(predictInput); Assert.IsNotNull(predictOutput); }

When I run all of the unit tests the all run green:

image

With the Perceptron created, I can now go back and change the code and figure out:

1) Why my weights across the XValues are the same (wrong!)

2) How to implement a more idomatic/recursive way of running rotations so I can remove the mutation

With my unit tests running green, I know I am covered in case I make a mistake

Terminator Program: With The Kinect 2

I got my hands on a Kinect2 last week so I decided to re-write the Terminator program using the Kinect2 api.  Microsoft made some major changes to the domain api (no more skeleton frame, now using a body) but the underlying logic is still the same.  Therefore, it was reasonably easy to port the code.  There is plenty of places in the V2 api that are not documented yet but because I did some work in the V1 api, I could still get things done.  For example, the V2 api documentation and code samples use event handlers to work with any new frame that arrives from the Kinect.  This lead to some pretty laggy code.  However, by using polling on a second thread, I was able to get the performance to where it needs to be.  Also, a minor annoyance is that you have to use Win8 with the Kinect 2.

So here is the Terminator application, Gen 2.  The UI is still just a series of UI controls:

1 <Window x:Class="ChickenSoftware.Terminator.Gen2.UI.MainWindow" 2 xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" 3 xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" 4 Title="MainWindow" Height="700" Width="650" Loaded="Window_Loaded"> 5 <Canvas Width="650" Height="700"> 6 <Image x:Name="kinectColorImage" Width="640" Height="480" /> 7 <Canvas x:Name="bodyCanvas" Width="640" Height="480" /> 8 <Button x:Name="takePhotoButton" Canvas.Left="10" 9 Canvas.Top="485" Height="40" Width="125" Click="takePhotoButton_Click">Take Photo</Button> 10 <TextBox x:Name="facialRecognitionTextBox" Canvas.Left="10" Canvas.Top="540" Width="125" Height="40" FontSize="8" /> 11 <Image x:Name="currentImage" Canvas.Left="165" Canvas.Top="485" Height="120" Width="170" /> 12 <Image x:Name="compareImage" Canvas.Left="410" Canvas.Top="485" Height="120" Width="170" /> 13 </Canvas> 14 </Window> 15

In the code behind, I set up some class-level variables.  The only real difference is that the photo is moving from 640/480 to 1920/1080:

1 KinectSensor _kinectSensor = null; 2 Boolean _isKinectDisplayActive = false; 3 Boolean _isTakingPicture = false; 4 WriteableBitmap _videoBitmap = null; 5 Int32 _width = 1920; 6 Int32 _height = 1080;

When the page is loaded, a new thread is spun up that handles rendering the Kinect data:

1 private void Window_Loaded(object sender, RoutedEventArgs e) 2 { 3 SetUpKinect(); 4 _isKinectDisplayActive = true; 5 Thread videoThread = new Thread(new ThreadStart(DisplayKinectData)); 6 videoThread.Start(); 7 }

Setting up the Kinect is a bit different (KinectSensor.GetDefault()) but intuitive:

1 internal void SetUpKinect() 2 { 3 _videoBitmap = new WriteableBitmap(1920, 1080, 96, 96, PixelFormats.Bgr32, null); 4 _kinectSensor = KinectSensor.GetDefault(); 5 _kinectSensor.Open(); 6 }

With the big change in the DisplayKinectData method

1 internal void DisplayKinectData() 2 { 3 var colorFrameSource = _kinectSensor.ColorFrameSource; 4 var colorFrameReader = colorFrameSource.OpenReader(); 5 var bodyFrameSource = _kinectSensor.BodyFrameSource; 6 var bodyFrameReader = bodyFrameSource.OpenReader(); 7 8 while (_isKinectDisplayActive) 9 { 10 using (var colorFrame = colorFrameReader.AcquireLatestFrame()) 11 { 12 if (colorFrame == null) continue; 13 using (var bodyFrame = bodyFrameReader.AcquireLatestFrame()) 14 { 15 if (bodyFrame == null) continue; 16 //Color 17 var colorFrameDescription = colorFrame.ColorFrameSource.CreateFrameDescription(ColorImageFormat.Bgra); 18 var bytesPerPixel = colorFrameDescription.BytesPerPixel; 19 var frameSize = colorFrameDescription.Width * colorFrameDescription.Height * bytesPerPixel; 20 var colorData = new byte[frameSize]; 21 if (colorFrame.RawColorImageFormat == ColorImageFormat.Bgra) 22 { 23 colorFrame.CopyRawFrameDataToArray(colorData); 24 } 25 else 26 { 27 colorFrame.CopyConvertedFrameDataToArray(colorData, ColorImageFormat.Bgra); 28 } 29 //Body 30 var bodies = new Body[bodyFrame.BodyCount]; 31 bodyFrame.GetAndRefreshBodyData(bodies); 32 var trackedBody = bodies.FirstOrDefault(b => b.IsTracked); 33 34 //Update 35 if (_isTakingPicture) 36 { 37 Dispatcher.Invoke(new Action(() => AnalyzePhoto(colorData))); 38 } 39 else 40 { 41 if (trackedBody == null) 42 { 43 Dispatcher.Invoke(new Action(() => UpdateDisplay(colorData))); 44 } 45 else 46 { 47 Dispatcher.Invoke(new Action(() => UpdateDisplay(colorData, trackedBody))); 48 } 49 } 50 } 51 } 52 } 53 } 54

I am using a frameReader and frameSource for both the color (the video image) and the body (the old skeleton).  The method to get the frame has changed –> I am using AquireLatestFrame().  It is nice that we are still using byte[] to hold the data.

With the data in the byte[] arrays, the display is updated.  There are two UpdateDisplay methods:

1 internal void UpdateDisplay(byte[] colorData) 2 { 3 var rectangle = new Int32Rect(0, 0, _width, _height); 4 _videoBitmap.WritePixels(rectangle, colorData, _width * 4, 0); 5 kinectColorImage.Source = _videoBitmap; 6 } 7 8 internal void UpdateDisplay(byte[] colorData, Body body) 9 { 10 UpdateDisplay(colorData); 11 var drawingGroup = new DrawingGroup(); 12 using (var drawingContext = drawingGroup.Open()) 13 { 14 var headPosition = body.Joints[JointType.Head].Position; 15 if (headPosition.Z < 0) 16 { 17 headPosition.Z = 0.1f; 18 } 19 var adjustedHeadPosition = _kinectSensor.CoordinateMapper.MapCameraPointToDepthSpace(headPosition); 20 bodyCanvas.Children.Clear(); 21 Rectangle headTarget = new Rectangle(); 22 headTarget.Fill = new SolidColorBrush(Colors.Red); 23 headTarget.Width = 10; 24 headTarget.Height = 10; 25 Canvas.SetLeft(headTarget, adjustedHeadPosition.X + 75); 26 Canvas.SetTop(headTarget, adjustedHeadPosition.Y); 27 bodyCanvas.Children.Add(headTarget); 28 } 29 }

This is pretty much like V1 where the video byte[] is being written to a WritableBitmap and the body is being drawn on the canvas.  Note that like V1, the coordinates of the body need to be adjusted to the color frame.  The API has a series of overloads that makes it easy to do the translation.

With the display working, I added in taking the photo, sending it to Azure blob storage, and having Sky Biometry analyze the results.  This code is identical to V1 with the connection strings for Azure and Sky Biometry broken out into their own methods and the sensitive values placed into the app.config:

1 internal void AnalyzePhoto(byte[] colorData) 2 { 3 var bitmapSource = BitmapSource.Create(_width, _height, 96, 96, PixelFormats.Bgr32, null, colorData, _width * 4); 4 JpegBitmapEncoder encoder = new JpegBitmapEncoder(); 5 encoder.Frames.Add(BitmapFrame.Create(bitmapSource)); 6 var photoImage = UploadPhotoImage(encoder); 7 CompareImages(photoImage); 8 _isTakingPicture = false; 9 }

1 internal PhotoImage UploadPhotoImage(JpegBitmapEncoder encoder) 2 { 3 using(MemoryStream memoryStream = new MemoryStream()) 4 { 5 encoder.Save(memoryStream); 6 var photoImage = new PhotoImage(Guid.NewGuid(), memoryStream.ToArray()); 7 8 var customerUniqueId = new Guid(ConfigurationManager.AppSettings["customerUniqueId"]); 9 var connectionString = GetAzureConnectionString(); 10 11 IPhotoImageProvider provider = new AzureStoragePhotoImageProvider(customerUniqueId, connectionString); 12 provider.InsertPhotoImage(photoImage); 13 memoryStream.Close(); 14 return photoImage; 15 } 16 }

1 internal void CompareImages(PhotoImage photoImage) 2 { 3 String skyBiometryUri = ConfigurationManager.AppSettings["skyBiometryUri"]; 4 String uid = ConfigurationManager.AppSettings["skyBiometryUid"]; 5 String apiKey = ConfigurationManager.AppSettings["skyBiometryApiKey"]; 6 String apiSecret = ConfigurationManager.AppSettings["skyBiometryApiSecret"]; 7 var imageComparer = new SkyBiometryImageComparer(skyBiometryUri, uid, apiKey, apiSecret); 8 9 String basePhotoUri = GetBasePhotoUri(); 10 String targetPhotoUri = GetTargetPhotoUri(photoImage); 11 currentImage.Source = new BitmapImage(new Uri(targetPhotoUri)); 12 compareImage.Source = new BitmapImage(new Uri(basePhotoUri)); 13 14 var matchValue = imageComparer.CalculateFacialRecognitionConfidence(basePhotoUri, targetPhotoUri); 15 facialRecognitionTextBox.Text = "Match Value Confience is: " + matchValue.Confidence.ToString(); 16 }

With the code in place, I can the run the Terminator Gen 2:

image

I think I am doing the Sky Biometry recognition incorrectly so I will look at that later.  In any event, working with the Kinect V2 was fairly easy because it was close enough to the V1 that the concepts could translate.  I look forward to adding the targeting system this weekend!!!

Neural Network Part 2: Perceptrons

I started working though the second chapter of McCaffrey’s book Neural Networks Using C# Succinctly to see if I could write the examples using F#.

McCaffrey’s code is tough to read though because of its emphasis on loops and global mutable variables.  I read though his description and this is how <I think> the Perceptron should be constructed.

The inputs are a series of independent variables (in this case age and income) and the output is a single dependent variable (in this case party affiliation).  The values have been encoded and normalized like in this post here.

An example of the input (from page 31 of his book) is:

image

Or in a more abstract manner:

image

In terms of data structures, individual inputs (each row) is placed into an array of floats and the output is a single float

image

I call this single set of inputs an “observation” (my words, not McCaffrey).

Looking at McCaffrey’s example for a perceptron Input-Output,

image

all of the variables you need are not included. Here is what you need:

image

Where A0 and B0 are the same as X0 and X1 respectively in his diagram.  Also, McCaffrey uses the word “Perceptron” to mean two different concepts: the entire system as a whole and the individual calculation for a given list of X and Bias.  I am a big believer of domain ubiquitous languages so I am calling the individual calculation a neuron.

Once you run these values through the neuron for the 1st observation, you might have to alter the Weights and Bias based on the (Y)result.  Therefore, the data structure coming out of the Neuron is

image

These values are feed into the adjustment function to alter the weights and bias with the output as

image

I am calling this process of taking the a single observation, the xWeights, , and the bias and turning them into a series of weights and bais as a “cycle” (my words, not McCaffrey)

image

 

The output of a cycle is then fed with  the next observation and the cycle repeats for as many observations as there are fed into the system. 

image

 

I am calling the process of running a cycle for each observation in the input dataset a rotation (my words, not McCaffrey) and that the perceptron runs rotations for an x number of times to train itself.

image

 

Finally, the Perceptron takes a new set of observations where the Y is not known and runs a Rotation once to predict what the Y will be.

So with that mental image in place, the coding became much easier.  Basically, there was a 1 to 1 correspondence of F# functions to each step laid out.  I started with an individual cycle

  1. type cycleInput = {xValues:float list;yExpected:float;mutable weights:float list;mutable bias:float;alpha:float}
  2.  
  3. let runNeuron (input:cycleInput) =
  4.     let valuesAndWeights = input.xValues |> List.zip input.weights
  5.     let output = valuesAndWeights
  6.                     |> List.map(fun (xValue, xWeight) -> xValue*xWeight)
  7.                     |> List.sumBy(fun x -> x)
  8.     output + input.bias
  9.  
  10. let runActivation input =
  11.     if input < 0.0 then -1.0 else 1.0

I used record types all over the place in this code just so I could keep things straight in my head.  McCaffrey uses ambiguously-named arrays and global variables.  Although this makes my code a bit more wordy (esp for functional people), I think the increased readability is worth the trade-off.

In any event, with the Neuron and Activation calc out of the way, I created the functions that adjust the weights and bias:

  1. let calculateWeightAdjustment(xValue, xWeight, alpha, delta) =
  2.     match delta > 0.0, xValue >= 0.0 with
  3.         | true,true -> xWeight – (alpha * delta * xValue)
  4.         | false,true -> xWeight + (alpha * delta * xValue)
  5.         | true,false -> xWeight – (alpha * delta * xValue)
  6.         | false,false -> xWeight + (alpha * delta * xValue)
  7.  
  8. let calculateBiasAdjustment(bias, alpha, delta) =
  9.     match delta > 0.0 with
  10.         | true -> bias – (alpha * delta)
  11.         | false -> bias + (alpha * delta)

This code is significantly different than the for, nested if that McCaffrey uses. 

image

I maintain using this kind of pattern matching makes the intention much easier to comprehend.  I also split out the adjustment of the weights and the adjustment of the bias into individual functions.

With these functions ready, I created an input and output record type and implemented the adjustment function

  1. let runAdjustment (input:adjustmentInput) =
  2.     match input.yExpected = input.yActual with
  3.         | true -> {weights=input.weights;bias=input.bias;yActual=input.yActual}
  4.         | false ->
  5.             let delta = input.yActual – input.yExpected
  6.             let valuesAndWeights = input.xValues |> List.zip input.weights
  7.             let weights' =  valuesAndWeights |> List.map(fun (xValue, xWeight) -> calculateWeightAdjustment(xValue,xWeight,input.alpha,delta))
  8.             let bias' = calculateBiasAdjustment(input.bias,input.alpha,delta)
  9.             {weights=weights';bias=bias';yActual=input.yActual}

There is not a corresponding method in McCaffrey’s code, rather he just does some Array.copy and mutates the global variables in the Update method.  I am not a fan of side-effect programming so I created a function that explicitly does the  modification.

And to wrap up the individual cycle:

  1. let runCycle (cycleInput:cycleInput) =
  2.     let neuronResult = runNeuron(cycleInput)
  3.     let activationResult = runActivation(neuronResult)
  4.     let adjustmentInput = {xValues=cycleInput.xValues;weights=cycleInput.weights;yExpected=cycleInput.yExpected;
  5.                             bias=cycleInput.bias;alpha=cycleInput.alpha;yActual=activationResult}
  6.     runAdjustment(adjustmentInput)

Up next is to run the cycle for each of the observations (called a rotation)

  1. type observation = {xValues:float list;yExpected:float}
  2. type rotationInput = {observations: observation list;mutable weights:float list;mutable bias:float;alpha:float}
  3. type trainingRotationOutput = {weights:float list; bias:float}
  4. type predictionRotationOutput = {observation: observation;yActual:float}
  5.  
  6. let runTrainingRotation(rotationInput: rotationInput)=
  7.     for i=0 to rotationInput.observations.Length do
  8.         let observation = rotationInput.observations.[i]
  9.         let neuronInput = {cycleInput.xValues=observation.xValues;cycleInput.yExpected=observation.yExpected;cycleInput.weights=rotationInput.weights;
  10.                             cycleInput.bias=rotationInput.bias;cycleInput.alpha=rotationInput.alpha}
  11.         let cycleOutput = runCycle(neuronInput)
  12.         rotationInput.weights <- cycleOutput.weights
  13.         rotationInput.bias <- cycleOutput.bias
  14.     {weights=rotationInput.weights; bias=rotationInput.bias}

Again, note the liberal use of records to keep the inputs and outputs clear.  I also created a prediction rotation that is designed to be run only once that does not alter the weights and bias.

  1. let runPredictionRotation(rotationInput: rotationInput)=
  2.     let output = new System.Collections.Generic.List<predictionRotationOutput>()
  3.     for i=0 to rotationInput.observations.Length do
  4.         let observation = rotationInput.observations.[i]
  5.         let neuronInput = {cycleInput.xValues=observation.xValues;cycleInput.yExpected=observation.yExpected;cycleInput.weights=rotationInput.weights;
  6.                             cycleInput.bias=rotationInput.bias;cycleInput.alpha=rotationInput.alpha}
  7.         let cycleOutput = runCycle(neuronInput)
  8.         let predictionRotationOutput = {observation=observation;yActual=cycleOutput.yActual}
  9.         output.Add(predictionRotationOutput)   
  10.     output

With the rotations done, the last step was to create the Perceptron to train and then predict:

  1. type perceptronInput = {observations: observation list;weights:float list;bias:float}
  2. type perceptronOutput = {weights:float list; bias:float}
  3.  
  4. let initializeWeights(xValues, randomSeedValue) =
  5.     let lo = -0.01
  6.     let hi = 0.01
  7.     let xWeight = (hi-lo) * randomSeedValue + lo
  8.     xValues |> List.map(fun w -> xWeight)
  9.  
  10. let initializeBias(randomSeedValue) =
  11.     let lo = -0.01
  12.     let hi = 0.01
  13.     (hi-lo) * randomSeedValue + lo
  14.  
  15. let runTraining(perceptronInput: perceptronInput, maxEpoches:int) =
  16.     let random = System.Random()
  17.     let alpha = 0.001
  18.     let baseObservation = perceptronInput.observations.[0]
  19.     let mutable weights = initializeWeights(baseObservation.xValues,random.NextDouble())       
  20.     let mutable bias = initializeBias(random.NextDouble())
  21.     let rotationList = [0..maxEpoches]
  22.     for i=0 to maxEpoches do
  23.         let rotationInput = {observations=perceptronInput.observations;weights=weights;bias=bias;alpha=alpha}
  24.         let rotationOutput = runTrainingRotation(rotationInput)
  25.         weights <- rotationOutput.weights
  26.         bias <- rotationOutput.bias
  27.     {weights=weights;bias=bias}
  28.  
  29. let runPrediction(perceptronInput: perceptronInput, weights: float list, bias: float) =
  30.     let random = System.Random()
  31.     let alpha = 0.001
  32.     let rotationInput = {observations=perceptronInput.observations;weights=weights;bias=bias;alpha=alpha}
  33.     runPredictionRotation(rotationInput)

 

Before I go too much further, I have a big code smell.  I am iterating and using the mutable keyword.  I am not sure how to take the results of a function that is applied to the 1st element in a sequence and then input that into the second.  I need to do that with the weights and bias data structures –> each time it is used in a expression, it need to change and feed into the next expression.  I think the answer is the List.Reduce, so I am going to pick this up after looking at that in more detail.  I also need to implement the shuffle method so that that cycles are not called in the same order across rotations….

Neural Networks

I picked up James McCaffrey’s Neural Networks Using C# a couple of weeks ago and decided to see if I could rewrite the code in F#.  Unfortunately, the source code is not available (as far as I could tell), so I did some C# then F# coding to see if I could get functional equivalence.

My first stop was chapter one.  I made the decision to get the F# code working for the sample data that McCaffrey provided first and then refactor it to a more general program that would work with inputs and values of different datasets.  My final upgrade will be use Deedle instead of any other data structure.  But first things first, I want to get the examples working so I fired up a script file and opened my REPL.

McCaffrey defines a sample dataset like this

  1. string[] sourceData = new string[] { "Sex Age Locale Income Politics",
  2.     "==============================================",
  3.     "Male 25 Rural 63,000.00 Conservative",
  4.     "Female 36 Suburban 55,000.00 Liberal", "Male 40 Urban 74,000.00 Moderate",
  5.     "Female 23 Rural 28,000.00 Liberal" };

He then creates a parser for the comma-delimited string values into a double[][].  I just created the dataset as a List of tuples.

  1. let chapter1TestData = [("Male",25.,"Rural",63000.00,"Conservative");
  2.                 ("Female",36.,"Suburban",55000.00,"Liberal");
  3.                 ("Male",40.,"Urban",74000.00,"Moderate");
  4.                 ("Female",23.,"Rural",28000.00,"Liberal")]

 

I did try an implementation using a record type but for reasons below, I am using Tuples.  With the equivalent data loaded into  the REPL, I tackled the first supporting function: MinMax.  Here is the C# code that McCaffrey wrote:

  1. static void MinMaxNormal(double[][] data, int column)
  2. {
  3.     int j = column;
  4.     double min = data[0][j];
  5.     double max = data[0][j];
  6.     for (int i = 0; i < data.Length; ++i)
  7.     {
  8.         if (data[i][j] < min) min = data[i][j];
  9.         if (data[i][j] > max) max = data[i][j];
  10.     }
  11.     double range = max – min;
  12.     if (range == 0.0) // ugly
  13.     { for (int i = 0; i < data.Length; ++i)
  14.         data[i][j] = 0.5;
  15.         return; }
  16.     for (int i = 0; i < data.Length; ++i)
  17.         data[i][j] = (data[i][j] – min) / range;
  18. }

and here is the equivalent F# code.

  1. let minMax (fullSet, i) =
  2.     let min = fullSet |> Seq.min
  3.     let max = fullSet |> Seq.max
  4.     (i-min)/(max-min)

 

Note that McCaffrey does not have any unit tests but when I ran the dummy data through the F# implementation, the results matched his screen shots so that will work well enough.  If you ever need a reason to use F#, consider those 2 code samples.  Granted McCaffrey’s code is more abstract because it can be any column in double array, but my counterpoint is that the function is really doing too much and it is trivial in F# to pick a given column.  Is there any doubt what the F# code is doing?  Is there any certainty of what the C# code is doing?

In any event, moving along to the next functions, McCaffrey created two functions that do all of the encoding of the string values to appropriate numeric ones.  Depending on if the value is a X value (independent) or Y value (dependent), there is a different encoding scheme:

  1.  static string EffectsEncoding(int index, int N)
  2.  {
  3.      // If N = 3 and index = 0 -> 1,0.
  4.      // If N = 3 and index = 1 -> 0,1.
  5.      // If N = 3 and index = 2 -> -1,-1.
  6.      if (N == 2)
  7.      // Special case.
  8.      { if (index == 0) return "-1"; else if (index == 1) return "1"; }
  9.      int[] values = new int[N - 1];
  10.      if (index == N – 1)
  11.      // Last item is all -1s.
  12.      { for (int i = 0; i < values.Length; ++i) values[i] = -1; }
  13.      else
  14.      {
  15.          values[index] = 1;
  16.          // 0 values are already there.
  17.      } string s = values[0].ToString();
  18.      for (int i = 1; i < values.Length; ++i) s += "," + values[i]; return s;
  19.  }
  20.  
  21.  static string DummyEncoding(int index, int N)
  22.  {
  23.      int[] values = new int[N]; values[index] = 1;
  24.      string s = values[0].ToString();
  25.      for (int i = 1; i < values.Length; ++i) s += "," + values[i];
  26.      return
  27. }

In my F# project, I decided to domain-specific encoding.  I plan to refactor this to something more abstract. 

  1. //Transform Sex
  2. let testData' = chapter1TestData |> Seq.map(fun (s,a,l,i,p) -> match s with
  3.                                                                | "Male"-> -1.0,a,l,i,p
  4.                                                              | "Female" -> 1.0,a,l,i,p
  5.                                                              | _ -> failwith "Invalid sex")
  6. //Normalize Age
  7. let testData'' =
  8.     let fullSet =  testData' |> Seq.map(fun (s,a,l,i,p) -> a)
  9.     testData' |> Seq.map(fun (s,a,l,i,p) -> s,minMax(fullSet,a),l,i,p)
  10.  
  11. //Transform Locale
  12. let testData''' = testData'' |> Seq.map(fun (s,a,l,i,p) -> match l with
  13.                                                                 | "Rural" -> s,a,1.,0.,i,p
  14.                                                                 | "Suburban" -> s,a,0.,1.,i,p
  15.                                                                 | "Urban" -> s,a,-1.,-1.,i,p
  16.                                                                 | _ -> failwith "Invalid locale")
  17. //Transform and Normalize Income
  18. let testData'''' =
  19.     let fullSet =  testData''' |> Seq.map(fun (s,a,l0,l1,i,p) -> i)
  20.     testData''' |> Seq.map(fun (s,a,l0,l1,i,p) -> s,a,l0,l1,minMax(fullSet,i),p)
  21.  
  22. //Transform Politics
  23. let testData''''' = testData'''' |> Seq.map(fun (s,a,l0,l1,i,p) -> match p with
  24.                                                                 | "Conservative" -> s,a,l0,l1,i,1.,0.,0.
  25.                                                                 | "Liberal" -> s,a,l0,l1,i,0.,1.,0.
  26.                                                                 | "Moderate" -> s,a,l0,l1,i,0.,0.,1.
  27.                                                                 | _ -> failwith "Invalid politics")

When I execute the script:

image

Which is the same as McCaffrey’s.

image

Note that he used Gaussian normalization on column 2 and I did Min/Max based on his advice in the book.

 

 

Terminator Program: Part 2

Following up on my last post, I decided to send the entire photograph to Sky Biometry and have them parse the photograph and identify individual people.  This ability is built right into their API.  For example, if you pass them this picture, you get the following json back.

image

I added the red highlight to show that Sky Biometry can recognize multiple people (it is an array of uids) and that each face tag has a center.x and center:y.  Reading the API documentation, this point is center of the face tag point and their point is a percentage of the photo width.

image

So I need to translate the center point of the skeleton from the Kinect to eqiv center point of the sky biometry recognition output and I should be able to identify individual people within the Kinect’s field of vision.  Going back to the Kinect code, I ditched the DrawBoxAroundHead method and altered the UpdateDisplay method like so

  1. private void UpdateDisplay(byte[] colorData, Skeleton[] skeletons)
  2. {
  3.     if (_videoBitmap == null)
  4.     {
  5.         _videoBitmap = new WriteableBitmap(640, 480, 96, 96, PixelFormats.Bgr32, null);
  6.     }
  7.     _videoBitmap.WritePixels(new Int32Rect(0, 0, 640, 480), colorData, 640 * 4, 0);
  8.     kinectColorImage.Source = _videoBitmap;
  9.     var selectedSkeleton = skeletons.FirstOrDefault(s => s.TrackingState == SkeletonTrackingState.Tracked);
  10.     if (selectedSkeleton != null)
  11.     {
  12.         var headPosition = selectedSkeleton.Joints[JointType.Head].Position;
  13.         var adjustedHeadPosition =
  14.             _sensor.CoordinateMapper.MapSkeletonPointToColorPoint(headPosition, ColorImageFormat.RgbResolution640x480Fps30);
  15.         var adjustedSkeletonPosition = _sensor.CoordinateMapper.MapSkeletonPointToColorPoint(selectedSkeleton.Position, ColorImageFormat.RgbResolution640x480Fps30);
  16.  
  17.         skeletonCanvas.Children.Clear();
  18.         Rectangle headRectangle = new Rectangle();
  19.         headRectangle.Fill = new SolidColorBrush(Colors.Blue);
  20.         headRectangle.Width = 10;
  21.         headRectangle.Height = 10;
  22.         Canvas.SetLeft(headRectangle, adjustedHeadPosition.X);
  23.         Canvas.SetTop(headRectangle, adjustedHeadPosition.Y);
  24.         skeletonCanvas.Children.Add(headRectangle);
  25.  
  26.         Rectangle skeletonRectangle = new Rectangle();
  27.         skeletonRectangle.Fill = new SolidColorBrush(Colors.Red);
  28.         skeletonRectangle.Width = 10;
  29.         skeletonRectangle.Height = 10;
  30.         Canvas.SetLeft(skeletonRectangle, adjustedHeadPosition.X);
  31.         Canvas.SetTop(skeletonRectangle, adjustedHeadPosition.Y);
  32.         skeletonCanvas.Children.Add(skeletonRectangle);
  33.  
  34.         String skeletonInfo = headPosition.X.ToString() + " : " + headPosition.Y.ToString() + " — ";
  35.         skeletonInfo = skeletonInfo + adjustedHeadPosition.X.ToString() + " : " + adjustedHeadPosition.Y.ToString() + " — ";
  36.         skeletonInfo = skeletonInfo + adjustedSkeletonPosition.X.ToString() + " : " + adjustedSkeletonPosition.Y.ToString();
  37.  
  38.         skeletonInfoTextBox.Text = skeletonInfo;
  39.  
  40.     }
  41. }

Notice that there are two rectangles because I was not sure if the Head.Position or the Skeleton.Position would match SkyBiometry.  Turns out that I want the Head.Position for SkyBiometry (besides, the terminator would want head shots only)

image

So I ditched the Skeleton.Position.  I then needed a way to translate the Head.Posotion.X to SkyBiometry.X and Head.Posotion.Y to SkyBiometry.Y.  Fortunately, I know the size of each photograph (640 X 480) so calculating the percent is an exercise of altering UpdateDisplay:

  1. private void UpdateDisplay(byte[] colorData, Skeleton[] skeletons)
  2. {
  3.     Int32 photoWidth = 640;
  4.     Int32 photoHeight = 480;
  5.  
  6.     if (_videoBitmap == null)
  7.     {
  8.         _videoBitmap = new WriteableBitmap(photoWidth, photoHeight, 96, 96, PixelFormats.Bgr32, null);
  9.     }
  10.     _videoBitmap.WritePixels(new Int32Rect(0, 0, photoWidth, photoHeight), colorData, photoWidth * 4, 0);
  11.     kinectColorImage.Source = _videoBitmap;
  12.     var selectedSkeleton = skeletons.FirstOrDefault(s => s.TrackingState == SkeletonTrackingState.Tracked);
  13.     if (selectedSkeleton != null)
  14.     {
  15.         var headPosition = selectedSkeleton.Joints[JointType.Head].Position;
  16.         var adjustedHeadPosition =
  17.             _sensor.CoordinateMapper.MapSkeletonPointToColorPoint(headPosition, ColorImageFormat.RgbResolution640x480Fps30);
  18.  
  19.         skeletonCanvas.Children.Clear();
  20.         Rectangle headRectangle = new Rectangle();
  21.         headRectangle.Fill = new SolidColorBrush(Colors.Blue);
  22.         headRectangle.Width = 10;
  23.         headRectangle.Height = 10;
  24.         Canvas.SetLeft(headRectangle, adjustedHeadPosition.X);
  25.         Canvas.SetTop(headRectangle, adjustedHeadPosition.Y);
  26.         skeletonCanvas.Children.Add(headRectangle);
  27.  
  28.         var skyBiometryX = ((float)adjustedHeadPosition.X / photoWidth)*100;
  29.         var skyBioMetryY = ((float)adjustedHeadPosition.Y / photoHeight)*100;
  30.  
  31.         String skeletonInfo = adjustedHeadPosition.X.ToString() + " : " + adjustedHeadPosition.Y.ToString() + " — ";
  32.         skeletonInfo = skeletonInfo + Math.Round(skyBiometryX,2).ToString() + " : " + Math.Round(skyBioMetryY,2).ToString();
  33.  
  34.         skeletonInfoTextBox.Text = skeletonInfo;
  35.  
  36.     }

And so now I have

image

The next step is to get the Kinect photo to Sky Biometry.  I decided to use Azure Blob Storage as my intermediately location.  I updated the architectural diagram like so:

image

At this point, it made sense to move the project over to F# so I could better concentrate on the work that needs to be done and also getting the important code out of the UI code behind.  I fired up a F# project in my solution added a couple different implementations of Storing Photos.  To keep things consistent, I created a data structure and an interface:

  1. namespace ChickenSoftware.Terminator.Core
  2.  
  3. open System
  4.  
  5. type public PhotoImage (uniqueId:Guid, imageBytes:byte[]) =
  6.     member this.UniqueId = uniqueId
  7.     member this.ImageBytes = imageBytes
  8.  
  9. type IPhotoImageProvider =
  10.     abstract member InsertPhotoImage : PhotoImage -> unit
  11.     abstract member DeletePhotoImage : Guid -> unit
  12.     abstract member GetPhotoImage : Guid -> PhotoImage

My 1st stop was to replicate what Miles did with the Save File Dialog box with a File System Provider.  It was very much like a C# implementation:

  1. namespace ChickenSoftware.Terminator.Core
  2.  
  3. open System
  4. open System.IO
  5. open System.Drawing
  6. open System.Drawing.Imaging
  7.  
  8. type LocalFileSystemPhotoImageProvider(folderPath: string) =
  9.  
  10.     member this.GetPhotoImageUri(uniqueIdentifier: Guid) =
  11.         let fileName = uniqueIdentifier.ToString() + ".jpg"
  12.         Path.Combine(folderPath, fileName)
  13.  
  14.     interface IPhotoImageProvider with
  15.         member this.InsertPhotoImage(photoImage: PhotoImage) =
  16.             let fullPath = this.GetPhotoImageUri(photoImage.UniqueId)
  17.             use memoryStream = new MemoryStream(photoImage.ImageBytes)
  18.             let image = Image.FromStream(memoryStream)
  19.             image.Save(fullPath)
  20.  
  21.         member this.DeletePhotoImage(uniqueIdentifier: Guid) =
  22.             let fullPath = this.GetPhotoImageUri(uniqueIdentifier)
  23.             File.Delete(fullPath)        
  24.  
  25.         member this.GetPhotoImage(uniqueIdentifier: Guid) =
  26.             let fullPath = this.GetPhotoImageUri(uniqueIdentifier)
  27.             use fileStream = new FileStream(fullPath,FileMode.Open)
  28.             let image = Image.FromStream(fileStream)
  29.             use memoryStream = new MemoryStream()
  30.             image.Save(memoryStream,ImageFormat.Jpeg)
  31.             new PhotoImage(uniqueIdentifier, memoryStream.ToArray())

To call the save method, I altered the SavePhoto method in the C# project to use a MemoryStream and not a FileStream:

  1. private void SavePhoto(byte[] colorData)
  2. {
  3.     var bitmapSource = BitmapSource.Create(640, 480, 96, 96, PixelFormats.Bgr32, null, colorData, 640 * 4);
  4.     JpegBitmapEncoder encoder = new JpegBitmapEncoder();
  5.     encoder.Frames.Add(BitmapFrame.Create(bitmapSource));
  6.     using (MemoryStream memoryStream = new MemoryStream())
  7.     {
  8.         encoder.Save(memoryStream);
  9.         PhotoImage photoImage = new PhotoImage(Guid.NewGuid(), memoryStream.ToArray());
  10.  
  11.         String folderUri = @"C:\Data";
  12.         IPhotoImageProvider provider = new LocalFileSystemPhotoImageProvider(folderUri);
  13.  
  14.         provider.InsertPhotoImage(photoImage);
  15.         memoryStream.Close();
  16.     }
  17.     _isTakingPicture = false;
  18. }

And sure enough, it saves the photo to disk:

image

One problem that took me 20 minutes to uncover is that if you get your file system path wrong, you get the unhelpful exception:

image

This has been well-bitched about on stack overflow so I won’t comment further. 

With the file system up and running, I turned my attention to Azure.  Like the File System provider, it is very close to a C# implementation

  1. namespace ChickenSoftware.Terminator.Core
  2.  
  3. open System
  4. open System.IO
  5. open Microsoft.WindowsAzure.Storage
  6. open Microsoft.WindowsAzure.Storage.Blob
  7.  
  8. type AzureStoragePhotoImageProvider(customerUniqueId: Guid, connectionString: string) =
  9.  
  10.     member this.GetBlobContainer(blobClient:Blob.CloudBlobClient) =
  11.         let container = blobClient.GetContainerReference(customerUniqueId.ToString())
  12.         if not (container.Exists()) then
  13.             container.CreateIfNotExists() |> ignore
  14.             let permissions = new BlobContainerPermissions()
  15.             permissions.PublicAccess <- BlobContainerPublicAccessType.Blob
  16.             container.SetPermissions(permissions)
  17.         container
  18.  
  19.     member this.GetBlockBlob(uniqueIdentifier: Guid) =
  20.         let storageAccount = CloudStorageAccount.Parse(connectionString)
  21.         let blobClient = storageAccount.CreateCloudBlobClient()
  22.         let container = this.GetBlobContainer(blobClient)
  23.         let photoUri = this.GetPhotoImageUri(uniqueIdentifier)
  24.         container.GetBlockBlobReference(photoUri)
  25.  
  26.     member this.GetPhotoImageUri(uniqueIdentifier: Guid) =
  27.         uniqueIdentifier.ToString() + ".jpg"
  28.  
  29.     interface IPhotoImageProvider with
  30.         member this.InsertPhotoImage(photoImage: PhotoImage) =
  31.             let blockBlob = this.GetBlockBlob(photoImage.UniqueId)
  32.             use memoryStream = new MemoryStream(photoImage.ImageBytes)
  33.             blockBlob.UploadFromStream(memoryStream)
  34.  
  35.         member this.DeletePhotoImage(uniqueIdentifier: Guid) =
  36.             let blockBlob = this.GetBlockBlob(uniqueIdentifier)
  37.             blockBlob.Delete()       
  38.  
  39.         member this.GetPhotoImage(uniqueIdentifier: Guid) =
  40.             let blockBlob = this.GetBlockBlob(uniqueIdentifier)
  41.             if blockBlob.Exists() then
  42.                 blockBlob.FetchAttributes()
  43.                 use memoryStream = new MemoryStream()
  44.                 blockBlob.DownloadToStream(memoryStream)
  45.                 let photoArray = memoryStream.ToArray()
  46.                 new PhotoImage(uniqueIdentifier,photoArray)
  47.             else
  48.                 failwith "photo not found"

And when I pop it into the WPF application,

  1. private void SavePhoto(byte[] colorData)
  2. {
  3.     var bitmapSource = BitmapSource.Create(640, 480, 96, 96, PixelFormats.Bgr32, null, colorData, 640 * 4);
  4.     JpegBitmapEncoder encoder = new JpegBitmapEncoder();
  5.     encoder.Frames.Add(BitmapFrame.Create(bitmapSource));
  6.     using (MemoryStream memoryStream = new MemoryStream())
  7.     {
  8.         encoder.Save(memoryStream);
  9.         PhotoImage photoImage = new PhotoImage(Guid.NewGuid(), memoryStream.ToArray());
  10.  
  11.         Guid customerUniqueId = new Guid("7282AF48-FB3D-489B-A572-2EFAE80D0A9E");
  12.         String connectionString =
  13.             "DefaultEndpointsProtocol=http;AccountName=XXX;AccountKey=XXX";
  14.         IPhotoImageProvider provider = new AzureStoragePhotoImageProvider(customerUniqueId, connectionString);
  15.  
  16.  
  17.         provider.InsertPhotoImage(photoImage);
  18.         memoryStream.Close();
  19.     }
  20.     _isTakingPicture = false;
  21. }

I can now write my images to Azure.

image

With that out of the way, I can now have SkyBiometry pick up my photo, analyze it, and push the results back.  I went ahead and added in the .fs module that I had already created for this blog post.  I then added FSharp.Data via NuGet and was ready to roll. In he Save photo event handler,after saving the photo to blob storage, it then calls Sky Biometry to compare against a base image that has already been trained:

  1. private void SavePhoto(byte[] colorData)
  2. {
  3.     var bitmapSource = BitmapSource.Create(640, 480, 96, 96, PixelFormats.Bgr32, null, colorData, 640 * 4);
  4.     JpegBitmapEncoder encoder = new JpegBitmapEncoder();
  5.     encoder.Frames.Add(BitmapFrame.Create(bitmapSource));
  6.     PhotoImage photoImage = UploadPhotoImage(encoder);
  7.  
  8.     String skyBiometryUri = "http://api.skybiometry.com&quot;;
  9.     String uid = "Kinect@ChickenFace";
  10.     String apiKey = "XXXX";
  11.     String apiSecret = "XXXX";
  12.  
  13.     var imageComparer = new SkyBiometryImageComparer(skyBiometryUri, uid, apiKey, apiSecret);
  14.     String basePhotoUri = "XXXX.jpg";
  15.     String targetPhotoUri = "XXXX/" + photoImage.UniqueId + ".jpg";
  16.  
  17.     currentImage.Source = new BitmapImage(new Uri(basePhotoUri));
  18.     compareImage.Source = new BitmapImage(new Uri(targetPhotoUri)); ;
  19.     
  20.     var matchValue = imageComparer.CalculateFacialRecognitionConfidence(basePhotoUri, targetPhotoUri);
  21.     FacialRecognitionTextBox.Text = "Match Value is: " + matchValue.ToString();
  22.     _isTakingPicture = false;
  23. }

And I am getting a result back from Sky Biometry.

image

Finally, I added in the SkyBiometry X and Y coordinates for the photo and compared to the calculated ones based on the Kinect Skeleton Tracking:

  1. currentImage.Source = new BitmapImage(new Uri(basePhotoUri));
  2. compareImage.Source = new BitmapImage(new Uri(targetPhotoUri)); ;
  3.  
  4. var matchValue = imageComparer.CalculateFacialRecognitionConfidence(basePhotoUri, targetPhotoUri);
  5.  
  6. var selectedSkeleton = skeletons.FirstOrDefault(s => s.TrackingState == SkeletonTrackingState.Tracked);
  7. if (selectedSkeleton != null)
  8. {
  9.     var headPosition = selectedSkeleton.Joints[JointType.Head].Position;
  10.     var adjustedHeadPosition =
  11.         _sensor.CoordinateMapper.MapSkeletonPointToColorPoint(headPosition, ColorImageFormat.RgbResolution640x480Fps30);
  12.  
  13.     var skyBiometryX = ((float)adjustedHeadPosition.X / 640) * 100;
  14.     var skyBioMetryY = ((float)adjustedHeadPosition.Y / 480) * 100;
  15.  
  16.     StringBuilder stringBuilder = new StringBuilder();
  17.     stringBuilder.Append("Match Value is: ");
  18.     stringBuilder.Append(matchValue.Confidence.ToString());
  19.     stringBuilder.Append("Sky Biometry X: ");
  20.     stringBuilder.Append(matchValue.X.ToString());
  21.     stringBuilder.Append("Sky Biometry Y: ");
  22.     stringBuilder.Append(matchValue.Y.ToString());
  23.     stringBuilder.Append("Kinect X: ");
  24.     stringBuilder.Append(Math.Round(skyBiometryX, 2).ToString());
  25.     stringBuilder.Append("Kinect Y: ");
  26.     stringBuilder.Append(Math.Round(skyBioMetryY, 2).ToString());
  27.     FacialRecognitionTextBox.Text = stringBuilder.ToString();
  28. }
  29.  
  30. _isTakingPicture = false;

And the results are encouraging –> it looks like I can use the X and Y to identify different people on the screen:

Match Value is: 53
Sky Biometry X: 10
Sky Biometry Y: 13.33

Kinect X: 47.5
Kinect Y: 39.79

Up next will be pointing the laser and the target…

 

 

 

Follow

Get every new post delivered to your Inbox.