F#, REPL Driven Development, and Scrum

Last week,  I did a book review of sorts on Scrum: The Art of Doing Twice The Work In Half The Time.  When I was reading the text, an interesting thought hit me several times.  As a pragmatic practitioner of Test Driven Development (TDD), which often goes hand in hand with Agile and Scrum ideas, I often wonder if I am doing something the best way.  I remember distinctly Robert C Martin talking about being your new CTO with the goal of all of the code working correctly all of the time and that he didn’t care if you used TDD, but he doesn’t know a better way.

image

I was thinking how lately I have been practicing REPL-driven development using F#.  If you are not familiar, REPL stands for “READ-EVALUATE-PRINT-LOOP” and has been the primary methodology of functional programmers and data scientists for years.  In RDD, I quickly prove out my idea tin the REPL to see if I make sense.  it is strictly happy path programming.  Once I think I have a good idea or a solution to whatever problem I am working on, I lift the code into a compiled assembly.  The data elements I used in the REPL then get ported over into my unit tests.  I typically use C# unit tests so that I can confirm that my FSharp code will interop nicely with any VB.NET/C# projects in the solution.  I then layer on code coverage to make sure I have covered all happy paths and then throw some fail cases at the code.

Thinking of this methodology, I think it is closer to Scrum than traditional TDD for a couple of reasons:

Fail Fast and Fix early.  You cannot prove out ideas any faster than in the REPL except for maybe a dry board.  Curly-braces and OO-centered languages like Java and C# are great for certain jobs, but the require much more ceremony and code for code’s sake.  As Sutherland points out, context-switching is a killer.  The less you have to worry about code (classes, moqs, etc..) the faster and better you will be at solving your problem.

Working Too Hard Makes More Work. One of the most startling things about using F# on real projects is that there is just not very much code.  I finished and looked around to see what I missed.  My unit tests were passing, code coverage was high, and there just wasn’t much code.  It was quite unsettling.  I now realize that lots of C#/Java code needs to be generated for real programming projects (exception handling, class hierarchies, design patterns, etc…).  But as the Dartmouth Basic Manual once said “typing is not a substitute for thinking”, all of this code begets more code.  It is a cycle of work that creates more work that F# does not have.

Duplication/Boilerplates/Templates. Complete and Total Waste So this one is pretty self-explanatory.  Many people  (myself included) think that Visual Studio needs better F# templates.  However, once you get good at writing F# code, you really don’t need them.  Maybe it is good that there aren’t many more?  In any event, you don’t use templates and boiler plates in the REPL…

 

The Wright Brothers and Scrum

I recently read two books that, on the cover have nothing to do with each other, but actually have very much similar lessons.  The first is David MacCullugh’s The Wright Brothers and Jeff Sutherland’s Scrum: The art of doing twice the work in half the time.

imageimage

Although the human-interest side of the story was kind of interesting to me, what really stood out was how the Wright brothers got their machine in the air.  If you are not familiar with the details of how they constructed the Wright Flyer, there are some pretty interesting points:

1) The Wright brothers were a small agile team that comprised of no fewer than two (the brothers themselves) and no more than seven.  I did not realize how important Charlie Taylor and William Tate was to different pieces of the project.

2) The Wright brothers spent the 1st part of their journey doing research, combing all of the scientific literature, and engaging with the current though leaders of the day in a very much open-source style where they would freely share knowledge but retain the final product for themselves.  This was in direct contrast to other teams that operated in silos and secrecy.

3) The Wright Brothers believed in doing one thing at a time well.  They realized that there were two major problems with heavier than air flight –> thrust  and balance.  They separated these two concerns and tackled the balance problem first.  Once they figured out how to make a glider stable in flight, they then tacked how to add a motor to it.

4) The Wright Brothers made hundreds of small incremental changes with each change able to stand for itself.  For example, they went out to Kitty Hawk in the summer of 1900, 1901, and 1902 with gliders before going out the forth time in 1903 with their airplane.  Each time, the designs got bigger and closer to the final goal.

5) The Wright brothers were willing to challenge conventional and commonly accepted “facts” when their evidence did not support it.  The Wright brothers relied heavily on the calculations of Lilienthal and Chanute to measure lift and drag.  After several failed experiments, the Wright brothers ditched those and went with their own, painstakingly researched, measurement tables.

3) The Wright Brothers were in direct competition with Samuel Langley’s airplane.  In contrast to the Wright Brother’s agile approach, Langley had a large team that operated in absolutely secrecy while taking massive (at the time) amounts of public funds.  When Langley finally rolled out his “final product” it failed miserably every time.

So what does the Wright Brother’s methodology have to do with Scrum?  Everything. If you look art the core tenants of Sutherland’s book,  most of them can be found in how the Wrights conquered the air.  I went though the end of each chapter of Scrum and pulled out some of the take-away points that direct match to how Orville and Wilber did things:

image

The Wright brothers were doing scrum a full hundred years before it became a thing. As amazing what they created, how they did it is really remarkable.  Interestingly for me, the “It’s the journey, not the destination” just rang home.  As I write this blog, it is Friday night and I am on my front porch.  A neighbor stopped by to say “hello”.  When  told her I was working on a blog post related to my profession, she said “Oh, I am sorry you are not doing anything fun tonight.”  And I said “But this is fun.”  and internally I was thinking “I wonder why so many people think work is not fun?  Why are some many people socialized that way?  I hope my kids don’t wind up like that.”

The Wright Brothers and Scrum

I recently read two books that, on the cover have nothing to do with each other, but actually have very much similar lessons.  The first is David MacCullugh’s The Wright Brothers and Jeff Sutherland’s Scrum: The art of doing twice the work in half the time.

imageimage

Although the human-interest side of the story was kind of interesting to me, what really stood out was how the Wright brothers got their machine in the air.  If you are not familiar with the details of how they constructed the Write Flyer, there are some pretty interesting points:

1) The Wright brothers were a small agile team that comprised of no fewer than two (the brothers themselves) and no more than seven.  I did not realize how important Charlie Taylor and William Tate was to different pieces of the project.

2) The Write brothers spent the 1st part of their journey doing research, combing all of the scientific literature, and engaging with the current though leaders of the day in a very much open-source style where they would freely share knowledge but retain the final product for themselves.  This was in direct contrast to other teams that operated in silos and secrecy.

3) The Wright Brothers believed in doing one thing at a time well.  They realized that there were two major problems with heavier than air flight –> thrust  and balance.  They separated these two concerns and tackled the balance problem first.  Once they figured out how to make a glider stable in flight, they then tacked how to add a motor to it.

4) The Wright Brothers made hundreds of small incremental changes with each change able to stand for itself.  For example, they went out to Kitty Hawk in the summer of 1900, 1901, and 1902 with gliders before going out the forth time in 1903 with their airplane.  Each time, the designs got bigger and closer to the final goal.

5) The Wright brothers were willing to challenge conventional and commonly accepted “facts” when their evidence did not support it.  The Wright brothers relied heavily on the calculations of Lilienthal and Chanute to measure lift and drag.  After several failed experiments, the Wright brothers ditched those and went with their own, painstakingly researched, measurement tables.

3) The Wright Brothers were in direct competition with Samuel Langley’s airplane.  In contrast to the Write Brother’s agile approach, Langley had a large team that operated in absolutely secrecy while taking massive (at the time) amounts of public funds.  When Langley finally rolled out his “final product” it failed miserably every time.

So what does the Wright Brother’s methodology have to do with Scrum?  Everything. If you look art the core tenants of Sutherland’s book,  most of them can be found in how the Wrights conquered the air.  I went though the end of each chapter of Scrum and pulled out some of the take-away points that direct match to how Orville and Wilber did things:

image

The Wright brothers were doing scrum a full hundred years before it became a thing. As amazing what they created, how they did it is really remarkable.  Interestingly for me, the “It’s the journey, not the destination” just rang home.  As I write this blog, it is Friday night and I am on my front porch.  A neighbor stopped by to say “hello”.  When  told her I was working on a blog post related to my profession, she said “Oh, I am sorry you are not doing anything fun tonight.”  And I said “But this is fun.”  and internally I was thinking “I wonder why so many people think work is not fun?  Why are some many people socialized that way?  I hope my kids don’t wind up like that.”

Sandcastle Help File Builder and FSharp

If you are going to write and release a professional-grade  .NET assembly, there are some things that need to be considered: logging, exception handling, and documentation.  For .NET components, Sandcastle Help File Builder is the go-to tool to generate documentation as either the old-school .chm file or as a web deploy. 

Consider an assembly that contains a Customer record type, an interface for a Customer Repository, and two implementations (In-Memory and ADO.NET)

1 type Customer = {id:int; firstName:string; lastName:string} 2 3 type ICusomerRepository = 4 abstract member GetCustomer : int -> Customer 5 abstract member InsertCustomer: Customer -> int 6 abstract member DeleteCustomer: int -> unit 7 8 type InMemoryCustomerRepository ()= 9 let customers = [ 10 {id=1; firstName = "First"; lastName = "Customer"} 11 {id=2; firstName = "Second"; lastName = "Customer"} 12 {id=3; firstName = "Third"; lastName = "Customer"}] 13 let customers' = new List<Customer>(customers) 14 15 interface ICusomerRepository with 16 member this.GetCustomer(id:int) = 17 customers' |> Seq.find(fun c -> c.id = id) 18 member this.InsertCustomer(customer: Customer) = 19 let nextId = customers'.Count 20 let customer' = {customer with id=nextId} 21 customers'.Add(customer') 22 nextId 23 member this.DeleteCustomer(id: int) = 24 let customer = customers |> Seq.find(fun c -> c.id = id) 25 customers'.Remove(customer) |> ignore 26 27 type SqlServerCustomerRepository (connectionString:string) = 28 interface ICusomerRepository with 29 member this.GetCustomer(id:int) = 30 use connection = new SqlConnection(connectionString) 31 let commandText = "Select * from customers where id = " + id.ToString() 32 use command = new SqlCommand(commandText, connection) 33 connection.Open() 34 use reader = command.ExecuteReader() 35 reader.Read() |> ignore 36 {id=reader.[0] :?> int; 37 firstName=reader.[1] :?> string; 38 lastName =reader.[2] :?> string} 39 40 member this.InsertCustomer(customer: Customer) = 41 use connection = new SqlConnection(connectionString) 42 let commandText = new StringBuilder() 43 commandText.Append("Insert customers values") |> ignore 44 commandText.Append(customer.firstName) |> ignore 45 commandText.Append(",") |> ignore 46 commandText.Append(customer.lastName) |> ignore 47 use command = new SqlCommand(commandText.ToString(), connection) 48 connection.Open() 49 command.ExecuteNonQuery() 50 51 member this.DeleteCustomer(id: int) = 52 use connection = new SqlConnection(connectionString) 53 let commandText = "Delete customers where id = " + id.ToString() 54 use command = new SqlCommand(commandText, connection) 55 connection.Open() 56 command.ExecuteNonQuery() |> ignore 57

To auto-generate XML code comments, you need to mark “XML documentation file” on the Build page of project properties:

image

With the .XML file created during the build, you can then fire up Sandcastle to point to the .XML file

image

With that, you can get some nice component documents based on your XML Code Comments.  Since I have not put any into my project yet, there is nothing in the docs.

image

So therein lies the rub.  I started entering XML comments (bare minimum) like so:

1 /// <summary> 2 /// Interface for Customer Repository implementations. 3 /// </summary> 4 type ICusomerRepository = 5 /// <summary> 6 /// Get a single validated customer. 7 /// </summary> 8 ///<param name="param0">The customer Id</param> 9 ///<returns>A validated Customer.</returns> 10 abstract member GetCustomer : int -> Customer 11 /// <summary> 12 /// Insert a single validated customer. 13 /// </summary> 14 ///<param name="param0">A validated customer.</param> 15 ///<returns>The Id of the customer, generated by the respository.</returns> 16 abstract member InsertCustomer: Customer -> int 17 /// <summary> 18 /// Deletes a single customer from the respository. 19 /// </summary> 20 ///<param name="param0">The customer Id</param> 21 abstract member DeleteCustomer: int -> unit

image

And you can see what happens.  The code base goes from 5 lines of readable code to 21 lines of clutter to make the help file.

One of the tenants of good code is that it is clean –> so we use SOLID principles, run FxCop, and the like.  Another tenant of good code is that it is uncluttered –> so we use FSharp, use ROP instead of structured exception handling, and avoid boilerplates and templating.  The problem is that we still can’t get away from clutter if we want to have good documentation.  Option A is to just drop documentation, a laudable but unrealistic goal, especially in a corporate environment.  Option B I am not sure on.  I am wondering if I create a separate file in the project just for the code comments.  That way the actual code is uncluttered and you can work with it undistracted and the XML still gets generated…

 

Carolina Code Camp Material

I had a great time presenting and attending Carolina Code Camp.  Hats off the organizers for making such a large event run without a hitch.  I especially want to thank Dan Thyer and Mike Linnen for the BuilderFaire.  What a great time!  The best part of the camp was meeting such smart and innovative people.

My own presentation materials can be found here: https://github.com/jamessdixon/2013CarolinaCodeCamp

Carolina Code Camp Prep

I am going down to Charlotte for the Carolina Code Camp this weekend.  I am presenting at two sessions:

1) Remote Control Lawnmowers (that I am building with my daughter).  I will show how to intercept and use PWM signals from a RC transmitter/receiver and how to turn those commands into the drive system of the robot.  Here is a couple of clips that we did recently:

 

 

I am also doing a code-only SOLID presentation.  This is pretty much the same presentation that I did at TRINUG’s DEV Craftsmanship SIG in February.

I am looking forward to a fun (half) day and learning some great stuff from the other hackers coders.

Why .NET Developers Don’t Unit Test

Short answer: It’s Microsoft’s fault.

Long answer:

I am going to describe the journey of Well Intention Programmer (WIP).  He started writing software code in the mid-90s using this book:

image

and when .NET came out he slid over to C#

image

Along the way, he has written several highly-regarded programs that always exceed expectations and came in under-budget in the the 1st release but struggled in subsequent releases.  So the WIP decides to learn about how to improve his code quality to help him with the old code and to deliver more timely 2.x releases.  He picks up theses books:

imageimage

And  learns that he shouldn’t change a single line of code without covering unit tests.  Eager to try unit testing out, he opens up some legacy code and finds this:

public List<String> GetRegions()
{
    List<String> regions = new List<string>();
    String connectionString = @"Server=.;Database=Northwind;Trusted_Connection=True;";

    using(SqlConnection connection = new SqlConnection(connectionString))
    {
        String commandText = "Select * from Region";
        using (SqlCommand command = new SqlCommand(commandText, connection))
        {
            connection.Open();
            SqlDataReader reader = command.ExecuteReader();
            while (reader.Read())
            {
                regions.Add(reader["RegionDescription"].ToString());
            }
        }
    }

    return regions;
}

So the 1st thing the WIP might want to refactor is the hard-coded connection string in the function.  Microsoft has been telling the WIP for years that this string belongs in the .config file.  And it really does – not so much because you are going to swap out your DBMS (never happens) but because you should have environment-specific settings (that you don’t have on your local workstation).

To protect himself from the change, the WIP right clicks on the method name in VS2010

image

and he gets this out of the box test snippet like so:

[TestClass()]
public class NorthinwindFactoryTests
{
    [TestMethod()]
    public void GetRegionsTest()
    {
        NorthinwindFactory target = new NorthinwindFactory(); // TODO: Initialize to an appropriate value
        List<string> expected = null; // TODO: Initialize to an appropriate value
        List<string> actual;
        actual = target.GetRegions();
        Assert.AreEqual(expected, actual);
        Assert.Inconclusive("Verify the correctness of this test method.");
    }
}

He then alters it like so, using the Microsoft snippet to guide him.

[TestMethod()]
public void GetRegionsTest()
{
    NorthinwindFactory target = new NorthinwindFactory();
    Int32 expected = 4;
    Int32 actual = target.GetRegions().Count;
    Assert.AreEqual(expected, actual);
}

When he runs it, the test passes.

He then adds a reference to System.Configuration

image

The WIP comments out the initial line and then add a functionally equivalent line using a .config file:

//String connectionString = @"Server=.;Database=Northwind;Trusted_Connection=True;";
String connectionString = ConfigurationManager.ConnectionStrings["NorthwindConnectionString"].ConnectionString;

He re-runs the unit test and it still passes.  Felling good, he checks the code back into source.  Unfortunately, when the build goes to TEST/QA, it breaks because they have a special .config they use and that build server doesn’t have that database installed.  After a couple of hours of wrangling, WIP gets the correct .config into all environments and includes Northwind as part of the build.

WIP then reads that unit tests shouldn’t have external dependencies like databases and .config files.  To unit test right, he reads, he should use a Mocking Framework and he heard that MOQ is the best out there (editor comment: he’s right).  So WIP downloads MOQ and tries to mock his ConfigurationManager:

image

After a couple of hours of researching, he decides that the ConfigurationManager cannot be mocked.  Unwilling to give up on mocking, he fires up Rhino Mocks.  Alas:

image

Still unwilling to give up, he hits F12 on the Configuration Manager to see if it uses an interface that he can mock.  Perhaps an IConfigurationManager?  Nope.

image

so now the WIP has spent several of hours and he is nowhere with Unit Tests.

So forgetting ConfigurationManager, he decided to mock SqlConnection.  It starts out well enough:

image

and then

image

So that is good.  he can create a mock connection like so but he can’t use it without casting.

image

image

So he just can’t set the connection string.  This violation of the Law Of Demeter makes it really hard.  All he wanted to do is set a String value called ConnectionString and instead he has to learn about the internal workings of the DbConnection class.

So the WIP decided that he needs to re-write the entire method and use Entity Framework.  After all, EF is the new new data access technology and it must be mockable.  So the WIP goes out to this great blog post and is disappointed to learn that EF is even harder to mock than ADO.NET classes.

The WIP then turns his attention to his UI and tried to unit test/mock the visual data controls in ASP.NET.  Already half-expecting the answer, the WIP is not surprised

image

WIP probably gives up.

So whose fault is it?

The books?  Low. The point of the books are to teach the language, not good coding practices.  And basic books that don’t have a UI to demonstrate a concept (versus using unit tests) would take attention away from the primary focus – learning the language.  The most successful books have real working, albeit simple, UI projects.

The developer?  Low. WIP is hard-working, want to do the right thing, and are going to try unit testing.  He also has deadlines.

Unit Tests Advertising? Low.  True that unit tests have generally been forced upon dev teams by people better paid, with cooler job titles if not smarter than them.  Also, these smarter people also likely introduce different, convoluted techniques of hundreds of lines long or that use parts of the .NET language spec that only Phds understand to work around the fact that lots of the .NET APIs are untestable.

The projects?  Medium.  Many .NET projects do not have any business logic – they are simply CRUD UIs on top of a database.  Unit tests can’t tell you if the Sql string returns the expected result.  And in lots of projects, that is all it has.  Also, in lots of .NET projects, any logic that could be tested is intermingled with the untestable code.  In the example above, the assumption tha GetRegions() is testable is false.

The API?  Absolutely. Every major class in Microsoft’s data stack are un-mockable and therefore un-unit testable.   Would it really kill MSFT to have an interface for these classes because perhaps, just perhaps, the classes won’t be only used in the way that they were thought of 15-20 years ago?  I get that Microsoft has been dragging code and outdated APIs though the different versions of .NET and it has to be frustrating to them that they can’t introduce changes easily.  However, until Microsoft starts making the classes that everyone uses Mockable, unit testing will not take off.  And it is not just ADO.NET, WinForms, WebForms, System.IO, etc… all are un-mockable.

So some people might ask.  Why not use VS2012 MSFT mocks?  Well, that is a clown show.  It flies in the face of iterative development – every new object, regenerate your mocks.  Also, the syntax pales in comparison to MOQ.  It is a compensation for poor class design that has been dragged forward – interest on technical debt if you will…  I think Microsoft should change their APIs, not build tools to get around their API’s failings.