Hector Correa's blog

Functional Asynchronicity Explained

hector@hectorcorrea.com

Over at the Pragmatic Bookshelf Trevor Burnham has a great post titled A CoffeeScript Intervention in which he shows several code snippets that demonstrate how CoffeeScript code is typically smaller and cleaner than the equivalent JavaScript code.

In particular I love the example under the Functional Asynchronicity section. In this section he shows a piece of code that trips most new JavaScript developers.

for (var i = 1; i <= 3; i++) {
  setTimeout(function() { console.log(i); }, 0);
}

When this code is executed it will output 4, 4, 4 to the console rather than 1, 2, 3 as most people guess the first time they work with JavaScript.

Although in his original blog post Trevor explains the reason for this behavior, in this blog I would like to expand a little bit more on the reasons why we get 4, 4, 4 rather than 1, 2, 3 since I believe they are fundamental to both understand and appreciate CoffeeScript.

There are two reasons why we get 4, 4, 4 in the previous code. The first reason is related to the way JavaScript’s setTimeout function works. The second reason has to do with the way JavaScript picks the value of i when the console.log(i) line is finally executed.

Understanding setTimeout in JavaScript

In JavaScript the setTimeout() function is used to execute a piece of code sometime in the future. In his book CoffeeScript: Accelerated JavaScript Development (p. 108) Trevor sums it up like this:

setTimeout always adds its target to the “event queue,” which isn’t invoked until after all the other code has run.

In our particular example this means that the function() { console.log(i); } (aka the target) will be queued up and executed when the for loop has ended -- not on each iteration.

This is true regardless of the timeout value used. Since we used a timeout value of 0 ms the target functions would be executed immediately after the loop has ended. If we had used a value of 1000 ms they would be executed 1000 ms after the loop ended. The important thing to recognize is that the targets won’t be executed until after the loop has ended.

This thread in stackoverflow.com gives a great overview on deferred execution in JavaScript and it’s worth a look if you are not very familiar with this concept.

The value of "i"

Now that we know that the calls to console.log won’t happen until after the loop has ended, we need to understand how does JavaScript determine what value of "i" would be used at that time.

We know that at the end of the loop the value of "i" would be 4. That’s easy enough to understand. However, what most JavaScript newcomers fail to grasp is that when we create an inner function that references an existing variable, the inner function will actually have access to the original variable when is eventually executed. Let’s spell this out.

The first thing to notice is that we are not passing the value of “i” to the setTimeout function. Instead we are creating and passing a function to setTimeout.

Secondly, the function that we are creating and passing to setTimeout references a variable “i” which exists in the context in which we are creating it. This is a key concept called closure and not found in most programming languages.

In his book JavaScript: The Good Parts (p. 38) Douglas Crockford defines closure as

the ability of a function to access the context in which it was created

In our case the new function (which just outputs the value of “i” to the console) was created in a context that has an “i” variable defined and therefore the new function will have access to the value of “i” whenever it is executed. Notice that this is very different than assuming that the new function will have the value of “i” at the time the new function was created!

We know from our review of setTimeout that the target (i.e. the new function that outputs "i" to the console) won’t be executed until after the for loop ends and that time the value of “i” will be 4. This explains why we got 4, 4, 4.

On his book Crockford also has a great example showing how a variable lingers after the original function has completed if the variable is referenced by any of the inner functions. In fact the whole context in which the inner function was created is what lingers. Definitively worth reading if you are not familiar with closures in JavaScript.

Yeah, but that’s still not what I wanted...

So now that you understand why JavaScript outputs 4, 4, 4 in our example rather than 1, 2, 3 you are probably wondering but how can I change this behavior.

Basically what you need to do is create a new context for the function() { console.log(i); } and make sure that in this new context the value of “i” is locked to the value of the iteration (1, 2, 3.)

Here is how Trevor does this on his blog post:

for (var i = 1; i <= 3; i++) {
  (function(i) {
    setTimeout(function() { console.log(i); }, 0);
  })(i);   // execute it!
}

In this example a new anonymous function with an “i” parameter is created and executed immediately on each iteration of the loop. Notice the “(i);” in line number 4 means we are executing it.

When this new anonymous function is executed on each iteration it calls setTimeout and creates a new function to output the value of “i” to the console - just as we did before. However, in this case the context in which the console.log will be executed will see the “i” defined as a parameter to this anonymous function -- not the “i” in the for loop!

In other words, the context in which the function() { console.log(i) } is created is now function(i) { … } rather than the original for loop. Therefore when console.log(i) runs it will pick the value of “i” from this new context and this new context receives a single value on each iteration.

Below is a slightly modified version of Trevor's code. In this new version I’ve renamed the variable “i” in the for loop to “x” to make it easier to see what we are trying to do:

for (var x = 1; x <= 3; x++) {
  (function(i) {
    setTimeout(function() { console.log(i); }, 0);
  })(x);
}

From this example it should be clearer that the we are creating a new variable “i” in the declaration of the anonymous function. The variable of the for loop is irrelevant because setTimeout and its target function will have a brand new context on each iteration and in this new context “i” will have the expected value (1, 2, 3) on each iteration.

The CoffeeScript Way

Trevor makes a great point on his post when he shows how much simpler the code to achieve the same result is in CoffeeScript:

for i in [1..3]
  do (i) ->
    setTimeout (-> console.log i), 0

In this case “do(i) ->” is doing the dirty work of creating a new function and executing it. Gotta love the simplicity of CoffeeScript syntax!

However, I doubt many CoffeeScript newcomers will really understand how this work unless they understand how JavaScript works behind the scenes. I hope this blog post helps to clarify what it’s really going on behind the scenes.

Sliding-in a Full Page Panel with CSS and jQuery

hector@hectorcorrea.com

In this blog post I show how to use CSS and jQuery to slide-in a panel full page on top of another and then slide back in the original content.

You can see a running version of the code presented in this blog post here. This demo page starts with a panel showing some part of the Lorem Ipsum text. When you click on the text a new panel displaying the Bacon Ipsum text slides-in on top of the previous text. Clicking on the new text slides the Lorem Ipsum back again. Nothing fancy but enough to show the slide-in functionality in a full page panel.

The HTML

The HTML used in this example is basically composed of two HTML divs. The first div (called mainDiv) includes the Lorem Ipsum text while the second div (called galleryDiv) includes the Bacon Ipsum.

<div id="mainDiv">Lorem Impsum text goes here</div>

<div id="galleryDiv">Bacon ipsum text goes here</div>

The CSS

The CSS to style these two divs is shown below. As you can see they are just plain divs using fixed positioning. The main difference between both styles is the background color (the mainDiv is beige and the galleryDiv is a tone of red.) Notice that initially the mainDiv is set to use all the real state (height and width are set to 100%).

The jQuery Code

The basics of the jQuery code to slide-in the one panel on top of another were taken from the excellent
Sliding Panel Photo Wall Gallery with jQuery example by Mary Lou at Codrops. I’ve simplified the code to make it easier to see what it takes to slide-in content on top of another back and forth.

When the page first loads we force the mainDiv to be visible and the galleryDiv invisible.

The next thing we do is to wire the mainDiv so that when the user clicks on it we slide-in the galleryDiv. The code for this is very simple too:

This code does the following:

Pushes the mainDiv to the back by setting its z-index to zero.
Pops the galleryDiv to the front by setting its z-index to something greater than zero, in our case to 100.
Sets the height of the galleryDiv to 100% using an animation (so that the slide-in effect is noticeable.)
Finally we collapse the mainDiv by setting its height to zero. This allows us to animate it when we slide-in back again.

The code to wire the galleryDiv to do the reverse is also very simple:

And that’s it

With just a few of lines of code we can easily slide-in content in our web pages. You can see this code running in this link. The page is self contained. View source and grab the code if you like it.

Update: A previous version of this code used the wrong kind of positioning on the mainDiv and galleryDiv styles and that resulted in weird behaviors when sliding content back and forth if the text was longer than what fits on the page (e.g. if the browser window was resized to be smaller than the text.) The current version that you are reading addresses this issue.

Future Directions for C# and Visual Basic

hector@hectorcorrea.com

These are my notes from Anders Hejlsberg session on Future Directions for C# and Visual Basic.

It's been more than 10 years since managed code was introduced back in PDC 2000. Every new release of C# has had one or two major themes:

Version 1: Managed Code was introduced
Version 2 2: Generics
Version 3: LINQ
Version 4: Dynamic features and language parity

The two major themes in the next release (version 5) are support for WinRT in Windows 8 and asynchronous programming.

Asynchronous Programming is becoming the norm in modern, connected applications.

About 10-15% of the WinRT API supports asynchronous operations. Sounds like a small percent but this percent is probably the most used stuff.

There are different asynchronous programming models in Windows 8 and .NET

WinRT uses AsyncOperation<T>
.NET uses Task<T>
JavaScript uses Promises

All of these models are objects representing "ongoing operations".

Callbacks turn your code inside out and are a big challenge but they have figured out how to automatically handle this so that asynchronous code "feel just like good old synchronous code".

The general pattern is:

public async SomeMethodAsync()
{
  // code before firing async operation
  await code to execute async
  // code to execute after async operation has completed
}

Use await operator to cooperatively yield control.

You can use Task.WhenAll() to run multiple tasks asynchronous in parallel!

It turns out that asynchronous method don't start new threads! I need to do more reading on this. Threading is only need when you run CPU intensive operations like a loop to perform math calculations.

Anders showed some JavaScript in which he consumed some C# code from within C# but he couldn't just use Task as-is and he had to bridge the two through IAsynchOperation (didn't quite get why)

Anders is obviously a big proponent of static languages and took several jabs at the dynamic nature of JavaScript. The funniest line was when he said:

“Sometimes you get lucky in JavaScript, you always get lucky in C#”

Another feature that is coming in C# is what is called Caller Info Attributes. These are built-in attributes that you can add to parameters to a method to receive the name of the caller file path, the caller line number and member name.

[CallerFilePath]
[CallerLineNumber]
[Caller/MemberName]

This is very nice for logging code.

Roslyn Project

Towards the end of the session Anders talked about what’s coming in C# 6.0 and a project named “The Roslyn project.” The basics of this project are:

Reimagine what compilers do
Compiler as a service
Open the compiler, make its information available to others (formatters, highlighters, refactoring, DSL)

With feature the syntax tree produced by the C# compiler becomes available to other tools so that they can peek inside the code with a true semantic knowledge of what the code does.

Anders shows a C# interactive prompt in which he was able to create a C# program on the fly, line by line, and execute each line as it went. This is nothing new to people that use JavaScript or Ruby but for a static language like C# this is huge! I was really impress on how clean the syntax to do this was.

The following picture (blurry) shows an example of what he did. In this code he declares a square function in a string and executes is in the program via a delegate. Wicked stuff.

Another example of using Roslyn that we showed was a feature to paste C# code as VB.NET code and vice-versa. This showed how the semantic of the original code was preserved.

There is a CTP available of Roslyn already.

The C# and VB.NET compilers are being rewritten in managed code.

8 Traits of Great Metro Style Apps

hector@hectorcorrea.com

These are my notes from Jensen Harris session at Build Windows conference on Tuesday September 13th, 2011. Jensen is the Director of Program Management for the Microsoft Windows User Experience Team.

Metro style design

There are no apps yet. Opportunity.
Sidebars are for the system, top and bottom are for the app
Win32 vs Metro apps
Content before Chrome
Apps re-imagined

Fast and Fluid

Touch faster than GUI, GUI faster than command line
Touch first approach
Beautiful, fast, and fluid vs crap
Most controls support animation
Touch has its own language, don’t mimic the mouse
Semantic zoom: not only does the content change, but also the meaning (e.g. show categories rather than details)
Don’t separate touch and mouse interfaces
A screen without touch is a broken screen - in a few years

Snap and Scale beautifully (form factor)

minimal: 1024x768
widescreen: 1366x768+
snap view (required)
portrait (optional)

Use the right contracts

Binds metro style apps together
Share, Search, and Picker
Share content with other apps (e-mail, social network)
Search: each app implements its own view
Picker: select a picture from file system or the web (sample: change user profile picture from a picture in Facebook without having to do the intermediate step of saving picture in my hard drive first)

Invest in a great Tile

Icons are yesterday’s way of representing apps
The Tile should be an extension of the app
You can also create tiles from content of the app (e.g. tile your favorite photos album or weather from SCE)

Feel connected

“Windows is alive with activity”
Live tiles
We don’t have folders, we have groups
Folders put stuff down one level, it’s hard to name them
People do love to make groups
Notifications are displayed for a small period of time, after that update the tile

Roam to the cloud

Every app gets per-user cloud storage for settings, state, and small amount of user content (via Live ID)
Applications preserve state as they go, do not require explicit save
Roam settings (favorite cities, last level on game played)

Metro style design principles

Other sessions to check out: 207, 211, 391, 395, 396, 405, 406

Simple Branching Strategies for Team Foundation Server

hector@hectorcorrea.com

In this blog post I describe some branching strategies that I've found useful when using traditional source control systems like Team Foundation Server (TFS).

Team Foundation Server has very good branching and merging capabilities that can help teams manage source code in a team environment. However, despite the fact that TFS provides excellent branching and merging capabilities, I usually recommend a simple approach to branching since the process of branching and merging can easily get convoluted and out of control even with the best of tools.

The basic idea

The way I approach branching in TFS is to have one main branch for all on-going development on the project. This is the branch where developers do their work on a daily basis. Once the team reaches the point where the code will be released to production then a second branch (a release branch) is created. This release branch is a copy of the code that will be pushed to production. The main branch stays active and most of the team will continue working of the main branch as they add new features to be released in a future version of the system. A few of the developers will make changes to the release branch to fix bugs that must be ironed out before the release goes out or even after it has been promoted to production.

The following picture shows an example of these concepts and how the branches are used over time.

In the previous picture we can see how the team worked on the main branch from week 1 through week 4 (the green bars.) At the end of week 4 the team decided this code had all the features needed for a version 1.0 release and created a release branch with these features.

During the next two weeks the team made a few changes to the code in the release branch, perhaps fixed a few show-stopper bugs before promoting to production. Meanwhile, the rest of the team continued working on the main branch on long term features to be released in version 2.0 (the blue bars.)

The team worked from week 5 through week 12 on the main branch (the blue bars.) During this time there were two hot fixes pushed to production from the release branch (green bars on week 9 and week 11.) The nice thing about the release branch concept is that it gives us total isolation from the on-going long term development on the main branch. Since the release branch is an exact copy of what's in production updating this branch when a hot-fix is needed is very safe and easy.

On week 12, the team decided that they were ready to promote the next major version of the system to production. A that time the main branch was branched out again to the release branch. From this point the cycle is repeated: hot fixes for version 2 are made on the release branch while all new development for version 3.0 is done on the main development branch (now the purple bars.)

Variations to this approach

In some teams we have made a small variation to this pattern in which a totally new branch is created every time a new major version of the system is deployed to production. For example, we could create a totally new branch on week 12 for release 2 rather than sticking the version 2.0 code on the existing release branch. The advantage of this variation is that, if at the end of week 12 we need to do a hot fix to version 1 (while version 2 is still in QA) we can easily make the fix on the original release 1 branch since that branch is still an exact copy of what's in production. This could be a life saver and worth consideration.

I first read about the concepts of the main branch and a release branch in the book Software Configuration Management Patterns: Effective Teamwork, Practical Integration by Stephen P. Berczuk and Brad Appleton. This book is a good read if you want more details on the approach that I describe in this post. Chapter 4 and 5 describe the Mainline and Active Development Line patterns while chapter 17 talks about the Release Line.

Merging - the dark side of branching

Regardless of how neat your branching strategy might be the real issues are found in the merging process. For example, in the previous picture I didn't indicate how changes made on weeks 5 and 6 on the release branch will be merged back to the main branch. The same is true on weeks 9 and 11 when hot fixes were applied to the release branch. If these changes are also needed in the main branch somebody will need to go perform this process.

There is nothing inherent to the branching strategy that I describe in this blog post that makes the merge process easier. Depending on the nature of the changes merging can be a messy process. The best that you can do with merging is to implement a workflow in which merging conflicting changes is minimized. However, since inevitably you will have conflicting changes to merge I find Jennifer Donnelly's advise on the subject rather valuable: merging should happen frequently and within context, in other words, by the person who made the change and as soon as possible. On small teams this is usually easy to do and prevents the merging process from becoming a disaster.

In closing...

The ideas that I've described in this blog are neither new nor the only options for branching. However, I've found them very useful for small teams working with centralized source control systems like Team Foundation Server. The fact that these ideas are simple to implement allow development teams to gain the most of branching (i.e. code isolation) without having to have a dedicated staff in charge of source control and merging.

Although I suspect these ideas apply to most traditional client-server source control systems I have only apply them with TFS. I am not sure however how well they fit with a distributed source control system like Git.

Running Visual Studio inside Mac OS X

hector@hectorcorrea.com

Lookie here...I am running Microsoft Visual Web Developer Express on my MacAir!

When I bought my MacAir last year to start learning Ruby and Ruby on Rails I wasn't sure how much I was going to really use it. Overtime, I've gotten familiar with it and little by little it has become my main machine at home to do pretty much everything including Ruby programming, managing my music, pictures, and documents, and web browsing. It's gotten to the point that the only reason I power up my Windows laptop at home is when I need to update one of my personal ASP.NET projects or research some C# topic in particular.

My Windows laptop at home is a nice small Lenovo 3000 v200 that is about 2 or 3 years old. It's a very nice laptop but compared with the MacAir it feels like a clunker. There isn't anything wrong it per-se, it just feels like an old machine.

As time went on I started pondering how to run Visual Studio inside my Mac so that I don' have to rely on two machines. There are many options to host virtual machines inside OS X including the built-in Boot Camp software that comes with the Mac, VMWare, and VirtualBox.

In my case I decided to use VirtualBox to create a Windows 7 virtual machine where I can install Microsoft Visual Web Developer Express 2010.

The following screenshot shows Visual Web Developer Express running inside a Windows 7 virtual machine hosted on OS X.

I decided to use VirtualBox over the other virtualization options for two main reasons. One is that it's free (that's hard to beat) but also because is multi-platform which allowed me to create the virtual machine on my Windows machine (where I have plenty of disk space) and once it was ready just transfer VM to my Mac and power it up there.

Although I configured the virtual machine that emulate a hard drive with 20 GB the VM itself only uses 12.5 GB with Windows 7 and Visual Web Developer Express installed. That was perfect for me as my MacAir has a very small solid state drive to begin with. Being able to create the virtual machine on a different machine with plenty of disk space was a big plus for me.

Now I am able to work on all my home projects on the same machine regardless of whether I want to program in Ruby or in ASP.NET. I am not quite ready to sell my Windows laptop yet, though. We'll see if that changes in the future.

Finder Error -36 on a NAS

hector@hectorcorrea.com

Ever since I got my MacAir late last year I've been having issues copying files from my Mac to my external storage device. My external storage device is a Buffalo HD-CELU2 DriveStation that can be accessed as a standard USB drive or as network attached storage (NAS.) I've been using this device as a NAS from my Windows computers for a couple of years with no problems but my Mac refused to work smoothly with it.

On my Mac I was able to read files from the NAS but I couldn't write files to it. When I tried to copy files to the NAS I got the dreaded and cryptic "Finder Error -36". There are several people reporting similar issues when using a NAS from their Macs and it took me several months before I found a solution that worked for me. Below is an example of the error that Finder will throw when I tried to copy a file (SamplePDF.pdf) to my NAS (hd-celu2-fce7).

The clues

One of the things that I noticed in dealing with this issue was that the Buffalo DriveStation worked smoothly when I connected it directly to my Mac's USB port (rather than accessing it through the network.) When I used the device through the USB port I was able to read and write files with no problems. This lead me to believe that the problem was somehow related to the way the Mac was accessing the device through the network rather than the drive itself. If the drive was formatted in a way that was incompatible with the Mac I would have the same problems no matter how I accessed it.

Here is a description of the issues that I noticed when I connected to the Buffalo DriveStation as a network drive:

My Mac computer can read files from the NAS with no problem.
My Mac cannot write files from the NAS. I get the infamous "Finder Error -36".
My Windows computer can read from the NAS and write files to it with no problem

Although I was able to read and write with no problems if I connected to the device through my USB port, using it this way defeated the purpose of having a NAS in the first place. I wanted to be able to connect to the device through my wireless network at home rather than having to drag the darn thing with me around the house.

I also noticed that the problem was not exclusive to Finder, I would also get an error when I tried to copy files to my NAS through the Terminal as shown in the following screenshot. In Terminal I was able to work around this problem by using -X switch on the copy command as shown in the following screenshot. Notice how the first copy failed but the second succeeded. The -X switch prevents the copy command from copying Extended Attributes or resource forks.

The solution

I found the solution to my problem at macwindows.com under TIP: Another take on smb.config fix for -36 file sharing error: edit nsmb.conf for "streams=no". You basically need to edit a configuration file to tell the Mac not to use streams when connecting to the NAS. In particular you need to add the following two lines to the /etc/nsmb.conf file:

[default]
[streams=no]

If the file does not exist on your machine (it didn't on mine) you will need to create it. Editing/creating this file is bit tricky since you need elevated permissions. The macwindows.com link has detailed steps on how to do this.

The aforementioned lines will tell the Mac OS X to turn off the use of NTFS streams when connecting via SMB to a device.

As it turns out, when the Mac connects to the Buffalo DriveStation through the network it uses a protocol called SMB. That's the reason you see "SMB://somename/somepath" in Finder when you connect to an external storage device. One of the things that I learned on the macwindows.com page is that SMB is a protocol to access drive and other devices (just like FTP or HTTP) and file nsmb.conf file configures the SMB client that comes with the Mac (see SMB vs. Samba, and what Apple uses for file sharing.)

An even simpler solution

I found an even simpler way to turn off streams in this article on Apple's web site: Mac OS X v10.5, v10.6: About named streams on SMB-mounted NAS, Mac OS X, and Windows servers; "-36" or "-50" alerts may appear. It turns out you can just create file on the device where you want to turn streams off by using a command like this:

touch /Volumes/SharedNAS/.com.apple.smb.streams.off

where SharedNAS is the name of your shared device. The Apple article also includes a great explanation of what the heck it means to "turn off streams" when connecting to an external device through SMB.

Are there any side effects?

I've been using my NAS device from my Mac with no problems for almost two weeks since I turned off streams with the solutions in this blog post. I can now read from and write to the NAS from both my Mac and Windows computers. The only side effect that I've seen so far is that, since streams are turned off, the Mac OS X creates two files on the NAS for every file that I copy to it. These double files are only visible when I access the NAS device from my Windows computer.

The aforementioned article on the Apple web site indicates that "named streams are used to store Mac OS X extended attributes and can be leveraged to avoid using AppleDouble files to store the data fork and the resource fork of legacy Mac files" so it makes sense that (1) the files are now being created since we turned off streams and (2) these double files are visible from a non OS X operating system.

The following screenshots show how the folder with the SamplePDF file is displayed in Finder and in Windows Explorer

This double file nuance is not a big deal for me specially given that for many months I couldn't even write files to my NAS from my Mac.

Drawing a Binary Tree in Ruby

hector@hectorcorrea.com

When I started learning Ruby last year I decided to implement a binary tree and some of its basic operations (insert, delete, walk, and search) just to get my feet wet on the language. Binary trees are a good exercise because you need to use several features of the language like conditional statements, loops, and classes while at the same time you are solving an interesting problem. The algorithm for binary trees is well documented in most introductory books on algorithms and on the web.

To make things a bit more challenging I also implemented a program to draw the binary tree so that people can see how a binary tree looks like when it has more than just a handful of nodes. In this particular case I wrote small a web application in Ruby that draws the binary tree using the new canvas element supported in HTML 5. You can see a live demo of the Ruby web site which allows you to draw binary trees with our own set of values or you can just let it draw binary trees with random values. For demo purposes trees on the demo site are limited to 250 nodes but you can draw much larger trees if you download the code and host it on your own.

This blog post gives short overview of the main methods to implement the binary tree, the algorithm to draw the binary tree, and the web site to demo the drawing algorithm.

Binary Tree or Binary Search Tree

Properly speaking the code in this blog post implements a Binary Search Tree (BST) which is a more specific version of a Binary Tree. Cormen et al. give a good explanation of this on their book Introduction to Algorithms. Binary trees are trees in which each node has at most two children, one of them called "left subtree" and the other called "right subtree" but there is no order enforced per-se (pp. 1178). Binary Search Trees on the other hand are always stored in a way as to satisfy the binary-search-tree property (pp. 287):

Let x be a node in a binary search tree. If y is a node in the left subtree of x, then y.key ≤ x.key. If y is a node in the right subtree of x, then y.key ≥ x.key.

I took a bit of a liberty on this blog post (and its related source code) and use both terms interchangeably. Please be aware that all of the references to binary search in this blog post are in fact references to binary search trees.

Implementing a Binary Search Tree

The procedures to implement the basic binary search tree operations like insert, delete, search, and walk are well documented in most introductory books on data structures and algorithms and on the web (e.g. www.algolist.net and wikipedia) and they are fairly easy to implement in Ruby.

The class that I implemented can be used as follows:

Out of the box we can use this BinaryTree class with numbers or strings or any other native Ruby type. We can also use it with other types as long as they implement comparable operations (i.e. def <=>).

Adding Nodes to a Binary Tree

The code to add nodes to the binary tree is shown below. This code first evaluates if we already have a root node, if we don't then we create a root node with the new value and we are done. If we already have a root node then scan the nodes of the tree (using the binary-search-tree property aforementioned, smaller values go to the left, equal or greater than values go to the right) until we reach a leaf node. Once we've reached a leaf node we update it to point to the new node.

Walking the Tree

One of the advantages of a binary search tree is that it's very easy to retrieve in order the values stored on it. This process is called "walking the tree in order". For example, if we have a tree with the following values: 100, 50, 200, and 150 walking the tree in order will give us: 50, 100, 150, and 200. Walking the tree can be implemented easily with a recursive function. However the recursive method, although elegant, it's not very efficient when the tree is expected to have a large number of nodes. The code that I implemented to walk the tree does not use recursion but instead uses an array as a stack to keep track of the nodes that it visits. This version is not as elegant as the recursive version found in most introductory books but it works well even when the tree has thousands of nodes.

Below is the code for walking the tree in order that I implemented.

While implementing this method I got to experience first hand the beauty and simplicity of one of Ruby's most touted features: blocks. You can see the use of blocks in the line yield node of the previous code. As the algorithm finds the next node to process it simple yields it to the caller program. This allows the code to walk the tree to be completely independent from the action that we want to perform on each node. For example, we can print each node to the console with the following code:

tree.walk { |node| puts "#{node.value}" }

In this example the block is a simple "puts" statement but we'll see a more sophisticated example of this when we review the code to draw the binary tree. This kind of decoupling between the iteration of the elements in the tree and the action to execute on each of them can be implemented in other programming languages via delegates or pointers but I was amazed on how simple and elegant it was to do this in Ruby.

How to Draw a Binary Tree

The algorithm to draw a binary tree that I implemented is pretty much a variation of the algorithm to walk the tree with the exception that we need to calculate the coordinates where each node needs to be placed as we traverse the nodes in order. Calculating the coordinates turned out to be an interesting challenge but a final simple solution.

The basics of this algorithm are as follow:

Draw the root node in the coordinates indicated.
Draw the left subtree to the left of the root node.
Draw the right subtree to the right of the root node.

To calculate the coordinate where the left subtree should be drawn we count how many children the right side of the left subtree has. We use this number to calculate the x-coordinate as a negative offset from the root. The y-coordinate is simple a positive offset from the root. Then we recurse this process for the left subtree and right subtrees. The following code snippet shows this code.

The procedure to calculate the coordinate where the right subtree should be drawn is just the reverse: we count how many children the left side of the right subtree has. We use this number to calculate the x-coordinate as a positive offset from the root. The y-coordinate is simple a positive offset from the root. Then we recurse this process for the left subtree and right subtrees. The following code snippet shows this code.

Like in the walk method, the three draw methods presented here only calculate where the node should be drawn but they don't actually draw the node. Instead they call a block with the appropriate parameters to perform the actual operation. You can see this in the line block.call(node.value, x, y, parent_x, parent_y) in these three routines.

The following code shows an example of calling the draw method on a simple tree and displaying on the console the coordinate where each of the nodes should be drawn

There is plenty of research on the best way to draw binary trees with lots of data. The book Graph Drawing: Algorithms for the Visualization of Graphs by Tollis et al. seems to be the reference that most research papers and presentations cite. On pages 43-45 they provide a more sophisticated version of the algorithm that I used plus several advanced algorithms for different scenarios.

Perhaps in the future I'll update my code to implement the enhancements suggested by Tollis et al. and also to remove the recursive calls from my implementation.

A Real Example of Drawing a Binary Tree on a Web Page

Once we have the coordinates where each node can be drawn we could use any graphic program to perform the actual drawing. Since I wrote this program as learning exercise I decided to write a small Ruby web application to do the actual drawing. In this web application I use the new HTML 5 canvas element as the drawing surface and a series of JavaScript calls to draw lines and circles on the appropriate coordinates.

The following code shows a simplified version of how I generate the JavaScript calls to draw the nodes (circles and labels) and lines connecting the nodes.

Notice that this code merely creates three arrays (circles, lines, labels) with some JavaScript calls. These arrays are later inserted on a web page so that when the user renders it the tree is drawn on their browser.

Source Code and Live Demo

If you want to the see this code running you can visit http://binarytree.heroku.com and generate a few binary trees with random data or draw a binary tree with your own set of data.

The demo site is limited to 250 nodes but on my local environment I've drawn trees with up to 10,000 nodes. One of the disadvantages of the drawing algorithm that I implemented is that it's not an area-efficient algorithm which leads to very wide drawings. You can see this effect in this tree with 500 nodes. A more dramatic example is a tree with 5,000 nodes that I generated with this code too. If I were to print this tree it would be 115 feet long by 1 foot tall (as a reference, a basketball court is 94 feet long!) You can download this tree from http://hectorcorrea.com/downloads/tree5000.png but be aware that it's almost 4MB in size (which is why I am not making it a hyperlink.)

You can find the entire Ruby source code to implement the binary tree and the binary tree drawer, as well as the demo web site on github at https://github.com/hectorcorrea/binary-tree. Feel free to check it out and let me know what you think.

Git Basics

hector@hectorcorrea.com

This blog post is a beginners' guide (written by a git beginner) on how to perform basic version control operations with git including initializing a repository, adding files to it, and using branches.

The goal of this post is to describe how to use git in your local machine for version control. In a future post I'll cover the basics of using git and github for collaboration with other developers.

Everything that I show on this blog is done through git's command line via the Mac OS X terminal. The steps should be very similar in other operating systems but I have not tested them outside of the Mac.

Is Git Installed on my Mac?

We can issue the following commands from the terminal to find out if git is installed and what version we have installed:

which git 
git --version

The command which git tells us if git is in our path and should return something like "/usr/local/git/bin/git". If it returns nothing it means that git is not in our path and probably not installed in our machine.

If git is installed, we can issue git --version to figure out what version we are using. The following screenshot shows what I got on my machine:

If git is not installed on your Mac you can use the pre-compiled installer to install it on your computer.

Creating a Git Repository

Once we have git installed on our machine creating a repository for version control is pretty simple. Let's start by creating a brand new test folder and asking git for its status.

mkdir testgit
cd testgit
git status

The previous call to git status will return something along the lines of "Not a git repository (or any of the parent directories)" which is OK since we haven't told git anything about this folder.

To put this folder under version control, or in git terms, to create a local repository, we can simply issue the git init command. This command will create several hidden folders and files inside our testgit folder. Git uses these folders and files to keep track of changes to files. If we issue git status now, we will see that git reports a status rather than an error like it did before we issued git init. The following screenshot shows the results of running these two commands on my machine:

Notice how git reports that a branch called master has been created and that there is nothing to commit at this point. This folder is now officially under version control and everything that we do inside of it will be monitored by git.

Adding Files to a Git Repository

Let's start by saving a text file named "file1.txt" into this folder and asking git again for the status of the folder. This time git will report something like the screenshot below

Notice that git is now reporting that the new file (file1.txt) exist in the folder but has not been added to the repository, or in other words, the file is not under version control yet.

Let's go ahead and issue the git add file1.txt command as suggested in the previous screenshot and then check the status of the repository again. The following screenshot shows what git reports now. Notice how git now reports that there is one new file to be committed.

By adding a file we let git know that this file now needs to be tracked for version control but it's content hasn't been added to the repository. We need to issue another git command in order to actually commit the file to the repository.

Committing Files to a Git Repository

To commit to the repository new files or changes to existing files we simply need to issue a command like git commit -a -m "first file added". The following screenshot shows the result of committing this file and getting the status of the repository again.

At this point we can add more files to the testgit folder, add them to the repository with git add and commit them with git commit and we are pretty much up and running with git.

Likewise, we can make changes to files already in the repository (e.g. to file1.txt) and commit them with git commit. We don't need to do anything special to be able to make changes to files already in the repository. We just edit the file and when we are ready we commit the changes. The following screenshot shows how an example of this.

A note for Microsoft Visual Source Safe (VSS) and Team Foundation System (TFS) users: In git, the term commit is similar to the concept of checking in a file in VSS and TFS. However the term checkout in git refers to the process of switching to another branch which is totally different from the meaning of in VSS and TFS. Keep this in mind.

View History of Changes

Now that we have committed a few changes to our repository we can use several git commands to review what has changed, when each change took place, and who made each change. For example, the git log command gives us a list of commits that have been performed on the current branch. From there we can use the git show command to find out what commits affected a particular file or the git whatchanged command to review what were the specific changes made to a file on a particular commit. We can also use the git blame command to find out who made changes to a file (you've got to love a version control system that has a blame command.)

There are a lot of options that can be used with any of these commands and an entire blog post could be written about any one of them. The best place to get more information is probably the git docs web site. Below is a screenshot of what git log displays in our sample after a few commits.

It's important to notice that aforementioned commands are command line tools and their results, although comprehensive, are not exactly easy to navigate or review since they are just plain text.

However, on the Mac OS X git comes with a graphic user interface to review the changes made to a particular branch. You can invoke this tool by issuing gitk on the terminal. Gitk displays the history of changes in a format that it's easier to navigate than plain text but in my opinion is still very rudimentary compared with what you can do in other version control systems like TFS. There are perhaps better tools out there that might be worth researching. Below is a screenshot of how gitk displays the history of changes in our repository. In this screenshot we can see that (1) on the second commit (2) file1.txt was changed and (3) the specific changes that were made to this file.

Using Branches

One of the biggest advantages of using a version control system for source code is that it allows you to make changes to your projects to test things out without having to worry about messing up a good working version of your system. By using branches you can create "sandboxes" from a good version of your source code, play on your sandbox until you are happy with your changes and then merge the changes back to the good version of your system...or you can also discard the sandbox altogether if you end up not being happy with the changes. As the git tutorial page says "branches are cheap and easy, so this is a good way to try something out."

When we first added a file to our repository git created a branch called master. You can see this by issuing the git branch command from the terminal.

If we want to create a new branch called "sandbox" from the current master branch we just need to issue git branch sandbox. This will create a new branch called "sandbox" with an exact copy of the code in the current master branch. After this if we issue the git branch command again we should see both branches listed as in the following screenshot. The asterisk next to the master branch means that we are currently on the master branch.

To switch to the sandbox branch we issue the git checkout sandbox command. If we issue the git branch again the asterisk will be next to the sandbox branch.

Any changes that we make to the files from this point forward will be localized to the sandbox branch until we switch back to the master branch. Let's say for example that we edit file1.txt and commit these changes using the regular git commit command. The following screenshot shows how the changes to file1.txt on the sandbox branch have been committed but the file still shows the original content on the master branch.

We can use the git merge command to merge the content of one branch with another. For example, let's assume that we are happy with the changes that we made on the sandbox branch and we want to bring them to the master branch. The following commands will merge the contents of the sandbox branch back into the master branch:

The merge command will automatically update on the master branch any files that were updated on the sandbox branch, it will also add any news files to the master branch that might have been added to the sandbox branch and delete any files that were deleted on the sandbox. You can use some additional arguments with the merge command to configure exactly what should be merged.

If the merge command detects that some of the files to be merged changed in both branches, git will do an automatic merge and ask you to verify the changes and perform a manual commit with git commit. Take a look at the git merge documentation for more details on this.

Summary

In this blog post we've seen how to get started with git. We went from creating a brand new repository, committing changes to it, reviewing the history of changes that we made, and created a separate branch to test changes before merging them back to the master branch.

As I mentioned at the beginning of this blog in a future post we'll explore how to use git to collaborate with others and work on remote repositories like github.

References

There are plenty of good references to find more information about git, the basic commands that we've reviewed in this blog entry, and advanced git topics. Below is a selected list that I've found useful.

This Git Tutorial is a great place to start. I learned a lot of what I posted on this blog in this tutorial.
More comprehensive documentation can be found on Git User's Manual.
The official git web site has also a ton of documentation that I've found useful.

Update:Lo and behold, today I found by accident the Git Community Book. This is a fantastic reference that explains in great detail some of the beginner and intermediate workflows. I highly recommend you take a look at it. It's too bad this book (although it lives under the http://git-scm.com/ domain) is not listed on the "documentation" section. Sigh.

Ruby Development on the Mac OS X

hector@hectorcorrea.com

A little more than a month ago I got myself a MacAir. Even though I’ve own computers since the mid 80s this is the first time that I buy a Machintosh. The reason I bough a Mac rather than a PC is because I wanted to start experimenting with the web development tools that exist for non-Windows environments. Although I’ve been a software developer for more than 15 years all my experience is with Microsoft Windows and I thought is about time I start experimenting with the tools that “the “other half of the world” is using.

Since I got a Mac most people would probably thing that I want to start writing applications for the iPhone or the Mac but in reality I want to experiment with the development tools for web applications. I will probably write an app or two for the iPhone or the Mac just to see what it’s like but my real motivation is web development using non-Windows tools.

This is not to say that I am abandoning development with Microsoft’s tools (or the Windows ecosystem) but rather I want to experience what else is out there. There are plenty of successful web sites (e.g. Google and Facebook) that are developed using non-Windows tools and I want to see what the development experience on those tools is like.

The programming language that I decided to learn first is Ruby along with the popular web development framework Ruby on Rails. Over the following months I will be posting my experiences developing applications on the Mac OS X using Ruby and Ruby on Rails from the perspective of somebody that is very familiar and comfortable with Microsoft/Windows development.

Why Ruby?

When I decided to learn a new language I wasn’t too thrilled to learn Perl, PHP, or Phyton. I don’t really know why, the thought of learning any of those languages just didn’t excite me despite the fact that they are the ones used some of the most successful sites including Google and Facebook.

But Ruby gave me a different vibe, people seem to be very passionate and excited about Ruby and Ruby on Rails. Both of them seem to be quite popular to build web applications which is what I am most interested in. In addition a lot of the ideas for Microsoft’s ASP.NET MVC (which is my main development platform these days) were taken from Rails which made me think that there is definitively a lot to be gained from learning both Ruby and Rails.

In his presentation 10 Things Every Java Programmer Should Know About Ruby Jim Weirich has this great quote:

“A language that doesn’t affect the way you think about programming is not worth knowing — Alan Perlis”

So What Does Ruby Offer? (or How I Learned to Stop Worrying and Love the Bomb)

The Ruby programming language is introduced like this in the ruby-lang.org web site:

“A dynamic, open source programming language with a focus on simplicity and productivity. It has an elegant syntax that is natural to read and easy to write.”

In digging deeper I found that Ruby is object oriented, provides garbage collection, borrows some interesting features from functional languages (which is something I’ve been meaning to look into), and it has indeed a clean syntax.

However, there were also a few things in Ruby that I wasn’t too excited about. In particular the fact that Ruby is an interpreted language and that it is not statically typed (although it is strongly typed.) After writing code in C# for so many years I had a hard time imaging myself using a programming language that didn't offer these features.

With my background in C# I found the slides for 10 Things Every Java Programmer Should Know About Ruby by Jim Weirich and Ruby for Java Programmers by Mike Bowler very interesting. Both of these presentations are packed with a lot information on what makes the Ruby language different from Java/C# and highlight its strengths. Some of the topics that they cover include:

The Dynamic nature of Ruby makes factories and mock objects trivial
Reflection is also taken to a new level (compared with what you can do in a statically typed language like Java or C#)
Nil is an object which means you never get a null pointer
Closures: an object that is a block of code

Jim’s slides also address my biggest concern (namely that Ruby is not statically typed) with a great quote from Bob Martin in which he mentions how the safety net provided by static typed languages is not as important when your code is covered with unit tests as those test would catch type errors for you. I am not sure I agree with this statement 100%. It is certainly an interesting point, though.

You can find these and other interesting articles at the ruby-doc.org web site.

Ruby on the Mac OS X

One of the nice things about OS X is that out of the box it comes with the runtime libraries for several programming languages including Perl, PHP, and Ruby.

This is really nice feature for me since that means I can start playing with Ruby without having too worry too much about where to get the binaries or how to compile the source code. In my case my Mac came with Ruby 1.8.7.

Beginning Ruby

I have found the book Beginning Ruby: From Novice to Professional by Peter Cooper a great resource to learn how to program in Ruby and get going.

Peter does a great job of covering everything from how to make sure Ruby is installed on your machine, how to install Ruby, very basic programming topics (e.g. what is a class) to more advanced topics like exception handling, database access, web development or Ruby-specific techniques like how to implement enumerators and the yield operator, mix-ins, and code blocks, development frameworks for Ruby like Rails and Sinatra.

I highly recommend this book to anyone starting with Ruby.

Development Tools for Ruby

The fact that the Ruby interpreter comes built in with the Mac OS X is great, but what about editors, debuggers, IDE, source control, and other development tools. For somebody like me used to a great IDE like Microsoft’s Visual Studio I was shocked to find that there is no “standard IDE” for Ruby on the Mac or any other operating system.

From what I’ve been able to gather, most Ruby developers are comfortable editing their code in a variety of text editors that provide color syntax, basic statement completion, and indentation but not much more. There does not seem to be a “visual debugger” like what Visual Studio provides for C#, there isn’t the concept of IntelliSense either. Despite this Ruby developers seem to be able to crank up code pretty fast.

Editors

You can download for free the Mac OS X development tool (called XCode) that is a full blown IDE and supports color syntax for Ruby but not visual debugger or much more integration.

I’ve found two text editors that seem to be both popular and powerful for Ruby development:

TextMate seems to be one of the favorite editors for Ruby (and Rails) development on the Mac as it provides really nice keyboard shortcuts for code completion, running unit tests, integration with source control (both SubVersion and Git) and other goodies. This has been my favorite editor so far. It is not free but I will probably end up buying a license in the next few days.
TextWrangler is a free text editor and supports color syntax for Ruby. Somehow I found the workflow for Ruby development not as good with this editor as with TextMate but you cannot beat the price.

Here is an screenshot of TextMate

There are plenty of other editors that people use on the OS X to edit Ruby code that are common among *nix users including Vim and Emacs. Yet, for somebody like me, with a Windows background, those editors look and feel like something from the stone age. They can be very powerful once you learn them but the learning curve might be very steep.

Visual Debugger

There is not a visual debugger like the one in Microsoft Visual Studio for Ruby on the OS X or other *nix environments. This has been one of the biggest shocks to me as I’ve gotten used to have one and still miss it.

One can make the argument that if your code is testable and properly structured (small methods, classes with single responsibility) the need for an advanced debugger is less critical. That might be true but I am still dumbfounded by the lack of one. I really hope that Ted Neward’s prediction for 2011 that “Apple starts feeling the pressure to deliver a developer experience that isn’t mired in mid-90’s metaphor” comes true.

I should clarify that there is a debugger in Ruby that allows you to step through your code but it is nothing like what I was used to in Visual Studio.

Source Control

Although more than two years old the article 7 Version Control Systems Reviewed in Smashing Magazine is a great resource to get an idea of what’s available and popular on the Mac OS X.

Of the source control systems mentioned in the article so far I have only experimented with Git and I am very pleased with it. As with the text editors I was a shocked with the lack of visual tools to do basic operations like check in and check out but this probably due in part to the lack of a standard IDE for Ruby development.

However, Git’s installation process on the OS X was incredible painless. I was pleasantly surprised on how well documented the process to install it and get started are. I did notice that Git seems very light-weight compared to a traditional source control system in Windows (e.g. Team Foundation or SourceSafe.)

It has taken me a bit to get used to depend on command line tools to check in and check out my code but at the same time I am amazed how simple it becomes after a few days of using it. I am even using GitHub to push my code “to the cloud” and keep a copy of it outside my machine.

In Summary

So after little bit more than a month of playing with Ruby on the OS X on my spare time, I am starting to get familiar with the language and the development tools. All of a sudden I find myself writing code without looking at my book (or the web), running my unit tests, making changes, and checking my changes to my local and remote source control repositories. I am not nearly as proficient with Ruby or the development tools on OS X as I am with C# or Visual Studio but I am starting to get to the point where I feel comfortable.

I am sure it will be a fun and interesting ride. I’ll keep blogging about my experiences of learning Ruby (and eventually Rails) and the development tools for OS X as I learn more.

Returning HTTP 404 in ASP.NET MVC

hector@hectorcorrea.com

A few weeks ago I noticed that when users of my site request a blog topic that does not exist although I was displaying a user friendly message indicating that the topic does not exist I was not returning the proper HTTP status code (404) to indicate to the user that the page was not found. This approach is typically not a problem if the user visiting my site is a human (users don’t really care about HTTP status codes) but it is important if the “user” visiting the site is a web crawler like the Googlebot or the Yahoo Slurp because for these non-human users the HTTP code is very important as it means something concrete to them.

In digging deeper into this issue I realized that I was doing what is known as a “soft 404” which is not good for web crawlers because it fools them into thinking that there is a valid page for an invalid URL. In my case my site was returning a status code 200 (which means OK) rather than a status code 404 that means “Not Found.” This is in fact so common that both, the Googlebot and the Yahoo Slurp, web sites talk about this problem here and here.

The Root of the Problem

The code that was causing this issue in my controller looked similar to this:

   1:  public class BlogController : Controller

   2:  {

   3:      public ActionResult Sample1(string topicName)

   4:      {

   5:          bool isTopicFound = SomeLogic(topicName);

   6:          if (!isTopicFound)

   7:          {

   8:              var route = new RouteValueDictionary(new { controller = "Error", action = "NotFound" });

   9:              return new RedirectToRouteResult(route);

  10:          }

  11:          // regular code to display topic

  12:      }

  13:  }

Although this code displays a nice user friendly page to the user (via the NotFound action of the Error controller) because I am using RedirectToRouteResult the browser gets an HTTP code 302 (Found) followed by a 200 (OK) which is not what I wanted. Below is a trace of a request like this with Fiddler. Notice how the original request to /Blog/Sample1 got a 302 response code indicating that something was found at a new URL and then the new URL (/Error/NotFound) in turn got an HTTP status code of 200 (OK) because this second URL indeed returned OK.

Throwing an HttpException

When I decided to update my ASP.NET MVC site to handle this issue correctly my first thought was to throw an HttpException from my controller, for example something along these lines:

   1:  public ActionResult Sample2(string topic)

   2:  {

   3:      bool isTopicFound = false; // SomeLogic(topicName);

   4:      if (!isTopicFound)

   5:      {

   6:          throw new HttpException(404, "Blog topic not found");

   7:      }

   8:      // regular code to display topic

   9:  }

The HttpException can either be handled in the controller’s OnException method or on the Application_Error (in Global.asax.) Whatever code we use we need to be sure we don't rely on RedirectToRouteResult like in the previous example or otherwise we'll be back on the HTTP 302 + HTTP 200 trap.

If the HttpException is not handled at all in the code then the customErrors section in the web.config could redirect the user to a default error page so they see a user friendly error message. In this case however the returned HTTP code is an HTTP 302 followed by an HTTP 200 just like in the previous example! This actually makes sense giving that the parameter in the customErrors section where the page is configured is clearly named “defaultRedirect”. However this does not address the problem that I am trying to solve.

The ASP.NET MVC way

As it turns in ASP.NET MVC you can directly change the status code of the response and still display whatever view you want to. The code is actually very simple as shown in the following example. Notice how the status code is forced to 404 (Not Found) and the “Error” view is selected.

   1:  public ActionResult Sample3(string topic)

   2:  {

   3:      bool isTopicFound = false; // SomeLogic(topicName);

   4:      if (!isTopicFound)

   5:      {

   6:          HttpContext.Response.StatusCode = 404;

   7:          HttpContext.Response.Clear();

   8:          return new ViewResult() { ViewName = "Error" };

   9:      }

  10:      // regular code to display topic

  11:  }

This code accomplishes both goals: the user sees a human friendly error (via the Error view) and the HTTP status code returned is 404 as show in the following Fiddler trace. Notice how the original request to /Blog/Sample3 returns a 404 code directly without doing a redirect.

Now when somebody tries to access a topic that does not exist on my blog they get both a nice error message and the correct HTTP status code. You can try this by hitting this link to a non-existing blog topic and looking with Fiddler the HTTP status code returned.

A Couple of Gotchas

While looking into this problem I noticed that throwing an HttpException from a controller is slightly different than throwing any other exception. For example, although you can decorate your controllers with the HandleError attribute and configure what view will be rendered in case of exceptions, HandleError does not handle HttpExceptions. This post on StackOverflow has more information on this issue.

The second gotcha that I ran into is that if you return an HTTP status code 404 and your view is smaller than 513 bytes Google's Chrome browser will not render your view but rather Chrome will render its own view. This issue has been documented before and it only happens on Chrome. Most likely this won't be an issue for you in production as most views are in fact larger than 512 bytes that by the time all the content of your page is considered but you should keep it in mind while testing as it might throw you off.

Log4net Thread-Safe but not Process-Safe

hector@hectorcorrea.com

A few days ago I was struggling with a problem with log4net in a web application. I have used log4net in many applications before with little or no problem but in this case even with a single user hitting the web site I was seeing strange logging behaviors. In my case some of the log entries were being logged but others were not.

The application in question is a C# ASP.NET MVC web application running under IIS. I am using the true and tried RollingFileAppender. It couldn't have been a more typical use of log4net and yet I was seeing a weird things.

Here are the symptoms that I was seeing:

Log4net was logging some entries but not others.
The issue was happening even when I was the only user on the site.
The issue was only happening in one of our development servers but not on my local box.

Since some entries were being logged but not others I was able to rule out the possibility that IIS might not have enough rights to write to the folder configured for the RollingFileAppender. This is a typical issue but not applicable in my case since if IIS wouldn't have enough rights nothing would be logged at all. In addition since the issue was happening even when I was the only user on the site I was sure it was not due to heavy utilization or some sort of race condition (which log4net typically handles nicely anyway.)

The fact that the issue was happening only on the development server with IIS 6 and not on my local machine with IIS 7 made me suspect that the different versions of IIS might the culprit but wasn't sure about that.

Turning log4net internal diagnostic

To begin troubleshooting I turned on log4net's diagnostic option to see if log4net was itself reporting problems that would give me a hint. If you are not familiar with this log4net feature check out the "How do I enable log4net internal debugging?" section in the log4net FAQ.

In my case I configured log4net to dump all of its diagnostic information to a file called "LogDiag.txt". After running my application and hitting the site a few times I went to see what was being logged in the LogDiag.txt file and lo and behold I noticed that there were three LogDiag.txt files being generated. The following picture shows how this looked on my machine:

I opened the "LogDiag.txt" file and there were no errors indicated logged in there. However, the other two files (the ones prefixed with a GUID) had the following error:

log4net:ERROR [RollingFileAppender] 
   Unable to acquire lock on file c:\dev\SandboxMvc\SandboxMvc\App_Data\LogFile. 
   The process cannot access the file 'c:\dev\SandboxMvc\SandboxMvc\App_Data\LogFile' 
   because it is being used by another process.
log4net:ERROR [RollingFileAppender] 
   OpenFile(c:\dev\SandboxMvc\SandboxMvc\App_Data\LogFile,True) call failed.

Houston, we have a problem...

OK so we indeed have a problem. Log4net is clearly telling me that it is having problems writing to the log file. That's good. At least I am not going crazy. But how come? How come am I having this problem now but not in other applications? How come it happens only on the web server but not in my local box?

The answer turned out to be the way IIS was configured on the server where the problem was happening. It turns out that IIS on that particular box is configured to use what is known as a web garden which means that IIS will launch multiple worker processes to handle requests. That was different from the configuration on my local box where IIS was setup to have a single worker process running.

The following picture shows how to configure IIS to use one or many worker processes. In this picture you can see that the "ASP.NET 4.0 Application Pool" is configured to use a maximum of three worker processes.

When the IIS application pool is configured to use more than one worker process (i.e. a web garden) what happens is that multiple processes will be spun by IIS to serve requests for your application. If you configured log4net to use a single file to log requests (as I did) then this means that multiple processes will attempt to write to a single file and locking issues will arise. Once I realized what the problem was finding a solution was quite easy.

The fact that there were three worker process running also explained why I saw three "LogDiag.txt" files on my machine. Log4net was smart enough to name each file differently (remember that two of them were prefixed with a GUID) so that it's internal logging didn't experience the same problem that I was having.

Before I discuss how you can work around this issue let me clarify that there is nothing wrong per-se with having a web garden in IIS. This configuration is actually good on a server-class machine. The discussion on when and how to configure a web garden is beyond the scope of this blog post however the lesson learned is that if you have a web garden you need to take this into account when configuring the RollingFileAppender.

How to address the issue

There are two ways that we can configure the RollingFileAppender to work in a web garden. One option is to configure it to use what is known as "minimal lock" on the log file. With this option log4net will only lock the file for the duration of each log operation. This option reduces the chances that more than one process will collide as they keep the file locked to a minimum. However, there is no guarantee that under load they might actually collide. Additionally there is a performance penalty in opening and closing the file on each log entry. I am not sure if the penalty is too high but it's worth testing if you decide to use this option. Below is an example of how to configure the RollingFileAppender to use minimal locking (notice the lockingModel setting in line 2.)

   1:  <appender name="LogFileAppender" type="log4net.Appender.RollingFileAppender,log4net">

   2:      <lockingModel type="log4net.Appender.FileAppender+MinimalLock" />

   3:      <param name="File" value=".\\App_Data\\LogFile" />

   4:      <param name="AppendToFile" value="true" />

   5:      <datePattern value=".yyyy-MM-dd'.txt'" />

   6:      <rollingStyle value="Composite" />

   7:      <maxSizeRollBackups value="5" />

   8:      <maximumFileSize value="100KB" />

   9:      <staticLogFileName value="true" />

  10:      <layout type="log4net.Layout.PatternLayout">

  11:          <conversionPattern value="[LOGENTRY] %date %-5level %logger{2} - %message %newline" />

  12:      </layout>

  13:  </appender>

The second option to take care of the problem is to make sure that each IIS worker process writes to its own file so that there are no collisions at all. With this option if the web garden is configured to use three worker processes then you'll have three simultaneous log files (one for each active worker process.) Each of them can use an exclusive lock on the file as these files are not shared across processes. This method is guaranteed not to have collisions and allows you to keep using the fast exclusive lock method. The disadvantage is that now you have multiple log files to look at rather than a single consolidated one. Below is an example on how to configure the RollingFileAppender to create one file for each process. Notice how line 2 now has been configured to use a PatternString to name the file and the inclusion of pattern %processid to make each file unique by each process ID.

   1:  <appender name="LogFileAppender" type="log4net.Appender.RollingFileAppender,log4net">

   2:      <file type="log4net.Util.PatternString" value=".\\App_Data\\Log[%processid]" />

   3:      <!--<lockingModel type="log4net.Appender.FileAppender+MinimalLock" />

   4:      <param name="File" value=".\\App_Data\\LogFile" />-->

   5:      <param name="AppendToFile" value="true" />

   6:      <datePattern value=".yyyy-MM-dd'.txt'" />

   7:      <rollingStyle value="Composite" />

   8:      <maxSizeRollBackups value="5" />

   9:      <maximumFileSize value="100KB" />

  10:      <staticLogFileName value="true" />

  11:      <layout type="log4net.Layout.PatternLayout">

  12:          <conversionPattern value="[LOGENTRY] %date %-5level %logger{2} - %message %newline" />

  13:      </layout>

  14:  </appender>

But I thought log4net was thread-safe?

Log4net is indeed thread-safe as it is clearly indicated on log4net's FAQ. A lot of people get confused on this because the documentation for the RollingFileAppender says that the appender is not thread safe. In reality the log4net code takes this into account ([1][2]) and makes sure logging is thread-safe. Some people have even run tests to prove this.

However, the fact that log4net is thread-safe does not mean that is process-safe to write to a single file. This is not log4net's fault, there is no such thing as the ability to have multiple processes write concurrently to the same text file. We use relational databases (not text files) when we need to have multiple processes write to a single repository of data.

In summary

If you are using log4net's RollingFileAppender in an environment where IIS is configured to use multiple worker processes (i.e. a web garden) you must ensure that the RollingFileAppender uses one file per worker process rather than a single file.

Encrypt and Decrypt a string in C#

hector@hectorcorrea.com

This blog posts presents an easy to use C# class to encrypt and decrypt strings in .NET.

Although the System.Security.Cryptography namespace in the .NET Framework provides a wealth of classes to encrypt and decrypt values, it seems that MSDN fails short of providing a good and simple example on how to use these classes for the most common request: encrypt a string.

The class presented in this blog allows you to encrypt and decrypt a string with the following lines of code:

   1:  string encryptionPassword = "supersecret";

   2:  string textToEncrypt = "the quick brown fox jumps over the lazy dog";

3:

   4:  string encrypted = Crypto.Encrypt(textToEncrypt, encryptionPassword);

   5:  string original = Crypto.Decrypt(encrypted, encryptionPassword);

Disclaimer: The code presented in this blog post is just one of the many ways to encrypt and decrypt strings in .NET. If you need to encrypt incredible confidential information (like the nuke launch codes) you should check specific books on Cryptography. Having said that, you probably arrived to this post because you are in search of a simple example on encryption with .NET. If that's the case then this post is for you.

Below is the code of a simple C# class to encrypt and decrypt strings. All in all it's 61 lines of code including curly braces. You can just copy and paste this code to your project and voilà you are ready to encrypt and decrypt strings. If you are bit curious the remainder of this post explains how the code works.

// Code based on the book "C# 3.0 in a nutshell by Joseph Albahari" (pages 630-632) // and from this StackOverflow post by somebody called Brett // http://stackoverflow.com/questions/202011/encrypt-decrypt-string-in-net/2791259#2791259 static public class Crypto { private static readonly byte[] salt = Encoding.ASCII.GetBytes("Ent3r your oWn S@lt v@lu# h#r3"); public static string Encrypt(string textToEncrypt, string encryptionPassword) { var algorithm = GetAlgorithm(encryptionPassword); byte[] encryptedBytes; using (ICryptoTransform encryptor = algorithm.CreateEncryptor(algorithm.Key, algorithm.IV)) { byte[] bytesToEncrypt = Encoding.UTF8.GetBytes(textToEncrypt); encryptedBytes = InMemoryCrypt(bytesToEncrypt, encryptor); } return Convert.ToBase64String(encryptedBytes); } public static string Decrypt(string encryptedText, string encryptionPassword) { var algorithm = GetAlgorithm(encryptionPassword); byte[] descryptedBytes; using (ICryptoTransform decryptor = algorithm.CreateDecryptor(algorithm.Key, algorithm.IV)) { byte[] encryptedBytes = Convert.FromBase64String(encryptedText); descryptedBytes = InMemoryCrypt(encryptedBytes, decryptor); } return Encoding.UTF8.GetString(descryptedBytes); } // Performs an in-memory encrypt/decrypt transformation on a byte array. private static byte[] InMemoryCrypt(byte[] data, ICryptoTransform transform) { MemoryStream memory = new MemoryStream(); using (Stream stream = new CryptoStream(memory, transform, CryptoStreamMode.Write)) { stream.Write(data, 0, data.Length); } return memory.ToArray(); } // Defines a RijndaelManaged algorithm and sets its key and Initialization Vector (IV) // values based on the encryptionPassword received. private static RijndaelManaged GetAlgorithm(string encryptionPassword) { // Create an encryption key from the encryptionPassword and salt. var key = new Rfc2898DeriveBytes(encryptionPassword, salt); // Declare that we are going to use the Rijndael algorithm with the key that we've just got. var algorithm = new RijndaelManaged(); int bytesForKey = algorithm.KeySize / 8; int bytesForIV = algorithm.BlockSize / 8; algorithm.Key = key.GetBytes(bytesForKey); algorithm.IV = key.GetBytes(bytesForIV); return algorithm; } }

The Crypto class

The Crypto class presented in this post has four methods. Methods Crypt and Decrypt are public and are the ones that your code calls to encrypt and decrypt values. The other two methods (InMemoryCrypt and GetAlgorithm) are used internally by the class.

Encrypt()

The process of encrypting values in .NET requires a few steps that are not evident to first comers. In a nutshell this process involves setting up the algorithm that will be used for encryption and pushing the data to encrypt through it. In our case since we want to encrypt strings and the .NET cryptography classes don't provide overloads for strings out of the box we'll need to take a few extra steps to account for this. The Encrypt() method in our Crypto class takes care of these steps. Let's review this method in detail.

   8:      public static string Encrypt(string textToEncrypt, string encryptionPassword)

   9:      {

  10:          var algorithm = GetAlgorithm(encryptionPassword);

11:

  12:          byte[] encryptedBytes;

  13:          using (ICryptoTransform encryptor = algorithm.CreateEncryptor(algorithm.Key, algorithm.IV))

  14:          {

  15:              byte[] bytesToEncrypt = Encoding.UTF8.GetBytes(textToEncrypt);

  16:              encryptedBytes = InMemoryCrypt(bytesToEncrypt, encryptor);

  17:          }

  18:          return Convert.ToBase64String(encryptedBytes);

  19:      }

The first thing we do in the Encrypt() method is define what algorithm we want to use for encryption (see line 10.) In our case the we will use the Rijndael algorithm (more on this later.) This algorithm class has the ability to provide either an encryptor or a decryptor. For the Encrypt() method we obviously need an encryptor (see line 13.)

The second thing we do is convert to bytes the string that we received with the value to encrypt (line 15.) This is an important step as the .NET encryption classes work with bytes, not strings.

As you probably already know strings in .NET are encoded in Unicode and as such they can hold characters from pretty much any language in the world. Some of these characters need more than one byte to be stored and therefore the number of bytes that you get on line 15 might be larger than the number of characters in your string.

If you only deal with ASCII values you probably don't give this much thought as most of the characters in your strings translate to a single byte (for example character "A" can be represented by one byte with value of 65 - its ASCII value.) However, strings in .NET can hold characters that need multiple bytes to be stored. For example, the Japanese character maps to three bytes with values 227, 130, and 166.

So now that we have defined an algorithm to use (along with its corresponding encryptor) and converted to bytes the information to encrypt we are ready to pass this information to the InMemoryCrypt method (line 16) to finally perform the encryption. We'll talk about the details of the InMemoryCrypt method in the section below but for now let's just say that it encrypts the bytes that we pass to it and gives us back the corresponding encrypted bytes.

Finally, on line 18 we convert the encrypted bytes back to a .NET string since that's what we want as an end result. This is the reverse process of what we did in line 15.

As you can see, although the Encrypt method is only a few lines long there is a lot going on. This is not exactly what I would call a self evident process and it is no wonder this is one of the most commonly asked questions in .NET message boards!

GetAlgorithm()

The GetAlgorithm() method declares what algorithm will be used for encryption and sets up the corresponding encryption keys. The .NET framework supports several algorithms and each of them has different strengths and weaknesses. On this particular example we are using the Rijndael algorithm. You can find a lot of information about this algorithm on the web but for the purpose of this blog post let's just say that it uses a pair of values (known as key and initialization vector) to encrypt a set of bytes. In our case both, the key and the initialization vector (IV), will be calculated from an "encryptionPassword" that we provide.

Let's look in detail at the code of this method.

  47:      private static RijndaelManaged GetAlgorithm(string encryptionPassword)

  48:      {

  49:          // Create an encryption key from the encryptionPassword and salt.

  50:          var key = new Rfc2898DeriveBytes(encryptionPassword, salt);

51:

  52:          // Declare that we are going to use the Rijndael algorithm with the key that we've just got.

  53:          var algorithm = new RijndaelManaged();

  54:          int bytesForKey = algorithm.KeySize / 8;

  55:          int bytesForIV = algorithm.BlockSize / 8;

  56:          algorithm.Key = key.GetBytes(bytesForKey);

  57:          algorithm.IV = key.GetBytes(bytesForIV);

  58:          return algorithm;

  59:      }

Line 50 shows the call to Rfc2898DeriveBytes to get a generator of pseudo-random bytes based on our "encryptionPassword" and "salt" values.

Line 53 declares an instance of the Rijndael algorithm and then on lines lines 56 and 57 we set the key and initialization vector values needed by this algorithm. Notice how we set these two values with as many bytes required depending on the KeySize and BlockSize of the algorithm.

Technically you could skip the call to Rfc2898DeriveBytes and manually set the values used for key and initialization vector. However if you do this you need to make sure that you have enough bytes to fill the key and initialization vector. This can be a problem if the "encryptionPassword" value was too short (say it's only 10 bytes long but the algorithm expects 32 bytes values.) The Rfc2898DeriveBytes takes care of this problem as it can generate enough bytes even if the "encryptionPassword" and "salt" are too short.

InMemoryCrypt()

The classes on the .NET Framework use streams to perform the encryption process. This allows the encryption of very large blocks of text while consuming very little memory. This is a nice feature of the framework as it allows you to encrypt entire documents without having to load the entire document in memory in one chunk. The drawback is that you must go through these streams even when you encrypt small sets of data like in our example.

The InMemoryCrypt() method (lines 35-43) hides the process of setting up the streams to read through the bytes to encrypt and dump the resulting bytes into another stream.

Encryption password and Salt

The "encryptionPassword" value that the Encrypt() and Decrypt() methods receive should be a value that only you know. This "encryptionPassword" alone is enough to decrypt a value encrypted with this code.

On the other hand the "salt" value defined on the class (line 6) and used in GetAlgorithm() on line 50 does not need to be kept secret as it is useless without the encryption password. It is important however that you use the exact same "salt" value to decrypt as you used to encrypt which is why I declared it as a static member of the class rather than make it a parameter to Encrypt and Decrypt. Feel free to update this value or make it configurable on your own implementation.

A few final words

The Decrypt() method (lines 21-32) is pretty much the reverse of the Encrypt() method so I won't elaborate on it.

There are many things to consider when choosing an encryption algorithm:

Some algorithms are one-way only meaning that they can encrypt a value but you cannot decrypt it. These algorithms are known as hashing algorithms and are perfect for storing password in a database.
Other algorithms do allow for encryption and decryption. Within these algorithms some of them are symmetrical (like Rijndael) meaning they use the same key for encryption and decryption whereas others are asymmetrical and use different keys known as private and public keys.

Keep this in mind when deciding what algorithm to use in your application.

References

As indicated at the top of the Crypto class most of the code for this blog post was taken from the excellent book C# 3.0 in a Nutshell by Joseph Albahari and Ben Albahari (pages 630-632) and from this post in StackOverflow by somebody called Brett.

There is a good summary of the different encryption options in the book "C# 3.0 in a Nutshell" (see pages 627-638)

AJAX calls with jQuery in ASP.NET MVC

hector@hectorcorrea.com

One of the nice things about ASP.NET MVC is the ability to easily integrate standard web libraries like jQuery in .NET web applications. In this blog post I am going to show how to execute AJAX calls with jQuery from an ASP.NET MVC application. The first example shows how to do it with an HTTP GET call and the second one uses an HTTP POST call. Both examples use JSON in the return type.

HTTP GET Example

This is probably the simplest example on how to execute an AJAX call with jQuery but it should be OK to show the general concept. In this page when the user clicks on the "click here" hyperlink a request is made to the server to retrieve the time on the server and display to the user. The AJAX part of this example is that the request to the server is executed without making a full page reload as it can be seen in the screenshot below (notice that the time the page was originally loaded does not change after I clicked the hyperlink and retrieved the server time.) Again, not terribly exciting but good enough for our purposes.

The MVC View

The MVC view that makes this possible is rather simple and it's composed of the following elements

A SCRIPT tag on the HEAD to include the jQuery library
A SCRIPT block to declare the code to execute the jQuery call
The HTML to render the page

To include jQuery in an ASP.NET MVC you just need to a add a SCRIPT tag to the head that looks like this (line 3):

   1:  <head id="Head1" runat="server">

   2:      <title>AJAX Sample 1</title>

   3:      <script src="<%: Url.Content("~/Scripts/jquery-1.4.1.js") %>" type="text/javascript"></script>

   4:  </head>

The HTML code for this page is also very simple as we are only displaying some text coming from the server (line 2), rendering a hyperlink to let the user invoke the AJAX call (line 4), and rendering a placeholder to display the value returned by the AJAX call (line 5). Notice that there is no JavaScript code in this HTML block. This is by design and I'll describe the reason for this in the next paragraphs (see Hijax section below.)

   1:      <h2>Ajax Sample 1</h2>

   2:      <p>Page loaded at: <%: System.DateTime.Now.ToString() %></p>

3:

   4:      <p><a href="GetServerTime" id="AjaxLink">Click here</a> to get current time.</p>

   5:      <div id="CurrentServerTimeDiv"></div>

Of course the interesting part of this example is the actual JavaScript block that wires the HTML anchor tag to JavaScript and makes the AJAX call.

   1:  <script type="text/javascript">

2:

   3:      $(document).ready(function () {

   4:          //Setup the AJAX call

   5:          $("#AjaxLink").click(function (event) {

   6:              event.preventDefault();

   7:              GetCurrentServerTime();

   8:          });

   9:      });

10:

  11:      function GetCurrentServerTime() {

  12:          // Make the call to the server

  13:          $.getJSON("GetServerTime", null, UpdateServerTime);

  14:      }

15:

  16:      function UpdateServerTime(result) {

  17:          // Update the page with the result

  18:          var message = "Time on the server is: " + result.ServerTime;

  19:          $("#CurrentServerTimeDiv").html(message);

  20:      }

21:

  22:  </script>

This JavaScript block has three main sections. The first section (lines 3-9) wires the HTML anchor tag with name "AjaxLink" to a JavaScript function called "GetCurrentServerTime()". It is customary to add this setup code to the ready event to make sure it's executed when the DOM of the page has been fully constructed even if not all of the resources for the page (e.g. images) have been downloaded. You can read more about this here. In this case we are replacing the "click" event of the anchor tag "AjaxLink" with a call to our own "GetCurrentServerTime()" JavaScript function.

The second section of the JavaScript block (lines 11-14) is the actual JavaScript function that will be executed when the user clicks on the link. This code executes a jQuery's getJSON call to URL "GetServerTime", passing no arguments (null), and executing function "UpdateServerTime" upon completion. When this code is executed our MVC controller will be hit. More on this later.

The third section of the JavaScript code (lines 16-19) is executed when our controller finishes its execution. In our case the controller returns an object with one property "ServerTime" that we use to construct a message to display on the placeholder "CurrentServerTimeDiv" that was defined in the HTML code.

The MVC Controller

When the user clicks on the hyperlink a request is sent through an HTTP GET to our controller. The controller will perform an operation (in this case just merely calculating the time on the server) and return this value to the view. The code for this controller is shown below:

   1:  // GET /AjaxSample/GetServerTime

   2:  public ActionResult GetServerTime()

   3:  {

   4:      var viewModel = new Models.AjaxSampleViewModel();

   5:      viewModel.ServerTime = DateTime.Now.ToString();

   6:      if (Request.IsAjaxRequest())

   7:      {

   8:          return Json(viewModel, JsonRequestBehavior.AllowGet);

   9:      }

  10:      // Fallback view if JavaScript is not supported by the browser

  11:      return View("ServerTime", viewModel);

  12:  }

When this code executes through an AJAX call (as in our case) it will return a JSON object to the client with the results. This is why our JavaScript code was able to use "result.ServerTime" to display the result. The following screenshot shows how the resulting object looks in JavaScript:

Hijax

You might be wondering why do we need a fallback clause in our controller (line 10-11) if the view explicitly makes an AJAX call. The reason for this precaution is in case the client browser does not support JavaScript or JavaScript is disabled. Although JavaScript is almost always enabled in most desktop and laptop computers that's not always the case in mobile devices.

This is also the reason we wired the AJAX call on the document ready event rather than directly on the HTML anchor tag in our view. For example, although we could have just as easily wired the AjaxLink anchor tag directly to the "GetCurrentServerTime" function that wouldn't have worked if JavaScript was disabled. Instead we wired the JavaScript call on the document ready event which is only executed if JavaScript is enabled. If JavaScript is disabled (and therefore the document ready event is not fired) the HTML anchor tag in our view is already wired to hit the "GetCurrentServer" URL which maps to the "GetServerTime" method in our controller.

This approach to JavaScript is known as Hijax and is a good way to provide progressive enhancement to your web pages.

HTTP POST Example

You can also execute AJAX calls using an HTTP POST request. For example, in the following screenshot the information in the textboxes (e-mail and comment) is sent through an asynchronous HTTP POST to the server.

The MVC View

Like in our previous example, the MVC view that makes this possible is very simple and it's composed of the following elements

A SCRIPT tag on the HEAD to include the jQuery library
A SCRIPT block to declare the code to execute the jQuery call
The HTML to render the page

The HTML code for this page is shown below. Notice how we declare an HTML form named "SaveCommentForm" (line 3) with two textbox controls (lines 4-6) and a submit button (line 7). Below the form there is a placeholder section called "NewComment" where we will display the results from the server after the user has added a new comment. Like in our previous example there is no JavaScript on this HTML code, we will be wire the JavaScript call when the web page is rendered on the browser using the Hijax method that we described before.

   1:  <p>Page loaded at: <%: System.DateTime.Now.ToString() %></p>

2:

   3:  <% using (Html.BeginForm("SaveComment", "AjaxSample", FormMethod.Post, new { id = "SaveCommentForm" })) { %>

   4:  Your e-mail address:<%: Html.TextBox("emailAddress")%><br />

   5:  Enter a comment:<br />

   6:  <%: Html.TextArea("comment")%><br />

   7:  <input type="submit" value="Save" />

   8:  <% } %>

9:

  10:  <div id="NewComment"> </div>

The JavaScript that powers this web page is very similar to the one that we used in our previous example and is shown below.

   1:  <script type="text/javascript">

2:

   3:      $(document).ready(function () {

   4:          //Setup the AJAX call

   5:          $("#SaveCommentForm").submit(function (event) {

   6:              event.preventDefault();

   7:              SaveComment(this);

   8:          });

   9:      });

10:

  11:      function SaveComment(form) {

  12:          // Make an AJAX call to the server

  13:          // notice that we request a JSON response

  14:          $.ajax({

  15:              url: form.action,

  16:              type: form.method,

  17:              dataType: "json",

  18:              data: $(form).serialize(),

  19:              success: CommentSaved

  20:          });

21:

  22:      }

23:

  24:      function CommentSaved(result) {

  25:          // Update the page with the result

  26:          var newComment = "Thank you for you comment " + result.emailAddress + "<br/><pre>" + result.comment + "</pre>";

  27:          var comments = newComment + "<br/>" + $("#NewComment").html();

  28:          $("#NewComment").html(comments);

  29:      }

30:

  31:  </script>

The first section of this JavaScript block (lines 3-9) uses the Hijax technique aforementioned to redirect the submit event of the "SaveCommentForm" to call a JavaScript function named "SaveComment".

The "SaveComment" function is defined on the second section of the JavaScript block (lines 11-20.) In this example we are issuing an HTTP POST (line 16) to the original location (line 15) that the HTML form would have on its own but we are making the call asynchronous (by using the AJAX function) and requesting that the result is delivered to us in JSON notation (line 17). When the asynchronous call completes we want jQuery to fire the "CommentSaved" function (line 19.) The following picture shows the values that are passed at runtime to the url, type, and data parameters in this AJAX call. Notice that the form.method is POST and form.action points to the "SaveComment" URL.

The third section of the JavaScript code (lines 24-30) is executed when our controller finishes its execution. In our case the controller returns a JSON object (similar to the HTTP GET example) that we use to update the page.

The MVC Controller

The controller to process our HTTP POST request is shown below. In essence this code is very similar to the one that we used before in the HTTP GET example. We process the request (although in this code I just indicate that we would need to do something with it) and then return a JSON object back to the called. Like in our previous example we return a full view as a fallback plan.

   1:  // POST /AjaxSample/SaveComment

   2:  [HttpPost]

   3:  public ActionResult SaveComment(string comment, string emailAddress)

   4:  {

   5:      //[Code to actually save the comment would go here]

   6:      if (Request.IsAjaxRequest())

   7:      {

   8:          return Json(new { comment = comment, emailAddress = emailAddress, savedAt = DateTime.Now.ToString() });

   9:      }

  10:      // Fallback view if JavaScript is not supported by the browser

  11:      return View("CommentSaved");

  12:  }

...and that's it

As you can see it is relatively straightforward to make AJAX calls using jQuery within ASP.NET MVC views to your controllers.

References

I took the idea of using Hijax for the jQuery calls from the book ASP.NET MVC in Action by Jeffrey Palermo et al. (pages 195-214)

In addition to the online documentation for jQuery, Joe Brinkman's e-book jQuery for ASP.NET Developers is a great place for beginners. If you are new to jQuery this is probably the best way to spend 7 bucks (other than buying a latte and a croissant at your local coffee shop!)

Steve Sanderson's book Pro ASP.NET MVC 2 Framework also has a good explanation on how to use jQuery in ASP.NET MVC applications (pages 517-559) that is worth checking out.

Password Recovery in an ASP.NET MVC Project

hector@hectorcorrea.com

While rewriting my personal web site with ASP.NET I noticed that although support for the ASP.NET Membership Provider comes included out of the box in a ASP.NET MVC project not all the options are fully implemented to the same extend that they are in a brand new ASP.NET WebForms project. For example, the option to reset your own password if you forgot your old one is not available out of the box in an ASP.NET MVC project.

Out of the box the following options are fully implemented in a ASP.NET MVC project:

Login in
Login out
Change your password
Create new user

Adding support for Password Recovery to an ASP.NET MVC project turned out to be pretty easy as the core functionality already exists in the Membership Provider and it's just a matter of calling it from your application.

The process that I implemented goes like this:

From the LogOn view users can go to the PasswordReset view
In the PasswordReset the user indicates his/her username and then they are sent to the QuestionAndAnswer view
In the QuestionAndAnswer view the user enters the answer to their own security question
Finally, the user is sent to the PasswordResetFinal view with a message indicating that their password has been reset and e-mailed to them.

The rest of this blog post describes the details on how this works. The source code is also available in the link at the bottom of this post.

New Views and Controllers

LogOn

The first thing that I did was update the LogOn view that comes with ASP.NET and added a link to start the Password Reset process. I wired this link to the a new method called PasswordReset in the AccountController.

Password Reset

Secondly I created the HTTP-GET PasswordReset method in the AccountController and a very simple view (PasswordReset.aspx) to allow the user to enter his/her username so that we can reset their password. The PasswordReset.aspx view is extremely simple as it only has a textbox where the user enter their user name.

public ActionResult PasswordReset()
{
if (!MembershipService.PasswordResetEnabled) throw new Exception("Password reset is not allowed");
return View();
}

I also implemented an HTTP-POST PasswordReset method in the AccountController to pick up the data and continue the process. This controller method decides whether the next step is to reset the password for this user or if we need to ask him/her a Password Recovery question before we reset their password. This step is required to honor the requiresQuestionAndAnswer configuration setting in the ASP.NET Membership Provider.

[HttpPost]
public ActionResult PasswordReset(string userName)
{
    if (!MembershipService.PasswordResetEnabled) throw new Exception("Password reset is not allowed");

    if (MembershipService.RequiresQuestionAndAnswer)
    {
        return RedirectToAction("QuestionAndAnswer", new { userName = userName } );
    }
    else
    {
        MembershipService.ResetPassword(userName, GetLoginUrl());
        return RedirectToAction("PasswordResetFinal", new { userName = userName });
    }
}

Password Question and Answer

If the Membership Provider is configured to require a question and answer before resetting a user's password then we route users to the QuestionAndAnswer view. This view is also very simple as it merely has two labels (one with the username and another with password question for the user) and a textbox where the user will enter the answer to their password question.

To support this QuestionAndAnswerView I implemented an HTTP-GET controller method that fetches the security question for the username entered in the PasswordReset view.

public ActionResult QuestionAndAnswer(string userName)
{
    if (!MembershipService.PasswordResetEnabled) throw new Exception("Password reset is not allowed");
    ViewData["UserName"] = userName;
    ViewData["Question"] = MembershipService.GetUserQuestion(userName);
    return View();
}

Finally I added an HTTP-POST method to support the QuestionAndAnswer. By the time we get to this HTTP-POST method we have all the information that we need to reset a user's password (namely the user name and the answer to the security question.) Hence this method is where the call to actually reset the user's password actually happens.

[HttpPost]
public ActionResult QuestionAndAnswer(string userName, string answer)
{
    if (!MembershipService.PasswordResetEnabled) throw new Exception("Password reset is not allowed");
    MembershipService.ResetPassword(userName, answer, GetLoginUrl());
    return RedirectToAction("PasswordResetFinal", new { userName = userName });
}

Password Reset Final

The last step in the process if a new view called PasswordResetFinal that just displays a message to the user telling him/her that a new password has been generated and e-mailed to them.

Changes to the Model

Every ASP.NET MVC project comes with a default model called AccountMembershipService to support the Views and Controllers that handle membership information. This AccountMembershipService is not much more than a wrapper for the ASP.NET MembershipProvider. Adding functionality to this model to support the password reset operation was very simple as the MembershipProvider already provides the core functions. The MembershipService referenced in the controller actions in the code actually point to an instance of this AccountMembershipService.

For example, the QuestionAndAnswer view calls the following method to to retrieve the security question for a user. Notice how this method's only job is to forward the calls to the Membership provider.

public string GetUserQuestion(string userName)
{
    MembershipUser user = _provider.GetUser(userName, false);
    if (user == null)
    {
        throw new Exception("User name not found");
    }
    else
    {
    return user.PasswordQuestion;
    }
}

All in all I added two methods to the AccountMembershipService (one to retrieve a user's security question and one to actually do the password reset) plus a few properties to expose a couple of features of the Membership provider (like the need for a security question) that were not exposed on the default implementation.

In Summary...

As I indicated at the beginning of this blog post, adding support for Password Recovery to an ASP.NET MVC project turned out to be pretty easy as the core of the functionality already exists in the Membership Provider.

You can download the source code of a brand new working ASP.NET MVC web site updated to support the password recovery functionality from here. Keep in mind that this code assumes that you already have the ASP.NET membership provider and its database configured in your system. The readme.txt file inside the download provides further instructions on how to run this sample.

Yield Return

hector@hectorcorrea.com

Even though the yield return statement was added to C# since version 2.0 (around 2004) I never quite understood how it really works. A few days ago while going through some tutorials I saw a sample of code that used yield return and decided to finally look under the covers and learn how it works.

How well do you understand yield return? Look at the code below and answer these two questions:

What is the output of this code?
Can you explain why?

class YieldSample

{

static void Main(string[] args)

{

foreach (int i in YieldSample.GetSomeIntegers())

{

Console.WriteLine("{0}", i);

}

Console.ReadLine();

}

public static IEnumerable<int> GetSomeIntegers()

{

yield return 100;

yield return 200;

yield return 300;

}

The answer to the first question is pretty simple: 100, 200, and 300.

The tricky part is explaining how come C# arrives to this output. How come C# knows that it needs to pick up the first value (100) in the first loop, 200 in the second, and 300 in the third one?

The reason for this behavior is that at compile time C# rewrites the code with the yield return into a state machine that has the logic to return the first value and prepare the code so that in the second call it picks up the second value, and so on and so forth. The pseudo-code for the generated code basically looks like this:

private bool GetNextElement()

{

switch ( state )

{

case 0: // first state of the machine

currentValue = 100;

state = 1; // prepare machine for next state

return true;

case 1: // second state of the machine

currentValue = 200

state = 2; // prepare machine for next state

return true;

case 2: // third state of the machine

currentValue = 300;

state = 3; // prepare machine for next state

return true;

case 3:

state = -1

return false;

}

While trying to understand how the yield return statement works I found two great articles that explain this in more detail. In particularly I liked Yield, and the C# state machine by Erik Forbes and C#: Dynamic Iterators with Yield Return by Paul Kimmel. These two articles actually gave me the clue to use Reflector to look at the code generated by the compiler.

If you are like me you are probably wondering how well this code generation works when the method is not as straightforward as this little example that just returns three hard coded values. The answer is pretty darn good. State machines have been studied by a lot of people and very well understood.

However, just for fun I wrote a few examples of my own with if and nested if statements around the yield return and reviewed the code that the compiler generated. The result was always the same: the resulting code returned the values as expected.

The Design of Design

hector@hectorcorrea.com

Earlier this month I received a draft manuscript of Frederick Brook’s latest book “The Design of Design – essays from a computer scientist”^[1] in which he reviews both what is design and how the design process works.

Although there is plenty literature on the concept of design and the design process Brooks feels there are several good reason to write another book on the subject.

The first reason that he mentions is that the design process has changed a lot since World War II and the new challenges have rarely been discussed. The new challenges that he mentions is that design is now a team activity rather than an individual one. Another challenge is that unlike designers of previous generations nowadays designers cannot longer built with their own hands what they design. Instead designs are captured in computer models and build by somebody else [p. xi]

The second reason why Brooks consider worthwhile writing a book on design is that “much mystery remains” on the subject and that this becomes “evident when we try to teach students how to design well” [p. xi] In this book Brooks uses his six decades of design experience to help expand the knowledge of how the design process works and how we teach it. In his book Brooks does not attempt to find or describe “a science of design” in fact he considers such a goal both impossible and misleading. Instead, the tone of the book is in the form of a few opinionated essays [p. xii]

The book is divided in six sections as follow

Section I: Models of Design
Section II: Collaboration and Telecollaboration
Section III: Design Perspectives
Section IV: A Computer Scientist’s Dream System for Designing Houses
Section V: Great Designers
Section VI: Trips through Design Spaces Case Studies

I found the first three sections of the book fascinating but was not as thrilled with the last three sections. In this review I am going to focus mostly on some selected chapters from the first three sections.

Section I – Models of Design

Chapter 1 - What is design?

On the first chapter Brooks addresses both the basic question of what is design and how the design process works. Brooks starts with the following definition of the verb design from the Oxford Dictionary:

“To form a plan or scheme of, to arrange or conceive in the mind … for later execution” [p. 4]

Brook uses a proposal from Dorothy Sayers to describe the design process as a three steps process in which an idea is conceived mentally outside time and space, then it’s implemented in time and space, and finally the implementation is used by someone else.

In the case of a musical composition the mental formulation (the idea) is first conceived in the composers mind then implemented in paper and ink, and finally the implementation is used by a conductor to execute it with an orchestra.

In a software project the same process happens when its designer conceives the idea of how the software will work, this idea is then implemented in source code, and finally the software is executed and consumed by users.

One of questions that Brooks tries to address in this book is whether there are invariant properties of the design process itself across multiple disciplines like the construction of buildings, software development, and other media. It’s evident that Brooks researched a wide variety of other authors and sources that have addressed the concept of design throughout history including philosophers, designers, and scientists.

Chapter 2 – How Engineers Think of Design – The Rational Model

In the second chapter Brooks describes what he calls the Rational Model for the design process. This model refers to an orderly and systematic step-by-step process to the design of things. He cites several sources that have proposed this design process including German mechanical engineers and people from the software engineering field. It’s important to highlight that Brooks does not support this model but considers useful to present it for several reasons that he discusses throughout the book.

In the Rational Model the designer starts with a primary goal and a series of secondary goals. After the goals have been defined we proceed to define utility functions (the weight of each goal against the entire design), constraints and resources so that the designer can proceed with a simple and systematic approach. This is how Brooks puts it:

“Now, so the Rational Model goes, the designer makes a design decision. Then within the design space narrowed by that decision he makes another decision. At each node he could have taken one or more other paths, so one can think of the process of design as the systematic exploration of a design tree” [p15]

In software engineering the Waterfall Model is the representation of the Rational Model in which different steps of the design process happen orderly and sequentially.

Although in software development the Waterfall Model has fallen out of fashion among developers as we have realized how it does not fit with the realities of software development, the Rational Model is still enticing to many people outside of the design process. Proof of this is how, in general as an industry, we still work on contracts that require a clear requirements definition upfront, accurate estimates and timelines, and fixed bid contracts. These steps are only possible in the Rational Model where the requirements are completed ahead of time, they accurate, and the developers know exactly how these requirements are going to be implemented.

Brooks make an interesting observation when he notes that the Rational Model is actually a very “natural” model and evidence of this is the fact that people in multiple disciplines going from mechanical engineering to software development have arrived to it [pp. 29-30]

Chapter 3 – What’s Wrong with the Rational Model?

Although the Rational Model is the first model presented in the book Brooks quickly discredits it in Chapter 3 as he explains that this is just an ideal process that although somehow describes how we think the design process ought to work it is not how it works in real life [p. 22]

Brooks dedicates this chapter of the book to highlight why the Rational Model (or the Waterfall Model as we know if in software development) does not work.

Brooks states that we don’t always know the goal at the onset. Most of the times the goal itself changes as the design evolve and new ideas are explored. Brooks indicates that “the hardest part of the design is deciding what to design” [p. 22]. He points out that not only the design process is iterative but also the design-goal-setting process is iterative. To anyone that has worked on software development this is probably obvious and we all have stories to tell on how we typically go back and forth with the customer to decide what they really need once the software product starts to take shape. Most of the times the end goal is different than the initial goal as both the designer and the customer learn more about the problem and re-evaluate the goals, constraints, and resources.

Another thing that works against the Rational Model is that it is very typical that the constraints and resources change as the design process evolves and we know more about the problem and the end solution that we are developing.

In the Rational Model it is assumed that we can easily evaluate in the design tree how much value a design decision adds to the entire design (“the goodness of a decision” so to speak.) In practice this is hardly the case as most decisions are interconnected to one another. For example, in software development a particular design decision might affect significantly performance, cost, and integration. Evaluating objectively how a particular decision affects the whole design can be close to impossible.

Finally, although the Rational Model has been proposed by people from multiple disciplines, the reality is that designers just don’t work this way. Designers go back and forth and change their own designs as they go. On page 257 when talking about his own experiences as a designer, Brooks states

“The boldest design decisions, whoever made them, have accounted for much of the goodness of the outcome. These bold decisions were due sometimes to vision, sometimes to desperation. They were always gambles, requiring extra investment in the hopes of getting a much better result.”

Brooks closes the third chapter with this sentence

“The Waterfall Model is wrong and harmful; we must outgrow it”

Chapter 5 – What are Better Design Process Models?

After describing what’s wrong with the Rational Model, in Chapter 5 Brooks argues the need to have a dominant model to replace it. His reasoning for advocating a particular dominant model as opposed to just let one surface organically is that he is afraid that if we don’t push for one model in particular we risk having another inappropriate model surface and dominate the landscape as the Waterfall Model did in software development.

In particular, Brooks suggest that we use Barry Boehm’s Spiral Model for building software as the dominant model. The Spiral Model suggests an iterative process that more closely resembles the way designers really work. In addition this model is easy to memorize, visualize, and should help with the eradication of the Waterfall Model.

If you are interested in a quick summary of the Spiral Model you can find one in Wikipedia. The original 1988 paper by Barry Boehm can be found at the IEEE web site.

Although Brooks does not reference newer development process model like the Unified Software Development Process and Scrum these are variations on the Spiral Model originally proposed by Boehm.

Section II - Collaboration and Telecollaboration

In the second section of the book, Brooks addresses two changes in design that have taken since 1900. In particular Brooks discusses the challenges of doing design work in teams (as opposed to solo designs) and collaboration in distributed teams via telecommunication tools (as opposed to collocated teams.)

Although most people would agree that collaboration is a good thing, Brooks points out that collaboration in design is not a self-evident truth and certainly not universally true. [p 64] Brooks points out that “most of the great works of the human kind have been made by one mind, or two working closely.” [p. 64]

In chapter 6 Brooks addresses the obvious question: If design by a single mind is so good why have we shifted to team design? Brooks attribute this shift first and foremost to the technology sophistication that has taken place in engineering (whether is construction or software engineering) in the last hundred years. With the level of specialization needed in most engineering areas people have been forced to become experts in certain areas and therefore we need to bring together experts from multiple areas to get the whole picture on any given project (say a new bridge, a new cell phone, or a new large software system.) The second reason Brooks talks about is that “team design becomes a necessity when it can accelerate delivery of a new product in a competitive environment” [p. 65] and this is paramount in this day and age when (right or wrong) being first to market is a driving force behind companies working on new designs.

Brooks points out that collaboration does not come without a cost. In chapter 6 he talks about the costs that one incurs when a team (rather than a single person) works on a problem. In particular he talks about the cost of partitioning the job among multiple people, teach teammates and learning from them, communication cost, and change control.

When it comes to design work however, Brooks things that biggest challenge is that the conceptual integrity of a design is much harder to maintain when a team works on the design. He talks about how solo designers keep design integrity subconsciously and how this can easily be affected when the design is split among multiple minds.

Since the reality is that nowadays team design is the norm rather than the exception, Brooks proceeds to give ideas and steps that can be used to help teams to achieve good designs. In particular he talks about having a system architect that is in charge of making sure the integrity of the design is achieved even though multiple members of the design team might have different ideas. In Brooks words “The architect serves during the entire design process as the agent, approver, and advocate for the user as well as for all the other stakeholders” [pp. 71-72].

Brooks considers the two-person team a special case that is a very good approach to design as it forces the designers to have to state why their decisions are taken and this in turn forces a quick validation among the team [pp. 81-82] The reason for this is that two people can easily work together and maintain design integrity but somehow this does not scale well with more than two people on the team.

Brooks points two aspects of the design process that benefit from team collaboration. He highlights the value that a team with multiple points of view brings forth it when it comes time comes to decide what to design (the needs of the customer) and also during design review sessions.

In Chapter 7 Brooks addresses the challenges that a distributed design team faces. Brooks is very much aware that “distributed design will only increase” [p. 93] and provides several ideas to help distributed team work. In particular he talks about the value that face to face time at the beginning of a project is crucial even when a team will eventually work distributed.

Section III – Design Perspectives

On Chapter 8 Brooks reviews the two different schools of thoughts of the Rationalists (like Rene Descartes) and Empiricists (like Galileo Galilei) and how they apply to design.

On Chapter 9 Brooks talks that team design makes the need of explicit models on what’s being designed more important than with solo designs. This is because designers in a team tend to assume that they are all sharing the same assumptions and in fact they might not. Brooks puts it like this: “If the team does not draft a common set of explicit assumptions, each designer will work with a distinct set of implicit ones” [p. 115] and then he adds “as soon as the designer starts to make explicit use models, trouble strikes: he is rudely confronted with how much he does not know. The very effort forces him to ask questions he might not otherwise have asked until much later. This is an unmitigated good” [p. 115]

On chapter 10 Brooks talks about resources and, on page 119, he summarizes the chapter rather succinctly:

“If a design, particularly a team design, is to have conceptual integrity, one should name the scare resource explicitly, track it publicly, control it firmly.”

There is far more insight and good advice for designers throughout the rest of the rest of the chapters of Section III of the book than I could cover on this blog post.

Section IV, V and VI

As I mentioned at beginning of this blog entry, I did not find the last three sections of the book as interesting as the first three sections.

In Section IV of the book (A Computer Scientist’s Dream System) Brooks describes his vision for “an ideal computer system for the architectural design of houses and other buildings. “ [p. 204] Chapter 17 presents a vision on how a person could convey a design to a machine (mind to machine) where as Chapter 18 presents the opposite, how a machine could convey to a person a design by using not only visual clues but also audio and tactile feedback.

In Section V Brooks addresses two different problems. In Chapter 18 he addresses the difficult question on how people and processes related to each other and the role of a process in design. This is a good chapter where Brooks describes the opposing views between the need of a design process to aim at predictability and how predictability is hardly a friend to great designs [p. 233] In Chapter 20 Brooks talks about his experience on how to developer great designers.

The last section of the book (Trips through Design Spaces: Case Studies) is dedicated to seven case studies in which Brook participated. This whole section was a big disappointment to me. I am not sure what Brook had in mind when he wrote this part of the book and what value he meant to add to the study of design when he included it on the book. This section includes case studies for the famous System/360 Architecture and the IBM Operating System/360 projects that Brooks managed in the 1960s and also some other personal projects including three personal constructions projects (!) to his own house as well as the design of a book on computer architectures.

Summary

All in all I think the first three sections of the book make the book a great reading and would recommend it to all software developers interested in improving and understanding the design process used to create software solutions.

References

[1] The Design of Design – Essays from a computer scientist; Frederick P. Brooks; Pearson Education; 2010 (Draft Manuscript as of 1/24/2010)

Flow: Second Generation Lean Product Development

hector@hectorcorrea.com

The last few weeks I've been reading the book "The Principles of Product Development Flow" by Donald Reinertsen^[1] in which he describes what he calls the second generation of Lead Product Development.

I've been reading the book from the perspective of software development products and this review will highlight examples on that area. However Reinertsen's book is about product development in general (e.g. the development of a new drug or a new cell phone) and not software in particular.

Reinertsen describes a paradigm for product development that although it could be called Lean Product Development he chooses to call Flow-Based Product Development (Flow) to highlight the differences that exist between lean manufacturing and lean product development.

Unlike lean manufacturing that primarily deals with predictable and repeatable tasks, homogeneous delay costs, and homogeneous task duration, in lean product development these situations are not the norm. In product development the tasks (almost by definition) are not repeatable and hardly ever predictable, delay costs are not homogeneous, and tasks durations have very high variations.

In the case of software development it's not uncommon for teams to be dealing with problems that they have never solved before (e.g. writing a brand new system) or and with technologies that are new to the team. Likewise, the cost of delaying the implementation of a new feature versus the cost of fixing a specific bug can be widely different.

In chapter one Reinertsen starts by describing what he calls twelve critical problems with traditional product development orthodoxy including: Failure to Correctly Quantify Economics, Blindness to Queues, Worship of Efficiency, Hostility to Variability, Institutionalization of Large Batch Sizes, Underutilization of Cadence, Managing Timelines instead of Queues, Absence of Work-In-Progress Constraints, and Centralized Control.

In the remaining chapters of the book Reinertsen presents eight ideas/themes that he suggests to address these problems.

Economics
Queues
Variability
Batch Size
WIP Constraints
Cadence, Synchronization, and Flow Control
Fast Feedback
Decentralized Control

In software development some of these ideas are already popular thanks in large to the wide spread adoption of agile software development processes. For example Scrum recommendation of short sprints helps reduce batch size. Likewise, the daily collaboration between developers and QA during the sprint as well as frequent product demonstrations advocated by Scrum allow for fast feedback to take place.

However, some of the other ideas that Reinertsen presents on his book (like Economics, Queues, WIP Constraints) are not mainstream in software development yet. Some of the new development processes like Kanban^[2][3] are making strides in these areas and this book helps to clarify why these changes are important in product development in general.

On Economics

On the topics of economics, Reinertsen starts with the premise that "we do product development to make money" (p.15) and yet most product development teams cannot correctly quantify economics on their products. In his view product developers currently focus on proxy variables to calculate economic impact rather than using the real economic objective which he calls life-cycle profits (p.29)

For example, a typical proxy variable is cycle-time. Most developers typically try to optimize cycle-time and make it shorter. On the surface this sounds all good. However, few product developers can correctly calculate how much a delay on cycle-time will actually cost in terms of life-cycle profits.

An example of this in software development is when a team cannot objectively calculate the value (in terms of profit) of working on feature A or feature B. If the team does not know the impact to the bottom line, How can they make a decision that makes the more sense for the company? Without knowing the real value of two features, the team might decide to work on feature A because it's more appealing or because the developers needed for feature B are currently busy wrapping something else. However, if the team knew that feature B will bring four times more profit to the company if delivered first they might act different and rearrange the team to complete feature B before feature A.

Reinertsen points out that having a common unit of measure solves many problems as it allows the decisions on what makes more sense to take place on an even keel rather than using the unit of measurement that is popular at the moment. The common measure that he proposes is profits.

For example, given two choices to implement a feature a team might decide to rush it and release it to market rather than use a more comprehensive approach and wait a few months. Either one of these choices could be the correct one at different points in the life cycle of a product. However, the right choice depends on the economics of this decision at a given point in time. Will the company make more money rushing the feature and being first too market or will it make more money delaying its implementation and doing a more comprehensive implementation? What about the "technical debt" that the team will incur by rushing a feature? Does this cost outweigh the benefits of being first to market? All of these things should be considered and measured with the same unit.

Reinertsen indicates that on his research teams tend to think that they know the economics of their decisions. However, when asking different people on the team to quantify (in economic terms) the value of the team decisions their responses vary on a 50 to 1 ratio. When the answers are in such wide range it's obvious that the team is not using a common unit of measurement.

Chapter two contains nineteen principles to help product developers make good decisions based on economics. Although Reinertsen does not provide the "silver-bullet" to make economic decisions the principles that he provides are a good place for product developers to start focusing including paying attention to Cost of Delay (p. 31), U-Curve Optimizations (p.35), Economics Trade-off (p. 37), Marginal Cost and Marginal Value (p.45),

On Queues and Batch Size

One of the areas where lean manufacturing is very different from product development is in the management of queues. In lean manufacturing the problem with queues is well understood and has been optimized for many years. Entire manufacturing books have been written on how to manage bottlenecks on assembly lines to prevent inventory accumulation (e.g. The Goal by Eli Goldratt.) However, there is little literature in software development that addresses the effects of work in progress (which is the software equivalent to inventory.) An example of work in progress in software development are features developed but not yet in production either because they are under development or developed but still being tested.

One of the reasons work in progress (WIP) is so poorly managed in product development is because it's almost invisible. In manufacturing you can walk around a plant and see work in progress lying on the floor. Although work in progress does not sits around on the shop floor in software development work in progress can be easily identified by looking at queues: queues of items ready to be coded, items ready to be tested, items ready to be released, and so on.

You might be wondering, so what? Why should I care about queues in software development? Although there is no cost associated with storing work in progress in software development Reinertsen indicates that work in progress cause several unwanted side effects in product development including: Longer Cycle Time, Increased Risk, More Variability, More Overhead, Lower Quality, and Less Motivation.

An example of this in software development is a queue of features coded but not tested. The larger the queue of features coded but not tested the larger the test cycle that the product will need to go before the features can be released. A large number of features to be tested and then released also translate in higher risk on the new version of the system.

One of the principles of queue theory is called Little's Formula. In layman terms Little's Formula says "average queue size determines average cycle time" (p. 112.) This means that you can reduce cycle time by reducing batch size. This is important as it means that before we go looking for complex solutions to reduce long cycle times (like adding more staff or looking for bottlenecks) we can start by reducing the batch size. Software teams that are already using Agile methods like Scrum have experienced this with the use of short Sprints (e.g. 2-week long cycles) rather than the old-fashion multi-month development cycles.

On his book Agile Management for Software Engineering^[4], David Anderson gives a comprehensive explanation of how inventory and lead time apply to software and it's another good reference to learn more on this subject.

On WIP Constraints

After making you aware of Queues and the importance of Batch Size, Reinertsen moves to Work In Progress Constraints (or WIP Constraints) and why they are important in product development. Reinertsen indicates that while limiting batch size helps by nature to reduce the amount of work in progress, it does not address the fundamental problem of random variations in flow common to product development (p. 143) As Reinertsen indicates at the beginning of his book, product development is subject to many variations not common in lean manufacturing. In the Chapter 6 Reinertsen describes ideas and principles applied in telecommunication networks to deal with high variability and how they can be applied to product development. Instead of using a Theory of Constraints as a manufacturing book probably would, Reinertsen describes Kanban systems instead.

There is an entire movement in the software development industry to raise awareness of the problems that work in progress (WIP) causes to software processes. You can take a look at their work at http://www.limitedwipsociety.org/ and also a new book in the works on Kanban for Technology Organizations^[2] by David Anderson.

In Summary

All in all I found Donald Reinertsen book very interesting and easy to read. Although the book is not specific to software development most of the concepts described on it apply to software development and would allow you to understand why the recommendations of agile, lean, or pull methods work and are important.

References

[1] Principles of Product Development Flow, Second Generation Lean Product Development; Donald Reinertsen; Celeritas Publishing; 2009.

[2] Kanban: Successful Change Management for Technology Organizations; David Anderson; Unpublished manuscript as of November 2009.

[3] See http://www.kanban101.com/ for a quick introduction to Kanban

[4] Agile Management for Software Engineering, Applying Theory of Constraints for Business Results; David Anderson; Prentice Hall; 2003.

Team Dynamics

hector@hectorcorrea.com

A few years ago during my ScrumMaster certification our instructor talked about the different development stages that teams go through. He used Tuckman's development model which includes four stages: Form, Storm, Norm, and Perform.

In this model teams start at the Form stage when they are first put together. This is the honeymoon period of new teams in which spirits are high, everybody is optimistic, but the work is just about to begin. Storm is the period after the honeymoon in which, once the work has commenced, team members try to find their place in the team. After the storm phase, teams go into the Norm stage as members start to find their place on the team and the team starts working as a cohesive unit. Finally, some teams reach another stage called Perform in which they become high performance teams.

I've always found this knowledge of incredible valuable. Knowing that teams go through these stages and that this is normal behavior is very important. As a team leader is also important to recognize that teams need different leadership styles as they go through these stages.

Blanchard et al. have a very nice chart where they explain the four stages and the kind of leadership recommended on each of them:

In the first stage (Form) the team needs a significant amount of direction (as the team is newly formed) but little support since they are typically excited just by being part of the new team.

As the team moves into the second stage (Storm) and tension among team members start to raise the leader need to act as a coach and provide both, direction and support for the team to get through the rough period of recognizing each other talents and trust one another.

The Storm stage is commonly misunderstood as a "bad stage" since during this phase team members tend to start feeling down as the discrepancies between the initial hopes and reality kicks [4], the initial politeness fade, people start to get more into the work and their roles and so start to argue about things that were left unsaid or not realized when they first met [3]. However as Blanchard, et al. indicates calling Storm stage a bad stage is like calling adolescence a bad period. Storm is a process that teams go through as they develop. As a leader you should be careful to help teams grow out of this stage instead trying to prevent this stage altogether or letting the team perpetuate this stage. The article in ChangeMinds.org [3] provides some guidelines on how to manage a team during this stage including:

"The manager here needs to assert their role and help draw out and resolve differences that might otherwise bubble along under the surface, causing continuing team cohesion problems."

and

"Storming can also be reduced by clarifying work goals and individual role and objectives. When people know what individual success means, they become more focused."

When teams start moving into the third stage (Norm) performance increases and the role of the leader is to keep supporting the team to make sure the trust among team members is kept. At this point the team has a clear idea of the task at hand and need little direction. The team is already making decisions and the leader should step back to let the team flourish.

Finally, as the team reaches the fourth phase (Perform) the leader's role shift into delegation which means letting the team make their own decisions and support itself. Teams that reach this stage are called high performing teams.

Beside Tuckman's group development stages other authors have proposed different stages but the main idea is the same. Teams go through phases through their lifetime.

An important thing to note is that teams can step back to a previous stage. For example, a team that has reached the Perform stage might go back to the Norm or Storm phase. This can be caused by changes in team members or changes in the team's goals. In software development teams this could be the result of adding new developers to a team or changing the goal of the team as a result of new market pressures. In a small team just adding a few more developers (say going from four to six developers) might be enough to disrupt the roles already established by the team. As a team leader you should be aware of this and act accordingly.

An unfortunately common occurrence in software teams is adding QA resources to a development team after the team has formed and bonded. Not only it is a bad practice to push the testing of the system towards the end of a cycle but adding a new role (that will find bugs in the code and "slow down" the development!) to a team already formed exacerbates the impact of this new role and increases the likelihood of taking a team from a norm or perform stage to a less productive stage like storm. A better approach in this case would be to include QA from the beginning and make it part of the development effort. If you don't have that luxury and have to add this role towards the end make sure you address both, functional and behavior changes that the new role will bring to the team.

The take away from this post is twofold. On one hand remember that all teams go through these development stages and that this is a normal process. On the other hand is important that you as a leader recognize the stage that your team is at and manage it accordingly.

There is a lot of material regarding the stages of a team and team productivity. A good reading is Alasdair White paper From Comfort Zone to Performance Management [5].

References

[1] A copy of Bruce Tuckman's original 1965 paper "Developmental Sequence in Small Groups" can be found here.

[2] Wikipedia http://en.wikipedia.org/wiki/Forming-storming-norming-performing

[3] ChangingMinds.org article http://changingminds.org/explanations/groups/form_storm_norm_perform.htm

[4] The chart and explanation of the different roles that the manager plays on each of the team development stages was taken from the The One Minute Manager Builds High Performing Teams by Ken Blanchard et al. (William Morrow, 1990, page 70.)

[5] Alasdair White's From Comfort Zone to Performance Management paper http://www.pm-solutions.com/Performance_ManagementApril2008.pdf

TransactionScope.Complete Maybe

hector@hectorcorrea.com

During a class this week I learned that the Complete() method in the TransactionScope class works different from what I have always thought. I've always thought that when you issue a call to the Complete() method the transaction is either committed or you get an exception if the transaction cannot be committed. In my mind once the call to Complete() returned it was a sure bet that the transaction was successfully committed. It turns out that is not necessarily true.

The following code snippet shows a typical usage of TransactionScope when connecting to a SQL Server database. The steps are basically start a transaction, issue a few SQL Commands, and call the complete method to commit the transaction. In a real application you will have more than one SQL Command and perhaps nested transactions but the idea is pretty much the same.

   1:  using (TransactionScope tx = new TransactionScope())

   2:  {

   3:     using (SqlConnection conn = new SqlConnection(connString))

   4:     {

   5:        conn.Open();

   6:        string sql ="INSERT INTO MyTable(ID,Name) VALUES(1,'Test')";

   7:        SqlCommand command = new SqlCommand(sql, conn);

   8:        command.ExecuteNonQuery();

   9:     } // conn

10:

  11:     Console.WriteLine("INSERT command has been issued but not committed.");

  12:     Console.WriteLine("Stop SQL Server and press [ENTER] to continue...");

  13:     Console.ReadLine();

14:

  15:     tx.Complete();

16:

  17:     Console.WriteLine("Yikes! tx.Complete() didn't throw an exception even though SQL Server has been stopped.");

  18:     Console.WriteLine("Press [ENTER] to continue...");

  19:     Console.ReadLine();

  20:  } // tx

In this particular example however I am purposely forcing an error condition by pausing the program right before the call to Complete, then manually shutting down SQL Server, and finally letting the program continue its execution. This is definitively not something that you will do in an application but the point of this example is to emulate what happens if the server goes down right before the call to tx.Complete(). My thought was that the program will throw an exception on the call to tx.Complete() but to my surprise that was not the case. The call to tx.Complete() executed and came back with no indication that an error has happened. Below is the output that this program gave me:

Notice how the lines after tx.Complete() were executed which means that no Exception was thrown even though SQL Server was down at the time of the call! How come this is possible? Does this mean that I have no way of knowing that the transaction succeeded or failed. This is kind of a big deal.

It turns out the real issue here is that the Complete() method does not work quite the way I thought it did. Here is what what MSDN says:

When you are satisfied that all operations within the scope are completed successfully, you should call this method only once to inform that transaction manager that the state across all resources is consistent, and the transaction can be committed. It is very good practice to put the call as the last statement in the using block. [...] The actual work of commit between the resources manager happens at the End Using statement if the TransactionScope object created the transaction. If it did not create the transaction, the commit occurs whenever Commit is called by the owner of the CommittableTransaction object. At that point the Transaction Manager calls the resource managers and informs them to either commit or rollback, based on whether this method was called on the TransactionScope object.

Notice that it clearly says you call this method to inform the transaction manager that the transaction can be committed. The actual commit happens at the end using statement. Indeed, if I let the program continue its execution I do get an exception when I reach to the end of the using statement. Below is an example of the Exception that I got:

With this knowledge, let's review what happened when the line tx.Complete() was executed and SQL Server was already shutdown. The call to tx.Complete() informed the transaction manager that as far as we were concerned the transaction could be committed. This is why that line executed with no problems. Next, when the transaction scope reached the end using statement and it tried to commit it detected that something was wrong and threw an exception indicating that "the transaction is in doubt". There was also an inner exception that elaborated that a "transport-level error has occurred" in this case as a result of .NET not being able to reach SQL Server to confirm if indeed SQL Server committed the transaction or not. The best TransactionScope could do is doubt it.

This is actually really good. If a similar situation where to happen in a production environment I would doubt that the transaction was committed too. In this particular example I know for sure it wasn't committed since I shutdown SQL Server at the right moment, but in real application there wouldn't be a way for us to know if we lost connectivity with SQL before or after the commit command was sent. For example, let's say that SQL Server didn't go down but rather that our program lost network connectivity with it. In that case we couldn't confirm that SQL Server committed the transaction but SQL Server might have received the commit command just before we lost connectivity and indeed committed the transaction successfully. In that case the best TransactionScope can do is doubt it and that's as correct as you can ask for.

To make matters more interesting you can use TransactionScope to handle nested transactions and transactions against resources other than databases. I am glad to know that whoever wrote this code in the framework already thought of these conditions so that I don't have to. Check out the MDSN documentation for the TransactionScope class and on Writting Transactional Applications for more information.