Hector Correa's blog

Flow: Second Generation Lean Product Development

hector@hectorcorrea.com

The last few weeks I've been reading the book "The Principles of Product Development Flow" by Donald Reinertsen^[1] in which he describes what he calls the second generation of Lead Product Development.

I've been reading the book from the perspective of software development products and this review will highlight examples on that area. However Reinertsen's book is about product development in general (e.g. the development of a new drug or a new cell phone) and not software in particular.

Reinertsen describes a paradigm for product development that although it could be called Lean Product Development he chooses to call Flow-Based Product Development (Flow) to highlight the differences that exist between lean manufacturing and lean product development.

Unlike lean manufacturing that primarily deals with predictable and repeatable tasks, homogeneous delay costs, and homogeneous task duration, in lean product development these situations are not the norm. In product development the tasks (almost by definition) are not repeatable and hardly ever predictable, delay costs are not homogeneous, and tasks durations have very high variations.

In the case of software development it's not uncommon for teams to be dealing with problems that they have never solved before (e.g. writing a brand new system) or and with technologies that are new to the team. Likewise, the cost of delaying the implementation of a new feature versus the cost of fixing a specific bug can be widely different.

In chapter one Reinertsen starts by describing what he calls twelve critical problems with traditional product development orthodoxy including: Failure to Correctly Quantify Economics, Blindness to Queues, Worship of Efficiency, Hostility to Variability, Institutionalization of Large Batch Sizes, Underutilization of Cadence, Managing Timelines instead of Queues, Absence of Work-In-Progress Constraints, and Centralized Control.

In the remaining chapters of the book Reinertsen presents eight ideas/themes that he suggests to address these problems.

Economics
Queues
Variability
Batch Size
WIP Constraints
Cadence, Synchronization, and Flow Control
Fast Feedback
Decentralized Control

In software development some of these ideas are already popular thanks in large to the wide spread adoption of agile software development processes. For example Scrum recommendation of short sprints helps reduce batch size. Likewise, the daily collaboration between developers and QA during the sprint as well as frequent product demonstrations advocated by Scrum allow for fast feedback to take place.

However, some of the other ideas that Reinertsen presents on his book (like Economics, Queues, WIP Constraints) are not mainstream in software development yet. Some of the new development processes like Kanban^[2][3] are making strides in these areas and this book helps to clarify why these changes are important in product development in general.

On Economics

On the topics of economics, Reinertsen starts with the premise that "we do product development to make money" (p.15) and yet most product development teams cannot correctly quantify economics on their products. In his view product developers currently focus on proxy variables to calculate economic impact rather than using the real economic objective which he calls life-cycle profits (p.29)

For example, a typical proxy variable is cycle-time. Most developers typically try to optimize cycle-time and make it shorter. On the surface this sounds all good. However, few product developers can correctly calculate how much a delay on cycle-time will actually cost in terms of life-cycle profits.

An example of this in software development is when a team cannot objectively calculate the value (in terms of profit) of working on feature A or feature B. If the team does not know the impact to the bottom line, How can they make a decision that makes the more sense for the company? Without knowing the real value of two features, the team might decide to work on feature A because it's more appealing or because the developers needed for feature B are currently busy wrapping something else. However, if the team knew that feature B will bring four times more profit to the company if delivered first they might act different and rearrange the team to complete feature B before feature A.

Reinertsen points out that having a common unit of measure solves many problems as it allows the decisions on what makes more sense to take place on an even keel rather than using the unit of measurement that is popular at the moment. The common measure that he proposes is profits.

For example, given two choices to implement a feature a team might decide to rush it and release it to market rather than use a more comprehensive approach and wait a few months. Either one of these choices could be the correct one at different points in the life cycle of a product. However, the right choice depends on the economics of this decision at a given point in time. Will the company make more money rushing the feature and being first too market or will it make more money delaying its implementation and doing a more comprehensive implementation? What about the "technical debt" that the team will incur by rushing a feature? Does this cost outweigh the benefits of being first to market? All of these things should be considered and measured with the same unit.

Reinertsen indicates that on his research teams tend to think that they know the economics of their decisions. However, when asking different people on the team to quantify (in economic terms) the value of the team decisions their responses vary on a 50 to 1 ratio. When the answers are in such wide range it's obvious that the team is not using a common unit of measurement.

Chapter two contains nineteen principles to help product developers make good decisions based on economics. Although Reinertsen does not provide the "silver-bullet" to make economic decisions the principles that he provides are a good place for product developers to start focusing including paying attention to Cost of Delay (p. 31), U-Curve Optimizations (p.35), Economics Trade-off (p. 37), Marginal Cost and Marginal Value (p.45),

On Queues and Batch Size

One of the areas where lean manufacturing is very different from product development is in the management of queues. In lean manufacturing the problem with queues is well understood and has been optimized for many years. Entire manufacturing books have been written on how to manage bottlenecks on assembly lines to prevent inventory accumulation (e.g. The Goal by Eli Goldratt.) However, there is little literature in software development that addresses the effects of work in progress (which is the software equivalent to inventory.) An example of work in progress in software development are features developed but not yet in production either because they are under development or developed but still being tested.

One of the reasons work in progress (WIP) is so poorly managed in product development is because it's almost invisible. In manufacturing you can walk around a plant and see work in progress lying on the floor. Although work in progress does not sits around on the shop floor in software development work in progress can be easily identified by looking at queues: queues of items ready to be coded, items ready to be tested, items ready to be released, and so on.

You might be wondering, so what? Why should I care about queues in software development? Although there is no cost associated with storing work in progress in software development Reinertsen indicates that work in progress cause several unwanted side effects in product development including: Longer Cycle Time, Increased Risk, More Variability, More Overhead, Lower Quality, and Less Motivation.

An example of this in software development is a queue of features coded but not tested. The larger the queue of features coded but not tested the larger the test cycle that the product will need to go before the features can be released. A large number of features to be tested and then released also translate in higher risk on the new version of the system.

One of the principles of queue theory is called Little's Formula. In layman terms Little's Formula says "average queue size determines average cycle time" (p. 112.) This means that you can reduce cycle time by reducing batch size. This is important as it means that before we go looking for complex solutions to reduce long cycle times (like adding more staff or looking for bottlenecks) we can start by reducing the batch size. Software teams that are already using Agile methods like Scrum have experienced this with the use of short Sprints (e.g. 2-week long cycles) rather than the old-fashion multi-month development cycles.

On his book Agile Management for Software Engineering^[4], David Anderson gives a comprehensive explanation of how inventory and lead time apply to software and it's another good reference to learn more on this subject.

On WIP Constraints

After making you aware of Queues and the importance of Batch Size, Reinertsen moves to Work In Progress Constraints (or WIP Constraints) and why they are important in product development. Reinertsen indicates that while limiting batch size helps by nature to reduce the amount of work in progress, it does not address the fundamental problem of random variations in flow common to product development (p. 143) As Reinertsen indicates at the beginning of his book, product development is subject to many variations not common in lean manufacturing. In the Chapter 6 Reinertsen describes ideas and principles applied in telecommunication networks to deal with high variability and how they can be applied to product development. Instead of using a Theory of Constraints as a manufacturing book probably would, Reinertsen describes Kanban systems instead.

There is an entire movement in the software development industry to raise awareness of the problems that work in progress (WIP) causes to software processes. You can take a look at their work at http://www.limitedwipsociety.org/ and also a new book in the works on Kanban for Technology Organizations^[2] by David Anderson.

In Summary

All in all I found Donald Reinertsen book very interesting and easy to read. Although the book is not specific to software development most of the concepts described on it apply to software development and would allow you to understand why the recommendations of agile, lean, or pull methods work and are important.

References

[1] Principles of Product Development Flow, Second Generation Lean Product Development; Donald Reinertsen; Celeritas Publishing; 2009.

[2] Kanban: Successful Change Management for Technology Organizations; David Anderson; Unpublished manuscript as of November 2009.

[3] See http://www.kanban101.com/ for a quick introduction to Kanban

[4] Agile Management for Software Engineering, Applying Theory of Constraints for Business Results; David Anderson; Prentice Hall; 2003.

Team Dynamics

hector@hectorcorrea.com

A few years ago during my ScrumMaster certification our instructor talked about the different development stages that teams go through. He used Tuckman's development model which includes four stages: Form, Storm, Norm, and Perform.

In this model teams start at the Form stage when they are first put together. This is the honeymoon period of new teams in which spirits are high, everybody is optimistic, but the work is just about to begin. Storm is the period after the honeymoon in which, once the work has commenced, team members try to find their place in the team. After the storm phase, teams go into the Norm stage as members start to find their place on the team and the team starts working as a cohesive unit. Finally, some teams reach another stage called Perform in which they become high performance teams.

I've always found this knowledge of incredible valuable. Knowing that teams go through these stages and that this is normal behavior is very important. As a team leader is also important to recognize that teams need different leadership styles as they go through these stages.

Blanchard et al. have a very nice chart where they explain the four stages and the kind of leadership recommended on each of them:

In the first stage (Form) the team needs a significant amount of direction (as the team is newly formed) but little support since they are typically excited just by being part of the new team.

As the team moves into the second stage (Storm) and tension among team members start to raise the leader need to act as a coach and provide both, direction and support for the team to get through the rough period of recognizing each other talents and trust one another.

The Storm stage is commonly misunderstood as a "bad stage" since during this phase team members tend to start feeling down as the discrepancies between the initial hopes and reality kicks [4], the initial politeness fade, people start to get more into the work and their roles and so start to argue about things that were left unsaid or not realized when they first met [3]. However as Blanchard, et al. indicates calling Storm stage a bad stage is like calling adolescence a bad period. Storm is a process that teams go through as they develop. As a leader you should be careful to help teams grow out of this stage instead trying to prevent this stage altogether or letting the team perpetuate this stage. The article in ChangeMinds.org [3] provides some guidelines on how to manage a team during this stage including:

"The manager here needs to assert their role and help draw out and resolve differences that might otherwise bubble along under the surface, causing continuing team cohesion problems."

and

"Storming can also be reduced by clarifying work goals and individual role and objectives. When people know what individual success means, they become more focused."

When teams start moving into the third stage (Norm) performance increases and the role of the leader is to keep supporting the team to make sure the trust among team members is kept. At this point the team has a clear idea of the task at hand and need little direction. The team is already making decisions and the leader should step back to let the team flourish.

Finally, as the team reaches the fourth phase (Perform) the leader's role shift into delegation which means letting the team make their own decisions and support itself. Teams that reach this stage are called high performing teams.

Beside Tuckman's group development stages other authors have proposed different stages but the main idea is the same. Teams go through phases through their lifetime.

An important thing to note is that teams can step back to a previous stage. For example, a team that has reached the Perform stage might go back to the Norm or Storm phase. This can be caused by changes in team members or changes in the team's goals. In software development teams this could be the result of adding new developers to a team or changing the goal of the team as a result of new market pressures. In a small team just adding a few more developers (say going from four to six developers) might be enough to disrupt the roles already established by the team. As a team leader you should be aware of this and act accordingly.

An unfortunately common occurrence in software teams is adding QA resources to a development team after the team has formed and bonded. Not only it is a bad practice to push the testing of the system towards the end of a cycle but adding a new role (that will find bugs in the code and "slow down" the development!) to a team already formed exacerbates the impact of this new role and increases the likelihood of taking a team from a norm or perform stage to a less productive stage like storm. A better approach in this case would be to include QA from the beginning and make it part of the development effort. If you don't have that luxury and have to add this role towards the end make sure you address both, functional and behavior changes that the new role will bring to the team.

The take away from this post is twofold. On one hand remember that all teams go through these development stages and that this is a normal process. On the other hand is important that you as a leader recognize the stage that your team is at and manage it accordingly.

There is a lot of material regarding the stages of a team and team productivity. A good reading is Alasdair White paper From Comfort Zone to Performance Management [5].

References

[1] A copy of Bruce Tuckman's original 1965 paper "Developmental Sequence in Small Groups" can be found here.

[2] Wikipedia http://en.wikipedia.org/wiki/Forming-storming-norming-performing

[3] ChangingMinds.org article http://changingminds.org/explanations/groups/form_storm_norm_perform.htm

[4] The chart and explanation of the different roles that the manager plays on each of the team development stages was taken from the The One Minute Manager Builds High Performing Teams by Ken Blanchard et al. (William Morrow, 1990, page 70.)

[5] Alasdair White's From Comfort Zone to Performance Management paper http://www.pm-solutions.com/Performance_ManagementApril2008.pdf

TransactionScope.Complete Maybe

hector@hectorcorrea.com

During a class this week I learned that the Complete() method in the TransactionScope class works different from what I have always thought. I've always thought that when you issue a call to the Complete() method the transaction is either committed or you get an exception if the transaction cannot be committed. In my mind once the call to Complete() returned it was a sure bet that the transaction was successfully committed. It turns out that is not necessarily true.

The following code snippet shows a typical usage of TransactionScope when connecting to a SQL Server database. The steps are basically start a transaction, issue a few SQL Commands, and call the complete method to commit the transaction. In a real application you will have more than one SQL Command and perhaps nested transactions but the idea is pretty much the same.

using (TransactionScope tx = new TransactionScope())

{

using (SqlConnection conn = new SqlConnection(connString))

{

conn.Open();

string sql ="INSERT INTO MyTable(ID,Name) VALUES(1,'Test')";

SqlCommand command = new SqlCommand(sql, conn);

command.ExecuteNonQuery();

} // conn

Console.WriteLine("INSERT command has been issued but not committed.");

Console.WriteLine("Stop SQL Server and press [ENTER] to continue...");

Console.ReadLine();

tx.Complete();

Console.WriteLine("Yikes! tx.Complete() didn't throw an exception even though SQL Server has been stopped.");

Console.WriteLine("Press [ENTER] to continue...");

Console.ReadLine();

} // tx

In this particular example however I am purposely forcing an error condition by pausing the program right before the call to Complete, then manually shutting down SQL Server, and finally letting the program continue its execution. This is definitively not something that you will do in an application but the point of this example is to emulate what happens if the server goes down right before the call to tx.Complete(). My thought was that the program will throw an exception on the call to tx.Complete() but to my surprise that was not the case. The call to tx.Complete() executed and came back with no indication that an error has happened. Below is the output that this program gave me:

Notice how the lines after tx.Complete() were executed which means that no Exception was thrown even though SQL Server was down at the time of the call! How come this is possible? Does this mean that I have no way of knowing that the transaction succeeded or failed. This is kind of a big deal.

It turns out the real issue here is that the Complete() method does not work quite the way I thought it did. Here is what what MSDN says:

When you are satisfied that all operations within the scope are completed successfully, you should call this method only once to inform that transaction manager that the state across all resources is consistent, and the transaction can be committed. It is very good practice to put the call as the last statement in the using block.

[...]

The actual work of commit between the resources manager happens at the End Using statement if the TransactionScope object created the transaction. If it did not create the transaction, the commit occurs whenever Commit is called by the owner of the CommittableTransaction object. At that point the Transaction Manager calls the resource managers and informs them to either commit or rollback, based on whether this method was called on the TransactionScope object.

Notice that it clearly says you call this method to inform the transaction manager that the transaction can be committed. The actual commit happens at the end using statement. Indeed, if I let the program continue its execution I do get an exception when I reach to the end of the using statement. Below is an example of the Exception that I got:

With this knowledge, let's review what happened when the line tx.Complete() was executed and SQL Server was already shutdown. The call to tx.Complete() informed the transaction manager that as far as we were concerned the transaction could be committed. This is why that line executed with no problems. Next, when the transaction scope reached the end using statement and it tried to commit it detected that something was wrong and threw an exception indicating that "the transaction is in doubt". There was also an inner exception that elaborated that a "transport-level error has occurred" in this case as a result of .NET not being able to reach SQL Server to confirm if indeed SQL Server committed the transaction or not. The best TransactionScope could do is doubt it.

This is actually really good. If a similar situation where to happen in a production environment I would doubt that the transaction was committed too. In this particular example I know for sure it wasn't committed since I shutdown SQL Server at the right moment, but in real application there wouldn't be a way for us to know if we lost connectivity with SQL before or after the commit command was sent. For example, let's say that SQL Server didn't go down but rather that our program lost network connectivity with it. In that case we couldn't confirm that SQL Server committed the transaction but SQL Server might have received the commit command just before we lost connectivity and indeed committed the transaction successfully. In that case the best TransactionScope can do is doubt it and that's as correct as you can ask for.

To make matters more interesting you can use TransactionScope to handle nested transactions and transactions against resources other than databases. I am glad to know that whoever wrote this code in the framework already thought of these conditions so that I don't have to. Check out the MDSN documentation for the TransactionScope class and on Writting Transactional Applications for more information.

The Model View Controller is dead, long live the Model View Controller

hector@hectorcorrea.com

The last few days I’ve been reading Microsoft .NET: Architecting Applications for the Enterprise by Dino Esposito and Andrea Saltarello. On Chapter 7 Dino and Andrea talk about the Presentation Layer and in particular they touch on the Model View Controller (MVC), Model View Presenter (MVP), and Presentation Model (PM) design patterns. I really like how they demystify MVC and MVP and put them in context. Dino and Andrea start describing how these design patterns have evolved over time and state that

“the definition of MVC that worked in the 1980s and even the 1990s might not work for today. As a result, probably nobody uses MVC today as its authors originally devised it.”

I completely agree with this statement and as a matter of fact this quote is the reason I stopped on the Presentation Layer chapter as I was browsing through the book even though I didn’t have an immediate need for Presentation Layer patterns. I was actually looking for Business Layer and Service Layer Patterns but that’s a topic for a future blog entry.

Another interesting observation that Dino and Andrea make is that “when MVC was initially developed it was intended to be used on the whole application and not just in the presentation layer”. However, since these days most developers intuitively design applications in layers as opposed to building monolithic applications as it might have been the norm in the 1980s, the role of the MVC pattern has shifted to a presentation layer pattern more than application level pattern.

Dino and Andrea do a great job of explaining the MVC pattern with a simple sequence diagram rather than using the quintessential chart with three circles and the words model, view and controller inside them. Below is a drawing of the sequence diagram that they use in the book:

From this diagram is easy to see that upon a user action (e.g. click of a button) the view talks to a controller, and the controller in turn talks to the model. It makes me wonder if this pattern should have been called View Controller Model pattern instead. The model notifies the view that changes have occurred in the model and the view in turn reads the new model. Another observation that Dino and Andrea make is that the model in MVC is what we call these days the business logic layer.

The View in MVC represents the controls that are drawn on the screen. In a .NET application they might represent Windows Forms controls or Web Controls. Ideally the view should be as simple as possible. The view should just draw the controls, intercept user actions, and forward them to a controller. In practice the view typically has some logic inside of it for handle things related to the presentation layer (e.g. data binding and sometimes even simple data validation.) There is a tradeoff between having the simplest of the views (one with no logic) and one with some logic on it. If the view has no logic the controller tends to be more involved but there is only one place with code logic on it. If the view has some logic on it the controller tens to be simpler but now you have logic in two places. This is neither right nor wrong. It’s just a decision that you as a developer will face and need to choose what is best for you given the particular application that you are building.

From the sequence diagram above you can also see how the Controller talks to the model (business layer) to take care of the request made by the user through the view. One of the main jobs of the controller is to script the required calls to the model. If the user’s action requires a new view the controller is also responsible for creating a new view, model, and controller and passing the control to this new MVC triad. If the user’s action does not require a new view then the controller just scripts the calls to the existing model. Something to keep in mind is that selecting which view to go to next can be a complex job in a real application. There is nothing intrinsic in MVC to make this simpler.

A common misunderstanding is that the controller in MVC separates the view from the model. However, as Dino and Andrea point out, and as you can see from the aforementioned sequence diagram, in MVC “the view knows the model directly and the model knows the view through the Observer relationship” Newer patterns (some of them derived from MVC) have addressed this issue and made other improvements to MVC to produce design patterns more suitable for today's applications. To put it in the words of Dino and Andrea

“It is a sharp and bold statement, but we have to make it: today classic MVC is gone. However, some of its variations are healthy and thrive. They are Model2 for the Web and MVP for both Web and Windows. In turn, and only more recently, MVP has undergone a facelift. In July 2006, Martin Fowler proposed to retire MVP entirely and replace it with two variations Passive View (PV) and Supervising Controller (SVC). “

Model 2: MVC for the web

Model2 is a new pattern derived from Model View Controller. Again, to quote Dino and Andrea “Model2 is just MVC adapted to a new medium – the Web – that just wasn’t around at the time MVC was first devised” Model2 was originally created for Java Server Pages but is now also used in the Microsoft’s ASP.NET MVC Framework which makes it the more relevant for .NET developers.

To review the differences between Model2 and the classic MVC let's look at a sequence diagram of the Model2:

There are a couple of things that become evident as soon as you compare this diagram with the classic MVC diagram that we reviewed first. The first thing is that the Model2 diagram is a little bit more complicated than MVC since there are more components involved. Secondly there is still a view, a model, and a controller embedded on it.

Another two of things that are also shown in this diagram that might not very evident at first are that (1) the view is rather passive in this pattern as it does not talk to anything and (2) in Model2 the controller (and not the view) is the one that talks to both the model and the view which is quite different from classic MVC.

The fact that the view becomes passive in Model2 means that is very much just a set of controls that are drawn based on some information (the “viewModel”) passed from the controller. ViewModel is a class with data like a dataset or a custom data class that the view uses to render HTML controls (remember that Model2 is targeted to the web.) The view in this case receives data through the viewModel but does not have access to the business layer (Model.) Although this is a subtle difference it makes the view in Model2 simpler than in MVC which is one of the goals on some frameworks like the Microsoft ASP.NET MVC framework.

So what about the new elements introduced in Model2 like Browser and Front Controller? Browser represents the users’ browser and, as the diagram indicates, its job is to submit HTTP POST requests and render HTML.

The Front Controller in ASP.NET Front Controller is implemented via an HTTP module that captures the requests, analyzes them and decides what controller should handle them. Scott Gu has a very good post on how this works in ASP.NET MVC.

Model View Presenter

As I mentioned before, another variation of the MVC design pattern is the Model View Presenter (MVP) pattern. MVP has in turn been superseded by two other patterns (Passive View and Supervising Controller) that are worth exploring but are beyond the scope of this blog entry. Dino and Andrea do a great job of explaining the differences between these three patterns on their book.

I highly recommend Dino and Andrea's book to anyone doing applications with .NET and that has not reviewed design patterns in a while. Dino and Andrea do a great good job describing well known design patterns for business, service, data access, and presentation layers in a modern perspective and targeted for .NET development.

In addition to their book, Dino has a very good MSDN article on ASP.NET Presentation Patterns that is worth checking out.

Rubber Band Reports with Pivot Tables (reprint)

hector@hectorcorrea.com

A few years ago I wrote an article for the Universal Thread Magazine on how to create reports using Excel's Pivot Tables. The article was titled Programming Rubber Band Reports with Pivot Tables and showed how to create Excel Pivot Tables from a Visual FoxPro (VFP) application.

Although the article is more than 6 years old I still get questions every now and then about the code samples that I mentioned in the article. The code samples mentioned in the article can be found at http://hectorcorrea.com/downloads/correa_pivottablesvfp.zip A VB.NET version of the code can be found at http://hectorcorrea.com/downloads/pivottable.zip

A PDF version of the article can also be found here.

Expose your data as an RSS Feed with .NET 3.5

hector@hectorcorrea.com

In talking with my one of my friends a couple of months ago I found that .NET 3.5 now has built-in classes to generate RSS Feeds. This was great news to me since both my personal blog (this site) and the DotWiki expose data as RSS Feeds but for the longest time I've been using an unsupported library that I downloaded from the web years ago.

Last month I updated the DotWiki to use these new .NET classes (instead of the old unsupported library) to expose recent changes to Wiki topics as an RSS Feed. Although I found the .NET classes to generate feeds rather comprehensive I had a hard time finding good practical examples on MSDN on how to use them^[*]. Luckly Bipin Joshi posted two great examples on how to generate RSS feeds and consume RSS feeds with these .NET classes.

Below is how the code in the DotWiki looks like to expose as an RSS Feed the list of changes in the last 24 hours. The complete code is available from the DotWiki source code page.

using System.ServiceModel.Syndication;

// Define the feed

SyndicationFeed feed = new SyndicationFeed();

feed.Title = new TextSyndicationContent("DotWiki");

feed.Description = new TextSyndicationContent("DotWiki RSS Feed");

// ... and the link for the site the feed represents

SyndicationLink link = new SyndicationLink();

link.Title = "DotWiki";

link.Uri = new Uri("http://code.hectorcorrea.com/DotWiki/");

feed.Links.Add(link);

// Load items that we want on the feed (more on this below)

feed.Items = LoadBlogItems();

// Prepare the response with the proper encoding and content type

Response.Clear();

Response.ContentEncoding = System.Text.Encoding.UTF8;

Response.ContentType = "text/xml";

// Dump the content of the feed as an RSS 2.0 feed into the response

XmlWriter rssWriter = XmlWriter.Create(Response.Output);

Rss20FeedFormatter rssFormatter = new Rss20FeedFormatter(feed);

rssFormatter.WriteTo(rssWriter);

rssWriter.Flush();

rssWriter.Close();

Response.End();

The code for LoadBlogItems which is the one that adds individual items to the feed is shown below. This method returns a generic list of SyndicationItems. Each element in the list contains the information about a topic that has been changed in the last 24 hours. Classes Topic and TopicList in this example are DotWiki specific classes but the rest is plain .NET code.

private List<SyndicationItem> LoadBlogItems()

{

List<SyndicationItem> items = new List<SyndicationItem>();

TopicList topics = TopicList.GetRecentChanges(GetLatest.Last24Hours);

foreach (TopicInfo topicInfo in topics)

{

Topic topic = Topic.GetTopic(topicInfo.Id);

// Define basic element about this topic

SyndicationItem item = new SyndicationItem();

item.Id = topic.Id.ToString();

item.Title = new TextSyndicationContent(topic.Name);

item.Content = new TextSyndicationContent(topic.Content);

item.LastUpdatedTime = topic.UpdatedOn;

string url = string.Format("http://code.hectorcorrea.com/DotWiki/?topic={0}", HttpUtility.UrlEncode(topic.Name));

item.AddPermalink(new Uri(url));

// ...and the author of the topic

SyndicationPerson author = new SyndicationPerson();

author.Email = "somebody@somesite.com";

item.Authors.Add(author);

items.Add(item);

}

return items;

}

The RSS Feed generated by the .NET classes seems to be fully compatible with popular feed readers like Bloglines and Google Reader. From reading the MSDN documentation on the System.ServiceModel.Syndication namespace it seems that there is support for Atom 1.0 in addition to RSS 2.0. I have only tested the RSS 2.0 formatter, though.

[*] update: Actually there is a rather nice MSDN code sample under the SyndicationFeed class documentation that shows how to expose the same feed as an Atom or RSS feed.

Multiple Endpoints for a WCF Service

hector@hectorcorrea.com

One of the nice features that Windows Communication Foundation (WCF) provides is the ability to expose a single service via multiple endpoints so that different client applications can consume the same service over different network protocols. For example the same service can be exposed via HTTP for external clients and via TCP for internal clients.

In WCF an endpoint refers to the combination of the address where service is available, the binding used to communicate to it, and the contract that the service exposes, these three parameters are commonly known as the "ABC" of the endpoint.

The address represents the URL where the service is to be found, for example http://localhost:8000/HelloWorld/HelloWorldService or net.tcp://localhost:9000/HelloWorld/HelloWorldService
The binding describes the transport protocols for the communication channel, for example HTTP vs TCP
The contract represents the functionality that the service exposes and it maps to a [ServiceContract] interface in your C# code.

An endpoint for a WCF Service can be configured via the app.config of the service or through code. In this post I am going to focus on how to configure a service via the app.config file to be available to multiple endpoints.

The Service

So let's assume we have a simple "Hello World" WCF service that implements IHelloWorld interface like the one shown in the following code snippet.

[ServiceContract(Namespace="http://hectorcorrea.com/wcf")]

public interface IHelloWorld

{

[OperationContract] string Greetings(string yourName);

[OperationContract] DateTime TheTimeIs();

}

public class HelloWorldService : IHelloWorld

{

public string Greetings(string yourName)

{

return string.Format("Hello {0}", String.IsNullOrEmpty(yourName) ? "Guest" : yourName);

}

public DateTime TheTimeIs()

{

return DateTime.Now;

}

To host this service you can use the following code. The first line creates a host object for the HelloWorldService and the second line opens a communication channel to receive and process requests for this service. The call to host.Open() in this example will read the settings from the app.config (more on this later) to decide what endpoints will be opened for the service. The rest of the code displays what endpoints were found in the app.config for this service and finally we use ReadLine to put the host in a wait state and let it process requests as they come.

ServiceHost host = new ServiceHost(typeof(TheService.HelloWorldService));

host.Open();

Console.WriteLine("Host has been started");

Console.WriteLine("Endpoints");

foreach (ServiceEndpoint endpoint in host.Description.Endpoints)

{

Console.WriteLine("Address: {0}", endpoint.Address);

Console.WriteLine("Binding: {0}", endpoint.Binding);

Console.WriteLine("Contract: {0}", endpoint.Contract.ContractType);

Console.WriteLine();

}

Console.WriteLine("Please [ENTER] to close the service host...");

Console.ReadLine();

host.Close();

Server Side Configuration

The following XML shows how the different endpoints for the TheService.HelloWorldService are defined in the app.config. This is the config file that lives on the server side. Notice the three endpoints for the contract TheService.IHelloWorld, these endpoints expose the service via basic HTTP binding, TCP binding, and WS binding and each of them has its own URL.

[[[xml:MultipleEndpointsService01.xml]]]

On the Client Side

Now that we have a WCF HelloWorld service implemented and a host running waiting for requests to be send let's turn to the client side and create a simple console application to consume it.

The following code will connect to the HelloWorld service via a proxy class called HelloWorldClient using the endpoint named "tcpEndpoint". It then makes a couple of calls to the Greetings and TheTimeIs methods in the service and finally closes the conection.

HelloWorldClient service = new HelloWorldClient("tcpEndpoint");

Console.WriteLine("Greetings: {0}", service.Greetings("Hector"));

Console.WriteLine("The time is: {0}", service.TheTimeIs());

service.Close();

Console.WriteLine("Channel closed.");

Console.WriteLine();

In this code the class HelloWorldClient is a proxy for the HelloWorld service that we created before. The process to generate this proxy class is beyond the scope of this post (see references at the bottom of this post for more information.) In a nutshell, this proxy is a class that implements a skinny version of the HelloWorldService that uses WCF to forward the request accross the wire to the actual server. Thanks to WCF we don't have to write any of the low level networking stuff to deal with HTTP or TCP since WCF wraps that functionality for us.

This proxy class makes the code in the client application looks like its calling a class running on the same machine as the client. By just looking at this code, you have no idea if the call to service.Greetings is to a class running on the same machine as the client or in a different machine across the country.

So how do we tell the client where to find the real HelloWorld service? As with the server side settings you can do it either via code or using a config file. In this post I am going to show how to do it using a config file (keep in mind that this client side config file is different config from the server side config file that we discussed before.)

Client Side Configuration

Like the server side configuration file that we reviewed before, the client side configuration also has endpoints and, as you can imagine, they are subset of the endpoints that the server exposes. The client side config tells the client application where to find the service while the server side config tells the server where to listen for requests.

The following XML shows how the endpoints are defined in the client side config. In this particular example we have all three endpoints for the HelloWorldService listed for illustration purposes, but we could have as well just one. You will notice that the definition of these endpoints is similar to the ones that we used for the server config file as they also have an address, a binding, and a contract. Yet, keep in mind that this is a different config file -- this config defines client side settings.

[[[xml:MultipleEndpointsService02.xml]]]

You will notice that the name (tcpEndpoint) of the second endpoint in this XML matches with the name of the endpoint that we indicated in the line

HelloWorldClient service = new HelloWorldClient("tcpEndpoint");

When this line is executed WCF will read the values of the tcpEndpoint from the client config file and open a TCP communication channel at the address indicated to communicate with a service that recognizes messages for the HelloWorldService contract.

After this the calls to service.Greetings() and service.TheTimeIs() are automatically forwarded to the real service. If our service is up and running and listening for request on the same endpoint (i.e. same address, binding, and contract) it will process the request and return the expected result.

You could change the first line to use the HTTP endpoint by just changing the name of the endpoint

HelloWorldClient service = new HelloWorldClient("httpEndpoint");

That's it, with one little change the client application will now use HTTP instead of TCP to send requests to the server but the rest of the code in the client application remains unchanged. As I mentioned at the beginning a typical scenario is to use TCP for internal clients and HTTP for external clients.

References

A great book on WCF and where I learned most the information in this blog post is Learning WCF by Michelle Leroux Bustamante (aka that indigo girl).

If you want more information on how to create the proxy class used on the client two good references are this MSDN link and Miguel Castro's article on how to do it manually.

Flickr Library in C#

hector@hectorcorrea.com

The last few days I've been working on a sample C# library to access Flickr pictures using Flickr's own API. This library is a light-weight C# class that provides two basic features: (1) retrieve the list of photosets in your account and (2) retrieve information about the photos in a photoset.

You can view a running version of this code at http://code.hectorcorrea.com/FlickrLibDemo It's nothing fancy but it should give you an idea of what the class can do for you.

Using this class you can retrieve the photosets in your Flickr account and display them in a combobox with a few lines of code like these:

Flickr flickr = new Flickr("YourApiKey", "YourUserID");

List<PhotoSetInfo > photosets = flickr.GetPhotoSets();

cboPhotoSets.DataSource = photosets;

cboPhotoSets.DataTextField = "Title";

cboPhotoSets.DataValueField = "PhotoSetID";

cboPhotoSets.DataBind();

Once you select a photoset you can retrieve information about the photos inside it (like title and URL) with the following code. In this example I am binding the list of photos to an ASP.NET datalist control to display them:

Flickr flickr = new Flickr("YourApiKey", "YourUserID");

List<PhotoInfo> photos = flickr.GetPhotosInSet(cboPhotoSets.SelectedValue);

listPhotos.DataSource = photos;

listPhotos.DataBind();

lblPhotosMsg.Text = (photos.Count == 0) ? "No photos in this photoset" : "";

The Flickr API supports multiple requests and response formats. I am using the REST format for requests and responses in my FlickrLib library. A typicall Flickr REST request looks like this:

http://api.flickr.com/services/rest/?method=methodName&parameter1=value1&parameter2=value2

where methodName is the name of the method to call, for example flickr.photosets.getList to return a list of photosets.

REST responses that Flickr provide are XML documents with the pertinent information for the request. For the flickr.photosets.getList method the reponse would be an XML document with a photosets node and several photoset children nodes. These XML responses are very easy to parse with LINQ to XML and turn them into C# strong-typed objects (like PhotoSetInfo and PhotoInfo) which then can be bound to .NET classes. That's exactly what the FlickrLib library does, it provides a wrapper to the Flickr API functionality so that you can easily consume it from your .NET applications.

References

You can download the C# source code for this class from http://hectorcorrea.com/downloads/FlickrLibDemo.zip

For information about the Flick API visit http://www.flickr.com/services/api/ there is a lot of information there on what the API provides and how to use it. There are tons of Flickr features that the API provide that you might find useful and that my little class does not expose.

You will need to obtain an API key from Flickr in order to use their API through this class or any other class. The API key is free but it is needed so that Flickr can prevent a single user from submitting too many requests to their site and take it down.

DevConnections 2008

hector@hectorcorrea.com

Below are the brief notes that I took while at DevConnections in Las Vegas (November/2008) This blog entry is mostly bullet points of things that I found interesting without much details. I'll elaborate on some of these topics in future blog entries.

Tuesday, November 11th

Opening Keynote

Scott Gu started with an overview of some of the new features released in the last year which included:

Dynamic data access (to build a database application with a wizard
Charting controls for ASP.NET
ASP.NET MVC - TDD friendly, closer to the metal (HTML) programming model, Not for everyone
Several Silverlight demos

Then he went to describe some of the new things coming in Visual Studio 2010 and ASP.NET 4.0 which included:

Visual Studio's UI will use WPF rather than Windows Form
Better support for release and deployment (e.g. support for multiple web.config files, one for each environment)
Better support for documentation and integration with TFS work items
Native support for SharePoint
TDD workflow
Distributed caching for ASP.NET apps (look for project Velocity)
NET continuum (web, desktop, RIA)

Chris Hammond has a very nice summary of this sesion.

Entity Framework in a World of Services and Processes

The Entity Data Model (.edmx) generated with the Entity Framework (EF) is serializable by default which makes it compatible with WCF out of the box.

Context in EF is the equivalent to a database connection in data access

Gotcha: You need to enumerate LINQ results before passing them back with the EF if you (as you should) surround your context with a using statement

using (EF context)

{

var blog = from x in ctx.Blogs select y;

return blogs.ToArray() // enumerate before end of using

}

In my opinion, the Entity Framework seems unfinished. Although EF makes really easy to ready data using an object model the interface to save data (inserts and updates) looks complicated. Several "hacks" were needed to save data. The problem seems to stem from the fact that the EF is very good understanding state but not inferring intent. Currently all operations apply to the entire object graph and there is no easy way to make operations more granular. This will be worked out in version 2.

ADO.NET Data Sets (formerly known as Astoria) makes data accessible through HTTP via a REST approach. In this approach the intent is inferred from the URL.

SOAP uses lots of verbs while REST few verbs (put/get/post/delete) but lots of nouns.

ASP.NET MVC - so what?

Scott Hanselman gave a very good presentation describing what MVC provides and does not when compared with traditional ASP.NET Web Forms.

The first thing he clarified is that ASP.NET is greater than MVC or WebForms. Secondly, both approaches are here to stay. They target different audiences.

ASP.NET with Web Forms hides HTML from the developer and adds a state subsystem to HTML.

MVC is a new project type in ASP.NET. It gives the developer more control over the HTML, lends to code easier to test (TDD friendly), but it is not for everyone. Unlike WebForms there are no event in MVC, that is there is no button’s click event, no page postback, no viewstate, and no page lifecycle.

If we lose so many “web form” features with MVC, isn’t it like going back to classic ASP? No, there was no separation of concerns in classic ASP (more on this below) the code was all lumped in a single page and it wasn’t easy to test.

You can mix and match WebForms and MVC in a single web site.

A typical URL in an MVC follows this format: http://controller/method/methodparameter, for example: http://products/retrieve/3

The name MVC comes from the Model View Controller pattern. Sites with MVC use a controller (C) to interact with a model (M) to get data and return views (V). This model enforces a clear separation of concerns between the model, the view, and the controller, for example, the controller does not know anything about HTML, that’s the job of the view. The view does not know anything about the database, that’s the job of the model.

The default view used by MVC uses WebForms. You can build your own or use third party views. Scott showed a kind of view using a third party product called NHAML that was quite different to a traditional WebForm way of constructing an HTML page.

LINQ to everything

The speakers gave a tour of all the different LINQ flavors to query a variety of data sources. Slides can be found at http://blogs.msdn.com/aconrad/

LINQ to objects - Provides a mechanism to query objects.

LINQ to SQL – Provides LINQ’s functionality but targeting a database (SQL Server only) rather than objects.

LINQ to Entities – A higher level of abstraction than LINQ to SQL, includes support for modeling (Entity Data Models) and supports multiple backend via plug-ins (although the only available at this point is for SQL Server)

LINQ to DataSet – So that you don’t los you investment on DataSets Microsoft provides this flavor. The tag line is: "Not having to say you're sorry" Supports everything from DataSets (e.g. offline access) with a few extra features (e.g. LINQ joins)

LINQ to XML – Provides a lightweight XML API (lighter than the DOM.) Several new XML classes are LINQ aware. For example, the new XDocument is LINQ aware but not the old XmlDocument.

Easier to generate XML documents with Linq to XML than traditional use of the DOM.

XDocument xdoc = new XDocument(

new XElement("contacts",

from c in contacts where country = "South Africa")

select new XElement("name")

VB.NET you can type XML directly as regular VB code (without having to add quotes) and embed expressions in the XML (similar to the way they are embedded in an ASP.NET) that get rendered at runtime to produce the final XML. This is really cool. Too bad we don’t have it in C#.

LINQ to ADO.NET Data Services - ADO.NET Data Services allow you to publish your data model as REST-based web services so that any HTTP client can consume them. For example you can use a URL like this http:/localhost/myservice.svc/customers("ALFKI")/Orders to retrieve orders for customer with key “ALFKI”. You can query ADO.NET Data Services with LINQ.

LINQ to Cloud Data Sources - At PDC earlier this year Microsoft unveiled two "cloud data sources": Microsoft Azure and SQL Data Services

LINQ to [insert your custom provider here] – You can write your own data source and make it LINQ friendly. Must implement IQueryable<T> and IQueryProvider. The two main challenges are (1) translate LINQ expressions to your native data source language and (2) transform query results to objects. They demo a LINQ to Twitter implementation.

Implementing RESTful Services with WCF 3.5 SP1

www.RobBagby.com

In 1993 Web = Content, today Web = Content + Capabilities. Web is a graph of linked resources. Resources support a fixed set of operations (HTTP verbs)

REST is an architectural style, not a specification

Play well with HTTP (honor response codes, use verbs) as we build other applications to run on the web. HTTP has proven to be good why not continuing using it.

New 3.5 binding webHttpBinding does not use SOAP envelope, it's RESTful friendly.

Use WebGet and WebInvoke attributes.

WCF REST Starter Kit (CodePlex project) has three components: New DLL Microsoft.ServiceModel.Web.dll, Visual Studio 2008 Templates and REST Samples.

Rob suggested four features our services should provide: (1) Use appropriated HTTP verbs, (2) Chose a representation, (3) Provide addressability, and (4) use valid HTTP response codes. These features are fully supported in the WCF REST Starter Kit via (1) WebGet attribute, (2) ResponseFormat parameter, (3) URITemplate, and (4) WebProtocolException class.

Wednesday, November 12th

From one server to two: Making the leap

I came late to this session so I don’t have much on the first part of it but I found it very interesting. The speaker (Richard Campbell) was very honest on what you get and what you don’t get when you start using more than one server to host your application.

You use load balancing to improve reliability and scalability, but that does not necessarily translate into performance. By using load balancing the application will be more resistant to hardware faults (reliability) and support more users (scalability) but it won’t become faster.

NLB adds overhead to the processing of a request, especially if you use server affinity.

You tell the cluster to drain a server, this will let the server finish processing whatever requests/sessions are currently on the server but it won’t take more requests. This work as expected when no affinity is used. When affinity is used draining a server can take hours since sessions don’t expire immediately and users could keep submitting requests on the same session and the machine won’t be free up until all sessions have been closed or expired.

If you use the no affinity approach objects need to be marked as serializable to be saved in the session cache. This is because with no affinity session is stored out of proc. It’s very important to keep session information to a minimal when storing it out of process.

Windows comes with a State Server that stores session (state) information in memory. It’s about twice as fast as storing information in SQL Server but it is a single point of failure. You can store session information in SQL Server as well and cluster SQL Server to prevent single point of failure. Third parties are worth considering.

He recommends that if you create a cluster, go all the way: use the no affinity approach.

When testing clusters make sure you test large time spans (10-15 minutes at least) to make sure you get a realistic picture.

Windows NLB detects if a machine goes down but it does not detect if IIS becomes non-responsive. You get what you paid for it.

WCF Durable Services

Juval Lowy started this session by talking about Custom Context in WCF and you can use this to pass out of band parameters in the header of a request. For example, if you need to have a way to identify who is making the call instead of passing an extra parameter on each method call you can store this information in the Context of the request and pass it back and forth.

There are three bindings that support this feature, look for bindings with the word “Context” on them, for example BasicHttpContextBinding.

He’s build a few wrapper classes that make using the context information a breeze. Download them from his site: http://idesign.com

After explaining this Juval moved to talk about durable sessions. He started by describing how in most client/server programming there is the implicit assumption that requests are short (milliseconds to minutes) Most client/server programming models assume this and work OK when the duration of the requests are short but fall short when requests take hours or days. On large time spans clients may come and go; hosts may come and go or move to different machines.

For long running requests you cannot keep service state in memory at all times, you need to persist it. Clients need to provide a state identifier so that the server can pull the correct state from the persistent storage. There are two basic was ways to pass this information around. One is to add an extra parameter to every call and pass the session identifier there but this is ugly and introduces “infrastructure” information to the business methods. A better approach is to pass this information in the context of the request.

You can add values to the context of a request from both, the client and the server. Server is the preferred approach.

If you change values on a static context on the server side make sure you make it thread safe (i.e. use lock)

To use durable services add DurableService attribute to your service class.

You can mark some of your methods with DurableOperation(CompleteInstance=true) to indicate that this method completes a session (e.g. the work to be done has been completed) and let the server know that when this method is called it can clear the session information for this caller. Once you end a session you need to recreate it (custom code required.)

Juval showed all his examples using a “File Persistent” provider to keep things simple. WCF provides a SQL Persistent provider. Look under the 3.5\SQL folder for the scripts to create the database.

Towards the end Juval closed the session by demoing “transactional memory” – this was very cool but I got lost in the process and my notes are not very clear. The basic idea was that you can implement transactions for operations that happen in memory. For example, execute this process but don’t change any values in memory if an exception happens in the process. Basically you decide when changes to values in memory are committed.

Building a WCF Router for Your Applications

This Michelle Leroux Bustamante talked how to create a WCF Service to route messages. There are two kinds of routers: Pass-Through routers and Processing Routers. This was by far the most advanced session that attended in the conference. Even though Michelle did a great job explaining the topic it was waaaaay more than I could understand.

She described several reasons why you might want to build your own router. A routing service can serve as a software load balancer (which is typically not very practical since routers are cheap these days) but it can also do routing based on the content of the message (requests from strict SLA clients go to the best servers) something that a hardware solution couldn’t provide. A router service is also useful if you want to support online/offline scenarios.

There are several considerations that you need to be aware when building a service router. When a router receives a message and forwards it to a different service it can change its header, content, both, or none. If a router A does not change the header information and forwards it to service A that does not use security the request will be accepted by service A. However if the Service A implements security at it will reject the message because in the header indicates that the message was originally for router A (not Service A.) If the router is smart enough it can change the header of the message to indicate that the receiving end is really Service A but this introduces another issue because now Service A will see that somebody has tampered with the original headers. Michelle explained ways to address these issues using supported WCF functionality. She even showed how to support different transport protocols between the client and the router (e.g. HTTP) and the router and the service (TCP.) None of this was trivial for the uneducated like me but nevertheless possible and well supported in WCF.

In general it seems that routing is very easy until you turn on security. It’s very easy to thing that it only took a couple of minutes to get it working until you realize that security is off. Once you turn security on then you’ll deal into the things that she described.

There were several interesting demos showing how messages would go back and forth to different servers depending on different algorithms including round robin, information on the message header or the message body.

LINQ to SQL

Dino Esposito presented a very good introduction to LINQ to SQL. LINQ to SQL provides an object model on which LINQ operates to read and write data to a database. It also provides an abstraction layer on top of connections, data readers, datasets, transactions, and such.

LINQ to SQL is SQL Server specific. Other vendors might support it in the future.

Classes generated in the object model used by LINQ to SQL have C# attributes that describe what table/column they represent in the database. These classes are partial and you can extend them, for example to add business logic.

Relationships between tables are implemented as composite classes in the object model provided by LINQ to SQL. For example, Customer.Invoices[0].DueDate and Invoice.Customer.FirstName

The entry point to the object model is the DataContext class. This is the class that tracks changes. This class is short-lived and it’s recreated on each ASP.NET request. Opening a DataContext does not necessarily open a database connection, the database will not be hit until the first read or write to the data is detected. The fact that this class is recreated on each request is an issue in n-tier applications. This is the issue mentioned on the “Entity Framework in a World of Services and Processes” session yesterday and that will not be addressed until V2.0 of the Entity Framework.

Something that I found curious is that for queries you use LINQ syntax (which looks a lot like SQL syntax) but for updated you need to create a new object, populate the values that you want to change, and then call some specific methods (like SubmitChanges) to push the data to the database. Like in the session about the Entity Framework yesterday it seems that they’ve done a great job when it comes to reading data but the procedure to perform updates seems kludgy. That whole concept that you clone an object and then attached to a different DataContext just does not look right. I am sure you can get used to it but it just makes the product seem unfinished.

Dino was extremely honest on some of the awkward things when working with LINQ to SQL (most of which also apply to EF.) For example if you turn of lazy loading (aka deferred instantiation) some of the children collections like Customer.Orders.Count will return zero. He also showed some of the hacks that he had to go through when trying to force an update when a property that didn’t match one to one to a field was used.

In general LINQ to SQL and the Entity Framework look like great ideas that one should keep an eye on. If I didn’t knew about the CSLA Framework that Rockford Lhotka provides for free I would be all over this stuff, but since Lhotka’s framework has been around for a while is a more mature product. You can use LINQ (and LINQ to SQL and I suppose even the Entity Framework) with CSLA but CSLA provides enough that I don’t see much value on LINQ to SQL and EF in their current state.

Thursday, November 13th

High Performance Computing (HPC)

Nick Landry presented this session. High Performance Computing is used in scenarios when a user needs to submit a massive job, where as load balancing is used to handle massive number of users. An HPC Cluster is similar to a super computer, more than a typical network cluster.

An example where HPC is used is to calculate stock projections. Calculating the stock projection for a single stock is a very intensive job and it gets even worse when we need to calculate projections for the entire portfolio of a large client. In this case a massive job (calculate stock projections for these 100 stocks) is submitted to HPC cluster in which multiple machines work on the same job.

Microsoft provides a product called Windows HPC Server 2008 (formerly known as “Compute Cluster Server) that runs on top of Windows 2008 to create a HPC Cluster. http://microsoft.com/hpc

Version 1 of Windows HCP accepted batch jobs only submitted via a command line. In Version 2 there is an API that allows you to submit jobs.

To get started you need Windows Server 2008 plus the HPC Pack. The HPC Pack is licensed by CPU and Microsoft’s offer is the cheapest of the products around. The HPC Pack is what you use to setup the “head node” or to join an existing cluster. The head node runs the scheduler and knows about other machines.

There are .NET libraries to submit jobs to a HPC Cluster. For example, a single GetProjectionsForAllMyStocks() call will be broken down into multiple tasks (for example one per available computer) and run in parallel.

Productive WCF

In this session Juval Lowy focused on how to use WCF more than the plumbing underneath. He started by talking about WcfSvcHost.exe to host WCF Services without having to write your own host wrapper code. You can use it to start a WCF library project. You can call it with a /client parameter to fire both server and client applications at the same time. You can also call WcfTestClient.exe program to call a particular server (similar to the ASP.NET Web Service test page)

Instrumentation. WCF has several performance counters that you can use to monitor WCF Services. Just turn on performance counters in your config file (under ServiceModel) and monitor them with perfmon like you would typically would.

Tracing and logging. “It’s an illusion that clients care about the details of an exception” – most clients application have “if returnValue == OK then do X else display message to the user” In reality there is very little that the client application can do with the error other than telling the user “sorry” – if there was something to do the app would have already done it. Don’t create custom exception. Use the already build in ones.

You can turn on tracing and logging via the configuration file or with the WCF Configuration tool (look under diagnosis.) Make sure you turn on “auto-flush” so that the logging information is saved even if you stop the debugger in the middle of a session.

The “activity ID” checkbox in the WCF Configuration tool refers to a “logical thread” that tracks the activity across all tiers (client, host, service, et cetera.) The TraceViewer utility can automatically merge the log file from the client and the server with this activity ID. This is really cool.

Operations Overloading. You can modify the proxy class to support multiple method overloading by tagging them different via an attribute. For example to alias Add(int, int) into AddInt and Add(float, float) into AddFloat.

DataSets and DataTables. You should not return DataSets in your services. They are not good for other platforms other than .NET and are bound to database schema. You should use arrays instead

Fault Debugging. By default exception information is not bubbled up to the client application. He showed some code on the server side where he throws exception using throw new FaultException(ExceptionDetail) where ExceptionDetail is a custom class. You need to turn on “include exception detail” at the service level for this to work.

Once you turn use this code you can catch FaultException<ExceptionDetail> in addition to a generic catch FaultException in your client application.

Using jQuery with ASP.NET

Rick Strahl gave a very good overview of what is jQuery, what it can do, and how to use it.

jQuery is conceptually like “LINQ to HTML” in the sense that it allows you to query the HTML document at runtime and returns collections of objects. Rick called it the new HTML API.

jQuery is open source and quite popular, there are 100s of plug-ins available for it. The download is pretty small: 16KB. It uses CSS 3 selector syntax (dot to select classes, pound sign to select IDs)

jQuery is open source but it also come as part of Visual Studio 2008 SP1.

Rick showed several examples on how to select rows inside a table and change the background color of the even rows to produce an alternate grid. He then added with a very simple line of code a OnMouseHover JavaScript to all rows of the table to highlight the current row as the user hovers the mouse. The library is very powerful and once you know a few simple commands you can start using it. Below are some examples. $() is the selector operator and is a shortcut for jQuery()

$(“#someElementID”).css(“color”,”blue”) changes the font to blue of some element

$(“.someClass, .someOtherClass”).hide() hides all elements of someClass and someOtherClass

$(“table tbody tr:even”).addClass(“gAlternate”).css(“{font-weight:bold}”) to assign gAlternate class to all even rows on a table and change their font to bold.

There were some neat examples on how to use client side sorting, drag and drop, pop-up screens with fading effects. All of this with pure JavaScript and cross-browser compatible.

Towards the end Rick also showed some examples on how to use jQuery to call ASP.NET pages AJAX-like. He also demoed some stuff on serializing objects using JSON, consuming WCF services with jQuery, and a bunch of other things.

PLINQ: LINQ but faster

Stephen Toub from Microsoft gave an outstanding presentation on Parallel LINQ which is something that Microsoft is working for .NET 4.0

PLINQ is for data in memory (i.e. LINQ to Objects, not LINQ to SQL)

The basic premise is that people are not taking advantage of multi-code machines because parallelism is (very) hard. Stephen showed a typical example how a loop that selects some values from a collection and sorts them can be done in parallel. The code is nasty to put it mildly, you need to use lock, synching results, creating multiple threads, and so on.

With PLINQ the idea is that you tell LINQ “run this in parallel” and let the framework do the work for you. All you would need to do is add AsParallel to a LINQ statement like in the following example: var v = from c in collection.AsParallel() when c.name > “x” select c.name, c.age

PLINQ is currently in CTP stage. The June CTP release works with .NET 3.5 Go to http://msdn.microsoft.com/concurrency

There are two kinds of collections that LINQ can use. Collections that implement IEnumerable and collections that implement IList. IEnumerable is the most primitive and since it not thread safe PLINQ results to brute force (lock, get current, move next) and therefore you might not benefit a lot of parallelism in these collections. IList on the other hand is thread safe and allows PLINQ to make “smarter” decision on how to break the job like using Range and Stripe partitions.

The benefit of parallelism is not linear. A machine with 2 cores might be 1.75x faster than a machine with one core and a machine with 4 cores might be 3x faster than the machine with one core. This is because there is some cost associated with breaking apart a job to be run in parallel and merge the results at the end.

If there is a lot of I/O in your process you can increase the degree of parallelism (DOP) to a large number than the number of processors since the processors might become idle as they wait of the data to be retrieved/saved.

When using AsParallel the order of the results is not guaranteed. You can add AsOrdered to force ordering but that also adds overhead.

There are several kinds of partitions used by PLINQ. Chunk and Hash for IEnumerable collections and Range and Stripe for IList collections. There are also several kind of merge procedures that PLINQ can use like Pipeline, Stop and Go, and Inverted. You have some control on what kind of partition and merge process is used but you are probably better off letting the framework make those decisions, similar to the way a database management system does.

Managing and Leading (The One Thing)

hector@hectorcorrea.com

A few years ago my friend Bill from Rebar Business Builders recommended me the book "The One Thing You Need to Know...About Great Managing, Great Leading, and Sustained Individual Success" by Marcus Buckingham. For a while I hesitated to buy this book, initially the title about "the one thing" kind of put me off but I ended up buying it and reading it.

Back then I knew I had enjoyed the book but until recently I didn't realize how much I took from it.

The first thing that the book helped me clarify was the difference between managing and leading. Not only are these two roles (and the talents needed to perform them) different but they are oposite. Marcus points that the main focus of great managers are his/her employees while the main focus of great leaders is a better future.

Here is how Marcus puts it in regards to management [p. 83]

"The One Thing that all great managers know about great managing is this: Discovery What Is Unique About Each Person and Capitalize on It."

and in regard to leadership [p. 132]

"The One Thing every great leader knows he must do is: Discover What Is Universal and Capitalize on It."

Marcus provides several examples on how great managers "see people as an end unto themselves" and "derive satisfaction from seeing tiny increments of growth in someone else". Their focus is to serve their employees first, find what is unique about each of them, care for them, and put them in situations where their individual talents shine. Great managers will go above and beyond to "rearrange the world" to take advantage of each employee's individual strengths. This in turn tends to drive productivity in employees and teams which is why companies need great managers.

The two most important talents that allow great managers to perform their role is the ability to coach people and perceive their individual differences.

On the other hand leaders have a different role. Great leaders "perform [their] role well only when they find a way to make many people, regardless of each person's uniqueness, excited by and confident in [a] better future" Leaders tend to be fascinated by the future and they believe they have what it takes to take people to this better future.

The two talents that all great leaders have are optimism and ego. Optimism because they are convinced they can overcome present challenges and that there is a better way to do something. Marcus points that the oposite of a leader is a pessimistic. Ego because they believe they have what it takes to get there. As Marcus puts it, "they are not humble in their assessment of their own abilities" -- yet, this does not mean they are brash or abrasive, as a matter of fact most of them are quite reserved. [p. 66-68]

On the Agile Management Blog, David Anderson has an interesting post on Lead Well and Prosper that echoes some of these thoughts on optimism and leadership.

I think having a clear idea of the different roles and talents needed for managing and leading is very important in software development teams, particularly as developers start getting promoted to positions in which they are expected to lead and/or manage. Each company has a different expectation on what a senior or lead developer should do. Make sure you know what your role is, what your skills are, and how to make sure you are in a position that you enjoy, can perform, and is the best for the team.

Some people are very good at managing but not at leading, and vice-versa. Some people are good at both, some are not good at either. This is something applies to you, your peers, the people that report to you, and also to the people that you report to.

Marcus goes as far as to claim that leaders are born, not made [p 69] He bases this claim on the fact that optimism and ego is something that people are born with, not something that leaders learn. I am not sure I agree (or disagree) with his statement -- it's an uncomfortable thought, but I am not sure it's wrong.

The book is very good in general and I highly recommend it to people that are getting into management or leadership positions.

References

Buckingham, Marcus. The One Thing You Need to Know...About Great Managing, Great Leading, and Sustained Individual Success. 2005. Free Press.

Skeletons, Mona Lisa, and Problem Solving

hector@hectorcorrea.com

One of the best practices when building applications that I've learned over the years is the value of always having a running version of the application available to show end-to-end progress on the system being developed. Although this is common sense it's easy to get caught on the day-to-day activities and lose track of these things.

A typical situation in which teams seem to forget the value of always having a running version of the application is when writting a brand new application. In this scenario is common to have a bunch of components half-way written and not ready for production and most development teams tend to be OK with this. I am OK with this as long as they all compile and can be run successfully to show the functionality implemented so far. It does not matter the system can only perform a sub-set of features that the final system will have. It should always run so that people can experience first hand how each of the modules interact with each other.

Walking Skeleton

In trying to explain this concept to a friend of mine I found that there is even a pattern that explains this idea when it comes to building new applications. Cockburn^[1] calls it a Walking Skeleton:

"A Walking Skeleton is a tiny implementation of the system that performs a small end-to-end function. It need not use the final architecture, but it should link together the main architectural components. The architecture and the functionality can then evolve in parallel."

As Cockburn points out a Walking Skeleton is different from a proof of concept (or spike) in the sense that the later is meant to be throw away code while the Walking Skeleton is meant to evolve over time.

This approach relies heavily on having a procedure in place to support the evolution of the code as the skeleton is fleshed out (pun intended.) Cockburn calls this strategy Incremental Rearchitecture.

I recommend that teams use unit testing to make sure the code that performs some basic features today remains fully functional tomorrow as new few features are implemented and the architecture of the system updated to accommodate new features and new ideas.

Mona Lisa

On his article The Neglected Practice of Iteration Jeff Patton talks about the difference between iterating and incrementing (no, they are not synonyms) and touches on the idea of a Walking Skeleton from a different angle.

Jeff uses a Mona Lisa drawing to describe the differences between an iterative and an incremental approach to software development. If we were to use a pure incremental approach to drawing the Mona Lisa we would start by drawing to perfection a section of this paiting (say her hands), then we'll draw completelly her arms, then her face, and so on and so forth. Using an iterative approach we would start by drawing a sketch of the Mona Lisa in pencil (that's our first iteration), then we'll do a second iteration in which we increment the detail and perhaps give some coloring to the drawing, then we'll do a third iteration and put the final touches (keep incrementing the detail) on the painting. His article has great images that show this graphically.

The point of Jeff's article is that you need both (iterations and increments) in software development. A lot of people thing these two words are synonyms and use a pure incremental approach in which they work one a single module of the software to perfection before moving to the next module. The pure incremental approach has the disadvantage that neither the user nor the developers get to see the complete picture until the system is complete...and by then is usually too late. By using an iterative and incremental approach both the users and the developers get a rough idea of the whole system early on and how each module relates to each other.

Depth-First versus Breadth-First Problem Solving

On their book Lean Software Development the Poppendiecks^[2] talk about two different ways to solve problems: Depth-First versus Breadth-First.

"Breadth-first problem solving might be thought of as funnel, while depth-first problem solving is more like a tunel. Breadth-first involves delaying commitments, while depth-first invokes making early commitments."

The breadth-first approach is similar to the Walking Skeleton in the sense that aim first to get an application running end-to-end and then incrementally update its architecture as more is known about the problem domain.

I think both approaches are complimentary, just like incremental and iterative are. There are times where breadth-first is preferred over depth-first and vice-versa. The Poppendiecks put it in these words:

"Notice that both breadth-first and depth-first approaches require expertise in the domain. A depth-first approach will work only if there was a correct selection of the area to zero in on. Getting this selection right requires two things: someone with the expertise to make the early decisions correctly and assurance that there will not be any changes that render these decisions obsolete. Lacking these two conditions, a breadth-first approach will lead to better results."

One of the observations that the Poppendiecks made on their book is that "most novice designers are biased toward the depth-first approach." The irony on this is that depth-first requires people to get the decision right early on, something that a novice designer (by definition) probably won't do. Further more, I believe this bias on novice designers is the reason they tend to apply a pure incremental approach instead of an iterative and incremental approach.

Breadth-first work better than depth-first when the knowledge of the business domain is expected to evolve (as it typically does in software development) and it is also very effective when the domain is stable (e.g. you have strong knowledge about the problem domain.) Again, in the words of the Poppendiecks:

"A breadth-first approach requires someone with expertise to understand how the details will most likely emerge and the savvy to know when the time to make commitments has arrived. However, the breadth-first approach does not need a stable domain; it is the approach of choice when the business domain is expected to evolve. It is also an effective approach when the domain is stable."

Personality types play a big factor in the way people use breadth-first or depth-first approach. This is an important consideration since there are probably people in your team that would favor one approach over the other. This is OK. Remember that these approaches are complimentary. When you are not sure about the domain use a breadth-first approach, once you know enough and are ready to make a commitment go deep.

References:

[1] Cockburn, Alistar. Crystal Clear A Human-Powered Methodology for Small Teams. 2005. Addison Wesley. Pages 49-53.

[2] Poppendieck, Mary et al. Lean Software Development An Agile Toolkit. 2003. Addison Wesley. Pages 60-62.

Static constructors and ASP.NET applications

hector@hectorcorrea.com

A few days ago I was running into an issue with a static class in an ASP.NET application. The issue (I thought) was that the static constructor was not being fired. It turns out the static constructor was being fired as it should but not as often as I thought it would be.

Static constructs are fired once per App Domain.

In a Windows application this means that the constructor is fired once everytime I launch the application and access the static class. If I close the application, run it again, and hit the static class the constructor will be fired again.

In a Web application that is also the case, but that does not translate to everytime I close and re-open the browser. Closing the browser and re-launching it does not terminate the application and re-launches it.

Here is a simplified example of the scenario that I ran into. The static constructor is highlighted:

public static class StaticClass
{
    static int _count;
    static StaticClass()
    {
        _count = 0;
    }
    public static int Count
    {
        get
        {
            _count++;
            return _count;
        }
    }
}

public partial class _Default : System.Web.UI.Page
{
    protected void Page_Load(object sender, EventArgs e)
    {
        // print the count
        lblText.Text = string.Format("Count={0}", StaticClass.Count);
    }
}

The very first time I run this code the count prints 1. If I refresh the page it prints 2. If I close the browser and run the application again, what would you expect it would print?

If you guessed 1 you guessed wrong (like I did.) It will print 3.

Closing the browser does not get rid of the static class since apparently the App Domain still is running under IIS. If you really want to force the constructor of the static class to be fired (and in this case reset the counter to 0) you need to kill the worker process in IIS (e.g. run iisreset)

In highsight this is obvious and, as one of my co-workers pointed out, we are lucky ASP.NET apps work like that! It would suck big time if I everytime a user closed their browser the application would be closed as well and the next user will need to pay the load time.

The moral of the story is keep in mind that you don't have much control on when the static constructor will be fired in an ASP.NET application. You have the guarantee that it will be fired the very first time the class is accessed in the application domain. This includes the very first time the class is accessed after the app is loaded into IIS, but it also means that if the worker process is recycled (and your app domain goes away with it,) the static constructor will be fired (again) the next time the class is hit when a new user visits your site. This is all good and the desired effect, but just keep it in mind.

Computer Museum (Paris)

hector@hectorcorrea.com

Early August I had the opportunity to visit the computer museum in Paris. The Musee de l'Informatique has a neat variety of computers in display that range from 1960s mainframe computers, early personal computer models (IBM, Apple, Commodore) from the 80s, and other computer related stuff including punch cards, old hard drives, mother boards, software, to name a few.

Here are a few pictures that I took while I was there.

First things first: a 1960's mainframe. The big cabinet to the left with a bunch of yellow wires (below the clock) is the memory bank: 32K of RAM! The other big cabinet to the right is the CPU. I need to remember this picture next time I complain that my laptop is "too heavy"

This IBM PC is almost identical to the first computer that I owned. Check it out, it had two (not one!) floppy drives :)

I also saw this Commodore computer. They didn't have a Commodore 64 which was the very first computer that I used but this one reminded me a lot of those days.

And of course, for those of you who started on the "dark side" there was also an Apple II.

Walk down on memory lane: Sidekick anyone? You've got to admit that there was no better way to edit Turbo Pascal 3.0 code than with a "terminate and stay resident" editor like Sidekick. High tech circa 1985.

A DOS developer?

Hard drives and floppy disks

DotWiki 2.0

hector@hectorcorrea.com

I've posted a new version of the DotWiki. This new version 2.0 is a complete rewrite using C# and it has several new features and a more attractive look and feel. Security is probably the most important new feature since it had been by far the most commonly requested feature since I released the original version in 2002.

Previous versions of the DotWiki had been updates to the original VB.NET code and I felt that a full rewrite was in order to clean up the code, remove unused features, reflect what I've learn since the original release, and add a few more features. Since I've been doing exclusively C# for the last few years I took the plunge and rewrote it in C#.

The look and feel is using the ASP.NET Design Starter Kit Templates that I wrote about a few months ago and that this very site uses.

Security is implemented using the ASP.NET Membership provider. It's amazing how easy was to secure the site using the built-in features for membership for creating new users, resetting your password, logging in, logging out, and such.

This new version also uses Rockford Lhotka's CSLA business objects framework. The business objects to read and write data are typical CSLA business objects. You don't need to know much (if anything) about CSLA to understand the code and yet it helps me tremendously to keep the code in logical tiers. Check out the DotWiki Design Documentation to get an idea of where CSLA business objects fit in the DotWiki.

This version still uses the FCKeditor to provide a what you see is what you get (WYSIWYG) experience to the user. I updated the code to use version 2.6 which supports the Safari browser.

I am continuosly amazed on how much free stuff there is for ASP.NET applications and how good it is. CSLA is a fantastic middle tier business object framework and is open source. The FCKeditor integrates seamlessly with ASP.NET and comes with all the JavaScript source code. The ASP.NET Membership provider is part of the .NET framework since version 2.0 and the design starter kits for ASP.NET applications have always been free.

The DotWiki itself is open source and I hope developers find the new version as useful (or more!) as previous versions.

You can download the source code from the DotWiki Source Code page.

Introduction to Scrum

hector@hectorcorrea.com

My article Introduction to Scrum has been published in the May/June 2008 issue of CoDe magazine and is also availabe on DevX.

As I said in this article

"Scrum is an agile process to manage software development projects. Scrum is not prescriptive on engineering practices, but rather it is a lightweight framework based on a few (common sense) guidelines for managing projects."

This issue of CoDe is dedicated to Agile Development.

Jeffrey Palermo has an interesting article on managing agile software projects that starts with quite a line:

"Everything right or wrong with a software project is management's fault"

Jeff then goes to talk about the falacy of fixed scope, the unspoken quality variable, hiring and firing, managing the customer, and other areas where management needs to act and how this is done in an agile project.

Jean-Paul S. Boodhoo provides a list of 16 steps to improve your development skills using techniques that are common on agile projects. I particularly like his approach of trying these techniques little by little. For example, start doing unit testing, get comfortable with it and then evaluate test driven development (TDD)

Scott Bellware talks about Behavior-Driven Development (BDD)

"TDD is often specifically about the design of code units and modules such as classes. BDD is aso concerned with unit design, but its focus addresses a much broader range of design concerns"

He talks about the common misunderstanding that Test Driven Development is about testing while in fact is about design.

All in all an interesting CoDe magazine issue.

ASP.NET Design Templates

hector@hectorcorrea.com

I've received several nice comments about the design of this site. I wish I could take all the credit for it but the truth is that I created it using the Visual C# Personal Design template that Microsoft provides.

The Design Templates web site provides several ASP.NET Design Starter Kit Templates that allow you to create web sites using some predefined look and feel options. When you install these templates they become available to you in Visual Studio when you create a new web site. This is how my Visual Studio 2005 Express looks like with the Personal Design Template installed.

For the UI impared developer like me these templates are amazing. They are ready to use, attractive, and allow us to learn by example. The one that I am using comes with master pages, themes, and site maps.

HTTP Handlers in .NET 2.0

hector@hectorcorrea.com

When I rewrote my web site late last year I knew I wanted to use friendly names for my blog pages instead of using URL parameters. For example, the URL to the post on Binary Trees in C# initially was this:

http://hectorcorrea.com/Blog.aspx?t=c8ff6dbb-284f-4bde-bf07-909f785529ae

I wanted something friendlier and I knew this could be done easily with an HTTP Handler in .NET 2.0 but it has been a very long time since I had written one. So I procrastinated.

A few weeks ago I bumped into a blog post by Timothy Khuori that described how easy it was to implement one. The code was not longer than 20 lines of code (including curly braces!) that I felt ashamed I hadn’t implemented it my site. I took Timothy’s code, stuffed it into my project, and voila, now I have friendly URLs for my blog posts. The URL for the Binary Trees in C# post now is:

http://hectorcorrea.com/blog/Binary-Tree-in-C-Sharp.aspx

Behing the scenes, my HTTP Handler still uses my original Blog.aspx page to serve the request but it preserves the original URL that the user submited. I updated Blog.aspx to parse the name of the blog topic to retrieve directly from the URL rather than expect it in the query string.

Beside being more human readable, these kinds of URLs are also better for tracking purposes. Since to the web server the request looks like the request for a specific page for each topic (rather than the generic blog.aspx with a parameter with the topic name) the web server can keep track of how many times each topic is requested. This means that I can get statistics in how popular a blog topic is using the tools that I already use to track how popular some pages from my site are.

Upgrading Your Software Development Tools

hector@hectorcorrea.com

Late last year (late November 2007 to be exact) Microsoft released Visual Studio 2008 with .NET 3.5 to production. This has generated a lot of buzz around the new features, what’s hot, what you should be looking into, and so on. The beta cycle for VS 2008 was long and very public. There has been a lot of articles and access to the new features for a while but now it’s here, now it’s real. People can go and buy VS 2008 today or download it from MSDN. Teams that have been just wondering how to go about the upgrade now need to answer the question: how do we upgrade?

Most software developers are technology driven so we love new tools and technologies. This seems to be a personality trait regardless of the technology that we chose. The desire to move to the latest and greatest version seems to be as strong in .NET developers today as it was for FoxPro and VB 6 developers in the late nineties. We crave the day when we can use new tools. Yet, we have a responsibility to our users in doing this responsibly. Upgrades can be hard to perform on medium to large systems that have large number of users in production. Like most things on these kinds of systems upgrades need to be planned in order to give the most to the customers.

Although the decision to upgrade or not to upgrade to a new version of development tools is very specific to every project, I usually tend to recommend upgrading on-going projects (with new features actively being developed) and thinking twice before upgrading projects that have been switched to maintenance mode (no new features are being developed.)

So why would you want to upgrade?

If updating a project to use the latest technology options for you environment takes time and money, why would you even consider want to upgrade? The fact that a new version of your development tools is available is hardly enough reason for a team to stop developing new features, spend days/weeks performing the upgrade, retest the system, and then continue with new development. There has to be some benefit for the customer for an upgrade to make sense.

New tools do come with benefits, but you need to be realistic about what will your project benefit from in the short and long term when upgrading to a new version. I tend to be wary of short term “magic” benefits that come with new development tools. One of the most common mistakes that teams do is promise (to themselves or to others) that productivity will increase incredibly with new version of the tool. Thinking that your application magically will be developed in half the time and will look much better than before just because you switched to VS 2008 and WPF is foolish. Assuming that the Windows Workflow engine in .NET 3.0 will solve all your workflow issues when you have never used a workflow engine before would also be a poor assumption. It takes time to get used to new tools and to learn what really works for your project and what does not.

In the case of Visual Studio 2008 there are a ton of benefits that apply to a wide variety of projects. A lot of the tools that Microsoft made available more than a year ago when it released .NET 3.0 are now finally easy to use for most of us. VS 2008 provides the (long awaited) designers to make the most of WPF, AJAX, WCF, and LINQ to name a few. These new technologies can indeed help your application look better, be more responsive to users, provide better communication with other applications, and such.

How to upgrade to the latest version?

Although the process to upgrade a project to use a new tool varies from project to project there are two main guidelines that I’ve found useful. One is to try the upgrade first in a test environment and with ample time (i.e. don’t try it the week before going to production.) The second is that you need to consider the entire project, not only the source code (How will the deployment be affected with the new tool? Will the users be affected? Will third party tools need to be upgraded as well?)

In my experience tests in the testing environment with the new version of the tool always work out and come out with glaring reviews. The issues with the new version of your tool usually show up once you’ve upgraded the entire system. The testing environment, although necessary, usually only covers a small portion of the system (e.g. the code but not the deployment) and some of the pitfalls might be in those other areas that will be upgraded later. This can be sort of a catch-22, you need to test the upgrade first to get an initial idea but you won’t get a good idea until you’ve upgraded the entire system. You can try to evaluate a wide range of areas of your system (code, deployment, third party tools, et cetera) but don’t hold your breath -- you might still run into issues once the entire app is upgraded. C’est la vie.

In the case of Visual Studio 2008 Microsoft did a tremendous benefit to the .NET developers by allowing projects to be upgraded to use VS 2008 and still target the .NET 2.0 runtime. This is a huge advantage for teams that need to take more time to upgrade since it allows them to upgrade the IDE but keep the runtimes and code the same. To make things even better once you've switched your project to use the .NET 3.5 runtime you can incorporate .NET 3.0 features little by little. For example, it is possible to have Windows Forms and WPF forms on the same application, which means that you don’t need to rewrite your entire user interface on day one. You can keep your existing Windows Forms and only upgrade to WPF one of two forms at a time, become comfortable with the new technology and plan how to (or if you want to) migrate the rest of the application to use WPF.

Estimates on Software Projects

hector@hectorcorrea.com

Estimates is one of the most controversial topics on software development, including the process to calculate them and the value that they add to the software development process.

Every team that I have worked with needs to provide estimates on how long it's going to take to complete a specific feature of a system. Yet, most teams seem to struggle with this activity. Developers enjoy attending a meeting to provide estimates as much as they enjoy visiting the dentist. Project owners typically complain how unreliable estimates on software projects are, to them is unbelievable that even that developers have been writing software for many years cannot give accurate estimates.

In my experience there are several reasons why providing estimates is so hard:

Poor specifications/Lack of domain knowledge
Technology challenges
Developers are optimistic
Software development is tough

Poor specifications

Poor specifications is the most commonly voiced reason for inaccurate estimates. And for a good reason. It is typical for developers to be asked to estimate features that they don't fully understand. The client requests a new screen to do X but there is a lot of unwritten expectations for this new feature that users will demand once they start using it but that developers don't know about. This is usually due to lack of business domain knowledge. Developers are trained on software development and very little (or nothing) in the specific business domain. If the software that is being written is for insurance billing is very unlikely that the developers will have enough education and training on insurance and/or accounting. They might have been working many years in software development but this might be the first time they are faced with a screen for insurance billing purposes.

Even when a decent amount of conversation between developers, business analysts, and product owners take place to explore what the requested feature will entail, there is inevitable a lot of things that are left unsaid that will be find out when the feature is developed and presented to the user. I've worked on companies where we received reams of paper with specifications that looked like they covered every possible detail about the feature to be built just to find out that (1) they are still incomplete or (2) they don't really represent what the customer wants. I've produced my fair share of specs that I thought were enough to provide accurate estimates just to find out that nope, there is still a lot left unsaid that will not be discovered until the software is written and presented to the users for feedback.

Technology challenges

Most software development groups try to stay up to date on technology and upgrade to new versions of compilers, databases, development environments, and such. This is usually a good thing since these new tools are meant to speed up the development process and make better use of existing hardware. But there is a cost associated with these upgrades. Developers need to spend time performing upgrades, learning new tools and ways of doing things, and this might not be cheap. Plus this happens on a regular basis. For example, since 2000 Microsoft has released three versions of C# (C# in 2000, C# 2.0 in 2005, and C# 3.0 in 2007) during the same time Microsoft has released three versions of SQL Server (SQL 2000, SQL 2005, and SQL 2008.)

You may be tempted to say 'the heck with it, we are not going to update anymore' but there is also a cost associated with this. For once, your system might not run on new computers or just look ugly (e.g. imagine a DOS application running on Windows.) Secondly, developers tend to be technology driven and you may lose some of your best developers if they see that they are falling behind in technology and their skills are not up to date.

Developers are optimistic

Most developers that I've met (myself included) are optimistic by nature. We are optimistic people and we like to please our customers. This has an impact on the estimates that we provide. We tend to provide estimates for the best case scenario even when experience tells us that the best case scenario hardly ever materializes in software development. Even when we try to account for the unexpected we do it in an optimistic way. Or as Brooks will say in the Mythical Man-Month [p. 15] "the assumption that all will go well has a probabilistic effect on the schedule."

Software development is tough

Software development is a tough activity. It requires dealing with abstractions, a lot of unknowns, on a tight schedule, and with new tools and technologies. Most activities in a business environments deal with physical limitations like size, weight, cost, counts, and other well understood constrains. Software on the other hand deals with "almost pure thought-stuff" [Brooks, p. 7] Estimates on an unconstrained environment like this are not trivial.

Do we need estimates?

Yes, we definitively do. Despite the grim picture that I might have painted so far, I strongly believe that estimates are valuable in software projects. Estimates are like price tags in the sense that they allow product owners make better decisions when it comes to deciding priorities for a software project. If there are four new features being requested by the users and three of them will take one week each but the fourth one looks like it is going to take two months alone, the product owner will be better able to decide what is best for the product. She might decide to complete the three one-week features and delay the large one for a better time, or vice-versa. But she will be able to make this decision having an idea on the size of the features.

In my experience there are two things that you can do to get the most out of estimates for software projects even when they are not accurate ..

One is to divide and conquer. Estimate small blocks of functionality so that even if estimates are not accurate they are not off by a lot either. If a task is estimated to take 2 days and it takes twice as much you can deal with it 2 days later and adjust your planning. However, if a task is estimated to take 2 months and it takes twice as much, four months later is probably too late to address the problem. Even more, if developers in a team can blow a 2 days estimated (for whatever reasons) imagine the likelihood of blowing a 2 months estimate.

The second thing that we can do is to constantly calibrate project expectations. To put it bluntly be realistic and use history as a guidance on how long stuff takes to be completed rather than be guided by hope. It is very common for people to say "this time it took me 2 weeks to complete X but with what I know now it would take me only 1 week next time." This hardly ever works out. For once, remember that developers are optimistic. We are bound to take this posture. If it took 2 weeks to do something there is probably a very good reason for it. You should use that number as your baseline in the future and only reduce the estimate once you have proven than it can be done in half as much. Koskela and Howell have a great paper in which they propose moving away from the thermostat model (in which an arbitrary productivity goal is set at the onset) into a scientific experimentation model in which we adjust based on previous results.

How to calculate estimates?

Since estimates are approximations, not promises, I prefer a light weight approach to calculate them rather than spending a whole lot of time on them. By light weight I mean the team getting together for no more than 3 hours to estimate what we think will be able to accomplish in the next two weeks. In my experience teams can come up with "good enough" estimates in relatively short term. David Anderson has a tongue-in-cheek post where he calls for people to stop estimating. In reality he calls for agile estimating rather than no estimates at all. In agile methods not a lot of time is spent calculating something that we cannot guarantee. It's better to spend that time producing software and delivering features to the user rather than promises.

Mike Cohn's book on Agile Estimating and Planning is a very good book on the subject. He does a great way of describing the use of story points and velocity to do long term planning on projects. Highly recommended if you are doing agile software development.

Joel Spolsky has an article on Evidence-Based Scheduling (EBS) in which he uses a Monte Carlo simulation to calculate the likelihood that a particular estimate will be met. Spolsky's approach sounds interesting but I have not tried it myself. The fact that uses historic values to calculate probabilities sounds promising. Still, I am afraid that it requires more time and effort than the value that it provides but I don't have any experience with it.

What to do about missed estimates?

Of the things that are controversial about estimates the one about missed estimates is probably the worst. It's almost taboo. Although everybody misses estimates but nobody wants to talk about it.

If you (or your team) misses an estimate, get over it. It happens. Don't let it bring you down. Review why you missed it and act on your findings. Perhaps the task is just more complicated than you thought -- it is perfectly OK to admit this. Maybe you need to break the task down in smaller chunks next time, or you need less distractions. What ever you do however, make sure to use the knowledge that it took longer than you expected next time you estimate a similar task. Don't tell yourself that "next time will be better because I've been there before." Remember: use history as a guidance rather than hope.

If somebody else misses an estimate see the previous paragraph. Talk with them to see why it was missed and act on it. Don't expect that because somebody thought something would take 5 days that is the time it will take. Remember that it is an estimate, a forecast.

There might be instances where a team or an individual developer constantly misses estimates by a long shot. If that is the case, there are perhaps larger issues that need to be addressed. Does the team/developer has the right tools for the job? Is the team/developer qualified for the job? Are the expectations for the team/developer realistic? Is it always the same team/individual? These are perfectly valid questions on a team/individual that is not performing at the pace that the business expects. The problem migh be the team, the expectations, or both.

The role of the product owner

In my experience most product owners learn the effectiveness of their team's estimates in a few cycles. Product owners are not necessarily as optimistics as developers and they can provide valuable feedback to the team after just a few rounds of estimates. This is great feedback that a lot of teams don't use. Product owners bring differerent experiences, business knowledge, and personality styles to the team that can be very useful when estimating and setting expectations.

The Scrum Daily Meeting

hector@hectorcorrea.com

We use Scrum at the office to manage our projects. One of the key practices of Scrum is the daily meeting (or daily Scrum.) The daily meeting is a short meeting (15 minutes in our case) held every day at the same time with the whole team (product owners, QA, and developers.)

The goal of the meeting is to keep everybody informed of the progress since the last meeting and make sure there are no roadblocks.

The rules for the daily meeting are simple: The meeting must be short, timeboxed, and every team member must report

What have you done since the last daily meeting?
What will you do from now until the next daily meeting?
Are there any roadblocks preventing you to move on?

Since in most of our projects we have remote participants I used to show a spreadsheet with our Sprint Backlog in a projector and have everybody report their status. The spreadsheet was shared real time with remote users and updates were seen by everybody.

This was a nice idea since it kept all participants (local and remote) involved in the meeting. However the spreadsheet approach had a huge disadvantage: the focus of the meeting became the spreadsheet rather than the status of the team.

With the spreadsheet approach developers tended to be very mechanical when reporting what they had done and what they will do next. It was very common for people to say "take 3 hours down from line 25 of the spreadsheet and 4 hours from line 28" rather than describing what they did (e.g. "I worked on the problem that prevent users outside the domain from logging in.")

Last year I decided to stop using the spreadsheet in our daily meetings. We still get together and report our status, but instead of looking at a spreadsheet some people bring a notepad with a list of things to report while others do it of memory.

This little change has made a huge difference in our daily meetings. The meetings are much more conversational now than they ever were. Since there is no central spreadsheet to look at participants now report to their peers rather than to the spreadsheet.

Now that the focus of the meeting has moved away from the spreadsheet it has been easier to make it a meeting where everybody gets informed on what's going on with the project. People really talk about what they are doing rather than just taking hours down from tasks. For example, since there is no spreadsheet during the meeting the participants need to describe what they are working on (e.g. "working on a bug in the login process") rather than just saying "working on bug report #20".)

We still keep track of the number of hours remaining in our Sprint Backlog items but is done outside of the daily meeting.