My article Introduction to Scrum has been published in the May/June 2008 issue of CoDe magazine.

As I said in this article

"Scrum is an agile process to manage software development projects. Scrum is not prescriptive on engineering practices, but rather it is a lightweight framework based on a few (common sense) guidelines for managing projects."

This issue of CoDe is dedicated to Agile Development.

Jeffrey Palermo has an interesting article on managing agile software projects that starts with quite a line:

"Everything right or wrong with a software project is management's fault"

Jeff then goes to talk about the falacy of fixed scope, the unspoken quality variable, hiring and firing, managing the customer, and other areas where management needs to act and how this is done in an agile project. 

Jean-Paul S. Boodhoo provides a list of 16 steps to improve your development skills using techniques that are common on agile projects. I particularly like his approach of trying these techniques little by little. For example, start doing unit testing, get comfortable with it and then evaluate test driven development (TDD)

Scott Bellware talks about Behavior-Driven Development (BDD)

"TDD is often specifically about the design of code units and modules such as classes. BDD is aso concerned with unit design, but its focus addresses a much broader range of design concerns"

He talks about the common misunderstanding that Test Driven Development is about testing while in fact is about design.

All in all an interesting CoDe magazine issue. 

I've received several nice comments about the design of this site. I wish I could take all the credit for it but the truth is that I created it using the Visual C# Personal Design template that Microsoft provides.

The Design Templates web site provides several ASP.NET Design Starter Kit Templates that allow you to create web sites using some predefined look and feel options. When you install these templates they become available to you in Visual Studio when you create a new web site. This is how my Visual Studio 2005 Express looks like with the Personal Design Template installed.

For the UI impared developer like me these templates are amazing. They are ready to use, attractive, and allow us to learn by example. The one that I am using comes with master pages, themes, and site maps.

So why would you want to upgrade?

How to upgrade to the latest version?

Estimates is one of the most controversial topics on software development, including the process to calculate them and the value that they add to the software development process.

Every team that I have worked with needs to provide estimates on how long it's going to take to complete a specific feature of a system. Yet, most teams seem to struggle with this activity. Developers enjoy attending a meeting to provide estimates as much as they enjoy visiting the dentist. Project owners typically complain how unreliable estimates on software projects are, to them is unbelievable that even that developers have been writing software for many years cannot give accurate estimates.

In my experience there are several reasons why providing estimates is so hard:

• Poor specifications/Lack of domain knowledge
• Technology challenges
• Developers are optimistic
• Software development is tough

Poor specifications

Poor specifications is the most commonly voiced reason for inaccurate estimates. And for a good reason. It is typical for developers to be asked to estimate features that they don't fully understand. The client requests a new screen to do X but there is a lot of unwritten expectations for this new feature that users will demand once they start using it but that developers don't know about. This is usually due to lack of business domain knowledge. Developers are trained on software development and very little (or nothing) in the specific business domain. If the software that is being written is for insurance billing is very unlikely that the developers will have enough education and training on insurance and/or accounting. They might have been working many years in software development but this might be the first time they are faced with a screen for insurance billing purposes.

Even when a decent amount of conversation between developers, business analysts, and product owners take place to explore what the requested feature will entail, there is inevitable a lot of things that are left unsaid that will be find out when the feature is developed and presented to the user. I've worked on companies where we received reams of paper with specifications that looked like they covered every possible detail about the feature to be built just to find out that (1) they are still incomplete or (2) they don't really represent what the customer wants. I've produced my fair share of specs that I thought were enough to provide accurate estimates just to find out that nope, there is still a lot left unsaid that will not be discovered until the software is written and presented to the users for feedback.

Technology challenges

Most software development groups try to stay up to date on technology and upgrade to new versions of compilers, databases, development environments, and such. This is usually a good thing since these new tools are meant to speed up the development process and make better use of existing hardware. But there is a cost associated with these upgrades. Developers need to spend time performing upgrades, learning new tools and ways of doing things, and this might not be cheap. Plus this happens on a regular basis. For example, since 2000 Microsoft has released three versions of C# (C# in 2000, C# 2.0 in 2005, and C# 3.0 in 2007) during the same time Microsoft has released three versions of SQL Server (SQL 2000, SQL 2005, and SQL 2008.)

You may be tempted to say 'the heck with it, we are not going to update anymore' but there is also a cost associated with this. For once, your system might not run on new computers or just look ugly (e.g. imagine a DOS application running on Windows.) Secondly, developers tend to be technology driven and you may lose some of your best developers if they see that they are falling behind in technology and their skills are not up to date.

Developers are optimistic

Most developers that I've met (myself included) are optimistic by nature. We are optimistic people and we like to please our customers. This has an impact on the estimates that we provide. We tend to provide estimates for the best case scenario even when experience tells us that the best case scenario hardly ever materializes in software development. Even when we try to account for the unexpected we do it in an optimistic way. Or as Brooks will say in the Mythical Man-Month [p. 15] "the assumption that all will go well has a probabilistic effect on the schedule."

Software development is tough

Software development is a tough activity. It requires dealing with abstractions, a lot of unknowns, on a tight schedule, and with new tools and technologies. Most activities in a business environments deal with physical limitations like size, weight, cost, counts, and other well understood constrains. Software on the other hand deals with "almost pure thought-stuff" [Brooks, p. 7] Estimates on an unconstrained environment like this are not trivial.

Do we need estimates?

Yes, we definitively do. Despite the grim picture that I might have painted so far, I strongly believe that estimates are valuable in software projects. Estimates are like price tags in the sense that they allow product owners make better decisions when it comes to deciding priorities for a software project. If there are four new features being requested by the users and three of them will take one week each but the fourth one looks like it is going to take two months alone, the product owner will be better able to decide what is best for the product. She might decide to complete the three one-week features and delay the large one for a better time, or vice-versa. But she will be able to make this decision having an idea on the size of the features.

In my experience there are two things that you can do to get the most out of estimates for software projects even when they are not accurate ..

One is to divide and conquer. Estimate small blocks of functionality so that even if estimates are not accurate they are not off by a lot either. If a task is estimated to take 2 days and it takes twice as much you can deal with it 2 days later and adjust your planning. However, if a task is estimated to take 2 months and it takes twice as much, four months later is probably too late to address the problem. Even more, if developers in a team can blow a 2 days estimated (for whatever reasons) imagine the likelihood of blowing a 2 months estimate.

The second thing that we can do is to constantly calibrate project expectations. To put it bluntly be realistic and use history as a guidance on how long stuff takes to be completed rather than be guided by hope. It is very common for people to say "this time it took me 2 weeks to complete X but with what I know now it would take me only 1 week next time." This hardly ever works out. For once, remember that developers are optimistic. We are bound to take this posture. If it took 2 weeks to do something there is probably a very good reason for it. You should use that number as your baseline in the future and only reduce the estimate once you have proven than it can be done in half as much. Koskela and Howell have a great paper in which they propose moving away from the thermostat model (in which an arbitrary productivity goal is set at the onset) into a scientific experimentation model in which we adjust based on previous results.

How to calculate estimates?

Since estimates are approximations, not promises, I prefer a light weight approach to calculate them rather than spending a whole lot of time on them. By light weight I mean the team getting together for no more than 3 hours to estimate what we think will be able to accomplish in the next two weeks. In my experience teams can come up with "good enough" estimates in relatively short term. David Anderson has a tongue-in-cheek post where he calls for people to stop estimating. In reality he calls for agile estimating rather than no estimates at all. In agile methods not a lot of time is spent calculating something that we cannot guarantee. It's better to spend that time producing software and delivering features to the user rather than promises.

Mike Cohn's book on Agile Estimating and Planning is a very good book on the subject. He does a great way of describing the use of story points and velocity to do long term planning on projects. Highly recommended if you are doing agile software development.

Joel Spolsky has an article on Evidence-Based Scheduling (EBS) in which he uses a Monte Carlo simulation to calculate the likelihood that a particular estimate will be met. Spolsky's approach sounds interesting but I have not tried it myself. The fact that uses historic values to calculate probabilities sounds promising. Still, I am afraid that it requires more time and effort than the value that it provides but I don't have any experience with it.

What to do about missed estimates?

Of the things that are controversial about estimates the one about missed estimates is probably the worst. It's almost taboo. Although everybody misses estimates but nobody wants to talk about it.

If you (or your team) misses an estimate, get over it. It happens. Don't let it bring you down. Review why you missed it and act on your findings. Perhaps the task is just more complicated than you thought -- it is perfectly OK to admit this. Maybe you need to break the task down in smaller chunks next time, or you need less distractions. What ever you do however, make sure to use the knowledge that it took longer than you expected next time you estimate a similar task. Don't tell yourself that "next time will be better because I've been there before." Remember: use history as a guidance rather than hope.

If somebody else misses an estimate see the previous paragraph. Talk with them to see why it was missed and act on it. Don't expect that because somebody thought something would take 5 days that is the time it will take. Remember that it is an estimate, a forecast.

There might be instances where a team or an individual developer constantly misses estimates by a long shot. If that is the case, there are perhaps larger issues that need to be addressed. Does the team/developer has the right tools for the job? Is the team/developer qualified for the job? Are the expectations for the team/developer realistic? Is it always the same team/individual? These are perfectly valid questions on a team/individual that is not performing at the pace that the business expects. The problem migh be the team, the expectations, or both.

The role of the product owner

In my experience most product owners learn the effectiveness of their team's estimates in a few cycles. Product owners are not necessarily as optimistics as developers and they can provide valuable feedback to the team after just a few rounds of estimates. This is great feedback that a lot of teams don't use. Product owners bring differerent experiences, business knowledge, and personality styles to the team that can be very useful when estimating and setting expectations.

We use Scrum at the office to manage our projects. One of the key practices of Scrum is the daily meeting (or daily Scrum.) The daily meeting is a short meeting (15 minutes in our case) held every day at the same time with the whole team (product owners, QA, and developers.)

The goal of the meeting is to keep everybody informed of the progress since the last meeting and make sure there are no roadblocks.

The rules for the daily meeting are simple: The meeting must be short, timeboxed, and every team member must report

1. What have you done since the last daily meeting?
2. What will you do from now until the next daily meeting?
3. Are there any roadblocks preventing you to move on?

Since in most of our projects we have remote participants I used to show a spreadsheet with our Sprint Backlog in a projector and have everybody report their status. The spreadsheet was shared real time with remote users and updates were seen by everybody.

This was a nice idea since it kept all participants (local and remote) involved in the meeting. However the spreadsheet approach had a huge disadvantage: the focus of the meeting became the spreadsheet rather than the status of the team.

With the spreadsheet approach developers tended to be very mechanical when reporting what they had done and what they will do next. It was very common for people to say "take 3 hours down from line 25 of the spreadsheet and 4 hours from line 28" rather than describing what they did (e.g. "I worked on the problem that prevent users outside the domain from logging in.")

Last year I decided to stop using the spreadsheet in our daily meetings. We still get together and report our status, but instead of looking at a spreadsheet some people bring a notepad with a list of things to report while others do it of memory.

This little change has made a huge difference in our daily meetings. The meetings are much more conversational now than they ever were. Since there is no central spreadsheet to look at participants now report to their peers rather than to the spreadsheet.

Now that the focus of the meeting has moved away from the spreadsheet it has been easier to make it a meeting where everybody gets informed on what's going on with the project. People really talk about what they are doing rather than just taking hours down from tasks. For example, since there is no spreadsheet during the meeting the participants need to describe what they are working on (e.g. "working on a bug in the login process") rather than just saying "working on bug report #20".)

We still keep track of the number of hours remaining in our Sprint Backlog items but is done outside of the daily meeting.

This weekend I started reading Managing the Dynamics of Change by Jerald M. Jellison. This book talks about people's emotional resistance to change and how to manage it. With my background on technology it's easy for me to look at things (including change) from the logical and rational point of view but, as the author points out, “successful implementation [of process change] ultimately comes down to getting people to begin doing things differently” and logic and communication is not always the only tool that is needed.

Jellison describes how performance goes through a J Curve when people is asked to change the way they do things. In this curve we start with certain performance level that immediately drops as a change initiative is started, after a short period performance levels starts to climb up again until it reaches and then passes the initial levels. He describes this as five stages of change:

• Stage 1: The Plateau – at the beginning, before any new effort gets underway
• Stage 2: The Cliff – as soon as the change begins, performance drops, people are fearful
• Stage 3: The Valley – things start to get better, employees turn cautiously optimistic
• Stage 4: The Ascent – performance improves at a faster pace
• Stage 5: The Mountaintop – performance passes the levels of Stage 1 and keeps going upwards

Most people go through the five stages, from the managers that come with the new idea to the people that will eventually implement it. However, as Jellison points out, “management is usually in Stage 4 and heading into Stage 5 by the time they decide an idea is good enough for everyone” but the rest of the team is just about to see the cliff!

There are two main points that I've taken from this book. One is that there is a strong emotional component to change and we should address it instead of pretend it does not exist. The second is the fact that most people go throught different stages and that they might do it at different times.

Jellison describes the typical approach of persuading people to embrace the new way of doing things. In this approach the new vision is laid out, the need for change explained, the plan detailed, and off we go. The idea is that if you communicate your message clearly people will follow you. He points out that persuasion only works with people that are already predisposed to the change or that are almost ready for it, but not the rest. If you don't address the emotional factor when implementing change you run the risk of getting “lip service” rather than real behavior change. People might tell you that they are going to do things the new way and not do it. The persuasive approach works very well once people reach stage 4 (i.e. they are not fearful and can co-relate the leader's vision with what they are doing) but it does not work in the first three stages.

Another approach that he proposes to manage change is what he calls activation. In this approach people get to try the new approach right away, in small increments, and in a safe environment, one in which people have positive experiences right away so that they move faster from resistance to conversion.

“Activation is a bottom-up approach to change. It gets employees to take actions in the new direction even though they have doubts and fears about doing so. Persuasion, by contrast, is a top-down model. Persuasion assumes you have to change employees' attitudes before they'll start to actually move in the new direction. [...] Persuasion is directed at thoughts and beliefs while activation works because it addresses people's feelings.”

He explains several techniques to facilitate change including

• Communicate at ground level
• Ask, don't tell
• Front-load rewards and
• Make it easy to start

Definitively a good book to read if you are in a leadership or management position.

I've been reading about Windows Presentation Foundation (WPF) for a while and, while at DevConnections this week, I've attended a few sessions on the topic and to this day I still have mixed feelings about this new technology.

On one hand I am really excited to see work done for smart clients and the user interface in Windows. The fact that WPF takes into account new graphics hardware available on most machines these days is really nice. WPF uses Direct X instead of GDI+ to render images on the display which turns into faster rendering even for more complex UIs. The Expression Blend product targeted for designers is also a nice addition that hopefully will help developers and UI designers to work together resulting in better applications for the user.

On the other hand I am disappointed Microsoft has lowered the abstraction layer instead of raising it. Most of the sessions and reading that I've seen on WPF focuses on how easy it is to do buttons, gradients, nested controls, stack panels, and such. Gosh! that's not very productive. I am eager to provide better and more polished user interfaces to our users but I cannot believe that in the year 2007 we are attending sessions on how to do gradients and nested controls. I wish MS had provided much richer environment on top of the basic functionality that they've created.

The lack of a data grid is another issue that I struggle with. The new controls allow you to create your own grid and bind it to data but why do we have to do that. Why isn't there a native data grid in VS 2008 is beyond me. Fortunately a lot of third parties are filling this vacuum and providing data grids for WPF and some of them are (at least for now) free.

I am really pleased to see that we can mix and match classic Windows forms with WPF in the same application. This would allow us to start migrating to the new UI paradigm slowly rather than starting from scratch.

One of the topics that I've been noticing for a while is how we (as developers) tend to worry about our software development process and try to educate ourselves on the topic while other project stakeholders don't seem to show as much interest on it.

For example, I often talk to other developers who are trying to implement Scrum, RUP, or other software development process at their companies and they fell that other project stakeholders (executives, business owners, analysts) are not as interested in the process as they should. It seems that it's always developers trying to drive these initiatives.

The most interesting part, however, is that process improvement is not an activity that should be left alone to developers, not even when is about software development process. If the software has an impact on the bottom line of the company (and it usually does) it should be an activity that is done in collaboration with executives, business owners, QA, and developers. Developers, as well intended as we might be, don't always see the whole picture and we are bound to make one-sided conclusions. Or as Buxton says

"the typical approach of software engineers having sole responsibility for new product development [is like] an ice hockey team with all goalies. No matter how good your goalies are, they aren’t going to beat even a bad team with the proper mix of players."

One of my co-coworkers had the great idea of taking a couple of business owners to our local Agile user group and they both seemed to have enjoyed it and learned a few things. Yet, that was the only time they attended one of these meetings.

I've given books on Scrum and Lean Software Development to some of our product owners with moderate success. Although they read it, it usually fails to spark ideas on what else we should be doing.

From what I've seen there are several reasons why most non-software developers don't pay as much attention to software development process improvement as they should:

• Software is just a small part of their job. They deal with it because they have to (e.g. the business needs a new software to help in the creation of X, but X is the real goal and software is just a necessary evil.) Although I've seen people not care too much about software development process even on software development companies where X is a piece of software!
• Everytime there is a conversation about process improvement it inevitably turns into a discussion about software tools and other geeky issues where they feel lost.
• They don't think there is anything wrong with the existing process.
• Althought they see problems with the existing process but they thing that's the way it is.
• They just haven't realized they could participate on it and that is on their best interest doing so.

This is an open invitation to all project stakeholders (executives, business owners, QA, developers, users) to jump into the software development process improvement in your company. Talk to your team members and see what you and others can do to develop better software and get it to your users more often.

Last week I finished reading The Mythical Man-Month by Frederick Brooks for the third (or so) time. Although this book was originally published in 1975 it is still a great reading for anyone involved in software development. The edition that I read is the 20th anniversary edition and has a few extra chapters with updated information.

There are two chapters that I particularly like in this book: The Mythical Man-Month and No Silver Bullet – Essence and Accident.

The Mythical Man-Month

In this chapter Brooks explains why using “man-month as a unit for measuring the size of a job is a dangerous and deceptive myth. It implies that men and months are interchangeable.” He goes on to explain that “men and months are interchangeable commodities only when a task can be partitioned among many workers with no communication among them. This is true for reaping wheat or picking cotton; it’s not even approximately true for systems programming” (emphasis on the original.)

This myth is a typical problem in our industry. I cannot enumerate how many times we’ve estimated a job to take X number of months by Y number of developers (say 6 months by 2 developers) and then somebody come and say “but if I give you twice as many people you’ll finish in half the time, right?” (say 3 months with 4 developers.) This hardly ever works out. Having more people in the team is not bad, but comes with a price to the team’s productivity mainly due to the communication and work allocation that has to happen among team members. There is a logical tendency to underestimate this burden because we tend to see men/women interchangeable with months.