Is the Waterfall Method Just a Way to Procrastinate?

This weekend, I finished reading the very good book The Now Habit: A Strategic Program for Overcoming Procrastination and Enjoying Guilt-Free Play, written by Neil Fiore. I really wish I had read this book before reading Getting Things Done: The Art of Stress-Free Productivity. The information in each book reinforces the other, but I think the information in The Now Habit would have provided a solid foundation to better understand the reasoning behind the strategies presented in GTD.

While reading The Now Habit, it was interesting to reflect on the information in the book as it related to the philosophies behind different software development methodologies, specifically Waterfall-like methods versus Agile-like methods. There were two main points of the book that stood out to me as related to software development.

Perfectionism
Fiore suggests that creative work requires workers to eliminate desires for perfection and allow themselves the opportunity to make mistakes. Perfectionism blocks workers from starting, making them afraid the initial steps they may be prepared to take aren't part of the "perfect" solution. Now compare the two different methodologies:

• Waterfall Method: Requirements and design must be "perfect" before starting any development work. Mistakes in earlier phases must be avoided, otherwise their cost explodes in later cycles.
  
• Agile Method: Development is undertaken with the understanding that initial ideas and design will need correction. Test-driven development provides a safety-net so mistakes cost less and can be corrected without fear of breaking other parts of the system. Working code is given preference to perfect code.

Frequent Rewards
Another suggestion Fiore makes is to plan work in short time intervals, giving yourself frequent rewards after completing discrete tasks needed to complete the project. Rewards that exist too far out into the future will only encourage a person to engage in activities that provide a more immediate reward (e.g. watching television). Comparing the methodologies once again:

• Waterfall Method: At best, the development cycle is broken up into short iterations allowing for frequent feedback during that cycle. However, such feedback rewards will only occur after the Requirements/Design phases. Any rewarding feedback from customers on the working product can only occur when requirements changes are much more costly.
• Agile Method: Short, frequent iterations reward the team with visible signs of progression. A short development iteration will have a proposed completion date near enough in the future to provide motivation to make the shorter deadline. The completion of each iteration builds momentum to move on and complete the next.

Should You Get A Master's Degree?

I'm currently a couple of semesters away from (hopefully) earning a Master's degree in Software Engineering. It has been a long process, one that I'm anxious to see come to an end. Working full time, while taking one class per semester, drags the process out much too long.

If I could travel back in time, I'd tell my younger self two things:

1. Google's stock won't be overpriced at $100 when they have their IPO.
   
2. Don't waste your time getting a Master's degree.

Waste is probably too strong of a word as I don't feel the entire process has had no benefit. I was able to take two programming classes I didn't take as an undergraduate (databases, compilers), and I've learned quite a bit working on my thesis project (more on that later). Having a Master's degree may provide me the opportunity to teach later in my career, something I always thought I might enjoy.

But overall, I feel that my time spent on course/project work could have been spent on things that would have improved my personal brand much better. Looking back, I could have built a website using Ruby on Rails, contributed to an open-source project or spent more time improving my writing skills by blogging regularly (at this current moment, writing a blog post is a way to procrastinate on my project). All of these activities would make me much more attractive to a potential employer than would having a Master's degree listed on a resume.

My thesis project is an attempt to build an automated refactoring tool for C++ programs. I knew going into the project that writing such a program was a fool's errand. But it seemed interesting at the time and I knew the project would be approved. Although I have quite a bit of work left, assuming I get as much functionality completed as I think I can, I'll be pretty pleased with the end result. It will be more of a prototype than a complete program. But one only has to do a Google search on parsing C++ to see why I think a prototype would be a pretty nice accomplishment.

So to answer my question, no, you probably shouldn't get a Master's degree. Instead spend your time learning/building something you find interesting, something that could impress a future employer or turn into your own business.

If you ignore my advice and still want to get a Master's degree, I would then offer you this advice. Get your degree immediately after you finish your undergraduate degree. Don't start after you are in the work force and have to take more than two years to finish. Also, wait until after you earn your degree to get married and have kids. :)

Problem Solved

In my last post, I talked about how include dependencies in our code base were causing lengthy build times. Well thanks to this entry on Jeff Atwood's blog, Coding Horror, our problem was solved overnight. Okay, our problem really isn't solved. We still have include dependency issues, but they no longer affect our build times, thanks to Incredibuild.

Jeff's mention of Incredibuild has completely transformed my day at work. A build that used to take over twenty-five minutes now takes about five minutes. I no longer have to worry about switching tasks/contexts to fill twenty-five minute voids throughout my day. I no longer have second thoughts about working in certain areas of the code, knowing that any small change would require a full rebuild. Refactoring can occur much faster and adds to the wave of motivation that has swept through our team since we began using this product.

Incredibuild is by far one of the best products I've ever worked with. Not only does it work as intended, but it was incredibly easy to setup. After reading Jeff's post, I downloaded the trial version and had the software working on two of my machines in less than fifteen minutes. A quick "You have to check this out..." email to my fellow team-members resulted in us having a seven machine build network by the end of the day. At that point, we started bugging management to approve purchasing the software and we are now on our way to becoming a much more productive team.

I am still trying to eliminate some of the worst include-dependencies/coupling in our code. But I no longer have to wait twenty-five minutes to get any feedback for my effort.

The Case of the Horrible Include Dependencies

One of my favorite programming books is Working Effectively with Legacy Code written by Michael Feathers. The book provides an arsenal of techniques to refactor legacy code and get it into a test harness.

One specific situation the author addresses is testing a C++ class that has a large number of include dependencies. Our product has many classes that suffer from this problem. Our main library consists of over 600 source (.cpp) files and changes to a single header file often result in a large number of files having to be recompiled. A full rebuild takes over thirty minutes and is a productivity and motivation killer. I've decided to attack this problem at its root and try to eliminate as many of the dependencies as possible. At the very least, I will reduce the number of times a full-rebuild is necessary and will hopefully see a noticeable reduction in compilation time in the future.

I started off by writing three Ruby scripts to collect data about the include dependencies for our product. The first script traverses our source tree, inserting the following line into each header file after the header guard:

#pragma message("including __FILE__")

The second script parses the compilation output, tabulating the number of times each header file is included and how many header files each .cpp file includes.

The final script is the inverse of the first and removes all of the #pragma directives.

The scripts were easy to write and they output the exact information I want, so I don't think a code analyzer tool would have saved me much time. The main downside to my approach is that I have to do a full rebuild anytime I want to get all of the current data. But since this project is something I'm doing on the side, rapid feedback isn't that important. I can just kick off the build and come back to it when I have some down time.

Since beginning this project two weeks ago, I have been able to significantly reduce the include dependencies on about half a dozen header files. For example, one header file that was being included 455 times is now included only eighteen times. That is quite a time saver the next time someone changes that header file.

It's very motivating to see the include dependencies shrink after each small improvement I make. In his book, Michael Feathers talks about situations where programmers feel the code the work on is beyond repair. He points out that little changes here and there begin to add up, and you slowly realize the situation isn't as bad as it seems. I think the small victories I've had so far point to continued success and will result in a much better code base and a much more productive team.

Great Experience with Unit Testing

This weekend I planned on stopping in at work for an hour or two to clean out some simple things on my TODO list. Unfortunately, upon opening my email, I was greeted with a message that our overnight build/run system had failed. In particular, a unit test suite for a library that I had created had failed to pass. The email also indicated that our application had significant changes in output from its previous run, something that wasn't expected for any changes that had been checked in the past day.

The unit test failure struck me as quite odd since the code in that library hadn't changed in over a year. I reran the test suite in my development environment, found the test that was failing and noticed that an output string I was expecting had changed. This failure gave me an immediate clue to where the problem was introduced, and with great satisfaction, I was able to track down the problem in no time.

As it turns out, my unit tests were not the only code expecting to see that same string value. Another major library in our system, which controls output functionality, had failed due to the changes that had been made. This library is much more complicated and does not yet have test coverage.

What I found significant about this experience, was that my unit tests had caught an error in functionality outside of the library they were testing. The library I wrote is pretty trivial, and I actually wrote the tests after writing the code (tut, tut) to try and introduce unit testing to my team. So an exercise that I felt was pretty academic at the time, had a significant practical payoff. Had I never written those tests, I would have spent much more time tracking down the change that had caused the problem.

Lesson learned: Unit tests not only test the primary functionality being tested. They can also test secondary functionality that the primary functionality depends on. Even if a test seems pointlessly trivial, its worth writing as it may uncover problems in other areas of the system.