How Do You Measure Success in Software Development?

The Standish Group Chaos Report (refer: Table 1) is an annual survey that highlights the percentage of software projects that were successful, those that failed, and the ones that experienced challenges. This report was based on a study of around 30,000 IT projects in the United States between 1994 and 2000, while in 2004 alone this figure shot up to 50,000 projects. The Standish Group Chaos report is frequently cited because of its large sample size and the fact that the study aims to strike a balance between small, medium and large scale projects.

The metrics that a project’s success is measured against include meeting the following key evaluation criteria:

• Adherence to specified requirements
• Compliance with initial budget estimations
• Timelines of agreed delivery

To be considered a successful project, a project must meet each of these three important criteria. The ones falling under challenged projects did not comply with one or more of the aforementioned factors, and failed projects met none of the three criteria.

In my opinion, whether or not a project is considered successful should not just hover around compliance to predefined goals, budgets and timelines. The all-important issue of having what we at 3Pillar call a Product Mindset, which focuses on creating quality-driven products from design to deployment to post-launch support, cannot be overlooked.

It has been observed by others that many projects in the Standish Group’s report that failed to meet the criteria identified above were still considered “successful.” And on the contrary, even meeting all three of the criteria listed above was not necessarily a reliable predictor of a project’s success after the initial project delivery.

Here are some key takeaways that came to mind for me after evaluating the Standish Group’s report:

• Creativity in Problem Solving is Immeasurable: Creativity is at the core of every software development process. While a lot of advances have been made in developing sophisticated tools for automatic code generation and implementation of frameworks or various class libraries, there are still challenges in terms of creativity that each new project faces.
• Estimates are (Often) SWAGs: Another trend that’s common in IT project delivery is that Project Managers – not ours of course – inflate time estimations by around 20%. Such decisions can come from disagreement over estimates, uncertainties and assumptions around functionalities, and planning for additions to scope that are imperative before a product is released.
• Software is Intangible – All the action that goes on behind the scenes of a software product often goes unnoticed. What do I mean by that? I mean that all an end user interacts with is a graphical user interface design, so that is all that they “see.” They don’t “see” the HTML, Javascript, Java, web service calls, API calls, Objective C code, or anything else that may be necessary to render the visual product. Only a good software development partner will involve a client in every stage of the full lifecycle product development. Releasing prototypes is one way of eradicating this ambiguity and moving one step closer to a successful project delivery.
• Software Complexities – Software development is a complex business. For instance, nine interleaved successive conditions may result in many different possibilities for testing individually for a thorough validation process. The complexities involved here are quite evident as the testing cycle will take weeks to be completed. In addition, there arises a need to repeat these tests each time a change is made. This will ultimately result in bugs residing for a longer duration, adversely affecting the performance of the software.

At the end of the day, the only surefire way to measure the success or failure of software projects is in their ability to provide return on investment by creating real business value. At 3Pillar Global, we follow an Adaptive Product Lifecycle Methodology  to ensure we the products we build are tied to a client’s business goals.

This post was adapted from an article by the author that first appeared in Today Software Magazine titled “The  Features of IT Projects.”