Software development death spiral

Studies show that around 70% of all software development projects fail being delivered on time, within budget, or most importantly, meeting their initial requirements. The cost overrun averages around 189% and the time overrun somewhere around 222%. If you dig a bit deeper, you realize that these numbers translate into billions of dollars spent every year trying to fix problems that could be prevented or minimized from the start.

The Software Development Life Cycle

Regardless of the software methodology, all software development boils down to five simple stages: Analysis, Design, Development, Test, and Release. These stages are commonly (and wrongfully) presented as a circle and that leads (non-technical) client to understand the software development as a constant flow with all stages equal in length and time.  This is particularly problematic when our development methodology is based on iterations.

If we focus on web applications, specifically PHP applications, the most common methodologies used for development are Waterfall and Agile (or a variation of these two). I will not go on explaining in detail the pros and cons of each one but, for the purpose of this article, it’s good to understand the basic differences between the two.

Waterfall is a sequential methodology where each stage only starts after the previous one is finished. This traditional methodology is still widely used today, specially for smaller projects where the requirements are clear and there’s a concrete vision of the project’s outcome.

Agile is an iterative methodology that focus on delivering complete features of the project in a very short period of time. Development teams that follow this methodology need to move quickly through all stages of the project’s life cycle, in a fixed amount of time, and then rinse and repeat.

One thing to keep in mind is that these methodologies exist to help develop better and faster applications. They provide structure and guidelines but should not be treated as fixed rules. Methodologies can be changed and adapted to each particular project. Whenever the methodology is stepping in the way of the development process it’s time to stop and change.

One good example of this is when a client has a project with a long list of requirements that are unclear or too risky. In these situations it’s better to adapt a Waterfall methodology, analyze the requirements with the client and create a partial list of essential features to implement (Minimum Viable Product). After moving through all the stages and create the MVP, switch to Agile and continue implementing the rest of the requirements.

The Software Development Death Spiral

Each stage of the development cycle is subject to problems that affect not only that particular stage but all the following ones. Usually, the outcome of these problems is an increase in development cost, time, or technical debt. These little “hiccups” in the process add up and, if left unchecked, may cause the death of the project.

Here’s a picture to illustrate what I mean.Detective Conan: Crimson Love Letter streaming

 Software Development Death Spiral

Analysis Stage

An application design is as good as its requirements. During the analysis stage it is crucial that the requirements are gathered and described in a clear and correct way. Communication is key during this stage because the client knows the business and the development team has the expertise to implement the application.

I lost count on how many times a client came to me with requirements that were something like “Just make me a clone of InsertSuccessfulAppNameHere but add this one little extra feature” or “Just make InsertDesktopAppNameHere run in a browser”. Other common issue I encounter many times is unrealistic client expectations. I had a client describe this feature to me: “So the user opens the app and we show him on map with his favorite shops and he can do a one-click buy and get the products delivered to him at home”. What do you mean about favorite shops? You realize that the user needs give us permission to use his current location? When exactly does the user gives us his payment details?

Design stage

Inadequate architecture design, wrong technology stack and inaccurate estimations are the most common problems in the design stage. Communication between client and development team is again crucial so that all technical requirements and future business expectations are factored in when selecting the technology stack and the architecture. Choosing the technology stack based solely on the experience of the current developers is a recipe for failure. Finally, estimating unclear requirements is another common issue that leads projects to fail both in time and budget.

Development stage

This is the stage where all the previous issues start causing real tangible problems. Unclear requirements lead to poor design which, in turn, leads to critical problems with development. Factor in the “missed” features that where not discussed before and the development will certainly fail. Up to now, I’ve been focusing a lot on issues caused mostly by poor communication from the client side, but from now on the development team is the one responsible for the major issues. One thing that all developers must understand is that if you reach the development stage, then it’s all on you. You accepted to develop an application that is poorly described. You did not alert the client that his decisions would have negative effects in the long run. You estimated time and cost to deliver a product that is unclear to you.

If all that was not enough, the development stage has problems of its own. If the development team is not mature and skilled, they will not follow industry best practices, they won’t bother with documentation, they won’t write tests and they won’t go out of their way to manage the ever-growing technical debt.

Test Stage

Most clients wrongfully expect to reach the end of any development stage with a bug-free application. Well, there’s no such thing. Even after testing and quality assurance, it’s estimated that an application carries up to 7 issues per 1000 lines of code. The worst and most common issue with the testing stage is having no testing stage at all. Sometimes because the project is not big enough to even bother to estimate it, other times because the development stage took too long and the team decided to cut the testing time short to keep up with deadlines. As a developer, ask yourself this: What is the benefit of delivering an unstable and unusable application?

Deployment Stage

The deployment stage is very straightforward. The application is ready so let’s go live, right? Wrong! This simplistic approach is the most common problem with deployment. I ran into problems trying to deploy legacy applications because there was no deployment instructions, no information about minimum version requirements, mismatches between testing servers and live servers, and no automation scripts for databases. It is the responsibility of the development team to make sure the deployment process can be done by anyone that was not involved in the development.

How to keep away from the death spiral

To conclude this article, I’m leaving below a list of recommendations for both the client and the development team. These recommendations don’t guarantee success, but I’m certain that they will lower the chances of failure.

For clients:

  • Make sure that all the project’s business requirements are clearly described. Not sure how to do it? Ask the help of a Business Analyst.
  • Involve your team when creating technical requirements and project specifications. You have the business know-how, they have the technical experience. Work together.
  • You will receive as good estimations as your requirements. If they are too generic or incomplete, then the estimation will be the same.
  • Ask for options on technologies and architecture. Research pros and cons. Make sure that everyone is clear about your growth expectations. Never choose a technology just because it’s the only one the developers know.
  • Avoid changing the requirements in the middle of the development stage. If your requirements are not clear or if there is a high chance of change, adopt an agile methodology and wait for the next iteration to add new features.
  • Make sure that there’s time estimated for testing and quality assurance. Expect working features from your developers but never expect that those features are bug-free.
  • Set clear and achievable deadlines. Your development team will sacrifice the project’s quality in favor of deadlines if you push them to it.

For Development Teams:

  • Expect proper requirements and help your client gather them if you have the know-how. Study the requirements, identify unclear or missing workflows and request descriptions for them.
  • Set clear expectations. Estimate realistically. Always under promise and over deliver. Do not accept projects where the client is setting technology limitations that you are not comfortable with.
  • Keep communicating with your client. Talk about your challenges and work together on solutions. Always alert your client when decisions are being made that will increase the technical debt of the project.
  • Make sure your development follows best practices. Continuous integration, code-styles, dependency management, documentation and tests need to be a part of your development. Never forget that you may end up maintaining the code you are writing.
  • Never skip on testing and quality assurance. It does not matter how beautiful is your code if it doesn’t do what was required initially. Your client is interested in results not in the process.
  • Implement deployment automation or, at the very least, have someone external deploy the application. It is the best way to make sure the deployment process is well documented.

Daniel R.

PHP developer and consultant. Enjoys solving problems, building apps and (occasionally) breaking stuff. Currently lives in Poland and works for an american startup focused on data-gathering.