Why big IT projects always go wrong

In 1975, a computer scientist named Fred Brooks published one of the seminal texts in the literature of computing. It had the intriguing title of The Mythical Man-Month and it consisted simply of a set of essays on the art of managing large software projects. Between its covers is distilled more wisdom about computing than is contained in any other volume, which is why it has never been out of print. And every government minister, civil servant and chief executive thinking about embarking on a large IT project should be obliged to read it – and answer a multiple-choice quiz afterwards.

How come? Fred Brooks was the guy who led the team that in the 1960s created the operating system for IBM's 360 range of mainframe computers. This was probably the largest non-military software project ever mounted, and it was of vital strategic importance to IBM, which then completely dominated the computer business. It also turned out to be vastly more complex than anyone – including Fred – anticipated, and it rapidly metamorphosed into a kind of death march.

The project fell further and further behind schedule. But because IBM was a rich company and OS/360 was so important, it was able to throw more and more resources (ie programmers) at the task. But as it did so, the problems got worse, not better. At which point Fred Brooks had his epiphany: he realised that every time he added a programmer to the team the project fell further behind.

In the end, however, the job was done. The death march ended, OS/360 was delivered and IBM went on to make a lot of money from it. Brooks, for his part, resigned from the company, became professor of computer science at the University of North Carolina in Chapel Hill and then sat down to write the book that made him famous. His aim was to distil into a set of elegant essays everything he had learned from the OS/360 experience. The striking title came from his epiphany – the realisation that man-months are a hopeless metric for assessing the size of a complex software project.

Why? Basically because a big software project involves two kinds of work: the actual writing of computer code; and co-ordinating the work of the dozens – or maybe hundreds – of programmers working on different parts of the overall system. Co-ordination represents an essential but unproductive overhead: and the more programmers you have, the bigger that overhead becomes. Hence Brooks's law: adding manpower to a late software project makes it later.

Over the years, fragments of Fred Brooks's wisdom have percolated into the consciousness of ministers, civil servants and chief executives. But only fragments. In Britain we are wearily familiar with the long, dreary catalogue of botched or outlandishly expensive government IT projects. This is not just a public sector problem, however. Research conducted by two Oxford academics and published in the Harvard Business Review suggests that the private sector has almost as much difficulty managing big software projects, and that some such projects have even endangered the survival of the companies that embarked upon them.

A case in point was the venerable clothing manufacturer Levi Strauss. In 2003 it was a global corporation, with operations in more than 110 countries but with an IT system that was an antiquated, "Balkanised" mix of incompatible country-specific systems. So its bosses decided to migrate to a single SAP system and hired a team of fancy consultants (from Deloitte) to lead the effort. "The risks seemed small," wrote the researchers. "The proposed budget was less than $5m." But very quickly things fell apart. One major customer, Walmart, required that the system interface with its supply chain management system, creating additional work. During the switchover to the new system, Levi Strauss was unable to fulfil orders and had to close its three US distribution centres for a week. In 2008, the company took a $192.5m charge against earnings to compensate for the botched project — and fired its chief information officer.

The Oxford researchers examined more than 1,400 big IT projects – comparing their budgets and estimated performance benefits with the actual costs and results. The average project cost $167m and the largest a whopping $33bn. The researchers' sample drew heavily on US-based projects but found little difference between them and European projects. Likewise, they found little difference between private companies and public agencies. One in six had a cost overrun of 200%.

The message is clear: if you run a big company or a government department and are contemplating a big IT product, ask yourself this question: can your company or your ministerial career survive if the project goes over budget by 40% or more, or if only 25-50% of the projected benefits are realised? If the answer is "no" go back to square one. And read Fred Brooks's lovely book.