Towards better public software projects

Ed Lazowska shared some observations about the failures in some recent large scale public (or quasi-public) software projects – notably the federal health portal. This project, like others, has had a massive budget, and undoubtedly has had hardworking and well-meaning people beavering away on it. And yet is is an operational failure so far.

The discussion on Facebook is worth reading, Ed is a super smart guy and has a set of very smart friends on Facebook. Ben Slivka points out that this is not a new pattern, many large scale government IT projects struggle. Ed and others point out it is not just large scale government projects, but large scale commercial projects as well. Rick Rashid points out agency problems and also the classic mistake of thinking that a “bespoke” solution is required, that off-the-shelf solutions are just not sufficient – Rick’s comments are very short but well worth reflecting on.

I’ve got some observations up there but I will repeat and expand here. First, there is a dramatic contrast between the economics and quality of this project, and that of internet startup projects. Internet startups survive on a fraction, a small fraction, of the budget allocated to the healthcare portal. And the traffic volumes that the healthcare portal aren’t crazy large, internet startups can scale up to handle these traffic levels reasonably quickly. This suggests that as a society we should try to figure out how to get some of the internet startup mojo applied to the health marketplace portal or other public infrastructure problems. Otherwise all that startup effort is going to be used to serve up animated gifs of cats wearing hats or gosh knows what else.

Second, rather than just trying to learn from internet startups and apply the lessons to a large public software project, perhaps we should create a different structure for these projects that encourages entrepreneurs to work hard on our public projects. Rather than have the government run the whole project, perhaps the government should define the goals and establish prizes, ala the XPrize foundation. Rather than committing a priori to a project and plan, let 10 or 100 teams chase the goal, and pick the best at the end of the project. Hand out contracts and award dollars at the end to the best projects and cherry pick the results. Have some consolation awards so that ultimately many teams are motivated to fight their way to the finish line. The government may be better setting goals and handing out awards rather than trying to build it all internally.

There are certainly problems with a prize-driven approach, and some thought needs to be applied to ensure that the most obvious pitfalls are avoided. Having a truly independent panel judge the results will be important. Preventing corruption and influence in the judging process will require some effort. Thinking through the IP issues and ensuring that the resultant IP is appropriately owned would be important. And many others.

We have proven that the way we are engineering these large public systems is costly and not very performant. We should discuss different ways to approach the problem, it seems stupid to keep on doing things the same way.