{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "@id": "https://www.vidyasource.com/blog/dont-go-chasing-waterfall/",
  "url": "https://www.vidyasource.com/blog/dont-go-chasing-waterfall/",
  "mainEntityOfPage": "https://www.vidyasource.com/blog/dont-go-chasing-waterfall/",
  "headline": "Don't Go Chasing Waterfall",
  "description": "Michael Daconta happily blames agile software development for the problems on the health care website. He's wrong. And yes, the title is a TLC reference.",
  "datePublished": "2013-11-11T00:00:00.000Z",
  "author": {
    "@type": "Person",
    "name": "Neil Chaudhuri",
    "jobTitle": "President",
    "url": "https://www.linkedin.com/in/neil-chaudhuri/",
    "sameAs": "https://www.linkedin.com/in/neil-chaudhuri/"
  },
  "publisher": {
    "@id": "https://www.vidyasource.com/#organization"
  },
  "image": "https://www.vidyasource.com/img/blog/waterfall.jpg",
  "keywords": [
    "Scrum",
    "Jenkins",
    "Agile",
    "Government",
    "Project Management",
    "Software Engineering",
    "Testing",
    "Continuous Integration"
  ],
  "articleBody": "As someone who spends a lot of time thinking about technology and software project management (and the possibility of a\n[Ghostbusters sequel](http://www.huffingtonpost.com/2013/11/07/ghostbusters-3-emma-stone-jonah-hill_n_4232686.html) but\nthat’s beside the point), I have been really frustrated by the poor quality of reporting by the media on the failures of\n[HealthCare.gov](https://www.healthcare.gov/). The flawed coverage has shifted from the enormous functionality and\nscalability problems to the blame game--contractors, government officials, the federal contracting process,\n[waterfall software development](http://www.washingtonpost.com/blogs/wonkblog/wp/2013/10/21/the-way-government-does-tech-is-outdated-and-risky/),\nand now apparently, agile software development as well.\n\nI contributed to a [GAO report](http://www.gao.gov/products/GAO-12-681) on applying agile methods in the federal\ngovernment. Check it out if for no other reason than to double its readership. And someone is going to try to blame\nagile methods for this mess? Oh. [It’s on now](http://www.youtube.com/watch?v=uvqJ1mTkEuY).\n\n[Michael Daconta](http://www.linkedin.com/profile/view?id=713784) recently published on GCN, the leading online publication for\nAmerican public sector IT. Despite patronizing praise for some agile practices, it’s pretty clear the accomplished Mr. Daconta\nis the epitome of the traditional waterfall executive. More than that, his hostility to agile software development is a\npreexisting condition (See what I did there?). He is eager to embrace the failure of the allegedly agile HealthCare.gov\ncontractors who developed the front end and the “data services hub” as vindication. In his zeal\nto proclaim “I told you so” to what he perceives to be haters in the media, Daconta’s piece ranges from simplistic to\ninaccurate.\n\nLet’s explore Mr. Daconta’s reasoning in detail.\n\n> I have seen some of the developer documentation, and it clearly discusses sprints, user stories and incremental testing — all of which are hallmarks of an agile process....The reality is that the developers did use agile, and the project failed miserably.\n\nMaybe the developers actually had a to-do list comprised of what they called “user stories” that were addressed during\n\"sprints,\" and maybe some kind of testing happened. Calling this agile is like calling me a [CrossFit](http://www.crossfit.com/)\nenthusiast because I did some pushups and [burpees](http://www.outsideonline.com/fitness/outside-fitness-center/Why-You-Need-to-Be-Doing-Burpees.html) yesterday.\n\nLet’s assume these teams were attempting Scrum since sprints and user stories are both Scrum-specific\nrather than generically agile. It is true Scrum has a to-do list called a backlog, features characterized as\n[user stories](http://www.mountaingoatsoftware.com/agile/user-stories) delivered in sprints, and\n“incremental testing.” A real Scrum effort by the HealthCare.gov teams would have these characteristics:\n\n - The consistent and immediate availability of a [Product Owner](http://www.mountaingoatsoftware.com/agile/scrum/product-owner)--\n a single person or more likely in this case a team of subject-matter experts in the government and private health insurance\n companies--available to answer domain-specific questions about the features to be delivered\n\n - Not just the mere existence of a backlog but a backlog comprised of\n [appropriately sized](http://www.mountaingoatsoftware.com/blog/how-can-we-get-the-best-estimates-of-story-size) user\n stories prioritized by the Product Owner so the most important features get done first\n\n - A Sprint Planning Meeting at the beginning of each sprint where the team commits to the  features they can deliver in\n the sprint and breaks them down into tasks estimated in hours\n\n - Daily standups where everyone on the team takes 15 minutes to catch up on status and, more importantly, lets everyone know what's\n  getting in the way of progress\n\n - A suite of fully automatable unit and integration tests for developers to feel confident their code does what they\n think it should do\n\n - A suite of fully automatable *acceptance* tests created by developers and testers with the Product Owner to verify the\n application works--correctly, securely, and at scale\n\n - A continuous integration infrastructure using [Jenkins](http://jenkins-ci.org/)\n or similar to build the application and run all these automated tests multiple times a day--including running all the\n unit tests anytime someone checks anything into [version control](http://git-scm.com/video/what-is-version-control)\n\n - Information radiators like [burndown charts](http://www.scrumalliance.org/community/articles/2013/august/burn-down-chart-%E2%80%93-an-effective-planning-and-tracki)\n and [velocity](http://www.scrumalliance.org/community/articles/2007/march/glossary-of-scrum-terms#1110) charts posted on the walls to enable\n everyone to assess project status at a glance and motivate course corrections as needed\n\n - A meeting held a few times per week among the different teams called a [Scrum of Scrums](http://www.scrumalliance.org/community/articles/2007/may/advice-on-conducting-the-scrum-of-scrums-meeting)\n so they can take a half hour to coordinate interface details among themselves\n\n - A demo at the end of the sprint--best given by the Product Owner--of the software completed during the sprint to\n anyone who wants to see it. I imagine a few people might care about the progress made on the public face of the President’s signature\n domestic achievement.\n\n - A Retrospective at the end of the sprint for the team to assess possibilities for process improvement\n\n - Most importantly, **the software is production-quality at the end of every sprint.**\n\nThat’s a lot of stuff.\n\nAll of these must be true, or really close to it, for development to be truly agile. But I am willing to bet Mr. Daconta\none year of insurance premiums for a gold plan on the federal health care exchange that almost none of these happened\nwith the development on HealthCare.gov.\n\n>\nI am currently involved in a Defense-related big data project where we are analyzing and decomposing customer requirements into system requirements....This will take a few months, but when it is done we will know exactly what we have to build….A complex back-end data services hub — a piece of software with zero actual, living, breathing end-users — has to be described in terms of “user” stories. Does something sound off-key to you?....Yes, I’m sure you can bend and twist user stories to address non-user based functionality, but should you?\n\nI cannot believe that someone in 2013 would be proud of taking “a few months” to create a PDF and not write\none line of code a client can use. Worse yet is Mr. Daconta’s naive understanding of how user stories work.\nHe insists *user* must imply a human being when in fact the user of a feature could be another component in the system.\nEvery non-trivial system has a lot of backend interactions going on, and agile methodologies have long taken that into\naccount. After all, a big-data professional like Mr. Daconta should know that Salesforce.com famously adopted Scrum\nthroughout its organization [very successfully](http://www.forbes.com/sites/stevedenning/2011/04/18/six-common-mistakes-that-salesforce-com-didnt-make/).\nDoes he really think everything Salesforce does is end-user-facing?\n\nIn the case of HealthCare.gov, let me make up a reasonable user story where the user isn’t a human being:\n\n*As the Subsidy Eligibility Component, I need to access IRS data to verify the income information provided by the applicant.*\n\nThat’s a big story that would need to be broken down into smaller component stories, but you get the idea.\nThe Subsidy Eligibility Component--something I made up--is a piece of software that’s going to do a calculation based\non business rules defined by the [Affordable Care Act](http://en.wikipedia.org/wiki/Patient_Protection_and_Affordable_Care_Act#Public_policy).\nIt demands the development of a feature that enables access to\nIRS data to do that calculation. The Subsidy Eligibility Component is the “user” of this feature. Does that really blow your mind?\n\nBut OK. Let’s say this whole users-not-always-humans thing is just way too weird. In his wonderful book\n*[Succeeding with Agile](http://www.mountaingoatsoftware.com/books/succeeding-with-agile-software-development-using-scrum)*,\nagile guru Mike Cohn points out something that even agile practitioners forget. Scrum doesn’t *make you* define\nfeatures as user stories. If you’re doing Scrum but a user story feels unnatural for a feature that humans never see, that's cool.\nYou can try an alternative. Cohn suggests the syntax they use in another agile methodology called\n[Feature-Driven Development](http://en.wikipedia.org/wiki/Feature-driven_development):\n\n*\\<action\\>\\<result\\>\\<object\\>*\n\nIn our case, the feature would simply become\n\n*Verify the income of the applicant.*\n\nNow was that so hard?\n\n>\nFor the state of Florida, I led a team in designing and developing a data integration hub that recently successfully passed its first milestone. The software design document was reviewed by the customer, Gartner, and the key software infrastructure vendor….In contrast...we see a site that failed a test of 200 to 300 people  when it was supposedly designed for 50,000 simultaneous users...Where are the design and architecture documents that show how the system was built to be scalable? Oh, agile processes don’t like design documents\n\nPersonally, I find nothing to brag about until working software\ngoes live and clients are happy using it. But Mr. Daconta’s [argument from authority](http://en.wikipedia.org/wiki/Argument_from_authority)\naside, he makes the same old facile claim [UML](http://en.wikipedia.org/wiki/Unified_Modeling_Language)-lovin’ waterfall\ndisciples have made forever. To say agile processes don’t care for design documents is simply, unequivocally wrong.\n\nWe believe instead that you should only do the *bare minimum design necessary* for you to get started delivering value,\nwhich is working, production-quality software. This could be as little as some drawings on a whiteboard created by the\nteam with the Product Owner. These drawings could even join the other information radiators on the wall for all to see.\n\nLightweight documentation is superior to heavy, UML-infected documentation for several reasons:\n\n - We consider time and money to be finite resources, so we don’t want to waste them on stuff that generates no\nvalue to the client.\n\n - We know from experience that circumstances--realization that a technology we chose is in fact unsuitable,\nchanges in priorities, external pressure of some kind, etc.--will eventually demand a change in the design. Why would\nyou spend say...oh, I don’t know...a few months and a lot of taxpayer money to develop a document that is going to be\nobsolete in less time than it took to create it?\n\n - Every developer knows *the best documentation is code*--namely unit and integration tests. Aside from, you know, testing\nthe code, tests convey exactly how the code works and fits together. They are also guaranteed to be current because they pass;\nif they don't, you have bigger problems. When I join a project, studying and running the tests helps me get up to speed much\nfaster than reading stacks of boring, obsolete PDF's.\n\n - I can never remember when to use arrows and when to use diamonds in a UML class diagram. Or why anyone cares.\n\nIf a client demands a rigorous design document upfront, as many government clients naively do, then create one because\nit is generating value--not for the software but for satisfying a contractual milestone. We can just drop a stack of PDF’s on their doorstep with\nwhat agile legend [Martin Fowler](http://martinfowler.com/) once called\n\"*[The Almighty Thud](http://martinfowler.com/distributedComputing/thud.html)*.” Just know a heavyweight design document does nothing to help\ndevelopers. In fact, it hurts developers by robbing them of the resources they need to do great things.\n\nBesides, Mr. Daconta should be aware that agile methodologies came along precisely because people were confused back in the\nday about why so many projects fail when they are documented so thoroughly. To suggest absence of documentation led to\nthe problems with HealthCare.gov is to cling to an ideology and ignore a history of data to the contrary.\n\n>\nIn examining some of the Business Service Description (BSD) documents, we see that key interfaces (like verify income and verify citizenship) were designed as synchronous instead of asynchronous interfaces….Frankly, for a site like this that requires high levels of reliability and scalability, a synchronous API design for the data hub is inexcusable.\n\n[Huh?](http://www.youtube.com/watch?v=tpsEdM74maA) I thought there were no design documents. Now Mr. Daconta uses some to\ncite another problem.\n\nAnyway, Mr. Daconta actually gets this one exactly right. So much is going on that all the operations on HealthCare.gov need to be [asynchronous and\nnon-blocking](http://stackoverflow.com/questions/7931537/whats-the-difference-between-asynchronous-non-blocking-event-base-architectu)\nfor the application to perform at scale.\n\nBut what the heck does this have to do with agile?\n\nThere is nothing--not waterfall, not agile, not UML, not [gluten-free](http://www.fda.gov/ForConsumers/ConsumerUpdates/ucm363069.htm),\nnot [Pilates](http://en.wikipedia.org/wiki/Pilates)--\nthat will save overwhelmed developers from making bad technical decisions.\n\nHowever, being agile will help you “fail fast.” Remember all that automated testing that happens in an agile project?\nYou are encouraged to experiment because the tests will fail *immediately* if you take the wrong path. If the\nHealthCare.gov developers who claimed to be agile actually were, their tests would’ve exposed the limits of a\nsynchronous design. It would be a setback, but because sprints are so short, it's only a few weeks at\nworst--far preferable to what’s happened since.\n\nOn a waterfall project, you test everything some time down the road--in theory. What happens in reality is that everyone\nstarts freaking out about how much work is left with the deadline fast approaching and so much time lost to generating UML.\nPeople scramble around the clock\njust to code *something* and claim all the features are delivered as contractually mandated. The testing that was planned\nnever happens.\n\nSound familiar, [Secretary Sibelius](http://www.hhs.gov/secretary/about/biography/)?\n\nThe developers of the front end and the data services hub of HealthCare.gov did something that is very common. They ran\na waterfall project cloaked in agile terminology and token application of some agile ideas. It is hard for me not\nto blame a seasoned, successful professional like Mr. Daconta for being as easily fooled by a weak disguise\nas [Lois Lane](http://en.wikipedia.org/wiki/Lois_Lane).\n\nMichael Daconta clearly has accomplished a great deal in public sector IT, but his agenda-driven, simplistic analysis\nof why supposedly agile teams failed on HealthCare.gov is beneath a senior executive of his stature. His analysis\nbetrays something between confident ignorance and callous disregard for facts.\n\nThe problems with the implementation of HealthCare.gov are complex, and they will be debated for a long time to come.\nBut there is one thing I know for sure.\n\nIf you are looking for a solution to the problems that have plagued public-sector IT for decades, and now most famously\nHealthCare.gov, [don’t go chasing waterfall](http://www.youtube.com/watch?v=8WEtxJ4-sh4)."
}