Considerations upon iterative and incremental development (Short History)

Pretext

Please don't shy away as non-developer from this subject matter. Such a development of processes is important to grasp and technicalities are aimed an at superficial understanding.


Reflections Upon the Brief History

Ever heard about Scrum or Agile? Spare the quick search for the Manifesto for Agile Software Development. Later within this paragraph we going to outline the history of what is abbreviated with IID. Iterative and incremental development. Next to Agile and Scrum. Sometimes there is thrown in the word Evolution as well. More on that later. One cannot pause but think that "Principles of Systems Science" is related to this article, next to "Seeing like a State". The former is worthwhile to read after grasping the history of iterative and incremental development and its main proponent of the waterfall method or model on meta level. The later, "Seeing like a State", gives rise to the notion why something like the waterfall method is put in place in the first place.

"Seeing like a State" will be discussed in the future in it's own article in detail, though it's subtitle can clarify already what it is about "How Certain Schemes to Improve the Human Condition Have Failed". It's a book by James C. Scott and it is written in a dry and unpersuasive manner, therefore even more important if you ask us. The only real critique of "Seeing like a State" comes from people being associated with loving a big state and being of a leftist persuasion. Hence negative examples of the state failing to build something are plenty in the Soviet Union, that's why leftists often overread the actual content of the book. However examples can be found everywhere.

Upon going through the history of 'iterative and incremental development' one cannot but think that the waterfall model, has simply something to do with the rice-board planning mentality. Which of course delivers abysmal results when ever implemented. Let's have look deeper down from within the history of IID. At a time where a model, which suited the needs of management and accountability demands of power hierarchies, lead to very poor results which in this case the US tax-payer paid. The DoD experienced rather huge and significant failures of projects based on software acquisition in harmony with a strict, document-driven, single-pass waterfall model - which government regulation required. (DoD-Std-2167)


Highly important quote

By the late 1980s, the DoD was experiencing significant failure in acquiring software based on the strict, document-driven, single-pass waterfall model that DoD-Std-2167 required. A 1999 review of failure rates of earlier DoD projects drew grave conclusion:

Quote
Quote "Of a total $37 billion for the sample set, 75% of the projects failed or were never used, and only 2% were used without extensive modification.”


A rather large sum to find out that the "current best practice" wasn't working. Management loved the idea, though everywhere within the projects reality simply demanded an approach more suitable to how things are. Not ought to be. When thinking about that sum one also has to note that, developed countries pay often a very huge developing price for even higher forms of development to continue improving the human condition. Developing countries can if they are smart, simply adopt the latest practices and implement them. One can see such implementation in African countries of which some never development a grid of banks throughout the country, but where very early on able to do monetary transaction via cellphone. Subsequently developing a nation is ever easier for the very underdeveloped nations. Here however often aid, plays a negative role and leads to a gridlock of no need for further improvement destroying necessary incentives and pain points for people and entire nations to get the very best out of themselves. Though we are digressing here. The previous points have been stated in a weak manner, future articles will nail home their points in detail. Just one more thing before we dive into the history of IID, one could summarize it as a bottom up vs. top-down approach and what is true for software development is also true for general idea generation. In this case, for idea generation one would not need IID, but simply a Zettelkasten.


So, without further ado let's dive into "Iterative and Incremental Development: A Brief History" which was an academic paper in the year 2003 of which this article draws heavily upon as an information source. What would later become known as the Manifesto for Agile Software Development began back in the 1930s. Here as a quick reminder:



Note: The following segments borrows heavily from 2003 article called "Iterative and Incremental Development: A Brief History".

Introduction

Many people see iterative and incremental development as a modern practice, it's application dates as far back as the middle of the 1950s. Prominent software-engineering thought leaders from each succeeding decade supported IID practices, and many large projects used them successfully. As agile methods become more popular some view iterative, evolutionary and incremental software development - which is a corner stone of these very methods - as a replacement of the waterfall model, though roots of its very practice go back decades. Students are familiar with this fact, though some commercial and other government organization remain oblivious. Descriptions of projects and individual contributions provides rather self-affirming evidence of IID's. Many examples come from the 1970s and 1980s - the most active but least known time of IID's history. The idea came independently from countless unnamed projects with contributions from thousands and the list here is anything but representative. It's not intended to diminish importance of contribution from unnamed projects.

Some attempted significant up-front specification work followed by incremental time-boxed development, while others were more classically evolutionary and feedback driven.

What all approaches had in common as a theme though was to avoid a single-pass sequential, document-driven, gated-step approach. Regarding terminology, some prefer to reserve the phrase "iterative development" merely for rework, in modern agile methods the term implies not just revisiting work, but also evolutionary advancement - a usage that dates from at least 1968.

Pre-1970

Walter Shewhart contributed towards IID in the 1930s, as a quality expert at Bell Labs, he proposed a series of short "plan-do-study-act" (PDSA) cycles for quality improvement. In the early 1940s, quality guru W. Edwards Deming began to push the PDSA concept, which he later described in 1982 in "Out of the Crises". Tom Gilb and Richard Zultner also explored PDSA application to software development in later works. X-15 a hypersonic jet which marked a milestone in the 1950s in applying IID whereby the practice was considered a major contribution to the X-15 success, even if it wasn't a software project.

Quote
It remains noteworthy because some personnel and therefore also IID experience, seeded NASA's early 1960s Project Mercury, which did apply IID in Software.

In addition to this some Project Mercury personnel seeded the IBM FSD (Federal System Division) another early IID proponent. Project Mercury ran with very short (half-day) iterations that were time boxed. The development team conducted a technical review of all changes, and, interestingly, applied the Extreme Programming practice of test-first development, planning and writing tests before each micro-increment. They were practicing top-down development with stubs.

Quote
We were doing incremental development as early as 1957, in Los Angeles, under the direction of Bernie Dimsdale [at IBM’s Service Bureau Corporation]. He was a colleague of John von Neumann, so perhaps he learned it there, or assumed it as totally natural. I do remember Herb Jacobs (primarily, though we all participated) developing a large simulation for Motorola, where the technique used was, as far as I can tell, indistinguishable from XP. When much of the same team was reassembled in Washington, DC in 1958 to develop Project Mercury, we had our own machine and the new Share Operating System, whose symbolic modification and assembly allowed us to build the system incrementally, which we did, with great success. Project Mercury was the seed bed out of which grew the IBM Federal Systems Division. Thus, that division started with a history and tradition of incremental development.

All of us, as far as I can remember, thought waterfalling of a huge project was rather stupid, or at least ignorant of the realities… I think what the waterfall description did for us was make us realize that we were doing something else, something unnamed except for “software development.”

Within a 1968 report from Brian Randell and F.W. Zurcher at the IBM T.J. Watson Research Center, they describe and recommend an iterative development which got later pushed by M. M. Lehman and again promoted the method in is internal report to IBM management on development recommendations. The recommendations were as following:

  • Within software-system design the basic approach recognizes the futility of separating design, evaluation, and documentation processes.
  • The very design process is structured by an expeditionary model seeded by a formal definition of the system, which provides a first, executable, functional model.
  • It gets tested and further expanded through a sequence of models, that develop an increasing amount of functionality and an increasing amount of detail as to how that function is to be executed.
  • Ultimately the model becomes the system.

Robert Glass added in the 1960s another notion of incremental development being a worthwhile undergoing, as it leads to a more thorough system shakedown, avoids, implementer and management discouragement.

The 1970s

The well know 1970 article "Managing the Development of Large Software System" Winston Royce gave his two cents on what would become known as the waterfall model, expressed within the constraints of government contracting at the time. Mistaken often as the paragon of single-pass waterfall, he recommended somewhat a different course of action that was devolved into the today known waterfall model. His recommendation was to do it twice.

Quote
If the computer program in question is being developed for the first time, arrange matters so that the version finally delivered to the customer for operational deployment is actually the second version insofar as critical design/operations areas are concerned.

He suggested that a 30-month project might have a 10-month pilot model and justified this necessity to account for unknown factors and novel elements. Thus we see hints of iterative development, feedback and adaptation in Winston Royce's article. His son a contributor to popular IID methods in the 1990s, said about his father and the paper:

Quote
He was always a proponent of iterative, incremental, evolutionary development.

The son elaborates further that only for the most straightforward projects the simple waterfall description would work. The rest deals with iterative practices with the serious restraint's 1960s and 1970s government-contracting models propose e.g. serious sets of constraints. This is a rather ironic insight, based around the influence this paper had as part of the bulwark promoting a strict sequential life cycle for large, complex projects. The next early reference is delivered by Harlan Mills, a 1970s software-engineering thought leader who worked at IBM FSD. His well know "Top-Down Programming in Large Systems" promotes iterative development. Less appreciated though was his life-cycle advice for building systems via iterated expansions:

Quote
… it is possible to generate a sequence of intermediate systems of code and functional subspecifications so that at every step, each [intermediate] system can be verified to be correct…

He suggested iterative refinement for the development phase, but did not mention avoiding a large up-front specification step nor did he specify iteration length. Without giving an emphasis on feedback and adaptation-driven development from each iteration. Though a decade later he raised those pints. Perhaps IID projects influenced him but this could not be verified. The IBM FSD division's story is fascinating for the extent of their very success with IID use on large, life-critical US Department of Defense space and avionics systems during this time. Take for instance 1 million lines of code for the command and control system for the first US Trident submarine. Though iteration cycles were longer than recommended today. IBM Management approved of IID methods. The IID approach was also a way to manage the complexity and risks of large-scale development, as Don O'Neil noted. In 1972 IBM FSD competitor TRW applied IID in a major project with 100 million $ for an Army Site Defense software for ballistic missile defense. This was developed in five iterations. Another mid 1970s extremely large application of IID at IBM FSD was the development of the Light Airborne Multipurpose System part of the US Navy's helicopter-to-ship weapon system. After a four-year 200-person-year effort involving millions of lines of code, the system was incrementally delivered in 45 time-boxed iterations (one month per iteration). A venture which is actually the earliest example found of a project that used an iteration length in the rage of one to six weeks, the length that current popular IID methods recommend. The project was successful. Harlan Mills wrote:

Quote
"Every one of those deliveries was on time and under budget."

In 1975 Vic Basili and Joe Turner published a paper about iterative enhancement that simply described classic IID. The basic idea behind iterative enhancement is to develop a software system incrementally, allowing the developer to take advantage of what was being learned during the development of earlier, incremental, deliverable versions of the system. Learning comes from both the development and use of the system, where possible. Key steps in the process were to start with a simple implementation of a subset of the software requirements and iteratively enhance the evolving sequence of versions until the full system is implemented. At each iteration, design modifications are made along with adding new functional capabilities.

Tom Gilb introduced the terms “evolution” and “evolutionary” to the process lexicon. The paper detailed successful IID application for the development of extendable compilers for a related variation of extendable compilers of application-specific programming languages for a variety of hardware architectures. It took 17 iterations over 20 months to form the base system for the development team. Each iteration gathered feedback from the user's and developer's point of view and used it to modify both the language requirements and design changes in future iterations. They also tracked measures such as coupling and cohesion, over multiple iterations. When Tom Gilb published Software Metrics he also coined the very term in which he discusses IID practice. That being evolutionary project management and introduction of the terms "evolution" and "evolutionary" to the process lexicon. This book being the earliest publictation that had a clear IID discussion and promotion especially of evolutionary delivery. Evolution producing the appearance of stability, as a complex system being most successful if it is implemented in small steps. If of course each step has a clear measure of successful achievement as well as "retreat" possibility to a prior successful step upon failure. This creates the opportunity to receive feedback from the real world before throwing in all resources intended for a system and leaves enough room for correction of possible design errors. The book was simply marking the arrival of a long-standing and within it passionate voice for evolutionary and iterative development. Tom Gib is one of the earliest and most active IID practitioners and promoters. Starting with it in the early 1960s and establishing throughout the years several IID milestones. His material was probably the very first with a clear and pronounced flavor of agile, light and adaptive iteration with quick results, similar to that of newer IID methods. By the year 1976 Harlan Mills had strengthened his message about IID:

Quote
  • Software development should be done incrementally, in stages with continuous user participation and replanning and with design-to-cost programming within each stage.
  • ...there are dangers, too, particularly in the conduct of these [waterfall] stages in sequence, and not in iteration—i.e., that development is done in an open loop, rather than a closed loop with user feedback between iterations. The danger in the sequence [waterfall approach] is that the project moves from being grand to being grandiose, and exceeds our human intellectual capabilities for management and control.

There is simply a pitfall via the waterfall model move within a closed loop moving from grand to grandiose, exceeding our human intellectual capabilities for management and control. Therefor software development should be done incrementally, in stages with continuous replanning and user participation. A design-to-cost programming approach is well advised as well.

Harlan Mills simply asked why do enterprises tolerate the frustrations and difficulties of such programs seen with waterfall development. Probably after reflecting several years of IID application at FSD.

Quote
...why do enterprises tolerate the frustrations and difficulties of such [waterfall] development?

1977 was the year, IBM FSD, incorporated the Trident IID approach, which lead to integrating all software components at the end of each iteration into its software-engineering practices, McHenry called this approach: "integration engineering"


This pushed the concept towards more then 2,500 FSD software engineers and sealed the ideas of IID as an alternative to the waterfall model and stimulated substantial interest within IBM's commercial divisions and senior customer ranks as well as among the company's competitors. Though not known to the majority of software professionals, another success IID delivered at the heart of the NASA space shuttle software, which primary avionics software system which IBM FSD built from 1977 to 1980. Ironically in hindsight there is almost an apologetic undertone for having to forego the "ideal" waterfall model for the IID approach.

Quote
Due to the size, complexity, and evolutionary [changing requirements] nature of the program, it was recognized early that the ideal software development life cycle [the waterfall model] could not be strictly applied...However, an implementation approach (based on small incremental releases) was devised for STS-1 which met the objectives by applying the ideal cycle to small elements of the overall software package on an iterative basis.

The shuttle project contained traditional IID practices that being for instance the following two next to the usual aspects.

  • time-boxed iterations in the eight-week-range
  • feedback-driven refinement of specifications

The first IID discussion in the popular press was in 1978 when Tom Gilb published a column in the UK's Computer Weekly appearing at the 6th of April.

Quote
Management does not require firm estimates of completion, time, and money for the entire project. Each [small iterative] step must meet one of the following criteria (priority order): either (a) give planned return on investment payback, or, if impossible, then (b) give breakeven (no loss); or, at least, (c) some positive user benefit measurably; or, at least (d) some user environment feedback and learning.

Stating that each management does not require firm estimates of completion, time, and money for the entire projects. There are however small iterative steps which must follow certain criteria which are now listed in priority order.

  • Give a planned return on investment payback
  • Give a breakeven (no loss)
  • Positive user benefit measurably
  • User environment feedback and learning

The next published discussion of IID practices was in 1984 and they are referring to a System Development Corp. which build an air defense system which began in 1977 and was completed in 1980. The project was a combination of significant up-front specifications with incremental development and builds. The project had to fit into the DoD single-pass waterfall standards, with testing and integration within the very last phase. Carolyn Wong commented on the unrealistic approach and the team's necessity for an incremental development.


Quote
The [waterfall] model was adopted because software development was guided by DoD standards… In reality, software development is a complex, continuous, iterative, and repetitive process. The [waterfall model] does not reflect this complexity.

The 1980s

In 1980 Weinberg wrote regarding IID in Adaptive Programming: The New Religion published in Computerworld. Summarizing his own article he stated:

Quote
"The fundamental idea was to build in small increments, with feedback cycles involving the customer for each."

Just a year later Tom Gilb wrote in more detail about evolutionary development. The same year Daniel McCracken and Michael Jackson argued against the "stultifying waterfall" in a chapter within the software engineering and design text edited by William Cotterman. The subordination of positioning IID methods at the time in comparison to the waterfall model were displayed by the title "A Minority Dissenting Position". Their arguments continued within "Life-Cycle Concept Considered Harmful" a 1982 twist on Edsger Dijkstra's late 1960s classic go-to statement considered harmful. The phrasing of "life cycle" as a synonym for the waterfall method during that period, suggest the unquestioned dominance of the concept. One is well advised to contrast this to the 1990s use of sequential life cycle or iterative life cycle. IID practice of evolutionary prototyping was the go-to method used throughout the 1980s efforts to create artificial intelligence systems. The IID practice of evolutionary prototyping was commonly used in 1980s efforts to create artificial intelligence systems.

William Swartout and Robert Balzer pushed in 1982 an iterative and evolutionary approach. That same year a 100 million $ military command and control project, provided the earliest reference to a very large application successfully built with evolutionary prototyping, an IID approach that does not usually include time-boxes iterations. This project was based on IBM's Customer Information Control System technology. In 1983 Grady Booch published "Software Engineering with Ada" in which he described an iterative process for growing an object-oriented system. The primary influence of the book was within the DoD development community. However, it's influence was more about being object-orientated than for its iterative advice. Grady Booch later 1990s books however which covered IID, found a large general audience and many adopters got a hold of IID methods over this influence.

The early 1980s were a rather active period for the attempted creation of artificial intelligence systems, expert systems and so on. This is especially true for Lisp machines. The most common approach in this community was evolutionary prototyping, an IID method. Tom Gilb wrote" Evolutionary Delivery versus the Waterfall Model" in the mid-1980s promoting more aggressive IID methods such as every few weeks delivery of tangible results for stakeholders. A landmark IID publication was Barry Boehm's "A Spiral Model of Software Development and Enhancement" which citation year is either 1985 or 1986 though it was written in the former year. Pushing forward development cycles by risk, IBM FSD had previously applied or even advocated variations of this very idea for example. The spiral model however did not formalize and make prominent the risk-driven-iterations concept and the necessity to use discrete steps of risk assessment in each iteration. 1986 was marked by Frederick Brooks, a thought leader in software-engineering of the 1970s and 1980s, when he published the classic "No Silver Bullet extolling the advantages of IID".

Quote
Nothing in the past decade has so radically changed my own practice, or its effectiveness [as incremental development].

Commenting on adopting a waterfall process, Brooks wrote:

Quote
Much of present-day software acquisition procedure specify a satisfactory system in advance, get bids for its construction, have it built, and install it. I think this assumption is fundamentally wrong, and that many software acquisition problems spring from that fallacy.
The Cleanroom method incorporated evolutionary development with more formal methods of specification and proof.

Frederick Brooks ended up summing up a decade of IID endorsements and promoting messages to military standards bodies and other organizations. Making his point very clear at the 1995 International Conference on Software Engineering:

Quote
"The waterfall model is wrong."

In 1986 David Parnas together with Paul Clements put forward "A Rational Desing Process: How and Why to Fake it". They stated they belief in the ideal of the waterfall model. Thorough, correct, and clear specifications before development. Listing the following statements

Quote
  • A system’s users seldom know exactly what they want and cannot articulate all they know.
  • Even if we could state all requirements, there are many details that we can only discover once we are well into implementation.
  • Even if we knew all these details, as humans, we can master only so much complexity.
  • Even if we could master all this complexity, external forces lead to changes in requirements, some of which may invalidate earlier decisions.


In 1987 TRW launched a four-year-long project to create the Command Center Processing and Display System Replacement or short CCPDS-R, a command and control system which relied upon IID methods. Walker Royce described the effort over 60 pages in detail. Team time-boxed six iterations, averaging around six months each. The approach was consistent with what would later become the Rational Unified Process to which Walker Royce contributed: Attention to high risks and the core architecture in the early iterations.

Bill Curtis and colleagues published On Building Software Process Models under the Lamppost which was an especially agile-relevant paper during the decade of the 1980s. His paper influenced 19 large projects. The authors identified that the prescriptive waterfall model was there to satisfy management accountability goals, but were far removed from how a project was successfully pulled off. The paper also noted that successful development emphasizes a cyclic learning process with an emphasis and broaden attention to people's skills, common vision, and communication problems, rather than viewing the projects common effort as sequential manufacturing process The author's state:

Quote
The conclusion that stands out most clearly from our field study observations is that the process of developing large software systems must be treated, at least in part, as a learning and communication process.

In 1987 within the IBM FSD Software Engineering Practices program, Harlan Mills and Dyer and Rick Linger continued the evolution of IID with the Cleanroom method, which combined more formal methods of specification and proof with an evolutionary approach towards development. This reflected Harlan Mills strong mathematical influences.


Highly important quote

By the late 1980s, the DoD was experiencing significant failure in acquiring software based on the strict, document-driven, single-pass waterfall model that DoD-Std-2167 required. A 1999 review of failure rates of earlier DoD projects drew grave conclusion:

Quote
"Of a total $37 billion for the sample set, 75% of the projects failed or were never used, and only 2% were used without extensive modification.”

Consequently, the waterfall model no longer was the standard, now IID was considered as a workable standard as well. As noted in a 1987 report from the Defense Science Board Task Force on Military Software. DoD-Std-2167 likewise needs a radical overhaul to reflect modern best practice. Draft 2167A is a step, but it does not go nearly far enough. As drafted, it continues to reinforce exactly the document-driven, specify-then-build approach that lies at the heart of so many DoD software problems…. In the decade since the waterfall model was developed, our discipline has come to recognize that [development] requires iteration between the designers and users. In a section titled "Professional Humility and Evolutionary Development" that the 2167's goals of getting the specifications accurate without incremental implementation and feedback was not possible, the report stated:

Quote
Experience with confidently specifying and painfully building mammoths has shown it to be simplest, safest, and even fastest to develop a complex software system by building a minimal version, putting it into actual use, and then adding functions [and other qualities] according to the priorities that emerge from actual use.
Evolutionary development is best technically, and it saves time and money.

DoD-Std-2167A which was the epitome of the waterfall specification. Yet, the authors actually wanted it to be an amendment, for life-cycle neutrality that allowed IID alternatives to the waterfall method:

Quote
This standard is not intended to specify or discourage the use of any particular software development method. The contractor is responsible for selecting software development methods (for example, rapid prototyping) that best support the achievement of contract requirements.

Despite the intent of the author, many justifiably interpreted the new standard as containing an implied preference for the waterfall model. Ironically in a conversation nearly a decade later the principal creator of DoD-Std-2167A expressed regret for setting the strict waterfall method-based standard. He simply had not had heard of the IDD approach and people he questioned and literature praised the waterfall model. With hindsight, he said we would have made a strong recommendation for IID rather than the waterfall model. In 1988 Tom Gilb published "Principles of Software Engineering Management" the first book with substantial chapters dedicated upon the discussion and promotion of IID. Within it, he reiterated and expanded on the IID material from Software Metrics. He went on to describe the Evo method, distinguished by frequent evolutionary delivery and with an emphasis on defining quantified measurable goals and then measuring the real-world actual results from each time-boxed short iteration.

Conclusion

Quote
“For every complex problem, there is a solution that is simple, neat, and wrong.”
H.L. Mencken

In the history of science it is the very norm that simplistic but inferior ideas first hold the dominant position, even without supporting evidence or results. Medicines four humors and related astrological diagnosis and prescription dominated Europe for more than a millennium. Software Development is a young field, thus it's no big suprise that simplified single-pass and document-driven waterfall model or "requirements, design, implementation" hold the sway during the first attempts creating proper work processes.

Quote
The brief history shows that IID concepts have been and are recommended best practice by prominent software-engineering thought leaders of each decade, associated with many successful large projects and recommended to standard boards.

Yet still the waterfall model is present in some commercial organizations, consulting companies, and standards bodies. They still promote a document-driven single-pass sequential life cycle as the ideal.

Comments 1

  • If your article aims to provide insights into the evolution of development processes, especially within the context of IID, Scrum, and Agile, it sounds like an interesting exploration. The principles of iterative and incremental development have indeed found application beyond software development, influencing various aspects of project management and problem-solving in different fields.