Waterfall Requirements in Agile Product Development

Datetime:2016-08-23 03:31:53          Topic: Agile Development           Share

Key takeaways

  • By nature, BRDs imply conclusiveness and are "contractual" in nature, and therefore, should not be used in iterative development
  • BRDs are frequently produced by "middle men" and weakly represent views and desires of End-Customer
  • Creating BRDs, done by a separate organizational layer, frequently leads to Local Optimization
  • In Agile Development, BRDs may lead to deterioration of relationships between Product Owner and Feature Team
  • In Agile Development, the use of BRDs leads to having false expectations and reduces involvement of End-Customer

Everyone is familiar with conventional Business Requirements Documents (BRDs) .

Usually, BRDs are written by Business Analysts – individuals representing a function specialty organizational tier – a layer that usually resides between end-customers and software engineers. BRDs explicitly talk about “what” a customer wants from a system, including functional and non-functional requirements. BRDs are comprehensive documents: heavy in content and with many supporting details. BRDs are based on far-reaching future plans that spans across multiple product development phases and releases. By design, a single BRD may represent an implementation effort that takes anywhere from a few to many months.

Historically, BRDs have been used in sequential/phase-based product development, where thorough upfront analysis and documentation would be followed by comprehensive system design and architecture, then followed by componentized (not feature-centric) coding/programming, further followed by QA, and finally - by UAT. In this process, commonly known as Waterfall, BRDs, once created, are passed down a chain from one organizational layer to another, for sign offs and approvals.

As such, BRDs represent a formal contract between different organizational units, within the same organization structure, primarily, between Customers and Technologists. Such contracts (or contractual agreements) typically contain hard assumptions about: ‘WHAT’ must be delivered, ‘WHEN’ it must be delivered and ‘HOW MUCH’ a delivery will cost.

Note: Please note, this post is not about what the role of Business Analyst becomes in Agile Software development or what type of contribution BA can make to Scrum team work.  A separate dialogue would be required to elaborate on this topic.

General shortcoming with BRDs

BRDs imply conclusiveness

Creating a comprehensive spec for software development implies that all best solutions are known upfront. This also assumes that throughout a development cycle customers should not change their minds about what they want and should not come up with better, brighter ideas. Any changes usually trigger, what is known as ‘change request process’ that is a standard protocol to deal with scope creep. During this process, Business Analysts or other individuals in similar capacity, produce voluminous documentation, usually, in a vacuum, finalizing requirements without direct input from technology.

In reality, and rather frequently, the best ideas and solutions come much later in the process when development phase is well underway. It is also not uncommon that customers change their minds about initially stated requirements after development begins. In cases like these, to justify BRD scope creep, a tedious and overly bureaucratic process, of change control is implemented – something that requires additional time and effort. By design, BRDs are meant to resist changes; anything that requires an update after BRD is finalized and signed off, carries a negative connotation. Lastly, having BRDs produced without initial participation of technology creates a lot of ‘wishful thinking’ and unrealistic expectations from customers that sometimes look for complex and expensive solutions.

BRDs are produced by “middle-men”

Typically, BRDs are produced by Business Analysts (BAs) – a group of people that represents a distinct functional layer, controlled by its own line management. Since BAs are not true customers, they may not have the best understanding of real business needs. Neither, BAs, are software developers, and therefore they may not know all nuances and specifics of technical options, dependencies or constraints. Effectively, BAs are “middle men” or translators that reside between business and technology, sometimes being imbedded on either side of the organization, yet still representing a separate functional layer. And while frequently BAs are imbedded on either side of the organization (customers or IT), they still represent a separate functional layer. Such additional translation layer frequently leads to miscommunication, misunderstanding, lengthening of communication feedback loops and increasing work cycle time.

Contractual behaviors caused by BRDs

By design, BRDs represent a contract between Business and Technology. This unofficial internal contract between different parts of the same organization splits it [organization] into “us” and “them”. Each organizational layer is focused on its own commitments, frequently losing a view of a larger picture and collective/shared strategic goals. Frequently driven by monetary incentives and other discretionary perks (e.g. next year hiring budget, performance driven bonus), every organizational layer attempts to do their best in delivering “their own piece” and then passing it on to another organizational layer that then becomes responsible for signing off on a receivable and producing their own deliverable that must, in turn, become a continuity of upstream receivable.

Producing BRDs leads to Local Optimization

Because each organizational layer is only focused on its own deliverable, it [layer] optimizes their internal process and dynamics only for its own, local success. This is called “local optimization”. In the case of BRD writing, it is frequently seen that a heavy document is produced way ahead of development begins. This is how an organizational layer that is responsible for producing documentation (e.g. Business Analysts), claims completion and their “local” success: “BRD is finalized and signed off”. In order for this to happen, an organizational layer that is solely in pursuit of completing its own, local goals, expands disproportionally, by hiring more and more people, in order to meet strict deadlines for BRD (in this case) creation. After documentation is signed off and consumed by another downstream organizational layer (e.g. architects, designers, developers, testers), a layer responsible for heavy documentation (e.g. BAs), becomes underutilized (idle) again. While being significantly expanded due to prior excessive hiring, and in order to look busy and justify its existence, this layer begins focusing on work (additional documentation, analysis) that has low business priority: a layer “locally optimizes” itself to justify its size and activities. If it does not, it would have to decrease itself, by off boarding its own people. This phenomena can be qualified as organizational waste (Lean term -“muda”).

Are there exceptions to general rules?

But just to be fair, there might be some exceptions, where more comprehensive documentation, done upfront, would be justifiable. Such exceptions are, rather, rare but for the sake of argument, here are some examples:

  • Organization has signed an external agreement with a third party (vendor), where statement of work (SOW) is clearly defined and legal contracts define a relationship. In cases like these, where work will be competed in sequential (waterfall) fashion, having a more detailed documentation might be necessary to avoid liability on both sides. Here, producing a light-weight requirement document could be acceptable. However, it is still not advisable to lock everything upfront: scope, budget and timeline. Most likely, scope and budget will become fixed (as oppose to agile development, where scope usually remains flexible) but if so, then timeline should remain flexible.
  • Organization/group/department takes on a limited-scope initiative that is expected to be complete within very short time frames. In cases like these, risks of going off track and encountering classic challenges of waterfall lifecycle are significantly lower.
  • Organization wants to decommission a legacy system and substitute it with another system (newer technology), and while preserving/replicating most of existing functionalities, there is no new development expected (prioritization is not required and time frames are not rigid). Unrealistic, but possible. Here documentation is used to capture findings made during reverse engineering of a legacy system.

Observed anomalies with the use of BRDs in Agile Software Development

Cases are known, when teams and their Product Owners have attempted to retain BRDs for agile development (Scrum or Kanban). Two main cases are worth describing, where this mistake becomes costly:

A team attempts sprinting and iteratively developing based on its “understanding of an entire BRD”

Given the fact that BRD is an all-inclusive, monolithic document that usually covers work of many weeks or months, decomposing it into smaller, independent requirements becomes very challenging. As a result, since requirements are not independent, a team cannot safely estimate and commit to a small, independent scope of work to be done in a short time frame(sprint). Needless to say, that Product Owner is reluctant to prioritize parts of BRD that cannot be decoupled from one another. Subsequently, a team ends up committing and executing on randomly selected parts of BRD that describe functionalities with upstream and downstream dependencies but cannot be classified as minimal marketable features (MMF) or potentially shippable product increments (PSPI) that a team can deliver independently, as vertical, cross-componentized slices. As a result, Product Owner and customers do not get to see a working product at the end of each sprint. This further creates challenges for a team with delivering independent, vertically sliced, cross-componentized features (a.k.a. minimal marketable features, or MMF) at the end of each sprint: Product Owner and customers do not get to see a working product at the end of each sprint. Finally, negotiations and trade-offs between a team and Product Owner halt, when delivering against BRD, as BRD represents all-or-none contractual agreement: a team ends up ‘faking’ incremental development. What is frequently seen is that teams spend a number of initial sprints, working on individual system components (e.g. database-only, or mid-tier only, or UI-only), with later sprints dedicated to massive integration and bug fixing. This translates into a team’s inability to deliver anything complete (“Done-Done”) to Product Owner and customers in any given sprint. Lack of cross-component, feature-centric work inside a sprint also discourages knowledge sharing and collaboration among team members

Product Owner shields a team from BRDs, while continuing to use BRDs with end customers

Here, Product Owner, along with a team, attempts to embrace principles of incremental development. As such, a backlog is maintained to support communication between PO and a team. This includes continuous backlog grooming, work prioritization and sprint commitments. End-of-sprint showcases attempt to deliver small, independent, cross-component-cutting, potentially releasable chunks of functionalities… This is all great…however…

However on the other side of communication chain: between Product Owner and customers – communication still takes place in the form of BRDs. What does this really mean? It means that Product Owner continues to live in the waterfall world with his customers, by accepting requirements from them in the form of heavy static, conclusive documentation and giving them [customers] hard, all-or-none, non-negotiable commitments ….At the same time, Product Owner still attempts to prioritize and negotiate work with delivery teams. Product Owner spends a lot of time and effort, trying to decompose heavy BRDs into small independent chunks of work (user stories) to feed a backlog. This is usually done in a silo, without a team’s involvement (to ‘spare’ their time), which creates many wrong assumptions and false expectations. It also makes Product Owner to work a double-shift.

While on the surface this willingness to shield a team and “absorb heat” may seem to be as a noble and courageous act by PO, at a deeper level this approach is wasteful and even dangerous. Effectively, Product Owner makes commitments and gives promises on behalf of delivery teams. PO creates an illusion for technology that customers appreciate transparency, are ready to accept work incrementally and, if necessary, willing re-negotiate scope and/or “trim a tail” of less critical work. PO also creates an illusion for customers that technology will deliver an entire requirement (as per BRD) by a certain date, and on budget. There is a set of false expectations on both sides because of miscommunication created by Product Owner. This problem may not be too obvious at the beginning of development cycle but does become very apparent as a release date approaches. This dysfunction is usually accompanied by inaccurate reporting: agile metrics, such as Velocity and Release Forecasting that are used in communication between a team and Product Owner get completely out of sync with RAGs that are used in communication between Product Owner and customers. Further, this translates in diminishing credibility of technology in the eyes of customers and deteriorated relationship between technology and Product Owner as well as Product Owner and customers. At the end of the day, nobody wins.

A much more effective and safe method to bring customers and technology closer together, while keeping Product Owner engaged in his critical role, is shared education. It is unreasonable to expect that by educating Product Owner(s) and team(s) only on agile principles, norms and frameworks, an organization can solve the problem of miscommunication and distrust (not uncommon!) between business and technology. The rest of an organization, beyond Product Owner cannot remain on sidelines; they also must be educated on core values and principles of agile product development.

While the key role of Product Owner still remains to be a voice of Customer, Product Owner should not put himself in an uncomfortable position of making competing commitments and giving unrealistic promises that will cause harm to other parties involved.

About the Author

Gene Gendel is Agile Coach and Organizational Change Agent. His focus is on helping organizations and teams with improving system design and organizational structure and overall efficiency. Gene engages at all organizational levels: senior leadership, mid-level management, teams and individuals.  To be effective as a coach, he use various coaching tools and techniques, based on circumstances, while also leveraging elements of training and mentoring, to supplement my coaching.

About List