Leveraging empirical process control theory

Scrum's foundations lie on empiricism, as opposed to following a pre-planned process. Empiricism in the philosophy of science, which emphasizes on knowledge acquisition through hypothesis, theories, experimentation, and the validation of results. For example, the scientific method is based on empiricism.

A hypothesis is an initial – though hopefully testable – assumption, when supporting data or information is lacking, about how or why something works the way it does. In contrast, a theory is an attempt to explain the observed phenomenon based on the data and the facts that are known. Hypothesis and theories both serve as a starting point to assess our understanding of the world around us.

Next, experiments are designed that test the hypothesis or validate whether the theories work. When the evidence supports the hypothesis or theories, the scientists (developers, in the case of Scrum) gain confidence in their understanding of the phenomenon under investigation. If the evidence does not support their theories, then the experimenters at least know what will not work, and they may gain insights as to why not, as well as some alternative approaches they might explore.

The main point behind Scrum's use of empiricism is that customers and end-users, more often than not, do not know what capabilities a new product or product enhancement needs to provide to support their needs. Likewise, the developers cannot know the best approach to developing the end product until the requirements are better exposed. Even after they understand the requirements, they may need to design and conduct experiments to discover the best way to implement the desired functionality.

Scrum Teams employ trial and error techniques to expose requirements by building and showing prototypes and incremental releases of new functionality to customers and end-users. Each new increment of functionality leads to new insights into what the product can and should be.

The Bible of Scrum is The Scrum Guide™, co-written and maintained to this day by Ken Schwaber and Jeff Sutherland. In the Scrum Guide, Schwaber and Sutherland introduce their three pillars of empirical process control: transparency, inspection, and adaptation. These three pillars are the foundation that supports Scrum development teams in their work.

The word transparency simply means that everything is out in the open. All product and project-related information is available at all times, including stakeholders, the CEO, customers, and development team members. Such information includes the following:

Inspection is an essential aspect of Scrum's empirical process by providing the data and information the development team needs to review and evaluate the current situation. The inspection includes any activity that serves to assess the quality of Scrum Artifacts, progress against the Sprint Goal, progress against the Sprint Backlog, impediments, team performance, and to detect any variance in the Sprint Plans. Scrum Artifacts include the Product Backlog, Sprint Backlog, and Increments, as defined in the next section of this chapter.

Adaption is the process of evolution in the empirical model. In other words, as the team continues to gather new data, they can assess both their goals and their progress to see if changes need to be made. If the customer's needs and priorities have changed, or the team is in danger of missing a Sprint Goal, adaption allows the team to make near-real-time course corrections to address these concerns.

Empiricism and the scientific process represent the technical aspects of how Scrum works, and why. But there is a softer side of Scrum that is equally important, and that is the role of the core values that make Scrum work at a human level. Core values are the subject of the next subsection.

Establishing Scrum's core values

Scrum is not limited to guiding the intellectual and physical aspects of designing and building products. However, there is a critically important softer side of Scrum that guides the development teams and other stakeholders through how they need to work together, expressed as a set of core values. The core values of Scrum include commitment, courage, focus, openness, and respect.

Commitment speaks to the team's desires and ability to evaluate the product backlog in terms of the goals established for each Sprint, scope the work, and make commitments to deliver. Making those commitments take courage. In the spirit of transparency, it also takes courage to speak out and let the truth be known with regards to all aspects of a team's work and the state of the product. The team must stay focused on delivering only the highest value items from the product backlog, and not allow themselves to get off track working on things that do not contribute to meeting the goals established for each Sprint.

Openness is another word for transparency, but not just limited to the team's performance and delivery of product increments. The concept of openness also applies to how team members work together. They must be open in their communications with other team members to work through sticky issues, without making things personal. Without respect, that degree of openness and honesty is not possible.

In Scrum, it's always about leveraging the collective skills and diversity of knowledge and experiences to make the team better as a group. Failure to respect others for their contributions is a surefire way to destroy the integrity of the team.

In summary, people are critical enablers in the implementation of Scrum. They must not only understand the core values; they must live them. Now that you've learned some of the core concepts that make Scrum what it is, we need to turn our attention to understanding some of the critical issues you will face when implementing Scrum. Let's start by discussing the pitfalls associated with partially implementing Scrum.

Partial Scrum is not Scrum

One notable comment to make about Scrum is that the original authors, Sutherland and Schwaber, are both adamant that all Scrum roles, Events, Artifacts, and rules are absolute. Though it is possible to find value in implementing only parts of Scrum, the final result is not Scrum.

I've included this issue based on real-life experience. One of my customers implemented concepts that were very similar to Scrum, including some of the terminology. However, without proper guidance, the development teams have to fill in the holes or gaps in the methodology. Not only does this cause stress throughout the organization, but it ultimately leads to project failures.

When organizations implement custom versions of Scrum, there is no definitive guide, outside the Scrum Guide, to go back to fix any problems that result. Also, human nature causes the authors of the revised Scrum methodologies to dig in and protect their approach. When things go wrong, by circumventing the Scrum processes, there is a high probability the executives and Scrum Teams won't know or won't understand the value of going back to the basics of Scrum.

Let me give you another real-life example of the impact of employing Scrum in unique or custom ways. In my situation, a program manager was used to seeing detailed requirements specifications for the products under his management. When the organization implemented an enterprise-wide but modified version of Scrum, there were no complete instructions provided on how to gather and use requirements under an iterative and incremental development process. As a result, the manager forced the contractor's development team to define all the User Stories at the beginning of the project. In effect, the program manager went back to old habits of the traditional model, such as defining all the requirements in advance of development. As you might imagine, this led to severe project schedule slippage as the customer's requirements and priorities kept changing.

Another critical issue faced by organizations that attempt to implement Scrum, without fully thinking about the consequences, is the impact that change will have on existing project teams and subcontractors working under approved project charters and legal contract agreements. We'll explore this issue in the next section.

Revising your contracts

When an organization makes a move to implement Scrum, the executives must communicate the new direction, and also appropriately modify all project charters and legal agreements that govern how employees and contractors work and deliver products. If an organization attempts to implement Scrum but fails to modify the contracts and charters, the teams have competing directives, and they will fail. The disparate directives are simply too broad to reconcile at the team level.

Project charters are directives from executive sponsors that guide the objectives of the project but also constrain the scope of work authorized for the project – usually expressed in the form of deliverable items, along with the budgets, schedules, resources, and quality constraints. A contract agreement provides similar information. However, this is a legal document with serious non-compliance consequences that cannot change without formal and written approval from the authorizing agent.

In a situation I was involved with, during a mandated mid-project implementation of Scrum-like practices, the customer's contract office never modified the contract agreements during the contract period. As a result, we had to adhere to the standard requirements for project schedules, reports, and documentation that were initially developed under the client's previous traditional software development practices. For reasons never passed down to me, the client's assigned internal program executives never changed the contracts.

Therefore, whether we wanted to or not, from a contractual perspective, we still had to develop a detailed project management plan and schedule plan. The client's program manager still insisted on providing a full definition of business and user requirements, as User Stories for the entire backlog, before starting any development work. The development team also had to provide all the documentation and ongoing reports required under the traditional SDLC model. In the end, the contracts prevented us from following the so-called mandated agile practices.

Part of the problem in this situation was that the organization attempted to deploy Scrum at an enterprise-scale without proper preparation, promotion, and training. Given the large size of the client's organization, a staged rollout, starting with some highly visible and useful pilots, would have helped the organization adjust and prepare for the changes that were coming their way.

I'll discuss the staged enterprise deployment concept in more detail in Chapter 7, Scrum of Scrum, which introduces the original Scrum scaling concepts. In the meantime, we need to move on and discuss the importance of providing complete visibility across all Scrum Team activities. That is the subject of the next subsection.

Making Scrum visible and transparent

Transparency means all Scrum products and project information is available to anyone who wants to see it. Teams that work in a single co-located working environment have an advantage as they can display all the product and Sprint information on whiteboards and flip charts within the room. However, when product development activities involve multiple teams, especially when those teams operate from geographically disparate locations, the organization may need to implement an electronic mechanism to distribute or otherwise make the information readily available.

Information Radiators is a common term used to describe displays of agile-based product and project information in a highly visible and available manner. Alistair Cockburn is credited for the initial description of Information Radiators in his book titled Agile Software Development (2001, 2007). Schwaber and Sutherland, in their book, titled Software in Thirty Days (2012), discuss the concept of Information Radiators as applied to Scrum.

The main point is that Product Owners and Scrum Teams do not hide what they are doing. Everything must be transparent and out in the open for inspection by anyone who believes they should have access to that information.

Information Radiators can provide access to virtually any type of product or development-related information, such as the following:

Project budgets still exist in Scrum, at least in the Scrum-based projects I've supported. By now, it should be pretty clear to you that it's impossible to define exact budgets for a large section of product development work, given the stochastic nature of the effort. In the next section, I'll present an alternative approach to budgeting that better fits the Scrum model for development.

Treating Scrum development as a fixed cost

This subject is not a part of Scrum, as outlined in The Scrum Guide, but helps further explain the economics of Scrum. Under the principles of project management, there is an expectation that funding allocations support the completion of a specific scope of work within an approved timeframe, with approved resources, and to an agreed level of quality. However, a planned budget is only reasonable if the development team can indeed specify all the work that is necessary to deliver a fully described set of product features. A project-based budget also assumes all requirements are fully defined and won't change throughout the project's duration.

Likewise, there can be no chance that the market conditions or desired features and functions will change during the duration of the budgeted effort. Finally, there can be no unexpected issues that come up to delay the project or cause more work.

Unfortunately, that's not real life. The stochastic nature of software and complex system development makes all these assumptions invalid. There are simply too many variables and unknowns to make such predictions.

Scrum turns the entire development paradigm on its head by planning and conducting work across very short intervals of time. The expectation is anything, and everything can and will change over time. So, we can only work off the information we have at the moment.

Through the three pillars of empirical process control – that is, transparency, inspection, and adaption – the team evaluates their progress against known requirements in the product backlog and identified impediments, and then adjusts their efforts to maximize their outcomes at the current stage of development. As long as the product's value justifies their efforts, the team can continue working against items contained within the product backlog.

For these reasons, I recommend that organizations look at Scrum-based development as a relatively fixed cost, at least in the short run. Periodically, the organization must review the level of effort being funded and adjust their budgets if something has changed in the value or delivery model. This strategy simplifies the organization's approach to justifying software and system development projects.

For example, if the value of a product justifies investments in seven development teams with an average of seven people each, and the work exists to support that level of investment, then the budget is fixed accordingly. Periodically, perhaps every quarter, and with completion of the current high-value items in the backlog, the executives and Product Owners can reassess the level of investment necessary to support continued development on emergent, high priority requirements.

Over time, a Product Owner may need a higher number or a smaller number of product teams supporting their products. A successful company will always be on the lookout for new product opportunities to grow the business, thus providing new opportunities for the developers.

Thrown objects don't stick

I will talk about enterprise-wide implementations of Scrum in the last two sections of the next chapter, but another issue I want to point out here is the tendency to throw a new IT methodology over the wall and hope it will all work out. In this case, it's the idea that Scrum will somehow stick if you just put it out there or mandate it through an edict without appropriate planning and follow-through. This strategy rarely works out.

This issue was the situation I described in the Revising your contracts subsection. Our client forced an enterprise-wide adoption of their version of Scrum. It came with almost no forewarning, no training, no modifications to corporate processes or contracts, and no discussion of revised roles and responsibilities across the chain of development-related responsibilities and processes. The client put out some simple training, really presentation-ware, with a proclamation that all project teams were to implement their new approach to development. The result is that they broke all their old software development practices before fully defining and implementing the new processes.

It took several years to sort some of these issues out, but in the meantime, another insidious problem began to seep in, and that was process bloat. Without proper training or proper contract guidance, some program and project managers felt compelled to mix what they knew from their experience with the traditional software development model with the new mandated agile practices. For example, project teams attempted to implement Scrum using the same project planning, detailed requirements gathering, gate reviews, documentation, and project scheduling strategies dictated by their contracts and cultural norms.

Scrum, as an agile framework, implements a lightweight set of processes and events to efficiently and quickly develop high-value products. Since the organization was not explicitly following Scrum, internal support organizations began to incorporate some of its legacy practices in the new agile paradigm. Given that Scrum is a framework and not a prescriptive methodology, that might make some sense on a situational and as-needed basis. But wholesale integration of traditional and agile concepts loses the point, and the result is not agile anymore, is it?

Now that I've discussed what can go wrong when Scrum is not applied correctly, let's discuss how to implement Scrum properly. We'll start with a discussion on the defined roles and responsibilities within Scrum, and then move on to discuss the events, artifacts, and approach to implementing Scrum at the project level.