What is Complexity and Why is it Useful?

This post originally appeared on Excella Consulting’s Blog.

This is an exciting time to be involved in the Agile community. A number of new models are emerging that allow us to understand our work better and frame our challenges more effectively. One of the most robust is the theory of Complex Adaptive Systems (CAS). A CAS is a collection of unique elements, like people and their tools, that interact in multiple ways, changing, learning, and adapting over time. CAS have several important characteristics:

  • Greater than the sum of their parts: a CAS is created by the interaction of its component pieces. It cannot be decomposed and must be considered as a whole.
  • Sensitive to time: the specific sequence of events within a CAS is an extremely important factor in shaping its future.
  • Sensitive to context: existing patterns and relationships are very influential; generalization is dangerous when working with a CAS.
  • Emergent futures: the future of a CAS is uncertain and unpredictable; new patterns emerge from the interaction of component elements.[1]

CAS theory has opened up new avenues for understanding in biology and the physical sciences, and is now being applied profitably to knowledge work.[2]

Complexity and Software Development

Software development is a form of knowledge work. We focus on non-routine problem solving, creative thinking, and the acquisition of new knowledge. That new knowledge is what allows software solutions to solve business problems in creative ways. Without it, we would be stuck automating existing routines, rather than developing new, innovative solutions that transform our clients’ ways of working.

CAS theory offers us a new perspective for managing this knowledge work. In a complex system, control—or influence—is exercised through constraints. Constraints limit the behavior of the individual actors in the system. I like to use examples from the natural world. In a forest, the limits of hot and cold temperatures in the summer and winter, the patterns of rainfall, and the features of the terrain would all be constraints. Along with many others, these constraints influence the animals and plants that dwell in the forest, helping to define how they live and interact. Similar patterns exist in a software team. In a team, the constraints might include a two-week sprint cycle, defined office hours, and the physical boundaries of the office. All these strongly influence how the team interacts and works together.

Triggering New and Innovative Ideas

That’s straightforward enough, but the magic of complex systems is that they are capable of spontaneously introducing new and innovative characteristics through the interaction of individuals within the boundaries of these constraints. In biological systems, this is what triggers evolutionary change and the introduction of new species. The classic example of this is the peppered moth; as the industrial revolution overtook England, soot covered the trees. The soot was a new constraint. Moths with darker coloration were better camouflaged in this environment. Birds didn’t eat them as often, so more of them survived to breed. Eventually, the whole population of moths became darker colored. This was an evolutionary change. When this process couples with an initial genetic mutation, new species can result; the different species of finches in the Galapagos are a well-known example.

The same dynamics can occur within a development team. New constraints can encourage different behavior, leading to evolutionary improvement in the work of the team. Shortening the length of sprints, for example, from three weeks to two will foster different behavior. Stories will become smaller, opportunities for feedback will occur more often, and the team may feel like it is moving more quickly. These differences may trigger fundamentally different ways of working within the team.

Finding the Balance

The challenge in doing this is that constraints have a sweet spot. They are more likely to trigger innovative approaches when they focus the team, but do not overly limit them. If the constraints are too loose, there will be too many degrees of freedom. Team members will lack focus and feel no need to harness new, creative ideas. However, in an overly constrained system, there will be insufficient flexibility and no freedom to introduce innovative concepts. In between these two extremes, constraints will channel the creative energies of team members and focus them on creative solutions.

We see this dynamic at work on effective agile teams. When the team members have a clear sense of their objective, and they are focused on delivering a small, discrete piece of value in the near term, innovation results. The constraints—the sprint goal, the two-week iteration, and a small set of valuable user stories—focus the team and energize them. Their creativity comes to the fore and shows in their work.

Complexity gives us a model that explains why agile works. It also helps us understand why some agile teams struggle. If a team hasn’t found the right balance of constraints—if they aren’t in the sweet spot between the two extremes—they won’t foster the creative energies of their team members and they will find it difficult to come up with new and innovative ideas.

Complexity suggests that the best way to enable the success of agile teams is to keep a close eye on the constraints governing their behavior and to deliberately manipulate them to encourage focus and foster creativity. One reason the Scrum framework has been so widely adopted is because it is an effective vehicle for this. Well-designed Kanban systems that employ visualization and work-in-progress limits are also extremely effective. In the next post in this series, I’ll look in more detail at these two approaches, contrast their use of constraints, and offer more insights for how use constraints in knowledge work.

Read the next post in this series here.

[1] For a detailed examination of these ideas, see Embracing Complexity by Jean G. Boulton, Peter M. Allen, and Cliff Bowman.

[2] For a great introduction, see the HBR article, “A Leader’s Framework for Decision Making” by David J. Snowden and Mary E. Boone.

Why is Complexity Useful?



Frogs are complex. I like frogs. From https://commons.wikimedia.org/wiki/File:Green-leopard-frog-in-swamp.jpg

Why is complexity useful? One of my colleagues recently challenged me to answer that question. I wasn’t sure how exactly to respond. I’ve become so accustomed to approaching situations with an eye towards complexity that it has become part of my mental fabric. I take its value as a given. I thought stepping back and reflecting on it might be valuable.

Complexity informs a lot of what I do. I use it to enhance my work with software teams; I try to apply it in my approach to historical analysis; and I find myself reflecting on the Cynefin framework when I’m confronted with a problem. I find it useful in all these situations.

Software teams are complex systems. They contain individuals who play different roles and have their own independent thoughts, desires, and motivations. Each team is unique. This makes it very difficult to develop a formulaic approach to helping them improve. There are, however, certain general concepts that I’ve found to be useful.

I accept Alicia Juarrero’s idea that we cannot cause innovation, but we can create an environment where it is more likely to occur. I try to create that kind of environment by encouraging team members to broaden their perspective on their own process and to momentarily step outside of it. I change the size and composition of working groups, run them through simulations, and impose short time boxes. Usually, these help foster a new shared frame that increases receptivity to potential improvements.

Those familiar with complexity will recognize that I’m altering constraints. I tailor the environment within which the team self-organizes. This triggers a change in perspective, allowing team members to look at their shared context in new ways. It is quite useful in generating ideas that are relevant and immediately applicable.

I also think about complexity when I’m doing historical analysis, but I approach it quite differently. Instead of manipulating constraints, I try to keep in mind the broad set of potential outcomes that were possible at a given moment in time. When looking into the past, we have a tendency to focus on known outcomes and ignore the possible alternatives. This leads to an unfortunate sense of inevitability, and we can easily fall victim to retrospective coherence—the belief that historical outcomes were largely predictable, for no other reason than because they occurred. Reality is far more complex.

If I keep the broader set of possibilities in mind when examining past decisions, I develop a deeper understanding of why certain choices were made, how they were justified, and what potential alternatives were considered. This provides a richer and more valuable narrative. If I can explain past decisions in context, then I there is more opportunity to draw out salient lessons for the future.

When approaching new decisions, I find myself often considering courses of action based on the Cynefin Framework. The framework itself is described in detail in the Cognitive Edge material. I have found it very useful for thinking about how best to approach a problem. It is especially useful in a group situation, with competing perspectives about how to proceed. Pausing for a moment to consider whether or not the problem is Complex or Complicated (rarely do discussions of Obvious problems last long) can be very valuable. In Chaotic situations, there’s a need to act right away, so pausing is better once the crisis has passed.

In see a use for complexity in all these situations. The more experience I gain with using it, the more I feel like complexity is “the way things are.” It’s less a tool and more of a model for seeing the world. I find it quite useful. I suspect you will as well.

Leadership Teams and Enabling Constraints

Michael Norton’s
Leadership Teams may be a smell” is a thought-provoking read. As I went through it, I found myself thinking of examples from my own experience that bear out his hypothesis. Here’s a summary of his idea:

Right now, I think of a leadership team as an organizational smell. Like a code smell, an organizational smell potentially indicates a deeper problem in the system. The smell itself, in this case the existence of a leadership team, is not technically a problem. A leadership team doesn’t prevent an organization from functioning. But the existence of a leadership team may indicate perceived weaknesses in the overall system. Maybe we’re attempting to compensate for these weaknesses by centralizing authority and decision making.

As I thought about the topic more deeply, I realized I didn’t fully agree. I don’t think a leadership team should be considered an organizational smell, but the attributes Michael Norton ascribes to them (centralized authority and decision-making) should be. I hope to build on his thinking.

Professional organizations are complex adaptive systems. They improve, adapt, and learn through constraints. Some constraints, like markets, act mostly from the outside; others are introduced internally. These two basic types of constraints interrelate and influence each other.

Beyond this categorization, there is another way to think of constraints. Some are “enabling” because they encourage creativity and self-organization. Enabling constraints help to channel activity and focus it. Examples abound in natural systems; the arteries of the circulatory system are one of the best; they channel nutrients to cells, allowing larger and more complex bodily structures. Arteries are analogous to the enabling constraints in a Kanban system, the work-in-progress limits and visual workflows, which help channel the delivery of value to customers.

Other constraints are imposed “top-down.” These constraints limit creativity and inhibit self-organization. They are generally human creations. Having to fill out a progress report every week, or mindlessly summarizing the day’s status to satisfy a Scrum Master are examples of top-down constraints.

The key organizational smell for me is not whether the leadership team exists, it is whether its members are using constraints well. Are they centralizing decision-making and imposing decisions? Or are they experimenting with structures and approaches that encourage teams to self-organize more effectively? Are they introducing top-down constraints that inhibit creativity? Or are they encouraging experimentation with enabling constraints?

I believe this holds even in the most challenging environments. If we consider the Cynefin Framework’s complex domain, for example, I agree with Norton that the optimal organizational design pattern is a decentralized network. There is value in having a mechanism in that network that watches for patterns and trends, so that positive experiments can be identified and exploited. This is what I would expect a leadership team to do and it can be done through effective use of constraints.


CIC on USS Independence

Fortunately, it is not just theory. This is the exact approach taken by the U.S. Navy’s Pacific Fleet in 1942. Existing shipboard information systems were being overwhelmed by new data coming from radars and radios, preventing effective decision-making in battle. Admiral Chester Nimitz, the fleet commander, recognized the problem, but refrained from imposing a solution. Instead, he and his staff introduced an enabling constraint; they ordered each ship of the fleet to develop their own solution. This triggered multiple parallel experiments throughout a large (fairly decentralized) network. The staff observed the results and selected the most successful experiments for future development, resulting in the revolutionary Combat Information Center (CIC).

To put it in terms of Cynefin, Nimitz and his staff successfully moved the problem of managing shipboard information from the Complex Domain to the Complicated one through the use of multiple parallel experiments and constraints. In doing so, they acted as an effective leadership team. I believe today’s leadership teams could have similar successes, so long as they keep the details of complexity in mind, and approach problems with an eye towards enabling constraints.






Constraints to Improve Flow

In my work with software teams, I often use traffic flow as an analogy. It is an extremely accessible way to describe the concepts of capacity, flow, and utilization. We all seem to have experienced traffic in one form or another and even the smallest localities have their share of traffic jams. Most people appreciate the value of the analogy immediately, which makes it easy to relate the use of metered on-ramps to work in progress (WIP) limits in a software system, an approach I like to take.


An officer acting as a constraint to improve overall flow. He was only moderately successful.

This familiar analogy can be extended. Recently, I’ve started realizing how constraints in traffic systems influence self-organization and flow. Some traffic constraints are obvious and common across geographies. We’ve all experienced lights at intersections, roundabouts (or traffic circles), stop signs, and yield signs. Some less obvious constraints, such as curbs, medians, and lane markers define the roadway. All of these together help to control the flow of traffic and make it easier for us to navigate to our destination.

Alicia Juarrero and others have broadened our understanding of complexity by emphasizing the impact constraints have on self-organization. She divides constraints into two major categories. Context-free constraints limit options, but lack contextual significance. It is often useful to think of these constraints as being imposed from above, or from another level of the system. Context-sensitive constraints, on the other hand, arise from within the system itself.

Within a traffic system or a software development process, both types of constraint can have an impact on flow. A stop sign is a good example of a context-free constraint. It controls flow by forcing vehicles to stop when they reach it. If the way is clear, then they can proceed; otherwise they wait. But the sign has no knowledge of the state of the system. You’re required to stop regardless of whether cars are approaching on the cross street.

A parallel in a software process might be a WIP limit on the “Development” column within a team’s Kanban. Although the choice of the particular value for the WIP limit is very sensitive to context, once it is chosen it acts like the stop sign, forcing work to wait until capacity is available regardless of the broader context.

Both the stop sign and the WIP limit can help improve overall flow. The stop sign exists to allow improved traffic flow along another axis, or to control flow through the intersection. The WIP limit prevents work from building up in Development, allowing more rapid feedback to that activity from downstream processes. These smooth out the flow in both systems.

Context-sensitive constraints also exist in traffic systems and software systems. These tend to be less obvious unless you’ve had a chance to immerse yourself in them. But even as a passenger, it quickly becomes apparent that the unwritten “rules of the road” are very different between India and Germany, or Italy and England. There are even subtle differences between Maryland and Virginia, where I tend to spend most of my time driving. This is because drivers in each of those locations have a specific set of expectations for how they believe they should behave in traffic. These expectations constrain their behavior, and they differ from place to place. Software teams have similar unwritten “rules” of behavior that influence their flow.

Together these context-free and context-sensitive constraints largely determine the overall flow of the system. Interesting relationships between the types of constraints can be seen. The German traffic system, for example, is relatively highly constrained by the “rules of the road.” German drivers go through an extensive certification process, giving them a strong common sense of how to behave on the roads. One result is that they tend to stay out of the passing lane unless they are passing. This context-sensitive constraint improves the flow of traffic on two-lane roads.

Contrast this with Virginia, where the use of the passing lane is far less disciplined. Drivers sit in it, inhibiting flow and triggering backups. To improve flow, traffic engineers end up adding new lanes. In effect, they’re substituting a context-free constraint (another lane) for a context-sensitive constraint (limited use of the passing lane) to achieve the same end (improved flow).

India is another good example. In Indian cities like Chennai, the “rules of the road” lead to the utilization of the entire expanse of the roadway. Lane markers are ignored and all spare capacity is utilized. This works well to fit lots of traffic into a confined space, but it can have a crippling impact on overall flow. Driving becomes a series of negotiations as each vehicle jockeys for just enough space to move forward.

To compensate for this, the city of Chennai has begun to introduce context-free constraints: large barriers have been installed along the sides and down the center of major roads. These limit the ability of traffic to enter and exit the roadway. They also prevent traffic travelling in one direction from flowing over into the oncoming lanes. Although some of my friends there might argue the point, I think they’ve improved flow.

Similar relationships between context-free and context-sensitive constraints exist in software systems. Some teams have developed a common understanding—a sense of how much work they can handle—and can do without WIP limits within their system. They maintain flow through their “rules of the road.” Other teams lack this capability and have sought to achieve the same end through WIP limits, a disciplined understanding of their velocity, or other means. 

Neither of these approaches is inherently more valuable. The key is understanding how the team is behaving today and determining how best to maintain an effective flow. If you regularly bog down and have variable flow, context-free constraints may be the best way to start. In other circumstances, working on team cohesion may be more effective. There are many approaches to that; I’ve found Lean Coffee to be an effective one. Informal conversations over coffee breaks are good too.

Unconferencing #NoEstimates

A little over a week ago, Paul Boos hosted the first Agile Dialogs Unconference. Our subject was inspired by the debate surrounding #NoEstimates. We hoped to get beyond some of the rhetoric and grapple with perspectives from both sides. I was hopeful that we would increase our overall understanding and generate some ideas for real improvements we could use with our teams. I think we achieved that goal.


A number of good ideas came out, and Paul has generated an accurate list. For me, there were three main lessons. The first is that when we debate #NoEstimates or #YesEstimates, we’re often talking about very different things. We paused several times in our discussions to ask ourselves what we really mean by “estimates.” There were some very diverse perspectives and we often needed to recalibrate. Differences in the definition of “estimates” influence how we approach the idea of #NoEstimates and what we think it means. Many of the arguments on Twitter seem to have their origin in these kinds of misunderstandings.

We were able to get past that in our dialog and draw out some valuable perspectives from both sides. From the #NoEstimates side, I think one of the most important lessons is that when we make an estimate—regardless of whether or not that is a traditional time estimate or one involving something more abstract like story points—we risk anchoring ourselves to a particular implementation approach. This may be fine.

But there are cases where it will lead us down an undesirable path and trigger rework or introduce technical debt. It can be difficult to avoid, as we often don’t realize the impacts of our assumptions until we have a chance to test them. Usually, that comes long after an estimate has been made. Teams that make estimates might want to keep this in mind.

Critics of #NoEstimates point out that estimates are often blamed for broader organizational dysfunctions. We discussed this at length and cited a number of examples from our own experience. Although estimates are a visible aspect of dysfunction (like when they’re used to transfer risk), that does not mean they are the cause. They may only be a surface symptom; discarding estimates may have no effect on these deeper issues and may only further complicate the problem. If you’re looking to discard estimates, you may want to investigate whether or not this applies to your environment.

As with many things, the benefit or harm of estimates appears to be very much dependent on context. By getting together and sharing our experiences, we helped broaden our perspectives and gain more understanding. I’m looking forward to more Agile Dialogs.

The Perils of “Obvious” Testing


The Cynefin Framework

Like many, I’ve been stunned by the VW Emissions Scandal. But I’ve also been fascinated. when I heard that software had been written to circumvent the EPA’s tests I immediately started wondering how it was done. I thought that would be an extremely complicated process, but that was because I failed to understand the nature of the testing.

As this article on the Washington Post points out, the EPA’s tests are very scripted. They follow a predictable and repeatable routine where speeds have to be kept within “two miles per hour of the required speed at any given moment.” With such a standardized approach, I immediately recognized how it would be comparatively simple to put the necessary monitors into the software to detect the start of such a test, adjust performance throughout, and guarantee a passing performance.

I’m glad the EPA quickly recognized that it needs to revise its testing approach. I hope that they take the deeper lesson from this experience and adjust their testing so that it is not “obvious.” I am deliberately using the term for the Cynefin Domain, because I think the sense-making framework can help us understand what happened and draw broader lessons for how to develop more effective tests.

The Obvious Cynefin Domain was previously called “simple.” It is tightly constrained; there are no degrees of freedom; and best practices can be developed and disseminated for dealing with problems in this space. This is an accurate description of the EPA’s testing approach and also applies to other forms of testing. It also helps explain what happened with the VW diesels, because when Obvious solutions fail, they do not fail mildly.

Instead, they tend to fail catastrophically by passing through the boundary into the Chaotic Domain. This shatters the existing dynamic. It is, I think, a perfect encapsulation of the VW Emissions Scandal; it was a catastrophic failure that undermined our faith in the automotive industry.

I have seen similar failures (on a much smaller scale) in software, where teams used scripted approaches to testing that failed to adequately model user behavior and expectations. The tests “passed,” but the user experience missed the mark. In these circumstances, the software “failed.” Some of these failures have been similarly catastrophic. How can we avoid similar types of failures in our own testing approaches?

One answer is to ensure that our tests are not so “obvious.” They must not be scripted and predictable. If they are, they invite “gaming” and will lead to failure as user expectations change (and they will, once they begin to use the software). Exploratory Testing is an excellent way to do this. Exploratory tests are not “obvious;” from a Cynefin perspective they are Complex, because the tester follows a “probe-sense-respond” paradigm as they work with the software. Assumptions about what it should and should not do are subordinated to learning what it actually does. This offers the potential for learning and discovery. Coupled with a rapid feedback cycle, it is a much better approach and will lead to better solutions.

Perhaps the EPA should consider something similar.  

Thoughts on Agile Coach Camp US

I attended Agile Coach Camp US last week. It was a wonderful experience and a great way to explore new ideas. Here are some of my personal highlights of the event.


Olaf Lewitz gave an excellent introduction to the concept of Temenos. It’s an approach that emphasizes creating space for effective conversations and mutual understanding so that we can become aware of our choices and take more deliberate (and positive) action in the future.

IMG_4302Olaf triggered a sharing exercise by asking us to tell stories from our lives where it felt difficult to say “no.” We began hesitantly, but soon we were relating our experiences, building on each other, and exposing how difficult it can be to turn down our friends, relatives, and colleagues.

While this was happening, we observed several things. The pace of the sharing accelerated. Sharing led to more sharing, more openness, and increased sense of connection with each other. As we discussed our experiences, I—and others as well—became more aware of options. Our choices to say “yes” or “no” were deliberate; we did not “have” to make them the way that we did.

That’s when I was struck by the power of the approach. Instead of drilling into why an event occurred or why a decision was made, we were being focused on options and future possibilities. We talked about our view of the past, but the analysis in my head did not dwell on the past; it was looking to the future. I felt that I had more choices than before. I felt that I could be more deliberate. It was a fascinating effect to observe.

It was made more powerful by the knowledge that the most successful retrospectives I’ve facilitated worked much the same way. I’ve always placed great emphasis on sharing diverse perspectives. I didn’t call these perspectives stories, but they worked similarly, and broadened our view of the past so that we become more aware of future options. From there, it is easier to make a deliberate choice about what to do in the future. I think I can make my approach even more effective with Temenos and want to learn more.

Remaining Curious

Sue Johnston led us through a fun conversation about the value of remaining curious as a coach. The most impressive aspect of this session was how we focused in on questions and questioning style. Questioning is an important aspect of curiosity, of course, but there are different ways to ask questions and we agreed that some are more effective than others.

IMG_4295One of the most interesting suggestions was to try to avoid the use of “why” questions. We ask “why” questions all the time; the “5 whys” is a well known technique. However, they can be very dangerous. “Why” questions can easily lead to blame. Consider the difference between these two questions:

Why did you do that?


What made it seem like that was a good idea?

These two are very different. The first question is framed in such a way that an individual (or a group) immediately feels responsible for doing something wrong (the “that”). Judgment is implicit. This will likely trigger hostility and/or fear. We risk shutting the conversation down.

The second question divorces the action (the “that”) from the individual or team. We operate from an assumption of best intentions and allow them to share their perspective. We remain open to the possibility that they know something we do not and that their choice might have been the best. This approach increases the potential for a broader level of shared understanding. The suggestion was that we should try to reframe “why” questions as “what” questions whenever possible.

Another valuable point was Sue’s introduction of the idea of the “arc” of a coaching conversation. This was presented as a series of different questions, each appropriate for a different moment in the arc.

  • In the beginning, we need to learn and so we ask, “What?”
  • Once we understand the situation better, we need to develop a better sense of the context and so we ask, “So What?”
  • With a sense of the context, we can move to helping draw out factors that may not be immediately obvious and so we ask, “What Else?”
  • Finally, to help someone understand what they can do with the knowledge gained, we ask “Now What?”

I thought this was an excellent little frame for thinking about coaching conversations that can help keep the focus on curiosity.

The Unprintable Work of Derek W. Wade

Derek ran a pair of sessions, both had great names that drew people in, but they’re inappropriate for a family blog.

In the first, we explored the power of venting frustrations in a positive way. We shared techniques we use to deal with frustration and affinitized them. Then we vented our frustrations, not as stories, but just as a word or phrase, going around the group several times. We had plenty of frustrations. Once we had a good list compiled, we shared a few stories and tried to relate them back to the brainstormed techniques. I think the best vision to come out was the concept of simultaneously drinking wine and juggling chainsaws. That gave us all a good laugh!

The second session explored techniques for dealing with leaders who seem resistant to change. We had all encountered people in powerful positions who appeared unwilling to support the improvement of their teams. Although it was not billed as such, the session was a great look into the dangers of fundamental attribution error. We spent a lot of time discussing how these individuals have their own valuable perspective and we need to gain a better understanding of it in order to communicate with them effectively.

Beneficial Conversations

In addition to the sessions, I had several really useful conversations which helped me see things in new ways.

Michael J. Tardiff and I IMG_4282had a fascinating exchange. We talked about infection and subversion as two different models for organizational change. We agreed that an infection model starts out localized and spreads once others see or hear about the value. With infection there is no inherent threat to existing power structures; an effective practice, like TDD, might spread this way. Subversion differs in that it deliberately seeks to undermine existing authority and is driven by interested parties. The two can be combined; they’re not mutually exclusive. But they have very different implications. Our talk got me thinking about the importance of accounting for power dynamics and vocabulary when we discuss organizational change.

Andrea Chiou reminded me of the value of creating a shared set of experiences for increasing empathy and accelerating understanding. The most effective workshops I’ve run all seem to have had some element of this, even if I didn’t intentionally create it.

Bryan Beecham explained his concept of “human refactoring” and used some very effective software code analogies to describe it. He believes we can use this approach to change our behavior and make better choices. I agree. I’m less confident in his goal to live for 200 years, but I wish him luck.