This is my talk from Lean Agile Scotland, “Complexity in Action – Organizational Learning in the U.S. Navy.”
This post originally appeared on Excella Consulting’s Blog
Lean Agile Scotland was an excellent conference. I’ve struggled with how to condense all my positive experiences into a single blog post; this may have to be the first of many. Over just three days, I was able to create new connections, participate in enlightening sessions, and start a number of thought-provoking conversations.
These are a few subjects that I can’t stop thinking about.
My talk fell on the third day of the conference. I adapted it slightly so that I could build on some of the prior sessions. Simon Wardley brought up doctrine in his opening keynote and Will Evans touched on the same subject in his talk on strategy. Doctrine is akin to a culture or an ethos, but more specific; it’s the set of assumptions embedded within a team that informs their behavior. Doctrine is important because it helps to drive decision-making; with a coherent doctrine, team members can make the correct decision for a specific context in the absence of instructions or communication.
I explained why this is beneficial for Agile teams (because I’ve yet to come across one that hasn’t cited the need for “better communication” and “fewer meetings”) and then described how the U.S. Navy went about developing a more coherent doctrine in the early years of the twentieth century by using regular learning cycles, self-organization, and heuristics. I think it worked pretty well. The slides are here, and a video should be coming soon.
Simon Wardley’s keynote was very good. I’d seen Wardley Maps on the internet before but never delved deep enough to really understand what they were or how they could be used. Simon explained them in detail, giving some entertaining background on how he came up with them.
Wardley Maps are a way to visualize a solution and the associated technologies. Two axes are used. The vertical represents where an item falls on the value chain, with “visible” or customer facing elements of it towards the top and “invisible” or internal aspects at the bottom. The horizontal axis reflects the level of “evolution” of the associated technologies. Evolution is usually represented in four categories: “genesis,” “custom build,” “product,” and “commodity/utility.” Each element of a solution can be mapped on these two axes and decisions can be made based on the current position and how those positions are likely to change. Technologies tend to progress along the horizontal axis, for example, from genesis to commodity/utility.
That is straightforward enough, but what got me really excited was the idea that different types of approaches, with different types of teams, are more applicable to different areas of the map. Simon described three basic types of teams, “pioneers,” “settlers,” and “town planners.” Each of these is more applicable to different levels of technological evolution. For discovery, we want “pioneering” teams who are open to alternative paths. To exploit new technologies, “settlers” are more applicable, and “town planners” are best in the commodity space. Simon explains it here.
Optionality and Uncertainty
Chris Matts presented his own take on these ideas. He tied together Wardley Maps, Cynefin, and Crossing the Chasm in a series of slides that spoke to his belief that Lean Agile Scotland brings together a “community of needs.” He contrasted this with a “community of solutions.” The main difference between the two can be expressed in one of his favorite ideas: optionality. In a community of needs, new options are created. The goal is an investigation and broader understanding through fostering increased connection and collaboration. This creates more options for each individual and for the community as a whole, fostering greater potential. For Chris, we are a bunch of “settlers.” In a community of solutions, answers are better defined; options are more constrained and context is less important.
Chris and I are currently having a dialog about options and their relationship with constraints. I believe we broadly agree, but I think there are some important differences on how the two interrelate. Can options create new constraints? Can constraints create new options? I believe the answer to both questions is yes, but we need to work through them and explore it a bit more. I’m excited to have the option to do so, and I expect my learning will be the subject of another post.
Melissa Perri took these concepts and made them more immediately relevant to Agile teams in her excellent keynote on “The Build Trap.” She’s seen the problem of teams building, building, and building without having sufficient focus on real value. A key element of this is discomfort with uncertainty. It is difficult to admit that we don’t know exactly how to solve problems for our users. But once we do, we can experiment and learn so that we discover the best approach. Melissa confronted this challenge directly.
Melissa’s and Chris’s sessions tie together because they both force us to think about the opportunities that arise from the limits of our knowledge. That spirit, and the commitment to learn from each other to create new options, is why I think the conference was such a great experience. I can’t wait to go back.
This post originally appeared on Excella Consulting’s Blog.
In the previous posts in this series, I introduced Complex Adaptive Systems (CAS) and described some basic approaches for using complexity to improve the work of Agile teams. In this post, I’ll delve a bit deeper and explain how complexity can help trigger the mindset shift necessary to create an effective environment for Agile teams.
But first, it’s important to revisit the main characteristics of CAS:
- Greater than the sum of their parts.
- Sensitive to time.
- Sensitive to context.
- Emergent futures.
Each of these has a bearing on the work of Agile teams.
Greater than the Sum of the Parts
Anyone who has been part of a good team has experienced this dynamic. Good teams become more than a collection of individuals. They create an environment that enhances the effectiveness of team members. They learn how to work together, to communicate seamlessly, and to enhance each other’s skills. Good teams become entities unto themselves; they are complex systems.
Although Agile emphasizes the need to develop cohesive, self-organizing teams, the importance of this is often underestimated. There is a lengthy history in modern management of treating individuals as “resources” and assuming that they can be moved from team to team without significant reductions in effectiveness. CAS theory illustrates how this idea rests on a false assumption. Teams cannot be reduced to their component pieces, taken apart, and reassembled. They need to be considered a whole with unique characteristics that are more than the sum of their components.
To be successful with Agile, we need to foster the development of good teams, create environments that allow them to flourish, and let them become effective at working together. Once they achieve that state, treat them as a team, not a collection of individuals. This means shifting the focus of management. Managers become responsible for creating the necessary environment; they shift from managing individuals to managing teams.
Sensitive to Time and Context
This work is aided by becoming familiar with how sensitive teams are to historical events and their current context. Just like CAS in general, Agile teams are a product of their pasts and current environmental influences. This has several important implications.
It is impossible to “rewind” the clock and return to a previous state in a complex system. Historical events and their impact become embedded in a team; they constrain future potential. Focus efforts like retrospectives on broadening everyone’s perspective and validating the diversity of views within the team; there will be no single “correct” view of the past. Acknowledging diverse perspectives will provide a solid foundation for future improvements and ensure they are informed by the experiences of the entire team.
Context exerts a similar influence. I have worked with dozens of Agile teams around the world, from Europe to Asia and North America. Each of those teams had a slightly different context—different team members, different products, different ways of working—and they all approached Agile slightly differently. To be most effective, I had to adjust my approach, and my message, to allow my efforts to resonate with them.
An effective environment for Agile teams needs to recognize this variability. Just as we cannot expect all individuals on a team to react and behave the same way, we cannot manage all teams the same way. What’s needed is a clear set of expectations—well-defined constraints—for how teams will be evaluated and managed. If those constraints are in the “sweet spot” then the teams will have the freedom to improve, grow, and self-organize, and the leaders of the organization will have enough visibility and control to develop the confidence they need. The key is to remember that the “sweet spot” will vary from one organization to another.
One of the most exciting aspects of Agile approaches is waiting until the “last responsible moment” to take action. It’s exciting because identifying the right moment can be a challenge, and because it allows us to defer commitment of energy and time until we have the latest (and best) information.
The future is unpredictable. We all know this. But in CAS, we assume the future will emerge, that unanticipated new ideas and opportunities will arise. By waiting until the “last responsible moment” we increase the probability that we will be able to take advantage of these when they do.
The challenge, of course, is that emergent futures are not predictable, and most organizations crave predictability as a means to reducing risk. From the perspective of CAS, this is quickly exposed as a paradox. If we invest in predictability to reduce risk, we inhibit our ability to defer commitment and seize emergent opportunities. The alternative is to reduce risk by minimizing unknowns. If we can go from starting work on a user story to delivering the associated functionality in two weeks, we keep risk very, very small. Agility offers new ways to reduce risk.
To make Agile work effectively, especially in an environment with multiple teams, a mindset shift is required. CAS theory helps us understand the nature of that shift and provides a new window into understanding why Agile approaches are so effective. In these four posts, I’ve barely scratched the surface. If you’d like to learn more about how to apply complexity in your organization, please contact me or one of the other coaches here at Excella.
 For a detailed examination of these ideas, see Embracing Complexity by Jean G. Boulton, Peter M. Allen, and Cliff Bowman.
This post originally appeared on Excella Consulting’s Blog.
In the earlier posts in this series, I introduced Complex Adaptive Systems (CAS), and contrasted Scrum and Kanban from a CAS perspective. I also briefly touched on the idea of heuristics; in this post, I’ll expand on these ideas and explain why heuristics are advantageous for decision-making in complex systems like software development teams.
A heuristic is a guide that aids decision-making. It is not a set rule, but it is a rule of thumb. I use a simple set of heuristics at my local grocery store to accelerate my checkout process. If I have a basket of items, I will use the self-checkout. If I have a cart, I’ll look for the checkout line that has the least number of people in it. If two lines have the same number of people, I’ll choose the one with the fewest items. These guidelines are simple heuristics and we all use techniques like these, even if we’re not always aware we’re doing so.
Heuristics are powerful because they accelerate the process of making decisions. We don’t have to spend time assessing all the potential options, performing a cost-benefit analysis, and ranking alternatives. Instead, we employ a specific set of guidelines to quickly arrive at what we believe to be the best option.
Why is this important and how does it relate to complex systems? In an earlier post, I noted how CAS are extremely sensitive to context. The specific ways in which the individuals in the system interact and the constraints that act upon them vary; the nature of these interactions also change over time, altering behavior within the system and its potential for change. This makes complex systems dynamic. To operate within them effectively, flexibility is needed.
Heuristics offer the necessary flexibility. Because heuristics are not set rules, they can be quickly tailored to specific contexts. I can adjust my grocery store heuristics based on circumstances. If I’ve got a bottle of wine in my basket, for example, I might opt for a checkout line over the self-checkout because I know that a cashier will be able to check my age and validate I’m over 21 faster than the attendant overseeing the self-checkout machines. If children are with me, I may look for the line with no candy, regardless of its length. Because I’m using heuristics, I remain sensitive to context and can tailor my decisions quickly and easily.
That’s the power of heuristics. They inform decisions without substituting for them. Heuristics enable human judgment by rapidly framing the problem and allowing our minds to focus on the specifics of the situation. This leads to quick decisions that factor in valuable contextual information without overly complicated and unnecessary rules.
What’s this got to do with Agile software development? Agile software teams make decisions every day. How can those teams ensure that these decisions happen rapidly and align with their overall goals? One way is to make sure they’re made collaboratively. If teams regularly talk about their work and the impediments they’re facing (like in the daily standup), then their collective knowledge and experience will be brought to bear. However, many decisions are made outside of those collaborative sessions. How can we ensure they’re made rapidly and reflect the team’s objectives?
Heuristics are the answer. By explicitly developing heuristics to guide their decisions, teams can make decisions more rapidly, ensure the outcomes align with their objectives, and leave room for individual judgment and creativity. An excellent heuristic at work in software development is the “rule of three,” a simple heuristic that suggests code can be copied twice, but should be refactored into a new procedure the third time it is used. Another common heuristic builds on the Agile principle emphasizing face-to-face conversation: whenever you have a question, ask it face-to-face; if that’s impossible, use a video conference; and if that’s not available, use the phone.
Explicitly defining heuristics can accelerate the decision-making in Agile teams. Because they are sensitive to context, heuristics can adapt to a variety of different circumstances; they won’t age and become obsolete like “agreements” or “rules.” Teams employing heuristics can make decisions more rapidly and be more confident that day-to-day decision-making aligns with their goals. By allowing team members to utilize their knowledge and skill, heuristics foster greater engagement and morale. They fit perfectly into a view of software development as a complex adaptive system.
Try defining some heuristics and see how they can improve the effectiveness of your team! Next in this series, I’ll write more about how you can improve the work of Agile teams by employing ideas from complexity.
This post originally appeared on Excella Consulting’s Blog.
In the first post in this series, I introduced the concept of Complex Adaptive Systems (CAS) and how constraints can trigger innovative ideas. In this post, I’ll investigate constraints in more detail and contrast how Kanban and Scrum use constraints to enable successful software teams.
Both Scrum and Kanban use constraints to limit work in progress (WIP), foster decentralized decision making, and focus the team on near-term deliverables. However, the two approaches have significant differences.
Constraints in Scrum and Kanban
|Purpose of Constraint||Scrum||Kanban|
|Limiting WIP||Sprint Backlog||WIP Limits|
|Near-term Deadline||Sprint Review||SLAs for Individual Work Items|
|Focused objective||Sprint Goal||Work Visualized on a Kanban Board|
|Prioritization Buffer||Sprint Backlog||“Ready” Queue|
|Roadmap||Release and Product Backlog||Product and Product Backlog|
In Scrum, the most important constraint is the timeboxed iteration, or Sprint, and the regular cadence of ceremonies. The Sprint limits WIP through the creation of a Sprint Backlog. This backlog is usually estimated in story points and sized based on past performance. If the team has a consistent velocity (e.g., 30 points per Sprint), capacity planning starts with that number and adjusts it based on circumstances.
The Sprint Planning ceremony ensures agreement on the content of the Sprint Backlog and its size. During the iteration, the backlog serves as a prioritization buffer. Outside of that buffer, the Product Owner can make as many reprioritization decisions as necessary, but the Sprint Backlog should only be adjusted in rare circumstances, so that the team can focus on delivering that Sprint’s Goal.
The Sprint Goal is a clear expression of value, the focus for that Sprint. Daily standup meetings allow the team to self-organize and determine how to progress towards the goal. The Sprint Review gives them—and their stakeholders—feedback on whether that goal was met, and informs the objective of the next iteration.
In Kanban, limited WIP and visualization are the main constraints. WIP can be limited in many different ways, but the most common is to cap the amount of work in any given state. A maximum of five stories might be permitted “In Progress,” or a maximum of three in “Test.” The WIP constraint in Kanban focuses the team on flow, and not on time.
Attention is focused on the work by use of a visual Kanban board. On the board, everyone can see what is being worked and what state it is in. This triggers self-organization; the board fosters a collaborative understanding of what is most important. Aligned and decentralized decision making flows naturally from this shared understanding. Scrum teams often leverage visualizations like this as well. Scrum does not explicitly call for them; Kanban does.
Rather than up-front estimates of size and complexity, Kanban uses Service Level Agreements (SLAs) based on historical cycle time. These become a constraint that encourages focus. Note that the SLA is not an estimate; it is a prediction based on past performance. SLAs operate on each and every individual item in a Kanban system, allowing a continual flow; work is not batched up into timeboxed iterations as in Scrum.
Kanban also uses a prioritization buffer, called a Ready Queue. This short backlog contains items that are ready to be worked on, but are not yet in progress. Teams and their Product Owners are free to alter the prioritization of items in the Ready Queue at any time, giving them flexibility to rapidly adjust based on new information.
Although there are no specific ceremonies in Kanban, it is customary to employ daily standups, regular grooming sessions, and demonstrations of new functionality. These can happen on a cadence, or they can be triggered by the flow of work. For example, a team might hold a grooming session once they have only two items in the Ready Queue; they might demonstrate new functionality after five items have been completed. An exception is the Sprint Planning event; Kanban teams have no need for them because there is no iteration.
Is one of these approaches superior to the other? Not in all circumstances; remember, CAS are very sensitive to context. However, if we examine the constraints employed by Scrum and Kanban, we gain some important insights. The ability of a CAS to support the creation of new and innovative ideas is called Emergent Potential. Greater Emergent Potential means a higher likelihood that creative approaches will be discovered; Emergent Potential is maximized in that “sweet spot” between constraints that are too restrictive and too loose.
Scrum is very portable and easily adopted because it provides a well-defined structure. The structure is intended to rest in that “sweet spot.” If it isn’t a perfect fit, teams should Inspect and Adapt, tailoring their specific Scrum implementation in light of experience to better fit their needs. However, this can be very difficult.
In situations where the business is unwilling or unable to leave the team focused for the duration of an iteration, the Scrum process can become a hindrance. If a team is responsible for production support, for example, it may be impossible to work effectively in Sprints. Support cannot always be nicely packaged up into iteration-sized chunks.
Mature Agile teams may also have difficulty with Scrum. The iteration boundaries and associated ceremonies can provide less value as the team becomes more effective at working together. In those situations, more flexibility may be needed to allow a team to maximize its potential.
In both of these situations Kanban may be a better fit because the constraints in a Kanban system are more sensitive to context. This gives the team greater flexibility and the promise of increased Emergent Potential.
If you’re uncertain about whether to employ Scrum or Kanban, consider these factors. Scrum, because it is a well-defined, portable framework, will work well for teams that are newly forming or just learning Agile. Scrum is easy to adopt and can quickly deliver value. More mature teams, who are already familiar with Agile, or who have deeply embedded ways of working, tend to be better off with Kanban. The increased contextual sensitivity of Kanban will offer those teams greater potential.
These guidelines are relatively simple and straightforward, but I’ve found them to be a nice set of heuristics that work well with Agile teams. Learn more about what heuristics are and how to apply them in the next post in this series.
This post originally appeared on Excella Consulting’s Blog.
This is an exciting time to be involved in the Agile community. A number of new models are emerging that allow us to understand our work better and frame our challenges more effectively. One of the most robust is the theory of Complex Adaptive Systems (CAS). A CAS is a collection of unique elements, like people and their tools, that interact in multiple ways, changing, learning, and adapting over time. CAS have several important characteristics:
- Greater than the sum of their parts: a CAS is created by the interaction of its component pieces. It cannot be decomposed and must be considered as a whole.
- Sensitive to time: the specific sequence of events within a CAS is an extremely important factor in shaping its future.
- Sensitive to context: existing patterns and relationships are very influential; generalization is dangerous when working with a CAS.
- Emergent futures: the future of a CAS is uncertain and unpredictable; new patterns emerge from the interaction of component elements.
CAS theory has opened up new avenues for understanding in biology and the physical sciences, and is now being applied profitably to knowledge work.
Software development is a form of knowledge work. We focus on non-routine problem solving, creative thinking, and the acquisition of new knowledge. That new knowledge is what allows software solutions to solve business problems in creative ways. Without it, we would be stuck automating existing routines, rather than developing new, innovative solutions that transform our clients’ ways of working.
CAS theory offers us a new perspective for managing this knowledge work. In a complex system, control—or influence—is exercised through constraints. Constraints limit the behavior of the individual actors in the system. I like to use examples from the natural world. In a forest, the limits of hot and cold temperatures in the summer and winter, the patterns of rainfall, and the features of the terrain would all be constraints. Along with many others, these constraints influence the animals and plants that dwell in the forest, helping to define how they live and interact. Similar patterns exist in a software team. In a team, the constraints might include a two-week sprint cycle, defined office hours, and the physical boundaries of the office. All these strongly influence how the team interacts and works together.
That’s straightforward enough, but the magic of complex systems is that they are capable of spontaneously introducing new and innovative characteristics through the interaction of individuals within the boundaries of these constraints. In biological systems, this is what triggers evolutionary change and the introduction of new species. The classic example of this is the peppered moth; as the industrial revolution overtook England, soot covered the trees. The soot was a new constraint. Moths with darker coloration were better camouflaged in this environment. Birds didn’t eat them as often, so more of them survived to breed. Eventually, the whole population of moths became darker colored. This was an evolutionary change. When this process couples with an initial genetic mutation, new species can result; the different species of finches in the Galapagos are a well-known example.
The same dynamics can occur within a development team. New constraints can encourage different behavior, leading to evolutionary improvement in the work of the team. Shortening the length of sprints, for example, from three weeks to two will foster different behavior. Stories will become smaller, opportunities for feedback will occur more often, and the team may feel like it is moving more quickly. These differences may trigger fundamentally different ways of working within the team.
The challenge in doing this is that constraints have a sweet spot. They are more likely to trigger innovative approaches when they focus the team, but do not overly limit them. If the constraints are too loose, there will be too many degrees of freedom. Team members will lack focus and feel no need to harness new, creative ideas. However, in an overly constrained system, there will be insufficient flexibility and no freedom to introduce innovative concepts. In between these two extremes, constraints will channel the creative energies of team members and focus them on creative solutions.
We see this dynamic at work on effective agile teams. When the team members have a clear sense of their objective, and they are focused on delivering a small, discrete piece of value in the near term, innovation results. The constraints—the sprint goal, the two-week iteration, and a small set of valuable user stories—focus the team and energize them. Their creativity comes to the fore and shows in their work.
Complexity gives us a model that explains why agile works. It also helps us understand why some agile teams struggle. If a team hasn’t found the right balance of constraints—if they aren’t in the sweet spot between the two extremes—they won’t foster the creative energies of their team members and they will find it difficult to come up with new and innovative ideas.
Complexity suggests that the best way to enable the success of agile teams is to keep a close eye on the constraints governing their behavior and to deliberately manipulate them to encourage focus and foster creativity. One reason the Scrum framework has been so widely adopted is because it is an effective vehicle for this. Well-designed Kanban systems that employ visualization and work-in-progress limits are also extremely effective. In the next post in this series, I’ll look in more detail at these two approaches, contrast their use of constraints, and offer more insights for how use constraints in knowledge work.
Read the next post in this series here.
 For a detailed examination of these ideas, see Embracing Complexity by Jean G. Boulton, Peter M. Allen, and Cliff Bowman.
Why is complexity useful? One of my colleagues recently challenged me to answer that question. I wasn’t sure how exactly to respond. I’ve become so accustomed to approaching situations with an eye towards complexity that it has become part of my mental fabric. I take its value as a given. I thought stepping back and reflecting on it might be valuable.
Complexity informs a lot of what I do. I use it to enhance my work with software teams; I try to apply it in my approach to historical analysis; and I find myself reflecting on the Cynefin framework when I’m confronted with a problem. I find it useful in all these situations.
Software teams are complex systems. They contain individuals who play different roles and have their own independent thoughts, desires, and motivations. Each team is unique. This makes it very difficult to develop a formulaic approach to helping them improve. There are, however, certain general concepts that I’ve found to be useful.
I accept Alicia Juarrero’s idea that we cannot cause innovation, but we can create an environment where it is more likely to occur. I try to create that kind of environment by encouraging team members to broaden their perspective on their own process and to momentarily step outside of it. I change the size and composition of working groups, run them through simulations, and impose short time boxes. Usually, these help foster a new shared frame that increases receptivity to potential improvements.
Those familiar with complexity will recognize that I’m altering constraints. I tailor the environment within which the team self-organizes. This triggers a change in perspective, allowing team members to look at their shared context in new ways. It is quite useful in generating ideas that are relevant and immediately applicable.
I also think about complexity when I’m doing historical analysis, but I approach it quite differently. Instead of manipulating constraints, I try to keep in mind the broad set of potential outcomes that were possible at a given moment in time. When looking into the past, we have a tendency to focus on known outcomes and ignore the possible alternatives. This leads to an unfortunate sense of inevitability, and we can easily fall victim to retrospective coherence—the belief that historical outcomes were largely predictable, for no other reason than because they occurred. Reality is far more complex.
If I keep the broader set of possibilities in mind when examining past decisions, I develop a deeper understanding of why certain choices were made, how they were justified, and what potential alternatives were considered. This provides a richer and more valuable narrative. If I can explain past decisions in context, then I there is more opportunity to draw out salient lessons for the future.
When approaching new decisions, I find myself often considering courses of action based on the Cynefin Framework. The framework itself is described in detail in the Cognitive Edge material. I have found it very useful for thinking about how best to approach a problem. It is especially useful in a group situation, with competing perspectives about how to proceed. Pausing for a moment to consider whether or not the problem is Complex or Complicated (rarely do discussions of Obvious problems last long) can be very valuable. In Chaotic situations, there’s a need to act right away, so pausing is better once the crisis has passed.
In see a use for complexity in all these situations. The more experience I gain with using it, the more I feel like complexity is “the way things are.” It’s less a tool and more of a model for seeing the world. I find it quite useful. I suspect you will as well.