Managing for serendipity

While best practice has its place in knowledge management, the fundamental principles on which the theory of best practice is based are often misplaced. Dave Snowden outlines several alternatives that that should help organisations to ensure that their KM initiative fully engages employees and allows for improved decision making and enhanced innovation capabilities.

Over the past two to three years I have asked well over 100 audiences at conferences, in company workshops and academic seminars a simple question: what spreads fastest in your organisation, stories of failure or stories of success? The inevitable answer is those of failure, and there are good reasons for this. Over the millennia the human race has come to realise that being equipped with several stories of failure is far more valuable than a single story of success. This implies that the common knowledge-management focus on best practice is in effect contrary to natural practice; an attempt to impose an idealistic structured process onto the natural activity of learning and knowledge transfer through a focus on efficiency at the cost of effectiveness.

The adoption of best practice implies that:

There is a best way to do something;
We can identify and codify what that thing is;
We can then get employees to follow best practice, and;
It is desirable that they should do so.

In this article, I will argue that in other than a very limited set of circumstances all four statements are false and that, in fact, best practice is simply entrained past practice. I will start by establishing some basic principles relating to human decision making to provide a framework to examine the above questions. I will then conclude that a major area of knowledge-management practice should be to create worse-practice systems on the grounds that they provide better and more resilient approaches to learning.

Human decision making: patterns and context

If, and it is a very big if, there is a stable and repeating relationship between cause and effect in a common context, then best practice can and should be mandated. Human social systems are uniquely able to create such stable contexts by agreeing and establishing conventions for matters such as payment systems and traffic regulations. In the pantheon of management-consultancy techniques this is the domain of business-process re-engineering and its use, together with the growing power of technology, improves reliability, etc, etc. Process re-engineering and legitimate best practice both rely on stable repeating relationships with information or appropriate materials available for each decision point, such decisions being rule based (eg, if X and Y but not Z then take action A). The model is a cybernetic one and cybernetic models are de facto applied to human decision making where humans exist in the process.

But this is where things usually go wrong, for two reasons. First, humans do not make rational, logical decisions based on information input. Instead they pattern match with either their own experience or collective experience expressed as stories. Often this isn’t even a best-fit pattern match but a first-fit pattern match.[1] The human brain is also subject to habituation; things that we do frequently create habitual patterns that both enable rapid decision making and entrain behaviour in such a manner that we literally do not see things that fail to match the patterns of our expectations.

Second, not all systems are ordered in the sense of repeating and empirically verifiable relationships between cause and effect. In complex systems, patterns emerge as a result of multiple interactions between agents and only repeat by accident; they are coherent in retrospect but not in advance. It is easy to be right with the benefit of hindsight, but to define best practice on the basis of past events in a complex system represents folly, especially as most matters relating to human and market behaviour are complex, not ordered. As understanding of complex systems is of increasing importance to knowledge management.[2]

Left to their own devices humans are remarkably good at dealing with this lack of order. Indeed, pattern recognition, the ability to manage patterns and our ability to store knowledge in the external scaffolding[3] that humans erect around their social systems, is at the heart of human intelligence. We use social networks and various other clues to guide our future behaviour; we do not work on the rule-based approach used by computers.

One of the basic validation techniques used by humans is to create the conditions for serendipity. A question asked to the same conference audiences: given a difficult project, one of those you couldn’t get anyone else to volunteer for, do you drawn down best practice from the organisation’s knowledge-management system or do you go and find five or six people you trust/respect and ask their advice and listen to their stories? The answer is not always for the latter option; there is always the occasional KM professional with a vested interest in arguing in favour of the former, but the (admittedly anecdotal) evidence is overwhelmingly in favour of the stories. We actively seek out multiple encounters to increase the probability of an emergent solution that does not just repeat the past, but also opens up new possibilities.

We also work very strongly on the basis of using shared context to determine confidence in future actions. A third, but this time rhetorical, question to my conference audiences serves to illustrate this. The audience is asked to imagine three scenarios:

First, a person you have known for years and worked with on many occasions, both good and bad, phones you up to ask a question. You know and trust the person and you have no inhibition answering. You know what they mean by the question and you know how they will understand the answer;
Second, another member of the organisation phones you up and asks the same question. You have no prior knowledge of this person and no experience working together; you have no shared context. Your first task is to create a context: you ask a question, respond to the answer, compare experiences and at the end you share knowledge, but the sharing is inhibited: ‘try this and then give me another call’; ‘if any of the following happens contact me straight away’; ‘I’ll come by and help you get started’;
Third, some idiot with the title chief knowledge officer comes along and asks you to write down what you know without any context.

The point is a very simple one: shared context is vital to knowledge exchange, and such context always involves some trusted human validation. This is not to say that codification of material in advance of need is not advantageous, but the effective reference is nearly always human. We do use written material – it represents reflective knowledge and has value – but we normally check out what is or isn’t relevant within a trusted network.

The process of codification of knowledge is one of abstraction. As we rise through successive levels of abstraction we have richer and richer conversations with fewer and fewer people. This is understood in the context of expert language by most knowledge-management practitioners, but what is often neglected is that, for humans, abstraction (and therefore shared context) includes common past experiences, beliefs and values. These common assumptions are rarely stated, because they are mostly taken for granted. One of the related problems with best practice is that when people communicate, they often forget the degree to which they have relied on shared common experience of which they are only partially aware. The all too frequent response is ‘I thought you knew that?’ or some variation on the theme.

Is there a best way to do something?

In an ordered system a ‘best’ way is theoretically possible, as we are dealing with repeating relationships between cause and effect. If we are dealing with a complex system there is no such repetition. Even in an ordered system the degree to which we understand the relationship between cause and effect determines the degree to which we can define best practice. This is true even of scientific knowledge where serendipity is frequently the cause of major breakthroughs, and where old knowledge frequently uses best practice to exclude new thinking. My favourite example of this latter tendency is the Longitude story, in which the clockmaker Harrison is ignored by the scientists of the day for over a decade because they are convinced that measuring the distance between the moon and the earth is best practice, and attempting to create a clock that keeps accurate time on board a ship represents an illegitimate approach from someone who is not a ‘real’ scientist. The Longitude story is repeated all too frequently in the day-to-day life of organisations. For complex systems, best practice is dangerous; for ordered systems it is valid, but not universally and only in very stable situations; in all other cases it is entrained past practice.

Can we codify knowledge?

Now let us assume that there is a situation in which there is a right way to do things, a way that is appropriate more than once and that can be discovered. The next question is, can it be codified in such a way that someone else can pick it up and use it? I am using the term ‘codify’ here in the sense of writing things down, as this is the most common approach to best practice. One of the basic rules of knowledge management is that we always know more than we can say and we will always say more than we can write down. The loss of content, but particularly context, involved in codification means that written knowledge is only ever a partial representation of what we know.

Will people follow best practice?

I remember when I was in primary school and a nine year old from the next class up was sent to read his essay to each class in school. The headmaster had decided that the essay, based on a fox hunt from the fox’s perspective, was a model that we all should follow. The essay produced several reactions. The sycophants in the class all proceeded to write essays about fox hunts from the fox’s perspective. Those subject to the tyranny of the green-eyed god speculated that his mother had written it, while a small group of unmentionables took him round the back of the bike shed for the treatment normally accorded to teacher’s pets. There is very little difference between the average eight year old and most employees in respect of their appreciation of something held up to them as best practice. Some of those who were involved in the project being touted as best practice may feel that essential facts have been left out. Others may resent the fact that key aspects of work that they contributed have been omitted. True, if someone I respect and trust does something or recommends something then it will achieve results, but that level of trust will never transfer to a ‘system’.

There is also a major question as to transferability of best practice. Weick and Sutcliffe argue that there are lessons from the behaviour of fire-fighting crews, aircraft carriers and the like relating to openness to failure than can be applied in industry.[4] This is idealistic to say the least. The context that creates the need for failure sharing in a crew of fire fighters is not common to most organisations. In a crisis, all organisations tend to increase levels of trust; it’s a natural human reaction. Yet to have an organisation maintain that level of openness on an ongoing basis means they would constantly be lighting fires. Context is the be-all and end-all of knowledge management.

Should people follow best practice?

Even if we can define best practice and assuming we can mandate and ensure conformance, there remains the question as to the desirability of such conformity. To return to my childhood experience of the essay on fox hunting, one reaction was to imitate the essay rather than to use it as an example to stimulate original writing. The worst offenders are those who follow best practice uncritically on the grounds that they cannot get fired for doing so. In one project I ran some years ago, removing artificial intelligence in a computer-based best-practice system enabled experts to apply their knowledge. A previous project had sought to capture expert knowledge and codify it into a system. The net result was that going with gut feel when this went against the computer recommendation was dangerous to your career, while following the computer recommendation meant that there was someone else to blame.

An alternative to best practice

Let me make something very clear: there is a legitimate and valid domain for best practice. Few of us would want any ambiguity or active learning in respect of internet payments or in safety procedures in a nuclear power plant, for instance. Best practice is an important knowledge-management function. Admittedly, it requires discipline, time and resources, and in many cases we simply cannot afford the associated costs for anything other than a limited number of cases. It’s rather like mission-critical software development, where two teams work in parallel on the same code and then compare results. It’s expensive, but for, say, an air-traffic management system, the costs is justified.

However, the range of circumstances in which we can really afford to invest in best practice is limited, even when it is appropriate, so we need to turn to other tools and techniques. It’s worth remembering that the primary purpose of knowledge management is to enable better decision making and to create the conditions for innovation; in turn, better decision making is contingent on active learning and innovation is dependent on the disruption of entrained patterns of thinking. In this final section I would like to look briefly at some of these alternatives, reflecting on current research and experimental consultancy within the Cynefin Centre.

Narrative databases

We normally learn by hearing stories from diverse sources, synthesising the learning with our current situation and determining a plan of action. Properly constructed narrative databases work on the basis of managed serendipity, enabling multiple and unexpected encounters with original anecdotal material. As such, they reflect natural learning processes, but with the advantage that we are not confined to people we can talk to as a source of stories. One growing area of application is for retired or retiring employees. Many will not write down what they know, but boy will they tell stories. Interestingly, many people entering this area cannot resist the desire to interpret people’s stories. They want to tell employees which stories they should hear and what those stories mean. A true narrative database uses only original material and searches this material based on abstract questions that discourage directed inquiries to create serendipitous encounter.[5] Narrative databases can be a first-entry knowledge-management system; observing the patterns of use can determine where investment in best practice might be best focused.

Social-network stimulation

Too many people focus on managing knowledge rather than on managing the channels through which knowledge flows. Just connecting or linking people can be a major knowledge-management activity. Mentors provide for this functionality but new tools now allow us to telescope five to six years of social networking down to five or six weeks, albeit with less density. Such programmes aim to create linkages where none exists, and are particularly useful during reorganisations and activities such as mergers and acquisitions. The key point to emphasise here is that the learning model is top down in respect of the heuristics and boundaries that govern the creation of the social network, but the membership of the network is self-generative and voluntary in nature. Attempts to engineer a network through design and allocation of staff to groups generally fail, as they create artificial relationships that are not sustainable. Self-selecting social-network stimulation replicates, but in a shortened timescale, a natural process.

As an aside, a lot of KM practice observes natural phenomena and then tries to abstract them into a formal process. Many communities of practice are set up on this basis. The problem with this is that the circumstances surrounding a particular natural process can never be fully known. We need to re-create the context to stimulate a similar occurrence, but as human systems are complex the stimulation will also produce a new pattern, hence the use of heuristics and boundaries to influence and direct the formation of those patterns.

Disruptive pattern breaking

A large amount of learning does not require us to communicate knowledge, be it best or good practice, but rather to disrupt established knowledge. I have argued elsewhere that formal communities of practice need regular and ritualised disruption to prevent entrained thinking. There is nothing quite so as conservative as a deep expert. We can also introduce disruption in a narrative database by introducing unexpected material, perhaps from history, which creates a new perspective when a story about a current situation is encountered. In more advanced cases under development we are starting to build experimental narrative filters in which the user is forced to see things from radically different perspectives for application in everything from foreign policy to sales practice. Providing new perspective can create new understanding and prevent negative pattern entrainment. Other advanced applications utilise game environments working with science-fiction writers and alternative histories to create a disruptive metaphor to allow people to encounter things indirectly through the metaphor rather than dealing with reality, which can often be painful. This process of displacement leads to another narrative technique based on ancient practice in which archetypal story forms, utilising archetypal characters that have emerged from the water-cooler stories of an organisation, can enable people to confess to mistakes without attribution of blame through the medium of stories told about the archetype.

Efficiency does not necessarily lead to effectiveness

The main focus in process-re-engineering and, to a degree, knowledge-management practice has been increased efficiency. The pursuit of efficiency lies at the heart of the concept of best practice: if there is a best way to do something, it is surely more efficient for all agents within a system to follow this process. Unfortunately, while efficiency does achieve effectiveness in mechanical or highly structured human systems, it does not in respect of the majority of human interactions, which, as previously stated, are complex in nature.

An interesting feature of complex systems, particularly in social insects, is that for a system to be effective there needs to be a degree of inefficiency in the operation of its agents. Humans are the same: the efficiency focus of best practice harms effectiveness because it assumes repeatable past patterns of cause and effect. Driving out inefficiencies increases vulnerability to new threats as the adaptive mechanism of the complex system has been withdrawn. Indeed, in using narrative we are building worst-practice systems that are both more popular in facilitating voluntary access and more effective in creating learning within an organisation.

Best practice has a space in knowledge management, but the space is small, highly specialised and generally expensive. On the other hand, creating a learning ecology that bounds but recognises diversity is another matter altogether. Here, the dynamics of human interaction and inquiry can be built to permit both better decision making and, though the active management of serendipity, enhanced levels of innovation.

References

1. Klein, G., Sources of Power: How People Make Decisions (MIT, 1998)

2. Snowden, D., ‘Complex aspects of knowing: Paradox and descriptive self-awareness’ in the Journal of Knowledge Management (Vol 6, No 2, May 2002)

3. Clark, A. Being There: Putting Brain, Body and the World Together Again (MIT, 1997)

4. Weick, K.E. & Sutcliffe, K.M., Managing the Unexpected (Jossey-Bass, 2001)

5. Snowden, D., ‘Narrative patterns: The perils and possibilities of using story in organisations’ in Knowledge Management (Vol 4, Iss 10, Ark Group, July/August 2001)

Dave Snowden is director of IBM’s Cynefin Centre for Organisational Complexity. He can be contacted at snowded@uk.ibm.com