Feature

posted 1 Oct 2009 in Volume 13 Issue 1

The status of lessons learning in organisations

A glance at the results of recent research, conducted by Knoco Ltd, into lessons learning objectives and processes at national and international organisations. Further analysis will follow in the October issue.

Methodology
In summer 2009, Knoco Ltd conducted a survey of the state of lessons learning in organisations, using an online questionnaire. The purpose of the survey was to gain some background data for the forthcoming book, The Lessons Learned Handbook: a Practical Knowledge-Based Approach to Learning from Experience, which is due to be completed at the end of 2009.

Seventy-four responses were received. The organisations represented fell into the following categories, with 11 respondents not identifying their organisation: academic (one); automotive (one); aviation (two); consulting and services (nine); engineering and construction (seven); insurance and banking (two); IT (four); legal (two); manufacturing and sales (five); military (four), mining (one); oil and gas (ten); pharmaceutical (four); public sector (seven); and, retail (one).

Prevalence of lessons learned systems
Seventy-six per cent of respondents said that their organisation had a lessons learned system in place in at least one major part of its activity. A further seven per cent were in the process of introducing one. Six per cent had previously had a lessons learned system, but had stopped, while 11 per cent have never had a system in place.

According to the research, lessons learned systems seem to be most common in the oil and gas, military and engineering and construction sectors (see Figure 2). Responses suggested they were also popular in mining and retail, although in this instance the numbers were too small to be sure.

The respondents were also asked which part of their business applied learnings and at least half applied lessons learned within the project context. Answers included: project management (24 responses); all activity (seven responses); software deployment and release (four responses); bidding and pitching (three responses); industrial safety occurrences (three responses); and, operations (two responses). Other respondents also used lessons learned within the following scenarios:

Service improvement and related IT projects;
During major, ongoing change programmes;
Before, during and after consulting assignments;
Monthly e-mail campaigns;
Lean or Six Sigma programmes;
Compiled as a part of most engagements; and
Supply chain.

Effectiveness of lessons learned systems
Next, the survey asked those respondents who had (or were introducing) a system, to rate its effectiveness, using a score between 0 (not at all effective) and 5 (excellent). The responses are shown in Figure 3.

Six per cent rated the system as 0 (not at all effective);
Six per cent rated the system as 1 (slightly effective);
Forty-eight per cent rated the system as 2 (moderately effective);
Fifteen per cent rated the system as 3 (good);
Eighteen per cent rated the system as 4 (very good); and
Six per cent rated the system as 5 (excellent).

Figure 4 shows the average effectiveness rating for each industry group, set within the range of ratings between high and low scores. The highest average ratings were from retail (one single score), consulting and services, manufacturing (with a very wide range), military, and oil and gas. The highest single ratings (excellent) were found in manufacturing and military.

Success factors
Participants who scored highly (3, 4 or 5), were asked to identify success factors that resulted in a high score. Reponses were relatively varied, with respondents citing consistency of application, honesty, gaining buy-in and governance (encompassing the process, systems and people aspects of KM) as key to successful deployment of lessons learned processes. Also essential were:

Ensuring participants in the review process covered a wide range of perspectives;
Devising effective action plans after the lessons learned activity;
Small group discussions coupled with larger group consolidation;
Constant communication;
Clear specification of what the evaluation needs to achieve (objectives);
High-level management and stakeholder involvement; and
Team evaluation and consultation.

Barriers
Participants who scored low (0, 1 or 2), were asked to identify the barriers which resulted in a low score. Several common factors were identified.

Senior management (11 responses) including:

Lack of senior-level support;
Lack of buy-in/commitment; and
Lack of ownership/leadership.

Cultural issues (10 responses) including:

Resistance to flagging ‘bad news’;
Not seen as central to performance management; and
An attitude of ‘this is how we have always done things, and we will not change’.

Lack of follow through and application (15 responses) including:

Not part of the process improvement process;
Poor dissemination and use of lessons learned; and
No recognised process.

Other barriers included:

Time issues;
Lack of governance;
Concerns about litigation; and
Lack of a central repository or access point.

Ranking the components of the lessons learned system
The respondents were given a list of components of a lessons learned system, and asked to identify whether they applied these components. The frequency of application of each component is shown in Figure 5.

The most common is the use of a defined process for identifying lessons from activity, with 46 of the respondents (80 per cent of those with a lessons learned system) operating in that way. The least common was the use of rewards to incentivise lessons submission.

Respondents were also asked to correlate these components with the effectiveness score for the KM system – for example, those systems that included the definition of actions arising from lessons learned scored approximately 3 on average. Those that did not had an average score of 2. Thus, it can be assumed that including such definitions and actions makes the process more effective.

Generally speaking, all except four components seemed to make a significant positive contribution to lessons learning. They can be grouped as follows:

Strong positive contribution

Actions defined arising from the lessons;
Clear high-level expectations from senior management that the lessons learned process will be applied;
A method to measure whether actions have been completed and lessons closed out;
A process for validating and agreeing the actions;
Accountable person or people assigned to complete the actions; and
A defined process for identifying lessons from activity.

Moderate positive contribution

A person or people to track the metrics;
An escalation method if the lesson or action needs to be addressed at a higher level;
A clear accountability for identifying lessons from activity;
A high-level sponsor of the lessons learned process;
Quality assurance of the process – for example, trained facilitation;
A method for disseminating the lessons; and
A lessons learned database, which can hold lessons from multiple projects or units.

Fairly neutral

Quality control of the lessons to ensure they are well written;
A method to measure whether lessons have been captured; and
A search function within the lesson database.

Strong negative contribution

Rewards to incentivise submission of lessons.

Missing components
When asked which components were missing from this list, respondents pointed to such elements as:

A knowledge baseline against which to understand whether or not a ‘lesson’ is something that should be widely known anyway;
A method for incorporating lessons into current practice;
A filtering process – quality control over lessons to be followed up on;
Improved accessibility to lessons; and
Communication – a lack of awareness.

Lessons identification method
Respondents were also asked to list the methods they use to identify lessons. Many indentified more than one method, although after actions reviews (17 responses) and other types of project-related review (28 responses) were popular choices. Such reviews included:

A specific chapter in the project completion report;
Board meetings at the end of a project or programme;
Formal processes associated with project execution;
Retrospects;
Post-project workshops; and
Post (or milestone) debriefing of client projects.

Other methods employed to share lessons learned were: external reviews and benchmarking (six responses); learning from incidents and events (five responses); individual or ad hoc submissions (seven responses); and other processes such as:

Reward and recognition;
Post-sales pitch reviews;
Information exchange with partners;
Online community discussions; and
Case studies, observation and rapid improvement events.

In summary
It seems evident from the responses to this survey that a large proportion of organisations are attempting to implement (or are intending to do so) a lessons learned system in some part of their business. However, less than half are actually satisfied with the effectiveness of this system.

No one industry sector can be shown to have ‘got lessons learning right’. Certainly, it seems more prevalent in the oil sector and the military, but even there satisfaction ratings are not uniformly high and survey responses are too few to be certain.

In most cases, lessons learned is being applied to project activity and project-related team dialogue processes, such as after action reviews and retrospects. These are commonly used to identify lessons, together with incident investigations, external evaluations and individual submissions.

However, there are many barriers to operating these processes, and even more difficulties in actually following through with the learning and making a difference to the work of the organisation. The issue of re-use and re-application of lessons is a constant theme in the responses.

It seems that effective lesson learning contains many elements, each of which has a positive impact on the success of the system; and, that a successful system needs to incorporate as many of the elements as possible. These elements are cultural as well as procedural. The most important things to get right seem to be:

Ensuring that lessons lead to action, and that these actions are followed through to application in future projects. It is probably the lack of follow through that causes the greatest frustration;

Constant involvement from senior management, with clear expectations that the lessons learned system will be applied. Without senior management attention, time for lessons learning is not prioritised, or it is treated as a tick-box activity;

Formalising, defining, embedding and consistently applying the system (and there are sub-issues here, for example accountabilities and avoiding the ‘tick box’ mentality); and

A supportive culture – this will be driven largely through the behaviours of leadership, and by the importance they place on lessons learning.

Learning lessons seems to be an activity that the majority of organisations seek to do, but it is something that seems to be very easy to do badly. If a company wishes to learn effectively, it needs to address lesson learning as a complete system. It must ensure that all elements of that system are in place – from identification through to re-application, and also that it has the full backing and attention of senior management. From this foundation, success should be possible.

For further information, a full list of questions or the complete survey results as a PDF file, visit the ‘Downloads’ page at http://www.knoco.com or contact Nick Milton at nick.milton@knoco.com.

Sidebar: Further insight
Respondents submitted further insight into their lessons learned capabilities and processes as part of a final, open question. Many of these are thought provoking and offer some insight into why lessons learned systems might not always be as effective or useful as they could be, as well as offering ideas as to how to make lessons learned processes work better.

“As always, the business does not provide the time to staff to learn from experience. There is always pressure to meet some deadline or get onto the next revenue driven piece of work.”

“Lessons learned is part of a broader process… and how this process is embedded within the organisation’s culture is perhaps what makes it work (or not).”

“Process take-up and adoption is very difficult. In the end it is all down to the individual. Few people seem inclined to participate either as a ‘pusher’ or ‘puller’. Pull is critical to effective lessons transfer. Systemic ‘pull’ only seems to happen under circumstances when such is demanded and followed through by leadership, or an individual feels out of their depth. Thus from the start there are big barriers to overcome in the lessons process. Many people also struggle to write clear lessons and unwittingly stray into tacit territory leaving the reader lost in terms of what the message is supposed to be.”

“Action on the items seems to be hit-and-miss; we have many great discussions, but follow-up is nonexistent at times.”

“Nobody likes to hear bad news. People can be very reluctant to accept that what they did was not optimal.”

“Documentation of lessons-learned is often lengthy and tedious to both write and read. There is a tendency, therefore, to use personal experience relayed at the start of a subsequent project, rather than referral to documentation or an archive.”

“Lessons learned is a good idea and helps pass knowledge onto new generations.”

“The key to lessons learnt is feeding them back into the organisation so that improvements can be made. I don’t see the point in having a database full of lessons learnt, that no one can access.”