Feature

posted 20 Jul 2004 in Volume 7 Issue 10

Achieving a healthy KM assessment

When implementing a KM programme, many organisations employ knowledge audits as a means of gauging the initiative’s success. Ana Neves shares her experience of a knowledge-audit exercise recently carried out at the NHS Modernisation Agency and reflects on the impact it made.

A knowledge audit can offer an evidence-based assessment of an organisation’s knowledge status. It can help frame knowledge strategy, assess progress, and facilitate strategic and operational buy-in. It can throw a spotlight on knowledge and information, infrastructure, processes and culture.

Many organisations accept that knowledge is the basis of innovation and that organisations need to innovate in order to get ahead of competitors. Whether or not your organisation has begun a knowledge-management programme, it is important to be aware of its knowledge-related requirements. It is also important that organisations find a way of assessing the impact of their knowledge initiatives.

The inertia exhibited by those organisations that have not yet warmed to the idea of managing their knowledge often comes from the inability to answer two critical questions: ‘where do I start?’ and ‘how do I measure progress?’ Other concerns often regard measuring the return on investment (ROI).

No matter what stage your organisation is at, a knowledge audit can help you. What follows is an introduction to knowledge audits (what they are and what their objectives may be) and reflects my own personal experience of carrying out a knowledge-audit exercise at the NHS Modernisation Agency.

Knowledge audits

A knowledge audit is a qualitative evaluation, a sound investigation into an organisation’s knowledge ‘health’. It is about understanding the knowledge required to deliver the organisation’s core business; being aware of the knowledge available and the processes to store, retrieve and share it; and identifying the missing knowledge and the processes to receive, generate and acquire it.

I would like to extend this definition, suggesting that a knowledge audit can equally be used to understand more about the organisational context (culture) and the organisational support (infrastructure) for the knowledge processes.

Knowledge audits can have different foci:

Knowledge and information;
Knowledge-related infrastructure;
Organisational processes;
Culture.

Knowledge audits that focus on knowledge and information are the most common. These audits aim to identify:

The knowledge and information the organisation requires to meet its business plan and strategic intents;
The knowledge and information currently held by the organisation and its individuals;
Any gaps in the organisation’s knowledge and information;
Existing knowledge and information sources (internal and external).

These audits take a huge amount of time and effort.

A knowledge audit that focuses on infrastructure is looking at the ‘things’ the organisation has in place that may impact on its knowledge processes. These audits should find the answers to questions such as: ‘does the organisation have explicit knowledge-management roles?’, ‘does the organisation have an electronic knowledge-sharing environment?’, ‘are the offices open-plan?’ and ‘is there a large percentage of home-workers in the organisation?’

Although these questions do not necessarily represent problems or difficulties, it is important that the organisation knows the answers and takes this information into account when creating its knowledge-management strategy.

If the knowledge audit is focusing on existing organisational processes, it will be trying to understand, for example:

How people learn together and innovate;
Where teams find the knowledge they need to do their job;
How teams share their experiences with each other;
If the organisation encourages people to share or to hoard knowledge.

Finally, a knowledge audit that focuses on culture hopes to identify whether:

People are willing to share;
People are keen to learn from each other;
People feel comfortable discussing past mistakes in order to improve future performance;
Trust predominates in the organisation.

There is no clear dividing line between these four focus areas. However, distinguishing your objectives is important in order to clarify expected outcomes and determine which methodology to adopt.

Knowledge audit in the NHS Modernisation Agency

The NHS Modernisation Agency has been considered as a catalyst for change in the NHS since it was created over three years ago. Established in April 2001 as a result of the NHS Plan, it helps to ensure that investment throughout the NHS is matched by the reforms necessary to provide its users with the highest quality of service.

Operating across all sectors of the NHS – acute trusts, primary care and mental health – and the whole performance spectrum, the system-redesign work of the agency is underpinned by the major principles of patient safety, quality leadership and workforce development.

The NHS Modernisation Agency’s vision is to become a world-class knowledge organisation. Knowledge focused on improving health and social care is the agency’s core competence and, by effectively managing this expertise, we can greatly improve the way care is provided in the UK.

The knowledge-management efforts at the NHS Modernisation Agency started in 2002. Since then, the organisation has started a KM/IT Forum, supported communities of practice, worked with teams in helping them to address their knowledge-related issues, used social-network analysis and piloted knowledge-harvesting techniques.

Despite the huge amount of work carried out, the organisation does not have an evaluation process for its KM efforts. Nor does it have a benchmark from which to assess the impact of this work.

The Department of Health and the National Health Service are, more than ever before, committed to evaluating and showing the benefit of the work they do. Every NHS organisation and every NHS programme must clearly show the impact it is having on patients, carers and staff. The implementation of a new KM strategy in the NHS Modernisation Agency is no exception.

Objectives

With this in mind, the audit’s objectives were to:

Understand how equipped the NHS Modernisation Agency was to become a world-class knowledge organisation;
Inform the organisation’s knowledge strategy;
Create a benchmark to assess future impact;
Understand the risks of failing to implement the knowledge strategy;
Identify areas needing extra attention;
Identify pockets of good practice.

A knowledge audit can equally be used to obtain people’s buy-in, and to demonstrate the need for action and resource allocation.

Preparation work

Preparation is crucial to a successful knowledge audit. The agency began by investigating the work of similar organisation through reading case studies and talking to people who had led similar exercises elsewhere.

The next step was to identify past, ongoing and planned projects within the agency that could have caused overlap or informed the audit. The agency identified initiatives such as a customer-satisfaction survey and an evaluation network, which helped decide upon audit questions and allowed the agency to act as knowledge brokers during the audit. This facilitated improved awareness and established connections, which demonstrated the effectiveness of the strategies advocated.

The final step of the preparation work was to engage internal and external support. Director-level approval and a small budget allowance were secured and colleagues from other teams and external consultants were brought in to help with the audit.

Methodology

With very little time to draft a strategy and with 40 per cent of our staff being peripatetic or home workers, focus groups were not possible, so we decided to go for 60-minute, face-to-face interviews with senior people from across the agency. We needed senior-management input to give us a feel for strategic planning in order to closely align our knowledge strategy to the strategic intent of the whole organisation.

The interviews were semi-structured, covering topics such as finding and retrieving knowledge and information, knowledge sharing, promoting innovation and spreading good practice, and skills and support. Most of the questions were open, although some were closed and invited interviewees to provide their opinion on a numbered scale.

The knowledge audit focused on processes and culture. However, the questions were all about organisational processes, for example, ‘how do you get hold of the information you need?’ and ‘how do you share the outcome of your work?’. We avoided questions like ‘do you trust your colleagues?’ and ‘do you like to learn from others?’ We deduced that the best way to understand the culture would be to carefully analyse the responses rather than pose direct questions. For instance, although we did not ask anything about people’s willingness to learn from each other, it became clear that people derive more pleasure from finding a solution by themselves. While many interviewees recognised they did not share as much as they could, none mentioned how willing others were to share their expertise.

Analysis and report

The final report included:

The main themes that came up during the interviews;
A list of problem areas, including associated risks;
A list of recommendations to address identified problem areas;
Identification of areas of good practice;
An assessment of the organisation’s readiness to become a world-class knowledge organisation.

The report was heavily populated with anonymous quotes gathered during the interviews, giving voice to the organisation as a whole rather than to the team that carried out the exercise. The report invited everyone to send their views to the team. In particular, it encouraged people to prioritise the list of recommendations.

Outcomes

When the agency’s chief executive received the final report he acted upon it immediately. He not only e-mailed the team with his appreciation, he also copied in the senior-management team, encouraging it to read the report and embed the recommendations into its business plans. Some private messages were also received congratulating the team and applauding their efforts to clearly identify problem areas with the agency’s KM strategy.

Lessons learnt

Although it was a conscious decision to only interview senior employees, we appreciate that interviewing front-line staff would have provided a more complete picture, especially in terms of the organisational culture. It would also have ensured organisation-wide buy-in.

Because of resources (people and time), in order to include a representative group of front-line staff in our audit, we would have had to consider other auditing methodologies: questionnaires and focus groups, for example. Questionnaires could also have helped to gather more quantitative data with which to populate the report.

Another lesson learnt through this exercise is not to underestimate the time required to analyse all the interviews. In order to truly appreciate their richness and to squeeze out the hidden organisational cultural hints, you need to devote sufficient time to analysis.

By engaging external consultants to help us draft the interview template and by carrying out the interviews ourselves, we were able to acquire new, useful skills and reduce the potential cost.

As a first attempt, this was an extremely positive exercise for the NHS Modernisation Agency and we plan to

repeat the exercise in one year’s time. We recognise that the findings represent the view of the senior team, rather than the majority, and have calculated this discrepancy into the strategy.

A job well done

The NHS Modernisation Agency undertook this audit after having experimented with and embedded a variety of ad hoc knowledge-management initiatives. It has been a useful exercise, as it confirmed the legitimacy of some of the ongoing initiatives and suggested new ways forward. The audit offered a mirror, reflecting back what the organisation, rather than the KM team, considered relevant.

However, I believe that this exercise would have been even more useful had it been carried out as a starting point for the conscious KM activity in the organisation to assist us when mapping out our KM goals. It would certainly have been easier to secure appropriate resources and even more commitment from senior management. What is more, we would now be in a position to evaluate the progress using the initial audit as a benchmark.

A knowledge-management programme will achieve maximum impact when all of its components are united by a common, known objective. A knowledge audit provides a context to define that objective and a basis from which to plan the implementation of each component.

Whether yours is a private or a public-sector organisation, it is worth considering a knowledge audit. Be clear about your audit objectives, ensure you have senior-management support, secure enough resources, do your homework and define your methodology before beginning the audit process. It might be the initial step on your KM journey, or the missing one on the journey you’ve already started, but be reassured that the benefits reaped will far outweigh the initial effort.

Acknowledgements

I would like to thank my colleague Annette Copper, who was a critical friend both during the audit and while writing this article.

Disclaimer

This article reflects the views of the author and not those of the NHS Modernisation Agency.

Ana Neves, knowledge network manager, NHS Modernisation Agency & editor, KMOL portal
ana.neves@npat.nhs.uk/editor@kmol.online.pt