Measuring the state of KM

Robert Hoss and Art Schlussel begin a series of articles on KM measurement by examining how to assess the state – or maturity – of KM. In the first instalment, they look at useful metrics and provide an overview of the US Army’s KM Maturity Indicator.

The degree to which an organisation effectively applies the art of knowledge management (KM) – creating, organising, applying and transferring knowledge to facilitate understanding and decision making – is an indicator of its KM maturity. Measuring the state of KM maturity provides a baseline from which to build KM. Developing metrics that assess KM impact and value is both essential and difficult. It is a topic that has often been studied, written about and debated. However, we have found little in the way of practical methods that are easy to understand and apply. This series will provide practical ways to measure the state of an organisation’s KM strategy, its initiatives and their impact.

We will explore practical ways to measure the state of KM in an organisation, examine accepted KM initiatives used throughout the KM community, and identify useful metrics for those initiatives from a military perspective – although civilian organisations will also benefit from this approach. Useful metrics are those that a chief knowledge officer (CKO) or knowledge management officer (KMO) can immediately apply to measure the state of KM at their organisation.

Metrics/measures and key performance indicators
Metrics, also known as ‘measures’, or key performance indicators (KPIs), assess the effect of a particular project or activity. The most important characteristic to consider when choosing or defining a KM performance metric is whether it indicates if knowledge is being shared and used. For example, a metric for a best-practice database might be the number of times the database has been accessed. A large number of accesses or ‘hits’ suggests that people are reading the document, but this does not definitively indicate whether it was useful to anyone or whether or not it improved operational efficiency or quality. A better metric would be to track database usage and ask a sampling of the users how it helped them1.

Organisations should measure what matters. Measuring for the sake of measuring is fruitless and a waste of time. It is important that measures and metrics be developed and collected for the purpose of continuous improvement of KM activities2. One method is to collect stories that explain metrics. For example – telling a story of how KM improved organisational efficiency by explaining how metrics were developed, collected and analysed, is extremely valuable. After data is collected, it is important to post the results and analyse them. When we can show leaders and employees that KM initiatives produce results, this will result in greater buy-in to the use of those initiatives.

Not all metrics are useful
Some metrics don’t really tell you anything of significance and will lead an organisation to false results. This article also lists metrics that should be avoided. An example of an ineffective metric is ‘time savings’, as it won’t necessarily lead to an employee using their ‘saved’ time productively. Typically, time savings are calculated as:

(x hours per employee saved) x (number of employees) x (cost per hour)

Leadership might respond with: “Saving me 4.6 hours of productivity per employee per week, means each employee gets to duck out of the office at noon on a Friday. Where’s the tangible benefit? Unless you show me how this leads to needing less unit resources per task and therefore a reduced headcount, I’m not going to see any real savings.3”

What gets measured matters and gets done
Knowledge is an intangible asset, but the impact of KM is measurable2. Measures need to link to the organisation’s strategy. Note: a successful KM programme usually involves multiple KM initiatives. There needs to be a blended approach using several of the KM initiatives listed in this article, plus others that are important to your organisation.

Metric performance should be compared to a baseline or benchmark
Using metrics without a standard (a benchmark or current level of performance) or target (goal) is like driving a car without a map, and without seeing the road. Before implementing any KM initiative, key metrics and a baseline against which performance may be measured during and after implementation, should be developed. Even if metrics development occurs after a KM initiative is underway, you will still need a baseline to answer the common questions, such as:

What kind of progress are we making?
When can we move to the next phase? and,
What do the users think?

As a result, the first metric collections may be expressly to establish the critical baseline for comparisons. However, do not use the fact that you lack a baseline as the reason not to use metrics. Many metrics projects have stalled at precisely this point, in the face of the potential effort of establishing a baseline4.

Know what your desired endstate should be before you begin your KM initiative
Why else would you perform the work? Look backwards from your end-state to find the metrics and indicators that will tell you when you have arrived at your destination. Should the indicators increase or decrease? Should they go up or down? Should they be more or less? Just as having a benchmark provides you with a starting point, having an end-state in mind gives you a place to call success.

Consider the culture when you choose your measures
What type of culture do you work in? Are metrics, measurement, and monitoring part of your organisational make-up? Do you already measure activities, cycle times, expenditures, and so on? These questions should be considered when developing ways to measure your KM initiatives and the results of those initiatives. If your organisation already has measurement mechanisms in place, try to integrate your KM metrics into the existing system. Tying them to a system everyone already understands and is comfortable with, will ease the pain of metrics gathering. If your organisation does not routinely perform measurements consider ways to gather the metrics that would be most palatable. Can they be automated, or performed as part of an existing process? Also, consider how metrics should be reported. Does your organisation value statistics and charts, or does it favour stories and examples that convey the essence of the initiative. Finding the right fit for capturing, analysing and reporting metrics is as important as the KM initiative itself, because metrics convey the results of the resources expended to achieve the desired end-state.

One size does not fit all, but…
A review of the KM metrics literature says that one size KM does not fit all organisations. Research reveals that there are common KM initiatives that most successful businesses with a KM programme use. This article looks at common KM initiatives and metrics to measure them. Those KM initiatives listed are not all inclusive and there are no guarantees that these measures are the most appropriate for your organisation. These metrics describe what you can do, not what you must do, or even should do. Select the KM initiatives and measures that matter to your stakeholders5. There are many more additional KM initiatives that can also be used. See Appendix C for more KM initiatives. [The appendices for this series can be obtained in PDF format by contacting the editor at kclifton@ark-group.com or viewed in HTML by visiting www.ikmagazine.com].

The easiest measures can be attained from process and IT applications, but just because we can easily obtain metrics on a KM initiative, it doesn’t mean that metric provides a useful way of measuring the maturity of that initiative.

Categorising measures
Measures can be categorised as hard (financial savings) and soft (intangible measures).

Hard (financial saving metrics):

Profitability, increased revenue or decrease in maintenance costs;
Money saved/cost reduction;
Time saved;
Quality improvements, number of errors avoided, cost avoidance;
Increased productivity;
Successful mission; and
Products successfully launched.

Soft (intangible measures):

Cost avoidance;
Customer satisfaction;
Quick problem resolution;
Professional development;
Improved employee satisfaction/improvement in morale;
Improved knowledge retention;
Capturing and retaining at-risk data;
Stories related to validated success;
Enhanced innovation;
Improved skills/competency; and
Trust.

Common measures
The following measures can be used for most KM initiatives.

System metrics
System metrics monitor the usefulness and responsiveness of supporting technology. They give an indirect indication of knowledge sharing and reuse, but can highlight which assets are the most popular and any usability problems that might exist and be limiting participation. They are the easiest to collect from software systems and include:

Page visits;
Contributions; and,
Number of members.

Output metrics
Output metrics measure characteristics at the project or task level, such as the effectiveness of lessons-learned information to future operations. Direct process output for users provides a picture of the extent to which personnel are drawn to actually using the knowledge system. For example:

Replies to discussions (online, e-mails, phone calls); and,
Documents downloaded and used.

Outcome metrics
Outcome metrics concern the impact of the KM project or initiative on the overall organisation. They measure large-scale characteristics, such as increased productivity1 and:

Time, money or lives saved;
Injuries prevented; and,
Changes in the way we do business

The Army Knowledge Management (AKM) Maturity Indicator
The AKM Maturity Indicator, (see page 20), is a tool used to examine KM maturity from an organisational perspective. It will provide an indication of where your organisation stands on a maturity level before you start examining common KM initiatives, which promote effective KM programmes.

The AKM Maturity Indicator is used to determine an organisation’s overall level of KM maturity relative to the adoption and use of the AKM principles;
The Indicator is composed of two axis;
The ‘X’ axis (horizontal) measures an organisation’s adoption and use of the AKM Principles (people/culture, process and technology) on a five-point scale, ranging from one (KM Novice) to five – (KM mature);
The ‘Y’ axis (vertical) applies the key elements of an integrated KM programme (culture, strategy, competency and metrics) against the AKM principles across the maturity level spectrum;
The AKM principles are both colour coded and typed in differing fonts in order for the user to easily determine which of the principles applies;
The AKM Maturity Indicator provides the means to evaluate its overall state of KM maturity;
An organisation can use the Indicator to better understand the types of behaviours more KM-mature organisations exhibit; and,
You can tick the boxes that apply to your organisation.

Most would agree that changing an organisation’s culture towards collaboration and sharing is difficult. That is because changing the culture means changing behaviours. One can get a sense of where an organisation is from a cultural perspective by observing these behaviours:

A ‘not invented here attitude’ versus a tendency to adopt good ideas, no matter where they were developed;
The knowledge-seeking behaviour of the staff. Whether they seek answers first and then invent answers if none are found, or they invent first and tend to reinvent the wheel;
Whether staff tend to think in terms of –
(a) ‘Only those who need to know get to know’;
(b) ‘Who needs to know what I know’; or,
(c) ‘I have a responsibility to share what I know so others may find and use it’;
Whether the organisation’s leaders personally participate in KM;
(a) Do they value knowledge and innovation or are they more focused on maintaining the status quo?
(b) Do they personally get involved in KM activities or is that something only done by the staff?
(c) Do they shape performance by asking questions in a way that provides an understanding of how the knowledge and information was obtained and used to develop answers? and,
(d) Are employees rewarded or acknowledged in some way for demonstrating KM behaviour? Are they penalised if they hoard or withhold information?

Understanding your culture will focus your attention on those activities that will be accepted by the organisation, and will better prepare you to create those change management initiatives that will drive behavior change and increase the organisation’s KM maturity.

In the next issue we will look at how you measure the maturity of your KM initiatives, from a number of viewpoints and perspectives.

Robert Hoss is a KM senior project analyst at the US Army War College. He can be contacted at robert-hoss@usarmy.mil

Art Schlussel is a KM senior project analyst at the US Army War College. He can be contacted at art.schlussel@usarmy.mil

References

Marine Corps Communication Electronics School (MCCES) Training Command., Standing operating procedure for the KM programme, December 2005;
Measuring the Impact of Knowledge Management, American Productivity & Quality Center (APQC), Consortium Learning Forum, 2003;
Patel, Sameer., Why ‘time saved’ and other such nebulous metrics are a cop out for Enterprise 2.0, retrieved on 6 April 2009 from http://www.pretzellogic.org/2009/03/why-time-saved-and-other-such-nebulous-metrics-are-a-cop-out-for-enterprise-20/;
AKM Collaborative Metrics Guide, Army CIO/G-6, EI Directorate, September 2004;
Metrics Guide for Knowledge Management Initiatives, Department of the Navy (DON) CIO, August 2001.