Feature

posted 28 Aug 2007 in Volume 11 Issue 1

EI workshop

Measured reaction

What cannot be measured cannot be managed. So what metrics should an organisation use to measure the success of its intranet?

By Stephen Musselwhite

Measurement is an essential part of the intranet-management process. It closes the loop on strategy implementation and ensures that the intranet achieves what it was established to do – and that you know that it is working accordingly to plan.

Well-honed metrics can help show you how to improve the efficiency of the content-management system (CMS), improve the usability of the intranet by identifying where you need to optimise both the interface (the information architecture) and content (readability of content, structure, level of detail), and focus the attention of the various stakeholders on the most important and rewarding intranet activities.

An intranet can provide measurements across all of the activities associated with the end-to-end management of content. But despite the obvious benefits of having a comprehensive programme of measurement in place, intranet measurement often doesn’t extend beyond the use of a software package (web analytics tool) to provide relatively basic data about the usage of the intranet. This is a lost opportunity. Careful thinking about what measurements are required is essential. Intranet managers must pose hypotheses that can be answered by the measurements and metrics.

However, measurement is supremely difficult. According to a recent survey conducted by the Web Analytics Association (www.webanalyticsassociation.org) and

Eric T. Peterson (full results available on www.webanalyticsdemystified.com), web analytics is poorly understood and the majority of people interacting with web data do not understand what the data means. Of the 1,000 international web-analytics professionals surveyed, 82 per cent claimed that web analytics were largely misunderstood within their organisation.

One of the reasons for this, other than the inherent complexity of some of the more advanced measures (and a reason for the frequent failure to act upon the analysis of those measurements) is that there is little common agreement on terms within the industry. Are ‘hits’ (the single most useless measurement of web-usage available) page views and visits interchangeable terms, different measures with the same meaning between vendors, or terms with different meanings across vendors?

This is often unclear in many organisations’ definitions. One of the aims of the Web Analytics Association is to standardise the terms, definitions and best practices used by the entire industry in order to improve understanding.

Understanding measurement

The collection and use of measurement information must be well understood by the intranet manager and translated into actions from the central intranet team and distributed content-management community if measurement activity is to positively impact on the intranet.

The intranet manager must understand not only the constituents and mechanics of measurement, but also what to measure and how to accurately interpret the data collected. This interpretation is more difficult than measuring internet website usage, not just because of the lack of formal agreement on term definitions, but also by a lack of comparative data available. The usage figures of other intranet sites is difficult to locate, so setting specific targets for usage based on industry or sector averages (in the same way that data exists to enable organisations to compare their expenditure on technology generally, for example) can be difficult.

Setting measurement targets in the strategy

Specific measurement targets drive the attainment of particular strategic goals. The effective measurement of the intranet’s strategic objectives has a central place in achieving the intranet strategy. Objectives must be measurable, of course, and it is vital that the ability to measure translates into realistic targets.

What to measure

What can be measured will depend on the tools and resources (skills and experience) available, but whatever the constraints, before thinking about specific measurement tools or techniques, it is vital to carefully consider what needs to be measured and what the outcome from that measurement activity will be.

As a minimum it is suggested that measurements are undertaken of the following:

The performance of the network and hardware;
The operation of the CMS (ie. how efficient is the end-to-end content-management process?);
The content-management process itself;
Usage of the intranet;
User engagement;
The usability of intranet content;
The use of search;
Output and efficiency of the roles and responsibilities;
Effectiveness of the content-management training.

Each of these areas will now be explored in a little more detail.

Performance of the network and hardware

The team that supports the network and hardware associated with the intranet will use a range of measurement techniques to ensure that performance of the hardware is optimised and maintained.

For the most part, these measurements require specialised technical understanding and, in themselves, are unlikely to be of great interest to the intranet manager on a day-to-day basis when performance of the network and hardware is satisfactory. But in the event of a failure of hardware or a general degradation of performance, such measurements will be of central importance as a fault-finding tool.

However, it should be noted that a common reason for a degradation in hardware performance is, quite simply, down to a failure to perform routine maintenance. Measuring the performance of the hardware helps to identify the maintenance routines that are required and helps ensure the ongoing health of the system.

It is important that both the live environment and the CMS hardware performance are measured and optimised. Poor performance of the live environment affects the users of the intranet and poor performance of the CMS impacts upon the content managers, too. Both challenge the credibility of the intranet.

Measuring the operation of the CMS

The end-to-end process for managing content must be efficient. Metrics and measurements can help to improve the efficiency of this process by focusing on where those inefficiencies (such as usability issues) lie.

Measurements on the overall level of usability and efficiency of the intranet CMS provide guidance on where the process might be inefficient and improvements can be made. Areas often requiring closer scrutiny include the time taken to create a standard content page; the time taken to create workflow, or to move content through the workflow process; and, the time taken to locate and edit a page once it has been published.

Measuring the content contained within the CMS is essential to understanding the health of the system: the following should be measured at appropriate intervals to identify the currency of content:

Total items stored within the CMS;
Items never changed;
Items not changed for more than two years;
Items not changed for more than one year;
Items in draft (in unpublished status);
Items in each stage of the workflow.

These metrics should be compared against the usage metrics to answer obvious questions, such as ‘are all those items of content that haven’t changed for two years ever read?’

Roles and responsibilities

To ensure the roles of content manager and content editor are being pursued efficiently, the following provide some basic metrics from the CMS:

Most active content manager;
Least active content manager;
Average time spent creating a page;
Average number of revisions in a page.

Both of the last metrics in this list are indicators of training requirements. Significantly longer than average times spent in page creation, for example, could indicate problems with the system, process or with staff understanding. Intranet managers need to be comfortable analysing data. But as part of their role, content managers need to be comfortable with interpreting the data provided by metrics tools, or using metrics tools and analysing the data output of those tools themselves.

Measuring intranet usage

Web analytics tools designed to measure the usage of web content provide the intranet manager with some valuable quantitative information on how and when (and sometimes where) users actually use the intranet. Web analytics are probably the first form of measurement considered when an assessment of how the intranet is being used is sought. When measuring intranet usage it is essential to balance purely quantitative information with qualitative data to provide context and test hypotheses generated from analysing the raw information.

Measuring user engagement

User engagement can be measured. Measuring user engagement enables the intranet manager to ensure content is provided in the format most preferred by users. The following all provide potential measures that, when used in the appropriate combination for the organisation, can be used to ascertain user engagement levels.

Duration – how long are certain pages viewed for?
Page views – the number of times a page is downloaded (as distinct from ‘hits’ – see glossary);
Average page views per session – how many different pages do users view every time they logon to the system?
Returning visits – do people come back to particular pages? If so, how often?
Pre/post information-architecture changes – what effect do changes to the system have on users’ viewing habits?

Assessing the engagement of user-generated content versus ‘officially content managed’ content is particularly valuable given the potential so many organisations face for an explosion of user-generated content. Quantitative information should be provided by web analytics and in-context workplace observation to provide a more accurate view of intranet employee-engagement. Focus groups could be arranged to provide a qualitative input, too.

What does good usage actually look like?

Setting realistic usage targets requires an understanding of what should be expected. Usage of the intranet will depend on many factors, starting with the content and tools provided over the intranet, and also including senior-management sponsorship of it. Is the intranet positioned as a critical tool or a distraction for employees? Are employees discouraged from using the intranet because it is seen as a ‘waste of time’ or encouraged to use it because it provides labour-saving tools? Other factors to consider include:

Is the intranet universally available?
Do employees have easy access to the intranet (are workshop-based kiosks provided for non-desk-bound staff?
Can employees access the intranet from home?
Are a range of access devices, such as handheld computers, supported?);
The nature of the work being undertaken by the employees. Are staff in call centres, for example, encouraged not to use the intranet or does it host constantly used applications?

A posting on Toby Ward’s Intranet Blog on 20 April 2007 entitled ‘Intranet Usage: What is Considered ‘Good’ Traffic?’ provides some broad metrics from some of the world’s largest and most successful intranets:

Nordea: Between 70 and 80 per cent of all employees visit the portal every day;
Hewlett-Packard: 95 per cent of employees use the intranet on a monthly basis;
British Airways: 94 per cent of all employees access the intranet every month;
IBM: 80 per cent of all employees access the intranet daily;
DaimlerChrysler: 70 per cent of all users in Germany – including 120,000 blue-collar workers – log in at least once per month;
Microsoft: 60 per cent of employees visit MSW once a day or more, and an additional 25 per cent use MSW at least a few times per week.

So, set targets for usage of:

The intranet homepage;
News areas (including traffic to specific news articles);
Personalisation features (usage of the capability to personalise and the subsequent use of the personalised content);
Use of user-generated content tools and the use of user-generated content;
Use of the key applications – contact directory, lexicon of terms.

Metrics must be interpreted with some caution, of course. Does an increase in page-view duration really indicate an improvement in engagement? Does an increase in visits denote a genuinely increased interest in the intranet or a usability issue?

Intranet usability

Quantitative data on usability supported by qualitative information drives usability enhancement. Declining intranet usage data provided by web analytics will provide clues to usability issues that can be more fully investigated via other more involved means.

The following measures are among the most useful in helping to identify usability issues:

Main paths through the site – are users navigating to the content they need in the most effective way?
Time spent on each page – are users spending enough time on a news-release page, for example, to actually read all of the content on that page?
Use of search and at what point in a path search is used – combine path taken with search use and the search term used to identify major outliers in the navigation;
The use of the back button (historically the most used key on the keyboard and a good indicator of where usability isn’t what it could be);
Paths through the site;
Advanced technology – consider using eye-tracking software to test placement of content and navigation, and use task-completion testing to get the most from eye-tracking software.

Use split testing techniques to test two versions of the same content with representative users. Compare the usability of the original version with an improved version against task completion metrics and general levels of satisfaction. The differences should convince any doubters of the benefit of the small investment in usability improvements.

Measuring content quality

Assumptions can be made on the quality of content through the analysis of quantitative data on page views, visit durations and returning visitors, but measuring the perceived quality of content relies upon qualitative methods. Feedback mechanisms are central to providing employee engagement:

Rate page – advocacy and evangelism (provide feedback on a page or star rating);
Page feedback – free text for descriptive comments;
Forward page to colleague – how often is this used?
What content is shared most often? How can this information be made more visible?

Measuring and improving readability

Good readability is a must for good quality content. Various approaches and tools to measuring readability exist and should be considered. The Flesch reading-ease index is perhaps the most well known and is a valuable tool for improving the quality of content across the intranet. Readability scores can be used, first to assess the readability of content across the intranet and, second, to demonstrate how more-readable content improves usability.

Measuring the use of search

Without good search metrics all you have is a bad search engine. Data focused on the use of search can reveal interesting information about users’ habits and illustrates the difficulties they may be encountering with the usability of the site (in particular its navigation).

In 2000, Jakob Nielsen postulated that 50 per cent of internet users are ‘search dominant’. More recent research by Katz and Byrne (2003) on the use of search or browse on online shopping sites found that selection depends on menu breadth and information scent. The research also suggested that users will settle on a strategy that best meets their need and that search or browse dominance is related to the design of the site.

There is no conclusive research on intranet users’ preference between browsing or searching for information. The intranet manager must, of course, aim to optimise both the information architecture to improve browsing and the accuracy of the search engine to improve retrieval of content and the discovery of related content. Questions to ask when considering search include:

How many times is search used?
How many times per user-session, on average, is search used?
What are the most common search terms?
What search terms receive zero results?
Has moving a previously difficult to locate item onto the homepage resulted in improved first-time location?

Measurement outcomes

Comprehensive measurement – particularly the first time – will no doubt result in a programme of action to make good identified shortcomings. The following list details some of the most likely outcomes:

Redesigns/information architecture realignment and adjustments;
Addition of terms to the taxonomy/ontology;
Restructuring of site layout;
Rationalisation of content – split test to prove less is more;
The benefits of collocation of related content (applications and supporting information). Look at usage of various pieces of content, then collocate most related content.

Summary and conclusions

Measurement is central to success in pretty much any endeavour – and in business you cannot manage what you do not measure. Measurement proves value, gains support and drives the intranet forward. To be truly effective, measurement must set baselines as part of the formalised strategic planning process for the intranet against which to identify successes, appropriately deal with failures and move the intranet forward.

Measurement can be a complex and time-consuming activity and, where resources permit, is best undertaken by a specialist. Content managers must be proficient in understanding the information provided by metrics and making appropriate decisions based on that information.

Stephen Musselwhite is intranet manager at UK-based insurance giant Prudential and also the author of the Ark Group special report, Intranet Strategy and Management. For more information about this report, please e-mail Adam Scrimshire, ascrimshire@ark-group.com. To contact Stephen, please e-mail stephen.musselwhite@ntlworld.com.