NEON illuminates the BBC

Knowledge sharing in the Information & Archives division

The changing pressures facing the broadcasting industry mean organisations have to constantly adjust to ensure employees have access to the latest information on demand. Andrea Wharton describes the experiences of the BBC and Smartlogik in implementing NEON a project estimated to be worth £5m and scheduled to last five years.

In a world where the only constant is change the BBC is eminently qualified to succeed. Over the past 50 or so years the corporation has proved time and again its ability not only to respond but to derive competitive advantage from the constantly evolving world of broadcasting. There was a time when television was available in discrete ‘chunks’ totalling eight hours a day or less. Now it is a 24-hours-a-day business. Radio was delivered to the nation on a ‘one-size-fits-all-basis’ via three national services but today a multiplicity of regional stations cater for local interests. The formality of panellists in evening dress on programmes such as ‘What’s My Line?’ have given way to the casually dressed uninhibited style of ‘Never Mind the Buzzcocks’. The list could go on.

There are some aspects of the BBC however that do not change and one of these is the corporation’s dedication to ensuring that its journalists researchers and programme makers have access to the reliable quality information that has enabled the BBC to earn and retain a worldwide reputation for excellence. But even here the BBC has had to make changes. At a time when licence fees had been pegged below inflation the emphasis for all departments within the BBC was on doing more with less. How then could the organisation continue to maintain its high standards in a world where the information resource to be managed was growing at an unprecedented rate and the pressure to respond more quickly was increasing all the time?

The BBC’s Information and Archives (I&A) division faced just such a dilemma in the1990s. The division had been formed in the mid-1990s by bringing together a number of historically separate archives – TV News Radio and World Service. The separate services had been extremely popular – each slightly tailored to their respective customers – but with the advent of the internal market at the corporation supporting them all became difficult to justify.

Merging the services reduced the amount of duplication that was inevitable with three separate press cuttings operations that had overlapping interests. A hard copy press cuttings library is labour intensive and therefore expensive to run as well as being relatively inconvenient and inflexible for the users of the facility.

With the paper-based cuttings service a journalist for example would phone or walk down to the library and ask for information about a particular subject. The researchers would find the correct classification term from their paper listings containing about 6 000 terms and then someone would go and look for the copy under that heading in a filing cabinet. The challenge therefore was not simply to reduce running costs but at the same time to improve the service to users who often now needed a faster response than was possible with the paper-based system; to eliminate the problems that arose when more than one person needed to work with the material at the same time and to enable those users who wanted to to do their own research. What the BBC really needed was an online system delivering press information to the desktop.

Not surprisingly one of the prime movers in this rethink was the BBC News the group of people for whom minutes and seconds really matter particularly when they are faced with a big news story that breaks just as one of the main bulletins is going on air. News was responsible for 60 per cent of the cuttings service usage and with such a powerful ally Information and Archives had the support it needed to go back to the drawing board and consider ways in which the press cuttings service could be delivered electronically direct to the user’s desktop.

The right approach

The most obvious alternative seemed to be one of the existing online services but a careful review of these convinced the BBC that the cost of providing access for thousands of desktops would be too high. Even more importantly the journalists and programme makers within the BBC were not impressed with what was on offer. The services available at that time assumed that users would be trained information scientists rather than the polyglot population of the BBC some of whom understood ‘information speak’ but most of whom did not. News and Current Affairs were already moving away from the traditional approach where programme makers used researchers. Rather the emphasis was on shifting to self-research as people saw the potential of using their existing desktop computers to help. This trend was already so strong that I&A discovered people were already beginning to use other services from their desktops as well as the internet.

If I&A did nothing it was obvious that users would find their own partial solution to the problem and the manual cutting service would gradually erode. With existing online service providers already out of the picture the BBC was left with a choice of outsourcing the entire activity to a third party or working in partnership with a supplier to ensure the essential continuity between the old cutting service and the new online facility and also to guarantee that the new service fitted the BBC’s requirements both in terms of its comprehensive nature and in being user friendly. The BBC decided that working with a supplier was the best option for it.

The project begins

Next began the arduous process of deciding on the supplier with whom to partner. The European Union purchasing process that had to be followed because of the size of the order meant that it took a full year to reach the stage of final negotiation with two short-listed candidates. At the beginning of 1998 the winner the Dialog Corporation Smartlogik’s forbear signed a contract with the BBC to develop and implement the NEON (News Information Online) project based on its Infosort technology. The project worth around £5m was to run over a five-year period. Within six months by the middle of 1998 the new service was up and running. At the start it was less than perfect. Negotiating to obtain electronic news feeds from publications ranging from The Times to Hello magazine and harmonising the new hardware and software with the existing BBC IT infrastructure all took time. Both Smartlogik and the BBC also realised that the news feeds had to be managed. Information gaps sometimes occurred that made internal BBC users extremely frustrated and it was not until a system was put in place to check that contracted information was being delivered at the appropriate time that this was resolved.

Commenting on the progress of the project Adam Lee I&A projects manager says: “The BBC News Service is the jewel in the corporation’s crown. Smartlogik was very much under the microsocope and although it did not get everything absolutely right first time it redeemed these early errors by the speed with which it responded to our criticisms and its enthusiasm and determination to build a ‘blue chip’ service. The service is now so popular among BBC users that we are constantly pushing our 5 000 user limit. To put it quite simply the system is mission critical. BBC journalists would not be able to do the first class job that they do without it.”

The keys to success

After a limited user trial lasting approximately three months NEON went live in April 1998 and by the summer of that year 4 000 users had been trained well ahead of the original schedule and already running at 80 per cent of the originally agreed capacity of 5 000 users.

This may sound like an overnight success but it was the months of careful planning and preparation that had finally paid off. It is something of a truism to say that successful projects are those that engage the end users in the whole process. The number of organisations that actually manage to do this is however relatively small. The BBC’s Information and Archives division was one of the few.

Working with the users Smartlogik and I&A worked to develop confidence in the new system from the outset particularly with the design of the user interface and the retention of classification and cataloguing terms wherever possible. (Classification terms were however reduced from 6 000 to around 1 400 making the system faster and easier for the varied user community to handle.) The intention was to provide a service that offered a level of continuity with the familiar press cuttings service but with the advantages of being available direct from the journalist’s desktop and with a user interface that would allow anyone information scientist or not to search effectively for the information they need.

Information and Archives also decided that it would take an innovative approach to training. It decided not to use conventional classroom methods and instead to allow a number of researchers to operate as ‘floor walkers’ to train users at their own desks during the course of the normal working day. This approach worked brilliantly and I&A could watch the system usage figures surge after the floor walkers had been around.

Information and Archives has had to manage passwords very carefully because of the success of NEON. As Lee says: “Initially anyone who asked was given a password and we found there were quite a few users who had only accessed the system once. We were able to reallocate these ‘slots’ on the system to people that had a greater requirement. In the last quarter of 2000 however we were really operating at full capacity with 20 per cent of NEON registered users accessing the system every day and an average of more than 3 000 hits in each 24-hour period. We are now being very strict about people who have passwords but have not used the system for some time. Even applying these rules however it will become very difficult to accommodate new users in the near future. But we have already delivered an infinitely better service to the journalists – fast convenient flexible and easy to use and it all costs less than the old paper-based cuttings service.”

What of the future?

The BBC archives contain more than press cuttings. There are audio recordings photographs video recordings and so on. In an ideal world users would like to have a single interface to all of these resources. For example a journalist researching a story about Cherie Blair wants the news stories available pictures and audio recordings from a single enquiry. Stills and content already go directly to the desktop but through different systems and the next stage is to deliver video audio and keyframes the same way.

Will it ever be possible to access all these resources via a single interface? Bill Thom managing director of Smartlogik’s Digital Media Division (DMD) confidently expects that it will. He says: “DMD is set to build on existing successes with text archives by working directly with customers and with partners who will implement our technology to enable media companies to manage all of their information assets – databases video and picture libraries and voice clips – through a single integrated system.”

Lee’s response is one of interest but tempered by the view that there is a balance to be achieved between the ideal and cost. Commenting on today’s priorities he says: “What journalists want today is easy and instant access to an information source in which they have confidence and at the end of the day to get their accurate story on the air before CNN! That’s the value of NEON to the BBC.”

Andrea Wharton is a co-founder of Wharton Information Systems. She can be contacted at: n_wharton@compuserve.com

A word on taxonomies…

Smartlogik’s Infosort technology used in the BBC NEON project and now marketed under the product name muscatstructure automatically structures information using rule bases devised by information scientists. It is in essence an automated taxonomy which allows thousands of incoming documents to be automatically and correctly categorised in background mode.

Classifying information makes it possible for the user to navigate over large amounts of data but guided as if by a map – the taxonomy terms or concepts are nodes and the links between them are the formal relationships. A typical navigation might take the user from a node ‘farm’ to related concepts such as ‘forestry’ ‘crops’ or ‘agricultural machinery’. Users come to understand how a domain of information is organised by actually carrying out the process of navigation and can then find exactly the area they are interested in and can keep coming back to it.

The taxonomy’s whole purpose is to enable documents to be usefully classified under related subjects in order that they can be found easily. The taxonomy is the means by which the users are connected to the documents.

The two basic rules for the information scientist building data location and retrieval systems are:

The taxonomy must be created with the needs of users in mind;
It must reflect the content and the types of document it will be used to classify.

The flavour of the BBC’s Information and Archives division taxonomy is very informal even colloquial. There are for example classifications for news stories about ‘unfaithful spouses’ ‘lucky escapes’ and so on. Each user can browse the subjects they need to research or type in words or phrases they think might be close (non-preferred terms) and the documents are then retrieved automatically by the formal classification term and delivered in real time to the screen on the journalist’s desk.

The problems encountered were mainly connected with the conversion of the paper classifications to an electronic system. There are many ways for inconsistencies to arise in paper listings and they show up very quickly when imported into an electronic form. For example individuals who have added to such a large structure have often included duplicates of existing terms and sometimes several duplicates of the same subject occur which makes good reliable retrieval very difficult.

The taxonomy produced was based on a structure that relied on ‘related’ connections. The very human nature of the subjects did not readily lend itself to hierarchical linking. For example if the information scientist were to try to create narrower terms for ‘obsessions and fetishes’ this could include just about anything. For example links to clothing hobbies uniforms would not be useful as these classifications are not always related to obsessions so they are not true ‘narrower’ terms. Much more useful are a list of related terms and in this case they include ‘tattooing’ ‘crazes’ ‘drag acts’ ‘rubber’ and ‘stalking’. So the relationship is much looser but provides what might well be useful pointers to other subjects although a standard broader/narrower relationship does not hold between them.

The implementation of the taxonomy has given journalists the ability to research vital programme information from anywhere in the world knowing the items retrieved are the most appropriate and the most up-to-date with the system being updated every second of every day with the latest news feeds.