Enterprise search: A commodity in the making?

Over the past decade, enterprise-search technologies have experienced significant evolution, with myriad tools and complex algorithms now part of the corporate mainstream. But as enterprise-search tools start to become commoditised, established vendors will have to continue to innovate if they want to compete on functionality rather than price. By Jason Schofield

As the race for the latest search technology hots up, the strategies of the major vendors are becoming more apparent. These strategies are primarily being driven by what is being hailed by some industry spectators as the commoditisation of enterprise-search technology.

But what exactly is meant by the term ‘commoditisation’? While this largely depends on who you talk to, the term is most commonly applied to the computer hardware industry, and can be defined as an increase in the number, availability and similarity of products of a certain type, which tends to drive down the price of products. New hardware or software usually begins life at the top of the IT heap, where it often generates healthy profits. It then starts the downward journey of becoming a commodity as the technology becomes widespread and standardised, with a subsequent reduction in its value. At this stage the most important criterion for customers is price.

So is the software industry on the verge of a shift in the way enterprise-search technologies are sold? The short answer is, yes. But to fully understand and appreciate the dynamics of this situation, it’s important to look at the strategic movements of the key vendors that dominate this space and to consider the impact of Google’s foray into this domain.

Google began in 1998 as an academic research project by Sergey Brin and Lawrence Page, who were at the time graduate students at Stanford University in California. Their success was to be born out of an algorithm called PageRank, which scores pages according to their relevance and importance. Each link from one page to another is counted as a ‘vote’ for the destination page and, using a variety of sophisticated mathematical techniques, each page is given a score that reflects its importance to a given search query. The rest, as they say, is history.

But it’s Google’s interest in the lucrative enterprise space that has been grabbing the headlines of late. The company recently launched a new version of its enterprise-search product, the Google Search Appliance (GSA), which uses the same code base and content-crawling technology as the software that powers Google.com.

“Google has made it easy to find random, esoteric information on the web,” says Dave Girouard, general manager of Google Enterprise. “But it’s easier to find information about the habitat of a Lynx spider on the internet than it is in many enterprises to find very important, basic company information such as a sales presentation from the last quarter. We view this as a paradox because it should be easier to find this information in the enterprise.”

Google’s brand recognition makes its appliance a popular entry on enterprise shortlists, according to Whit Andrews, a research director at Gartner. “Its low price, term licence and simple deployment models are best used in tactical external or intranet installations where content need not be indexed directly from dynamic repositories,” he wrote in an evaluation of the enterprise-search market in May 2004.

But there are many who dismiss Google and general web search as not relevant to more complex corporate information-management challenges. Despite being able to crawl and index documents in more than 250 file formats, the GSA is unable to access file servers and database records or integrate directly with document-management or content-management systems. Documents must also be accessible via HTTP or HTTPS.

According to Andrews, where the GSA falls short of some of its more sophisticated competition is in its relevancy analysis of queries and document structure semantics. This is a view echoed by John Lervik, chief executive officer of Fast Search and Transfer (FAST). “In the enterprise environment, one must rely on a variety of search query structures to ensure accurate results. Simply relying upon Boolean logic is self limiting.”

“I think it is inevitable that some customers will find the brand image generated by Google attractive,” says Simon Atkinson, UK managing director of Verity. “However, the demands of enterprise search are very different to the demands of web search. The needs of enterprise search customers, especially in terms of functionality and security mean that Google is unlikely to be a major threat initially. The concept of putting the software on a box and just plugging it in over-simplifies the problem.”

“I think the criticisms of Google are generally accurate,” says Kate Noerr, CEO of metasearch specialist MuseGlobal. “Google is OK at a simple level, but I would never use it for what I call a professional search.” Steve Papa, CEO of Endeca, agrees that Google’s approach is not best suited to the rigours of corporate search. “The relevancy ranking algorithms that make Google’s web search so powerful do not translate well into the enterprise environment,” he says.

Google is not taking such criticisms lightly and is reportedly working hard to broaden the searching and indexing capabilities of its enterprise search appliance to include non-web data repositories. “We want to index and allow users to search on the broadest set of enterprise documents, while retaining the simplicity of use that people have come to expect from Google,” said Girouard in an interview with IDG. However, the company has yet to specify the data sources the GSA might support in the future.

An area in which Google can certainly hold its own is price. The Google Search Appliance is sold as a two-year licence, with hardware, software, product updates and support all included in the price. The GSA is available in three models. Pricing for the stand-alone GB-1001 model ranges between $32,000 for a 150,000-document index and $175,000 for a 1.5 million-document index.

“This ‘one price’ model eliminates expensive annual maintenance fees and removes the threat of cost over-runs,” says Girouard. With maintenance fees typically 20 per cent to 25 per cent of the licence fee, he adds that organisations should be wary of the hidden costs of existing enterprise solutions. “The total cost of ownership of an enterprise-search solution is, in many cases, several times the initial licence fee.”

Clients that are benefiting from Google’s model include Boeing, Cisco and Xerox. Morgan Stanley, meanwhile, has been using the GSA for over a year to provide intranet search for more than 25,000 employees around the world, searching an index of 2.2 million documents from 200 different intranet web services. “Since its introduction, search traffic has increased by a factor of 11,” says the company’s managing director, Benjamin Fried.

Of course, Google is not the only new player to have entered the enterprise search space. Technology giant IBM recently launched its own enterprise-search engine, DB2 Information Integrator. This follows the company’s acquisition of enterprise-content-integration vendor Venetica in August 2004.

Lervik welcomes both new entrants. “Google’s success in the internet search space has increased people’s curiosity, resulting in an increased demand for enterprise-search solutions,” he says. “The entrance of a player such as IBM, meanwhile, validates the space and increases the visibility and awareness of the need for enterprise search, so this is good for all players in this segment.”

Atkinson agrees that the involvement of platform vendors could “ignite” the enterprise-search market by drawing more attention to the technology and he remains largely unperturbed by the threat of the likes of IBM, Microsoft, Oracle and SAP. “Several platform vendors have had offerings in this space for a number of years. But whatever success they have at selling enterprise search, it will only represent a small fraction of their overall revenues. It is therefore unlikely they will dominate the market.”

Similarly, while many would not question Google’s commitment to the enterprise-search marketplace, the GSA remains a small part of the company’s overall business. In 2003, revenue from licensing and other non-advertising sources accounted for $45.3m of Google’s $961.9m in revenue, which equates to about five per cent.

But there is certainly room for growth if the analysts are to be believed. A recent study from research firm Ovum, entitled Knowledge Discovery, states that the enterprise-search marketplace has begun to emerge from a two-year period of stagnation. While the sector was worth just over £257m in 2003, Ovum predicts that it will reach just under £450m by 2008.

This growth will be fuelled, in part, by a growing number of legislative regulations and requirements that demand a clearly defined, integrated approach to information management. In the US, and sometimes beyond, organisations must get to grips with the demands of the Gramm-Leach-Bliley Act, the Patriot Act and the Sarbanes-Oxley (SOX) Act; closer to home, Basel II, the Freedom of Information Act (FOIA) and the Environmental Information Regulations are all equally pressing. Many of these regulations require companies to access records of every communication entering the organisation, which is spurring interest in enterprise-search systems.

In the UK, e-government initiatives and the FOIA are forcing public-sector bodies to provide citizens with better access to information. “Search and categorisation technologies play an important role in helping individuals to locate accurate information quickly,” says Iain Fletcher, alliance director for Europe at Convera. “Within government bodies, search and categorisation will also provide the backbone for information and knowledge management, allowing users to locate information within a variety of potentially distributed repositories, using free-form search or navigation via a variety of classifications that are tailored to specific needs.”

Fletcher adds that automatic categorisation will assist public bodies to be compliant with the UK’s e-Government Metadata Standards, which requires all public-sector websites and intranets to be meta-tagged according to the international Dublin Core model. “Without this technology it is unlikely that any government body will be physically able to be compliant with the requirement to associate metadata with all documents and records,” he says.

The continued innovation of the established enterprise-search players will also play an important role in the market’s expansion. In fact, as the basic elements of search solutions become commoditised and are no longer seen as a valuable technology, those vendors not willing or able to innovate and branch out into other disciplines will likely have an uncertain future.

In the face of increased competition, and in a bid to raise the bar for new entrants, Verity has made great strides in recent years and is one of few vendors that have been able to maintain profitability for six consecutive years. Recent acquisitions include Cardiff Software, an e-forms data capture specialist, and NativeMinds, a provider of online customer-service software for creating virtual representatives.

FAST, meanwhile, has won a place in the 2004 Deloitte Technology Fast 500 EMEA, a ranking of the fastest-growing technology companies in Europe, the Middle East and Africa. Organisations are ranked according to their average annual revenue performance over the past five years, with FAST placed 14th. Earlier in 2004, the company launched the Enterprise Search Platform, while at the same time delivering several new Search Derivative Applications that support business processes. These processes may be departmental (marketing, human resources, etc), vertical (telecommunications, government, manufacturing, etc) or application specific (enterprise-resource planning, business intelligence, etc).

Lervik warns that there is much room for continued innovation in enterprise search. “Do not confuse a growing awareness of the value of enterprise search with the space becoming commoditised.”

Endeca also sees room for innovation and is increasing its capabilities in information access and navigation by adding new analytical and visualisation capabilities. “This will allow a new breed of applications that blur the lines of BI and search, giving casual and business users not only the ability to find a collection of information or data but also to aggregate it, perform ad hoc analysis, present it in graphical form and then share it with others,” says Papa.

No article on search would be complete without discussing the latest development of so-called ‘desktop-search’ solutions, which promise to revolutionise the way users search for information on their local hard drives.

In what is seen as a pre-emptive strike aimed at nullifying expected improvements to Microsoft’s in-built search functionality in its next version of Windows, Google recently launched a beta version of its free Google Desktop Search tool. The device is a small 400 kb applet that uses keyword searches to retrieve data from files and e-mails held locally on PCs. It also indexes Internet Explorer’s web cache. Software makers such as Blinkx and Copernic Technologies already offer free desktop-search tools.

A more surprising development, however, is the launch of a desktop-search product by enterprise-search specialist Autonomy. The IDOL Enterprise Desktop Search uses a technique called ‘implicit querying’, which means it runs constantly in the background looking for data that could be relevant to a user’s work. This data is then listed in ‘Active Folders’ that are automatically updated. “Active Folders constantly monitor what I’m doing, searching for relevant material and pushing it to me,” says Ian Black, managing director of Autonomy’s Aungate division.

It’s important to note that Autonomy’s desktop-search solution can search enterprise applications such as document-management and CRM systems. This functionality puts it in a different class to most, if not all, other desktop-search applications. As a result, the application comes at a premium in that it must be run in conjunction with Autonomy’s back-end IDOL search software. Pricing for this, says Black, depends on the size of the company and which parts of the IT systems are searched. Autonomy is targeting the product at companies that have already implemented that system.

There can be little doubt then that enterprise-search technology is evolving rapidly. At the lower end of the market search technologies have become commoditised with various companies offering solutions that compete predominantly on price rather than functionality. At the other end of the spectrum, established enterprise-search vendors have responded by expanding their offerings into areas such as collaboration, BI and knowledge management, with improved support for taxonomies, search analytics and personalisation.

Only time will tell whether the two approaches are mutually exclusive.