Indexing Is Not Enough

We all use search. Every day. Most of us use one search engine only. Type a keyword or two, get a result list, look at the results – most of us never go beyond the first three results. If not satisfied, redo with another keyword combination. Most of us never use advanced search forms. Over the past 20 years, we’ve all been primed to this interaction pattern.

Search traditionally works by crawling data, creating an index, providing for a search interface, typically a single field search bar. Trillions of person-hours have been invested by the leading contestants to get the relevance of the result set right. It’s the first three results that matter.

Back in 2004-2009, when we operated with local.ch the largest homegrown search engine here in Switzerland, our key metrics were 1.23 keywords as input. That is not enough to spell “Bern Restaurant”. It’s either “Be Restaurant” – no we don’t want to be a restaurant … or “Bern Rest” – ok, the saying is that’s what Bernese people do the whole day anyhow (me being one of them). Today with better type ahead, this has grown to a query length of about two words.

The core principle remains: You need to go to information.

Done efficiently, it is a wonderful way to access information. Large search providers and more focused providers such as Enterprise Search vendors have added layers of sophistication to the approach (e.g. ingestion optimization, profiling, relevance ranking, and many more strategies). The basic approach, though, remains the same.

In a world of too much information, this approach has its limits.

You need to know what you look for. Only if you can describe - to some level of precision - what you’re looking for, will the result list be somewhat meaningful for your request. While it is straightforward to use “that restaurant” for a search, it is less so for more complex situations. Say you join a new organization. By definition, you do not know what others, before you have written, say, about products. How do you start your search not even knowing the product terms?  Or you try to find, within a large organization, who knows what about a customer situation? Most organizations have disjunct information systems – multiple CRMs, service management tools, file shares, and more. How do you find the relevant information for a quick remedy?

A different approach is required to get to the next level: instead of you going to information, the right insights need to come to you at just about the moment you need them to get your work done. You turn information provision upside down.

What is required for this to happen: you need computationally aware informational objects. Sure, the data item will not develop cognition about itself. But the construed concept of the informational object must express in computational terms the concept and the notion of the underlying information/data.

In essence, the information/data needs to be expanded into a concept of an informational object with lots of (probabilistic) meta information to get some level of cognition of its content and meaning. Besides, the system needs a good understanding of your current informational needs, in other words, it needs a good profile of you.

To extract such insights of information/data and you as a user, the recent advances in AI are the catalyst of this transformation. An essential element is to approach this transformation to an informational concept or object not as yet another relational schema but as a continuous probabilistic re-compute: Information changes, situational changes, user preference changes.

The first three Bernese restaurants we rendered back in the days were not, in absolute terms, the best. They were the ones we thought – well, as the ranking algorithm we put in place provided – were the most relevant to your search and matching your search profile.Today we extend this concept and think of this like a continuously cooked bouillabaisse and from which the system picks a bowl when it’s time for lunch, automatically.

This information revolution that is about to take place is well summed up in this quote from Anthony Mullen et al.:

In the 15th century, Copernicus introduced the shift from an earth-centric to a sun-centric view of the solar system. There is a Copernican shift underway in how enterprises handle data. The approach shifts the emphasis from relational schemas as the center of the “representation” universe to concept and object models expressed across semantic and machine learning technologies.

Anthony Mullen, Magnus Revang, Stephen Emmott, Erick Brethenoux, Bern Elliot, Jessica Ekholm2021 Strategic Roadmap for Enterprise AI: Natural Language Architecture, Gartner, December 2020

Read Gartner's Magic Quadrant for Insight Engines, where Squirro has been recognized as a Visionary.