Gartner defines Insights Engines as enterprise search that provides more-natural access to information for knowledge workers and other constituents in ways that enterprise search has not

The Rise of Insight

European Data Market Study

For all the value that is assigned to data in the modern economy – and in the EU alone the data economy is projected to reach €739 billion by 2020 according to the European Commission report, Final Results of the European Data Market Study Measuring the Size and Trends of the EU Data Economy – in and of itself that data is worthless. What actually gives data the incredible value that it holds, is the meaningful insight that allows an organization to use that data to improve their business in some way.

Finding and then extracting this insight from the massive volumes of data held by modern enterprises is, therefore, becoming a number one priority for many organizations. According to a 2005 study by IDC, The Hidden Costs of Information Work, employees spend around 9.6 hours searching for relevant information every single week. The situation has almost certainly worsened considering the on-going and significant growth in data and information.

Locating the relevant, necessary and contextual information, efficiently and quickly is vital in today’s knowledge economy. It enables businesses to meet the demands of the market and their customers. This means that organizations have a growing and genuine need for an Insights Engine to improve collaboration and make the lives of their knowledge workers easier.

What Exactly is an Insights Engine and What Can it Deliver?

An “Insights Engine” is a relevantly new term, coined by industry analyst group Gartner and one which essentially replaces enterprise search. Gartner has defined insight engines as enterprise search that provides “more-natural access to information for knowledge workers and other constituents in ways that enterprise search has not.”

As with many terms in technology, exact definitions of Insights Engine can differ, depending on who you ask. However, there is a general consensus that an Insights Engine will include cognitive processing and Machine Learning (ML) summarized under the term Artificial Intelligence (AI).

These will all help facilitate a much more contextual, intuitive, and personalized search for information in the enterprise.

The results that come from using an Insights Engine are markedly better than those delivered by traditional enterprise search, a technology that has been somewhat exposed over the past five years. The right solution will deliver results – content and recommendations – that users will need next, whether they even know that yet or not.

Because the technology within an Insights Engine can work with unstructured data – social media content, call transcripts, premium external data, video, audio, and much more – the insight extracted and delivered is far more valuable. Unstructured data tends to be where the most insight is found, and most enterprise search options are simply unable to work with data of this type.

Gartner has predicted that by 2022, information will proactively find more employees more often, thereby providing the insight needed to progress actions, decisions, and reduce reactive searching by 20%

The Challenges with the Insights Engine

Although the potential of an Insights Engine to transform an organization is vast, deploying such technology does not come without its challenges. There can be a certain conservatism when it comes to implementing new technology, especially in particular industries. Some of the challenges associated with insights engines include:


This is important across virtually all enterprise functions. But given that insights engines give knowledge workers access to a wide range of company data, it is especially pertinent to have the proper security measures in place. A breach of data security is a big deal for any organization, so any Insights Engine should have access-authorization features to help prevent this from happening.


Organizations often have data stored in a wide range of locations, geographies, and systems. From ERP to CRM, and ECM to network drives, the data sources are varied and many. Any Insights Engine must therefore easily integrate with all of these sources. Failure to do so will significantly reduce the effectiveness of any insights engine.

Size and Scale of Big Data

Big data is named as such, because there is so much of it and it comes in a wide array of files and formats. While insights engines do work better with large volumes of data, the volume of available data is growing all the time, with the proliferation of mobile devices and the Internet of Things. This means that organizations must be mindful to work with a solution that can scale and manage big data sufficiently.

The Winning Difference

“We wanted our search function to be fast, accurate, and to move away from traditional approaches — to include context and intent. The way people use search has changed and it’s essential for us as a company to have a search function that meets those changing needs, delivering smarter and more relevant results to our customers.”

Michel Küng, Webmaster, Helvetia

Squirro Cognitive Search

AI is at the core of any Insights Engine and is also what powers Squirro Cognitive Search. This is a solution that was acknowledged in the updated (Q2 2018) version of a recent Forrester report, “The Forrester Wave™: Cognitive Search And Knowledge Discovery Solutions, Q2 2017

The report stated that “smartly, Squirro leverages open source Elasticsearch as part of its platform, enabling it to focus on business search applications and enterprise features”, and then goes on to say that “enterprises that need immediate customer 360° insights or service will appreciate Squirro’s out-of-the-box readiness for fast implementation times.”

In 2018, The Helvetia Group, one of Europe’s largest insurance firms, replaced Google Search Appliance (GSA) with Squirro Cognitive Search as it sought to deliver better search results to its five million customers across Europe. GSA had reached its end of lifecycle and The Helvetia Group wanted a solution that allowed for a more cognitive approach to search, anticipating customers’ requirements and incorporating free text, context, and relevance when delivering results.

The Prerequisites to Getting Started with an Insights Engine

For any organization yet to fully utilize an Insights Engine, the time to start is now. However, when getting started, the organization should always consider the following requirements:

1. People

A key tenant of an Insights Engine is that it must be utilized by business users, not IT teams. An Insights Engine delivers a single source of truth and its success hinges on being accessed by all knowledge workers. In fact, results to queries will be presented precisely according to a particular user and their role. Knowledge workers will almost certainly be proficient at creating and working with many types of different queries, but organizations may want to provide some guidance here.

2. Data

Insights Engine are best positioned to work with large data sets. Big data encompasses what is referred to as the 3v’s – volume, velocity, and variety. Volume alludes to the sheer amount of data, which is almost unfathomably large in 2019. Velocity is the measure of how fast the data is coming into an organization, while variety means the breadth of different file types and formats that data comes in. Big data is highly recommended to see the best results from an insights engine.

3. Infrastructure

An Insights Engine should always be easy to implement, without the need for a costly and/or time-consuming setup process. Depending on the business needs that the solution is solving, there may well be a requirement for connectors to the different and disparate data sources, such as data lakes, file servers, enterprise systems, as well as other internal and external sources, but there shouldn’t be any large-scale infrastructure requirements.

4. Futureproofing

Once an organization is using and reaping the benefits from an Insights Engine, it will be almost impossible to work without it. But given this importance, it means that organizations have to guard against the technology becoming obsolete. To ensure continued investment, the Insights Engine must demonstrate strong ROI to the business, empowering knowledge workers but also providing tangible, clear and measurable benefits – reduced search time, increased customer response times etc…

What are the Next Steps?

An Insights Engine brings relevance and context to every interaction between an employee, the organization they work for, and the subject they are researching. The era of knowledge workers spending days every week just looking for information are long gone Insights engines are at the forefront of this change in how people seek information and how that information is delivered to them.

Squirro Cognitive Search is a solution that makes good on the Insights Engine potential, delivering a year-on-year reduction in search time of up to 90% and an automatically generated 360°client cockpit for better decision-making. Such technology is fast becoming the bedrock on which the modern enterprise is based.

To better understand how you can prepare for and implement an Insights Engine in your environment, providing your team with increased efficiency and your clients with improved services and support, please contact us. We’d be happy to schedule a 30-minute call to discuss your situations, your goals, and how an Insights Engine would help.

Contact Us