What is an Insight Engine?

What is an Insight Engine is a short episode that focuses on answering this one question...

With the aim of ensuring that the universe is accurately informed about what an Insight Engine is, Squirro invited a key member of its team: Dr. Moritz Müller to talk to us about the software and its powerful impact.

Take a listen to learn firsthand about the term, its market relevance, its potential, and capabilities; as well as receive candid, expert advice on the opportunities and challenges that accompany the introduction of an Insight Engine into a corporate organization.

Full Transcript

Welcome. My name is Lauren Hawker Zafer, and as Head of Education and Training here at Squirro, I welcome you to our podcast series and educational session today. In this particular episode on “What is an insight engine?”, we've been graced with the presence of a long-standing member, and talented Squirro engineer, Dr. Moritz Müller. Moritz has been with Squirro for the past three and a half years and currently resides in Singapore as the Head of Product. Welcome, Moritz!

Moritz Müller
Thank you, Lauren, and good to be here and good to be speaking to you.

Lauren Hawker Zafer
Excellent. So again, thank you for being with us today. It would be great if we could start the conversation just by finding out what brought you to this point in your professional career, and maybe a little bit about what you personally enjoy most about working in this exciting space.

Moritz Müller
Yeah, this is an excellent question. And I had just been reflecting on exactly that topic, this week. So, I think it really was a number of coincidences that moved me to work in the insight engine space. Coming from an academic background in Geophysics, I started, after getting a Ph.D., to work in financial services and data analytics and kicked it off in a consulting company before switching over to Squirro. And in the meantime, I've seen so many different use cases around data analytics, where Squirro was used for data analysis, be it unstructured or structured with all kinds of components from very basic algorithms that can be super powerful to more advanced stuff like machine learning and NLP.

And it is super interesting to see how you can actually generate benefits and help users and business users by revealing certain structures or certain concepts in this data. And, I see it on a daily basis that whenever we speak with clients or potential clients, it's really interesting to see how the AI part helps them to make more of their data and at the end of the day also receive better results and attract more business.

Lauren Hawker Zafer
Interesting to hear. So obviously, you've touched upon the space that we're working in, and I want to chase that rabbit as we start to turn to our focal point today, which is, of course, as we've highlighted at the start, the insight engine. So, let's begin to unravel the layers together, I would say, and try to paint a vividly realistic picture of what an insight engine is. With that, if we start maybe from an etymological point of view, how would you define the term insight engine? Or maybe simply put, what should we understand when we hear these two words?

Moritz Müller
That is a good question because that is exactly what we need to be very clear on before we start talking about it. For me, an insight engine is a technology component that allows you to make, essentially, the most out of your data. You should have an insight engine at the center of your whole data architecture. In big organizations, you have tons of data distributed in different silos; they can be in many places. What I have observed in the past two years, is that, right now, a lot of banks are getting traction in understanding that having data kept only in data silos is preventing them from solving certain data analytic problems. So they started to introduce data lakes, make data available inside of organizations, whatever we want to build on top of them.

Once you've established this, you should use an insight engine to enrich the data. Because typically, you will want to make sure that client information is tagged to the news; that it's enriched with the same concept across all the different kinds of data sources. And what an insight engine can do, is not only allow you to be part of the data lake or to be in a data lake, but it adds to the data lake by enriching it with key information that you need for any kind of project, analytics, notification, or recommendation that you later would want to build on top of this. And that is exactly what the insight engine gives you - it gives you that framework to do the enrichments, and to build on top of it.

Lauren Hawker Zafer
Interesting. So if we're talking about this framework, and you feel the necessity for this framework to enrich this data, I mean, I can see the relevance myself, but I'm also starting to question the relevance that it plays in today's market. Why now, what's so significant about this insight engine? Apart from the component that you've mentioned, that it should be a centerpiece or that architectural component, what is the necessity to enrich this data? I mean, why should an organization invest in an insight engine?

Moritz Müller
What we see is that, very often, our clients or innovation managers that drive digital transformation in organizations, face the challenge that they have a very clear idea of what they want to do, what data they need, and what they want to build from this. But unfortunately, too often, a majority of the budget is spent not on the actual analytics part, the value generation, that on laying the foundation of getting the data connecting to the data sources, making sure you have entitlements that links correctly, making sure you have understood which clients are mentioned in this data or tagged to this data, and where it comes from. An insight engine allows you to do all of this, and you don't do it just once, for a specific project, but you build it on your whole data.

You can build a data fabric that gives you a head start the moment you want to solve specific data problems. For me, the relevance of the insights engine is fundamental, because you don't have to worry, as an innovation manager, that you need to get data, you'll need to have it enriched, you need to manage the entitlements, but you can right away start with the core problem and solve the actual use case that is at the core of the question.

Lauren Hawker Zafer
So would you say that an innovation manager is the key target for an insight engine?

Moritz Müller
That is a good question. I think you need to really look at this from different angles. There is definitely a key component for a Chief Data Officer, also the Chief Innovation Officer, or Chief Technology Officer. They need to adopt the concept of the insight engine to be at the center of their data architecture, on whatever they do. So in the bigger picture, they need to clearly have an insight engine as part of the architecture. But the inside engine itself is, of course, not only open for a CTO, CIO, and so on and so forth. But you also have a significant value add for business stakeholders, product managers, or data scientists; all of these have different perspectives. And I'm happy to go into further detail. But let's just look, for instance, as a product manager, if you are a product manager inside of a big organization, you need to have a platform to allow you to build your technical product instead of starting from scratch. You can use an insight engine, there are all the components that attribute to build on top of this and manage, for instance, your model life cycles, and so on.

Lauren Hawker Zafer
So from what I'm hearing you find that it is a key component to build upon?

Moritz Müller
Absolutely. It is a key component to build upon. Let's say you are a data scientist that analyzes data, you build a model, you build analytics - you don't want to have something that you do only once, but that is continuously analyzing newly incoming data. Let's think, for instance, about news or emails. You want to continuously run models on this. And an insight engine can give you the tools that you need to completely productize this setup: to set up a model, run it in a live dashboard that is exposed to the business stakeholders - that's why it's so important to the business. They can give it to the actual relationship managers, for instance, to get live analytics on top of the data. And for the data scientists, the fantastic part is that they can just work now with the business stakeholder again, to refine the model and have a quick and easy way to release an updated version of the model. All of those are capabilities that an insight engine should provide.

Lauren Hawker Zafer
Excellent. So I think that obviously, yourself, when you're touching upon the smoothness, the accessibility, and relevance that is provided for these key individuals in an organization, and you're touching upon some of the capabilities that is provided for a data scientist, that's provided for a product manager, then I think it would be really interesting for us as an audience to understand the actual critical capabilities of the engine and maybe what the technical setup is. I think it would be wonderful if you could elaborate on that, so that we can really start to envision here, what the setup looks like, and what it is capable of.

Moritz Müller
To cover all aspects of an insight engine will probably take quite a lot of time. But I think the best way to look at it is to be very concrete. What are some of the key functionalities that you need? So, one of the big dangers that I observed when talking to business stakeholders, data owners, and data scientists in financial services is that you need to make sure that your insight engine is modular. You don't want to go in with a platform approach, saying everything needs to be done on my insight engine platform. You need to make sure you provide the interfaces, for instance, to plug in any kind of external model, the data scientists will have their very personal preferences for the tools they use to build models. So, you need to guarantee that the models they build can be run as part of an enrichment process of an insight engine.

At the same time, an instance engine should also give you a tool for business analysts to build models without being exposed to too much code. That can enable them a head start, whenever they start to build specific models. Imagine they can just go in and label sentences and click on "build a model out of this" and immediately understand "what I'm doing here makes actual sense because I can build this." They can later on still engage with the data scientists. This is another key component. Then you should guarantee that the whole architecture is built such that you can connect to the data sources, so you need to have the respective APIs. You'll need to make sure you own the data models. The insight engine must fit directly in the whole data architecture of an organization. You must connect the data lake if you are not the data lake itself, you must connect to external sources. You cannot just expose all the data to anyone that wants to work with this, you need to guarantee that only the people that have the entitlement to see the data are exposed. At the same time, you want to make sure that the outcome of the results can be surfaced with any kind of dashboard and integrate the respective dashboards - or the versions that you need to visualize the data - can integrate into the respective established solutions and software that's used.

Let's say you generate customer analytics, on top of call reports, on top of the news, on top of documents, and on top of structured data. You want to make sure that this information is serviced, for instance, in a CRM system. It is a very bad experience nowadays that any business person has to go to three different kinds of terminals, places to find information. An insight engine needs to guarantee that it's performing this by providing information where it's needed because only then it is actionable. So there are so many components that are part of this, all of them really matter.

Lauren Hawker Zafer
That is a really wonderful overview that you've provided us with: the components that you've touched upon, and the fact that it should be modular. You've spoken to us about the necessity for it to enrich data, display results. Also, dashboarding and delivering data to various touchpoints. How does Squirro align with those capabilities that you have highlighted there?

Moritz Müller
Of course, it's not always easy to conform to all of that. We've tried to provide all of those functionalities. And sometimes it's not easy for us. When we want it to be, for instance, modular, it is very clear that as a product company, you want to keep a user or platform as long as possible. I'm personally convinced that if you do this you actually prevent adoption. And as a software company adoption is key. So even if it can only be one step, the actual enrichment step, or even if a client wants to do with us, purely the whole NLP components or machine learning classification components, we should be happy about this.

And that is why - as head of product - my focus is really to guarantee that we are modular, that there is no fear from the business that they buy into a solution that makes them too dependent. I believe that by being modular and enabling the business to connect to any kind of service they already have, and also to be able - for data scientists - to provide their customized models and still run them on our insight engine, we can provide a significant value to all the product managers. Because they don't need to worry about the dependencies, they don't need to worry about the fact that suddenly, a new data scientist comes in and says “I have a much better model, but I cannot deploy it on the insight engine”. And at the same time, we guarantee that if the business comes in now and says, "oh, we are switching our CRM system from A to B, from Salesforce to Dynamics - or the other way around," we can still guarantee that our plugins and dashboards work, and that is the key component for us. And we are trying to drive this.

I also believe that in the future, there will be a lot of talk about low code or no-code environments, and for me, the key or value there is it gives you a head start. And that head start is key. It's not necessarily allowing you to have more sophisticated models. Too often I have business users or business stakeholders come to me and say, "look, we have a very clear idea of the problem. - the business has a clear understanding of what they want. - “but I don't have the capacity for data scientists to kick off complex processes." And having such a tool like we have, for instance, what we call our AI studio, where you can start off building models as a business analyst is fundamentally important to unblock people that want to explore data analytics. And because we keep the whole setup modular, data scientists can still take that initially labeled data and that initial model, later on, and evolve it to an even better model, and manage it over time, because models drift and have to stay up to date. And for me, being aware of this component and managing it, and keeping it moderate, I think is something that we have now understood. And that is also how we should go about it.

Lauren Hawker Zafer
You've mentioned maybe some of the restraints or reservations that you come into contact with when talking to businesses, and people involved in the decision of whether they want to implement an insight engine, and maybe also trying to understand the sort of head start it gives them if they were to implement it. You've convinced me, with your argument, and obviously, the factors that you've touched upon, but let's say I were in the position of having to sell the technology myself, having that understanding that you have provided - to my internal stakeholders and budget holders. What fundamental do I need to understand and what do I really need to get them to understand? I mean, is it the emphasis that this tool will give you a head start? It's unblocking these resource constraints that you've mentioned, obviously, with concerns around having a data scientist to carry on, etc.? What advice would you give those having to do this, Moritz?

Moritz Müller
There are quite a few learnings that we had in the past years working with clients. I think what is fundamentally important is, whenever we talk about insight engines is that you come across the word AI, machine learning, and it is very much about expectation. And I can tell you there can be very different expectations. Expectations from business stakeholders, data scientists, from product managers. The key part is to be aware of those, to be aware of how those expectations can be handled with the insight engine from the technology point of view. And very often, our users come to us and say, "look, we have a problem that we want to solve with machine learning," but they don't even have enough data for machine learning.

So for me, the key advice is: an insight engine gives you the tools to unblock these people and keep all the expectations and control. So make sure that you have such a key framework. Because if you build a product internally, you need to deal with data: all the tasks of data governance, data enrichments, entitlements, all modeling governments as well, and so on and so forth. You need to do them for your project. But instead of doing it just once for your project, put the whole thing in a framework, which can be an insight engine, so that you can leverage everything again on the next project. And I think that is, for me, a key advantage of using a tool such as Squirro for instance, that you start to do it in such a systematic way that it is not unique to the project that you're working on. But you can use all the puzzle pieces that you put together there, to give you a head start for the next analytics project. And I think that you will convince every stakeholder, be it on the business or data side, by telling them and convincing them that once you do it systematically, you profit further from this thing.

Lauren Hawker Zafer
So, I think that that's really good advice to ensure that we can really manage the expectations of all people that will be involved in the implementation and the whole project itself. And obviously highlighting that it can provide a framework and that they can also leverage it in the future is something that should be convincing for a lot of individuals in the organization.

Now, in a podcast series, we always like to address topics realistically, and candidly, as well. So taking this topic and ensuring that we can address it from all angles, I'd like to maybe look at it from a more reactive perspective. There are always challenges and obviously, you've just spoken about the learnings that you've undergone in the last few years, at being not only a product owner, but obviously heavily involved in the implementation, and conversations with clients, the delivery of such products. And from that, you have a wealth of experience. So I want to ask you, what are the key challenges that we also need to highlight to other stakeholders or project members when we're thinking about the implementation and setup of an insight engine in an organization?

Moritz Müller

There are a few key challenges. It really depends on which role you play, which are the main challenges that you will face? For business stakeholders, it might be how do I justify the return on investment on an insight engine? A single project alone might not do the job, you need to maybe look at the bigger picture saying the next one will be much cheaper because we do it properly. And if you look at the data scientists, it might be the same, you might need to do quite some convincing work to convince the data scientists that his buy-in into such a setup can help him, later on, to deploy and manage models much quicker, because he has a whole framework for this. And at the same time, you want to make sure that when you talk to the business, they understand why you are addressing it in a systematic way by leveraging an insight engine and not just building a very customized solution. These are all things that one needs to look at.

There can be all kinds of challenges coming that way. I think the key part for me is not to go in and solve the problem by saying "this is the technology we need to use." But be open to it. An insight engine gives you the tools to solve the problem in the best way that the engineers realize that, manage that you get the buy-in to deliver the best solution and the best outcome from it. And you just need to demonstrate this once. Because you will then also convince the budget holders that their investment has been very wise because, in all the future projects, they will have much more time to spend on the actual data analytics part because the framework has been established for deliveries.

Lauren Hawker Zafer
Do you think that there's ever an element of fear of a lack of ownership? Do you see that as well? That the possibility of an emerging technology - like an insight engine might take away the ownership of certain tasks for certain individuals?

Moritz Müller
There is of course a risk in terms of ownership. I think one of the things that are often forgotten is when you adopt an insight engine, it's not something that you deploy once, and then it does the job forever. You need to make sure it's maintained. By maintained, I mean you need to make sure that the data connections keep running; there might be some changes; that your enrichments are consistent; that you have the same enrichments across the different datasets, then you guaranteed that the models - that can be very project-specific or use case-specific - that the models that are typically drifting are maintained. So you need ownership, you very clearly need ownership. But by introducing this with an insight engine, because you have a central place where the components are being maintained, I think it allows you to control the efforts much better. And if the efforts are well defined, and there's a clear framework, the individual persons have much more time available to focus on their actual task, be it model building for data scientists or finding out about the relevant client activities for users.

Lauren Hawker Zafer
Excellent, yeah! I'm sad that we have to start to usher this conversation to a close, Moritz. We've covered a plethora of key benefits and considerations, and now we should all have a wonderful understanding of what an insight engine is. I want to take this opportunity to thank you for sharing such valuable first-hand information, Moritz. And I'd also like to ask you if you want to share any final words that you feel we've maybe not exchanged or conveyed in this session up until now with the audience?

Moritz Müller
Yeah, first of all, thank you as well for having me. And thank you for asking some very concrete questions. I think our clients and stakeholders out there can learn a lot from this. I think one key aspect that I want to give along is the following: if you look at the money spent on these data projects, then for every dollar spent on building a solution, nowadays, you spend up to $100, on maintenance, implementation, and so on. That is something you need to be aware of. And I also am fully convinced that the insight engine allows you to reduce that number that is required for maintenance and implementation, because everything is streamlined, and you have a whole framework in place to do these kinds of things. That might be, in the long term, one of the key benefits for your organization that you have as an outcome of adopting an insight engine approach.

Lauren Hawker Zafer
Wonderful! I think that that's really good parting advice.

I want to thank you, and the audience too, for listening today. If you would like to find out more about the critical capabilities of the insight engine, then head over to the Squirro Academy on learn.squirro.com and access our educational material.