Across industries, organizations are drowning in information, struggling to extract the full value of the data they possess. Generative AI’s arrival is timely, but it too faces limitations, particularly in its ability to understand and interpret data accurately. To operate effectively, it needs a clear understanding of the content and context of the data it draws on.
By systematically tagging content with relevant identifiers, data classification can help create the structure and clarity that enterprise GenAI platforms needs to generate accurate insights from data. Growing data volumes are, however, pushing traditional data classification approaches, e.g., manual or machine-learning-based tagging, to their limits. These methods often fall short because their tagging labels may not align with the organization's internal knowledge, which is typically stored in their taxonomy and ontology management systems.
It’s against this backdrop that we created the Squirro Classifier, an AI-driven solution that streamlines data classification for the Squirro Enterprise GenAI Platform. This article dives into the Squirro Classifier's value proposition, exploring why it matters, how it works, and what tangible benefits it brings organizations.
Why Data Classification Matters in Enterprise GenAI
The democratization of AI has raised the bar for search accuracy. We’ve grown used to standard search, which offers users a list of dozens of potential results to pick from. Today’s GenAI platforms, by contrast, are expected to deliver the right answer in one shot. To succeed, they need a much more nuanced understanding of both the search query and of the data on which they will base its answer.
Particularly in enterprise settings, retrieval augmented generation (RAG), has established itself as one way to achieve this. RAG augments user queries with additional context like user's profile, preferences, previous queries asked etc. It then searches internal and external data sources based on the refined query before passing the top retrieved content on to the large language model to synthesize the final answer.
Ultimately, the quality of the generated output hinges on the relevancy of the relevance and accuracy of the top retrieved information, in the context of the user's query – as they say: garbage in, garbage out. Providing the LLM with the most relevant input improves its ability to deliver highest-quality output.
And this is one area where data classification plays an important role. By categorizing information based on relevance, accuracy, and context, data classification reduces noise, improves retrieval precision, and minimizes hallucinations. This leads to more coherent, factually grounded responses, making RAG-powered applications more reliable and effective.
Known Entity Extraction and Machine Learning Classifiers
Until now, the Squirro platform enabled data classification using two methods. Known entity extraction (KEE), the first of the two, allows organizations to extract specific entities, such as company and individual names, from unstructured data. For example, when a report on top-performing investment funds is uploaded, KEE can be used to tag the report with the fund names and financial institutions it mentions during data ingestion. This ensures the report is surfaced when relevant to a user query about a specific fund.
The second method for data classification uses machine learning. Squirro’s AI Studio no-code platform makes it easy for clients to develop machine learning classifiers capable of classifying documents based on their content, allowing them to train machine learning models to classify content based on sentiment, topic, or other attributes. As with KEE, this provides additional context and understanding, enabling the enterprise GenAI platform to generate more relevant outputs.
The Fusion of RAG and Knowledge Graphs
Perhaps the most important recent development in improving accuracy and contextual understanding of generative AI came with the fusion of RAG and knowledge graphs, giving rise to GraphRAG. By structuring information into a network of nodes that captures relationships, hierarchies, and real-world context, knowledge graphs enable RAG to deliver more precise and consistent outputs, leading to deterministic information retrieval. Unlike RAG alone, which may deliver slightly different answers to the same question, knowledge graphs help ensure that the exact data is retrieved every time.
Through our acquisition of Synaptica, a leading provider of enterprise taxonomy and ontology management systems (TOMS), we are able to tightly integrate enterprise taxonomies and knowledge graphs into our Enterprise GenAI Platform. The solution was, however, still missing a critical component needed to relate the vast volumes of unstructured enterprise data our customers have to contend with – internal documents, market reports, emails, etc. – with their granular and highly curated enterprise taxonomies.
What’s New About the Squirro Classifier?
The new Squirro Classifier bridges the gap between unstructured data and structured knowledge. It achieves this by streamlining the classification of documents against existing enterprise taxonomies. Creating a data classification pipeline from scratch and maintaining it as new tags are added to the enterprise taxonomy takes expert work. The Squirro Classifier lets organizations benefit from the precision of the knowledge graph that their own expert taxonomists have already curated.
Building on our existing platform capabilities, data enrichment pipelines, AI Studio, and our taxonomy and ontology management system, the Squirro Classifier lets users train and publish specific classifiers, which then become part of the data pipeline workflows.
For organizations that have spent decades meticulously curating their enterprise taxonomy, the Squirro Classifier offers a simple way to increase the potential of their enterprise GenAI initiatives while also increasing the ROI on past efforts in taxonomy building. It’s no wonder that we are seeing growing market traction for this enterprise GenAI and knowledge graphs with the classifier as the bridge between the two.
What’s Next For the Squirro Classifier?
Going forward, the Squirro Classifier will be extended to support greenfield scenarios – organizations seeking to leverage highly accurate GenAI that lack a carefully curated knowledge graph. By automating the generation of candidate taxonomies for internal subject matter experts to validate, it will help shorten the time that it takes to generate a knowledge graph and reduce the time-to-value of graph-informed enterprise RAG deployments.
Are you ready to revolutionize your data categorization? Contact us today to learn more about the Squirro Classifier and how our data classification solutions can benefit your organization.
