Darwin At His Best: The GenAI Race

Charles Darwin is renowned for his theory of evolution and natural selection, concepts that seem particularly apt when observing the GenAI space.

Looking at the GenAI space I get that same feeling. What looks easy at the start is rough in detail and very selective. Let me explain: 

Scaling a GenAI solution to large-scale production levels requires meticulous alignment of various components

Each step (and a few more) is essential for a full production rollout, and without prior experience, each comes with a steep, costly, and time-consuming learning curve.

Let's examine these steps: 

- Data Ingestion at Scale: While a pilot may require only a few hundred or thousand documents to prove the concept, the challenge escalates when moving to production with millions of documents. Ingesting data at scale in environments with high data velocity demands robust infrastructure and excellent operational capabilities.

- RAG Search: Sounds simple, is complex. All too often the systems out there produce seemingly good but factually incorrect results. After all search is probabilistic, so are LLMs. Multiplied probabilities don’t yield better results. So, you need to work hard on the retrieval part to really get good results, e.g. by factoring in better enriched data to produce more accurate result sets before exposing them to an LLM to produce a natural language looking result (and by using the LLM what it is good for, that is text comprehension and text generation and not what it is not good for, that is search). 

- Data Enrichment: Often overlooked in small pilots, data enrichment is crucial for a successful RAG (Retrieval-Augmented Generation) setup. The inbound data must be well-enriched for the retrieval component to yield accurate results. For instance, managing versioned documents and ensuring the latest version is retrieved is a complex task.

- Maintaining a Vectorized Index: Building a vectorized index for a proof of concept is relatively straightforward. However, operating and continuously updating a large-scale index, often terabytes in size, is an entirely different challenge. And simultaneously, you need to master the art of updating a large (TBs of data) index while querying that same index in that same moment. 

- Security: Security is critical in any enterprise setup. Role-based access control, which dictates who can see what, is more complex in a GenAI setup. Ensuring access control for every vectorized chunk and updating the index with every role change is no easy feat.

- Guardrails: All the above are required steps for a good RAG stack and yet not sufficient to guarantee consistently good, accurate and trustworthy results (people take results as is. Why? That’s for another blog post). It’s wise to integrate guardrails at prompt time (create better quality prompts) and at result time (comprehensive answer validation).

- Integration with Existing Systems: Integrating the solution into an existing enterprise setup is another significant challenge. Simply adding another dashboard is not useful. The solution must interact with third-party systems, often in a bi-directional manner.

- Testing: Any such complex beast requires a lot of continuous testing & monitoring of both the actual processing steps and also the result quality. Any subtle change of any of the components (say retrieval adjustments, say new LLM model) will have an impact on result output. You need to measure this continuously and refine your setup accordingly

- Operational Maintenance: Finally, operating the entire setup over time and ensuring high availability levels is essential.

- Measurable ROI: A common oversight is the lack of a clear-cut business case detailing the cost-benefit ratio of any such rollout (they are impressive if well done*) By mid-2025, CFOs will start asking tough questions: "Where's the bang for my buck?" 

Delivering reliable, accurate, and transparent AI at scale is not easy, and is costly, and yet, if done well, it will yield massive benefits. At Squirro we’ve been doing this for a while - we happily share our experience. Have a look at our Knowledge Hub resources  or simply get in touch! 

PS: They are impressive, indeed, if you go beyond search and see this AI transformation as the biggest opportunity since the dawn of the (commercial) Internet of disintermediation and reintermediation of entire value chains. We will discuss this in a forthcoming blog post. 

Would you like to get real-world examples and insider knowledge from industry experts? Tune in to our latest webinar recoding below where we dive into the latest on reliable, accurate, and transparent AI at scale.