Alright. We do have an exciting topic today. We have a lot that we wanna unpack with you. So I'm gonna kick us off. My name is Juanita Oguin. I lead product marketing here at Coveo, and I'm really excited to unpack this topic with you. But I'm also really excited to have our VP of machine learning, Sebastian Paquette, here with us today on our presentation. Seth, thanks so much for being here. We know you're you're busy building all the AI, in the background. So welcome. Thanks, Juanita. Today, we are going to talk about the best retrieval method for your RAG and LLM applications. Before we jump in, I do wanna do a little housekeeping. So, for those of you joining us, we do wanna hear from you over this next sixty minutes or so. So feel free to submit your questions or your thoughts in the chat or the q and a. We will be recording this session, and you will get a copy of this so that you can, you know, digest it a little bit further because we are gonna cover a lot of a lot of great things today. So with that, again, please submit your questions and chat with us along the way. We'll monitor it. I'm gonna jump right in. And for those of you that are in this space day in and day out, you know we have to start by talking about the Gen AI hype cycle. In particular, I wanna talk about where where it started and where it's at today. Back in the middle of last year, we saw some amazing forecasts. McKinsey projecting that there would be up to four trillion dollars in value and benefits across multiple different use cases. So we saw that excitement. We see it to this day. Everyone really wants to deploy this amazing technology in as many places as possible. Fast forward just a few months, so to early this year, we start to see a little bit of challenge. We see concerns over AI data quality, some old sayings that we're all familiar with, garbage in, garbage out, started to emerge. There were questions about whether generative AI and data quality could coexist. And then fast forwarding a bit to this fall, we start to really see investors, the market really question, is this big bet going to pay off? So really a true hype cycle, and I know you are all all feeling it, probably on a daily basis. We have to ask the question though of why. Why is it this case? Why are we here today? And I think you can agree with us. It is much harder than it looks to deploy Gen AI, and, also, anyone can show a nice demo. Right? A demo isn't necessarily reality. So I'm gonna invite Seb to unpack this a little bit further and talk to us about what's going on in this space. Thanks, Juanita. And, I'm gonna talk about, to to give you a sense of why is it harder, give you, some challenges that we've seen in the field, with enterprises that try to build the, their Gen AI experiences. The first one we've seen is that companies, add multiple, siloed POCs going on in parallel at the same time. Many of these initiatives were redoing some of the data retrieval part to get access to documents. So at the end of the day, they were wasting a lot of time and resources duplicating efforts between these different side of POCs. We also, when they were trying to move from POCs to, to their production environment, they ended up having extra requirements and some of them being around security. So if we're if you're deploying an AI models, then you need to adhere to some really strict regulations, that can be complex to implement within an enterprise. And at the same time, multiple enterprises have content that is secured, meaning that not all users have access to all the content. So, being able to, propagate this, the access rights on the documents to their generative experiences is, can be time consuming and complex to do. Then they do have a lot of integration complexity that can happen, because they need to connect to multiple data sources, each of them using different APIs, different data formats. So just connecting to all these sources can be really time consuming if you want to scale from okay. You manually copy the data for your POCs and demos, but then you need to really plug the system together, which gets really, more complex. And afterwards, you need to scale your POCs. And just scalability is a challenge by itself as you need to be able to in the to, provide and retrieve from millions of documents, from hundreds of sources. You may add thousands or millions even of users, using your system. And just trying to scale all the components, can be really time consuming. And this is one of the reason we've seen a lot of POCs lag over time and not reaching production fast. And finally, an an important challenge is, data retrieval, which is really complex when you have to retrieve content from multiple data sources. You have to build complex data pipelines. And at the end of the day, it can be really hard to, have good relevancy, for each specific question. And if you don't have good relevancy, then it will really hinder your AI generation, making it less accurate, for your end users. So these Barca, a sample of the challenges that companies can face when they are trying to move from their demos and POCs to their production environment. And one of the approaches they've been told is really useful is to use the retrieval augmented generation approach, which is, really important. But first, let's start by explaining what it is. So if you look on the left, if you have a question, then you can ask a retrieval system to retrieve the information you have, in your enterprise and to use this information from the retrieval system to augment the prompt with, your enterprise information. And this is really important if you want to ground the LLM generation into your enterprise knowledge and reduce the hallucinations. If you look at these, there's three main components, the retrieval augmented generation components. If we look at the generation part, most companies have access to the same LLMs that are out there. So it's not really where you're gonna get a big differentiation. As for prompting, there's a lot of, prompt engineering, techniques out there. You can have best practices for that. It's not where you're gonna spend most of your time, but, you it's quite easy to, build demos, because prompt is not that hard to write, at first, and then you can good get nice generation. The complexity arrives when you're really trying to, to deploy a a strong retrieval system within your enterprise and get access to all your documents. And this is really where, the RAG solutions with will differentiate from one another is actually the strong and the, the capabilities that your retrieval system can provide. It's really where you're gonna get, the differentiation between different RAG approaches. If we, look a little bit closer, if you want to real to build a RAG system, what you're gonna need, if we start from the left, is first secured connectivity. You're gonna need to connect to all your sources. You're gonna need multiple connectors that can understand these sources and extract the information from it. You're gonna need to be able to do that frequently so that you can, your content can stay fresh within your rack system. And with the content, you also need to index the access rights on these documents, which can be complex, but it's mandatory if you want to be able to make sure that your generation doesn't use information the user doesn't have access to. Once you get all this information coming from your, secure connectivity layer, then you're gonna need to, parse it, chunk your documents in different in smaller sections. You're gonna need to index that in an efficient retrieval system. You're gonna need to generate vectors and generate embeddings for, your different, text passages so that you can have a semantic search on top of it. So all of this needs to be done at indexing time so that your retrieval system has a rich information to provide strong retrieval. Then you need a strong retrieval system that can mix different ranking, approaches using keywords, using semantic understanding, even using AI to provide the best results possible, for, for any specific question. At the same time that you want AI relevancy, so your system being able to, improve by itself, get better over time, get more personalized. You also want to keep some control over the over the retrieval so that your retrieval is really, tailored to your specific business needs. So you need a way to create and maintain business ranking rules and such. All this blue part is what is required, and it's only subset of it, for a strong retrieval system. Then you can do the augmented generation part. So you can create prompts using the information provided by your retrieval system. At this point, you will need to maintain the prompts because the prompts are different for each experience, but they are also different if you change the LLM version, which the LLM providers are asking you to often. Then you need to generate, the answer itself, which will require you to manage the LMM deployments, making sure that you can generate the answer, that you manage the no answers. And most of the time, you would like to stream the answer back so that the end user can see in the UI the answer being written without waiting for the full answer to be completed. So this shows the main components of a RAG, system. And, where you're gonna spend most of your time is actually on the blue one on the left, if you want to rebuild everything. But is it where you really want to spend your time? I think most enterprises would like to spend their time in building the generative experience that is really specific to their enterprises, and that will really bring value to their enterprise and to be able to, use a strong retrieval system and not have to rebuild it, manually. If we go even deeper into what it is, a strong search retrieval system and what you need to to do to build it, I won't go into the full list. I explained many of them, but you will need a a a unified index with a strong ranking, layer so that you can have the best, information possible for your generative experiences. You're gonna need a lot of secured connectivity to all your different, sources systems that is able to be refreshed frequently. You still want control. So you still want filters, boost, bury, ranking functions. You still want the flexibility to control it and the AI power to make so that the system can be as best as possible, can improve over time, and can be personalized for each user based on its, real time actions right now on the website or his historic historical, information. And at the end of the day, you want to measure all of this. You want to make sure that your investment is actually paying off, that the system works well. So you're gonna need analytics and reporting to make sure that everything works well. And this to build all of this, is quite complex, and we should know. And I I pass the ball to, Wanita for explaining it to you why. Thanks, yes. This is a lot, but it's important to acknowledge and understand what it takes, right, to build these systems. For those of you that are not familiar with Coveo, we've been doing this for about twenty years focused on the toughest information management, product catalog enrichment, AI development over the last decade. So we really have a ton of experience doing this day in and day out for the largest enterprise companies you can think of. Coveo is one platform, an AI search and generative experience platform that gets used across enterprise use cases. You can see some of those here, website, commerce, service, workplace. And just to give you an extent of our size and our scale, we have over seven hundred enterprise subscription customers, a team about seven hundred to seven hundred and fifty spread across Canada, the US, Europe, and Asia. As I mentioned, you know, we'd like to think Gen AI is, you know, new, but AI has been around for a long time, and we've been focused on that for the last ten years years or so. Do wanna share we really do take, you know, our work seriously, including, being part of the one percent pledge where we give back our time, resources, and volunteer volunteer hours. And as you can see with all these logos across the bottom, we really do work with some of the most, known and renowned technology ecosystems from SAP to Salesforce, AWS, mock alliance, and more. And if you're still questioning, you know, we what do you what do you have to say or share about this? Look. We've been in this space for a very long time, and our customers and the market trust us and rank us consecutively as leaders for several years. And so here's just a snippet of some of our analysts, including IDC, Gartner, and Forrester. So we're really proud of the work we've been doing, but also wanna make sure that we're sharing that back with you all to help guide you in your different LLMs. We talk about retrieval. We talk about RAG, but we need to be talking about search. And I wanted to take a moment to explain why search in case you're wondering. And the fact of the matter is that search is not what it used to be. What you see this image is an illustration of what a modern day search experience and search page looks like. Yes. You're all familiar with the search box, but this is really an intent box. It's where customers go when they are seeking information and answers. But each of these blue boxes and each of these different components of a search page are essentially powered by different types of AI models. Of course, you have a generated answer. You have recommendations. You have next best questions. You have the ability to do follow-up questions. People sometimes don't wanna search, so you wanna recommend relevant things to them as well as give them that dynamic filtering and faceting on the left hand side to be more precise about what they're looking for. So just to kind of double down on this a little bit, you know, search really has changed, and it is the primary way customers start and begin their research. They go to your sites. They trust your sites. They're trying to find that information. It's both the technology and experience that is there to connect people to what they're looking for no matter where it's at within your organization. It's also a listening opportunity, and I feel a lot of people don't, consider search in this way. But isn't this a very direct way that your customers are telling you what they want and need? It also represents a high intent inbound opportunity. When we do an external report, we pull external workers. And from a survey of four thousand people across the US and the UK, we learned that forty two percent of them go directly to a brand's site when they are seeking help or information. Forty two percent. So you might not think your site is important, but it really is, and you don't wanna waste that high intent inbound opportunity. You wanna convert that, right, and connect people to what they need. It's also your opportunity to be relevant to your customers, your prospects, your stakeholder, whoever is searching. This is, you know, your your chance to really show them what you have to offer and deliver. And it's also an opportunity to have one retrieval method across the enterprise, and I'd like to double down on this one a little bit more. If we think about an enterprise and we break it down into different departmental organizations and consider that our departmental teams actually represent different parts of the buying journey, different parts of the digital customer experience, we can see that we have really, different teams that are managing parts of the customer experience. You have marketing who is managing your websites, really concerned about conversions and pipeline. If you have a commerce or transactional component on your sites, your commerce teams are really worried about that average order size value. Once a prospect becomes a customer, you know, they're coming to your sites and your teams are really focused on just delivering great customer satisfaction, resolving those cases as quickly as possible. And then your your agents who are employees as well as your other knowledge workers are also there, right, selling to, supporting, you know, these end customers. And so different teams are managing different parts, sometimes only focus on their own areas. And what happens is that because they're focused on their own areas and have their own budgets, they get to decide which technology they're going to buy to support their different needs. Now this ends up resulting in that siloed approach, siloed retrieval, siloed search. A lot of these platforms really can't go beyond their own content and knowledge. And all of this, you know, internal siloed tech knowledge and content really is what, you know, end customers experience as fragment fragmented, you know, frustrations, can't find what you need, can't get answers. Not really a good state for your enterprise and your brand. And, of course, these days, we're being told just add AI on top of these things. But adding AI on top of a siloed, fragmented, fractured system and reality is really not gonna solve the problem. And we know that this is something that is worth solving and considering when we look at external statistics like this one. Three point eight trillion dollars of revenue is at risk this year alone due to bad customer experiences. Now I like this number because we typically talk about productivity, reduced cost, CSAT when it comes to CX, but this is revenue. Three point eight trillion of revenue is at risk due to those bad experiences. So there is definitely a risk in not getting this right and not considering an alternative to how you can really create those unified intelligent experiences across your digital properties. So I'll pass it back over to Seb to take us through some possible solutions to how you can do this today. Seb, over to you. Thank you. And, we're we're gonna first start by looking at, our own experience building a generative answering solution. We built what we, call COVID relevance generative answering, which is a solution where, when people are asking a question within a search interface, they get a generated answer out of it, on top of the search results. I'm gonna show you, why we were able to deploy that, really fast, on public websites, and be confident on the quality of the answers. This is because we didn't, start from a white page. We started with one of the strongest, search platform out there. So we had all the connectivity, all the indexing components, all the relevance AI models to make sure that the, information is accurate. All the other, UI, components or all the other components necessary to show the answers within our customers' applications, and everything afterwards to track what their what their users were doing. So what we did with generative answering is that we simply added a small component to actually use the the return information and to call an alum to generate an answer. This is because we were on top of a strong search platform that we were able to get accurate answers and to be able to have a good answer rate for these questions. If I it a little bit and and put it a little bit simpler, so we add the retrieval part, which is the left part where we were able to securely connect to, different content sources. We add a really powerful unified, hybrid index that can do, semantic search and ethical search and a really strong relevance layer that can mix and match the different ranking, capabilities. And then at the center, we have the we have the API that returns the best passages or any specific questions. And then we built on top of it the relevance generative answering component that can create the prompts, call the LLM to generate an answer, and also include some of the UI components and eval evaluation tools necessary to, provide the answer to the end user and to evaluate its efficiency. The advantages of this managed solution is that all the complexity is taken by Coveo. We manage the prompts. We change them to each LLM versions. We manage the LLM scalability so that it can, it can scale with with your, number of users. We're streaming the answers. We're providing the UI. So this allowed our customers, our new customers, to take from four to six weeks, to have a generative answering solution. And if it was already a Coveo customer, so if their documents were already in index in Coveo, then we're talking about a ninety minute setup to add generative answering in a search page and start testing it. So this is really, really fast time to value, and it provides a lot of benefits to our customers. But to show you a little bit, what it looks like, let me give you some examples. So here at, Dell, we're powering the search support page, where if a customer is asking a question, you get a generated answer that, directly answered a question on top of the search results. So if there's if the answer is not complete enough or if there's no answers, you still have the search results to look at. You're also in a rich search interface, so you still have the filters on the left. If you filter, for example, the the specific product you have, then you're gonna get a different answer. So everything is really integrated with, where the user are searching for information. This shows that we can deploy generative answering with a high volume of searches, with complex content. In this case, it was complex hardware and software manuals where we needed to extract the information from to be able to answer the questions. And they also have, websites in a lot of countries and a lot of languages. So we deployed, multilingual support where we can generate answers in French, German, Spanish, and multiple other languages, for them to support their, worldwide users. Another example is Xero, a company building and accounting software, where within their search interface, we, generate we deployed the generative answering solution, which rich answer that shows really well formatted answers that answer many of their questions, directly within, within the app. And since they have, app, we also deployed the in product experience where if you hit the help help button on Xero app, you're gonna get a generated answer directly within the app so that users can have access to all the information directly within the app where they work, and they have the generated answers specific to their context, what they were trying to do in their specific workflow. So this is really powerful to to bring the information directly where the users are working at. These are simply two examples of where we deploy generative answering. And what is I think we can use it in many more use cases. Yes. We can. And for those of you that are customers on the call, you know that as well. The way that we're doing this and the very builder out of the box based approach that we are providing this managed answering solution means that it can be deployed quite quickly across any of your search pages or applications. And we like to use the keyword agnostic because as I mentioned before, a lot of the AIs and Gen AIs and Copilot, you see you're very much limited to the system that, you know, introduced them, and they really are limited in how they can be used. With agnostic generative answering, you're able to provide answers leveraging your existing enterprise content using it within your SaaS based applications on your employee portals or intranets. If you have on your employee portals or intranets. If you have a case submission flow or process, consider, you know, providing generative answers there so that you're not sending cases to an agent that can easily be solved with your existing content. You can consider it for your chatbots, for your agents as well. Yes. Copilots and such might give summaries, but they might be may not provide the complex answers to products or things that agents need support on. And lastly, we even have this available for any product, catalog or commerce type situations where people might want, maybe, detail on ingredients in a product or more guidance on how to use different types of, products or that they're purchasing. We and we just wanted to share again, this is with our managed solution, the more out of the box one, just a few examples of our success from our different customers. The first is Xero, which Seb just showed you a few different scenarios. By deploying generative answering on their Xero central support site, they were able to increase self-service success by twenty percent. Now all of these are incremental benefits. They were already getting great outcomes. They got additional incremental benefits with generative answering on top. Similarly, with f five networks for their community portal, able to improve self-service success by eleven percent. And, right, what this means is if a user is able to self-service, that means less, you know, simple or or less repeat questions to your contact center. So a natural cost savings component there. Interestingly, for ForestPoint, they did have a case submission flow and deployed this generative answering or answers within that case submission flow, again, trying to deflect those cases to the contact center, especially the ones that can be, again, resolved with your existing content. And finally, SAP Concur, which I know many of you are familiar with. This was one of our most, you know, impressive results. They shared with us that they were able to decrease their cost to serve by eight million euros while reducing thirty percent, reducing their cases by thirty percent. So these are very impressive results by pretty renowned companies, and I think what Bayer is calling out is that they are deploying this on their public sites to their customers. You can actually access and test this yourself, and that just shows the ease, the trust, and the quality that they're, you know, putting out there for their customers. And a and a and a side benefit is, you know, their customers are, you know, happy that these companies are supporting some of the latest cutting edge technologies that are out there. So with that, we, you know, we we talk about our managed answering. We talked about generative answering, which is our application of Gen AI for search pages and such, which we just covered. But we hear that there's other Gen AI applications that companies wanna build. So, Seb, what are we hearing, and what do we have to offer here for those that wanna build something a little bit more customer, a little bit more business specific? Yeah. Sure. So as Juanita said, we got really good results with our generative answering solution on search pages. But at the same time, our customers told us that, yes, it's really nice. We have really good, return on it. But at the same time, we have other Gen AI experiences that we have more trouble building and putting them into production. We would have to have, we have ideas about custom Gen AI apps to generate an email, generate an article in their own format. They wanted to be able to provide to agents, chatbots, or co pilots, whatever you want to call them, the information of their enterprise so that they can be better answering questions with, their, their own in the enterprise information. And they they ask us, okay. How can we use the information provided, already in Coveo and the power of the Coveo retrieval system to power these other Gen AI apps? And this is what we're we're providing to, these customers now. As I said, we started with a fully managed solution, and we looked at it and and and say, okay. What is really the the art part about it? What takes time to build? And and what will we will really make a difference to the other Gen AI apps? And this is why we, decided to provide what our current solution is using. So the passage retrieval where we have the best rank passages for any question, this is actually quite hard to get to. But we know we have a really good quality of frame passages, and this is why we were getting so, good results with relevance generative answering. Now we're providing it to our customers so that you can see on the right that they can build their own custom radic RAG experience out of this rich information. They can pro they can build their own prompts. They're generate their l use their own LLM, build their own experiences with it. But they build it on top of a solution that provides really strong ranking, fresh data, and secured access to their information out of the box. To give you a better sense of what the passage retrieval is, let me show you, show it in action. So if you have a question, for example, what are the advantages of generative answering? If you call the passage retrieval API, what you're gonna get out of the API is actually a list of passages coming from different documents and rank from the best to, to the list. That's good to answer the question. This passage retrieval API, returns passages in a format that is easily integrated within the LLM. As you can see here, it can return passages from different documents or even or from the same documents. If you look closely, number two and five are actually the same documents, but two different passages from the same document. But they're ranked differently. So we say, okay. The first part of this document is more important than the second one. And this information, you can then easily provide it to your own LLM and to do whatever experience you want to generate out of it. Here, I'm showing you an example where I'm using, meta, LAMA three model to generate an answer. But here, you can use your own LLM, provide build your own Gen AI experience out of the passages. But being able to get access to the the right information in a format that is easily integratable within, an LLM prompt is will gain you a lot of time and give you a better accuracy and answer rate. So why consider the Kovio passageway to all API? You don't want to spend all your time building a search system to get the best information out of your enterprise. You would be better to use a trusted leader in search for the last two decades. That can provide all the enterprise, grade, capabilities like hybrid indexing, like AI models that can tune your relevancy, that can personalize the results to each user, that provides secure connectivity and native integrations to, many enterprise systems, a system where you can still have full control over the relevancy and what it returns using business rules to make sure that you you have the answers that really fit your business objectives. And at the end of the day, really get rank passages that are LLM friendly, and easy to use within your GenAI experiences. The speed you're gonna get by using it is tremendous, and you will be way more successful with your GenAI experiences. And if we step back a little bit and and don't look at one siloed app, we said at the beginning that one issue was that companies were building silos, including the retrieval part. If you you, build your Gen AI all your Gen AI experiences on top of the same retrieval platform, you will gain even more efficiency. You will have, generative answering on search pages. You can build custom GenAI apps, with the your enterprise knowledge really efficiently, and you could even provide, your enterprise knowledge within Copilot. We can provide you even code example to, include your enterprise knowledge within Microsoft Copilot, within Einstein, within Salesforce Agent Force, within Amazon queue, within Bedrock agents, any any other. But if you're using the same virtual platform for all these experiences, you're gonna spend your time and energy in in really building what makes your enterprise successful, so what makes your GenAI special, what makes your, services really specific to your enterprise, and you you will not, lose your time and energy to try to rebuild a retrieval system for each of these applications. To give you a sense of what our customers are doing with the passage retrieval API, here are five examples of customers using it right now. The first one wanted to use information provided by Coveo, but they also add other, information, internal information about their customers, and they want themselves to manage, combining the information together to generate a more personalized answer. The second customer spent a lot of time and energy building their own LMS specific to the enterprise, specific to the type of, tasks they want to do, and they wanted to, be able to ground them, ground this specific LLM into their enterprise knowledge. So with the passenger tool API, they were able to use Coveo to get access to the information and to plug it with their own element. The third one was doing a chatbot. Again, wanted to the chatbot was able to do action in their HR system, but they needed the information so that they they can use the information they have internally to answer their employees' questions. And they got really, really nice results with, with Coveo. The fourth one was trying to do something a little bit more specific. But based on the answer generated, they wanted to actually add product recommendations on top of it using the information provided by Coveo, but also their own knowledge of their, their customers. And the fifth one, is a consulting company that had, to answer different, requests for proposals, And, they were able rapidly to build an automated RFP response tool based on the passage retrieval API where they can send all the questions and get, and get the answers generated and inputted directly within their tool. So this shows really different use cases, different reasons why these these customers were using, are using the passage retrieval API. But all of them have something in common is that, yes, they wanted to build something specific for the enterprise, but they are really gaining a lot of momentum by using the Coveo passage retrieval API and not spending all their time trying to access the information. So if I summarize, what's the value of Coveo PassEdge virtual API? The first one you're gonna get is accuracy and precision. The information you're gonna get is precise, accurate, and you're gonna have a lot of control to make sure that it is. You will get an enterprise grade system that can support all the different security standards that can make sure that all the documents, keep their, access rights and that the generation will, will only generate using information the user has access to. At the end of the day, you're gonna increase a lot your operational efficiency. You're gonna, invest your resources in the things that really differentiate your enterprise, that really provide value to your customers, either internal or external. And if you're using the same, retrieval system for all your GenAI apps, you're gonna get a really accelerated time to Barca, not only for one GenAI app, but for all of them, at the same time. And this is really, really, important to make sure that you drive, important business outcomes. To give you a specific case, a specific case study, and to to show you the importance actually of having a a good retrieval system, Here's an example of a software organization that was building an HR chatbot using an internal search engine, and, they wanted to try out the passage retrieval API. Only in two weeks, they were able to implement the passage retrieval API to change their search engine to use, the passage retrieval API and to test it on four hundred test cases. What they got out of it is, tremendous results. First of all, if we compare only the retrieval part, Coveo was returning twenty two percent more often the right information to answer the questions. But what's what's even more important is that even when both solutions were returning the d n e the right information to answer the questions, we get a seventy three percent increase in generated answer accuracy because Coveo is not only returning the full document or full text of the document. We're returning precise passages that are ranked, and passages from different documents that are ranked together to provide the best answer in a format that the LLM understands well and the LLM can use efficiently to generate good answers. So even when both solutions are returning the the the information from the same documents, Since Coveo was more precise with the passage retrieval API, we got a really, really huge increase in, answer accuracy. I think we have come full circle now. Thank you for that amazing example, Sebb and sharing more about the passage retrieval API. We are nearing the end of our presentation, so you can still submit questions if you have them. If you've already submitted, we'll get to them shortly. So, you know, the the title and the topic of today's presentation was the best retrieval method for your RAG and LLM applications. The fact of the matter is our view is you actually have two options for doing this. You have our managed generative answering solution as well as our you know, more custom passage retrieval API, but both powered by state of the art retrieval architecture, which hopefully throughout the presentation, you got a sense of what it takes to build something like this that is both effective, powerful, and accurate. And just as a summary on the managed answering our relevance generative answering solution, this is that more out of the box, you know, generative answering use case, fast time to value, little maintenance. We do a lot of the heavy lifting for you, but we still provide you, robust analytics, evaluation and troubleshooting troubleshooting tools so you can dive deeper into why why you're getting certain answers. So So I put a few examples of, you know, when and where you would use these, but really for more general search and knowledge discovery use cases, you know, some of those search pages, properties, across your systems and applications. And on the right hand side, you have the more custom passive passive retrieval API where you're still able to use that best of breed retrieval capability. You still have one retrieval mechanism that you can deploy across any and all of your Gen AI applications. We know you're building many types of Gen AI applications across the enterprise. Here, you will have to do a little bit more investment, obviously, to build your prompts, the management, all of that. And so, again, put a few examples of where you would use that. Maybe you're trying out Agent Force. You're trying out some different co pilots. This is that more custom flexible solution that you are looking to build and manage on your own. With that, we do want to give you two offers before you go. Our team will post these in the chat. The first is for those of you looking for that managed answering solution, you wanna use that more out of the box. We do have a limited time offer until the end of the year where you can use this capability on your websites or within your internal employee portals with a fast deployment. So a really great offer. You saw the results and some of the customers using it today. And on the right hand side, our password retrieval API is actually currently in an early access program. So if you have some ideas, you're trying to build your own GenAI, apps, you're not seeing the best results, and you want to try this password retrieval API, product and capability, we can get you into this program and have you start using it to see the results, right, and see this thing go into live production. So really two options, two offers for you here. With that, we'd like to thank you for your time. Thank you for your attention. We know there's a lot going on even though we're near the end of this year. Seb, thank you so much for all of your insights and your guidance. We will now open it up to a few questions. And apologies. I'm gonna be looking I'll be, doing a couple of things here. The first step for you is, around the, vector database side of the house. So a lot of customers or enterprises are hearing that vector databases are really the answer. What would you say about why a vector database isn't enough, and what is it lacking? It really depends on the different vector databases, but, many of them, don't include a strong connectivity layer. So all the connectors and many of them or most of them don't provide secured access to, your documents, so being able to index the security access rights at the same time as the content. But more import importantly, the vector databases that are only doing vector similarity search, we already know it's not enough. It's not precise enough for all use cases. So you need a really richer, richer retrieval capabilities, that can mix in, multiple ranking factors. So I think, the vector database is is nice, but it's only one component of a full retrieval solution. And, it's not complete by itself, and you need more than that. Thank you. I'm gonna, ask you another one here, which is sorry. Give me a second. So we talked about how not all rag is created equal. So since there is no rag standard, do you have any tips for assessing the different RAG frameworks? How can companies make sense of this? A good question. I would say as I said at the beginning, I think the generation part and you have, libraries and tools that will allow you to orchestrate a rack infrastructure. But as I said, if you choose, one of them and you choose, your prompting engineering, techniques or tools, at the end of the day, I think you need to really evaluate the quality of your retrieval system and make sure that, from the get go, it will support your enterprise requirements so that you don't build a system on top of something that when you will try to translate it to your enterprise or your the the department in your enterprise that is, doing the production software, that it won't be blocked because, it's not using an enterprise grade system at that point. And and out there, I think you you have a lot of search. You have a lot of analyst reports like we showed that are comparing different retrieval and search systems together so that you can analyze which ones better fit your, use cases. But I think it's really important that you choose your virtual system first and then integrate it within your right infrastructure. Thank you for that. I'm gonna go to a couple here in the q and a. The first is, does Coveo benchmark against real world sample datasets such as NDCG, I'm probably not saying that right, and publish any metrics on this? Yeah. We don't have readily available benchmarks. The strength of Coveo is return decks at the enterprise, systems with rich enterprise systems. What from what we've seen, most of them are unique. But, it's something we will look, look for if we can, provide results on public benchmarks. But currently, we have put our efforts into delivering, the best system for our customers. But, we know in it could help to compare the different virtual solutions. Thank you. I think what you're saying there too is, it's tough to benchmark against sample data, and we're doing it on real enterprise client data and delivering those unique, business outcomes we shared a bit earlier. Yeah. Exactly. And I think these, these business outcomes are the real testament of the quality of the the conveyor virtual system. Fair enough. I don't know if this is a cheeky question or not, but the question is, will the passage retrieval API be free in the future? Be free? No. But but at the same time, I think it brings value. And if you consider free being that the system will pay itself, I think with the results we've shown, all the customers that bought generative answering got a ROI way bigger than the price of Coveo. So, it's paying itself, I should say, but it's not free from the start. Thank you for that, and great answer. I'm gonna ask you just one more. This is a little bit of a open question for you. I mean, given what's going on with JennyI, RAG, LLMs, all this hype, like, what are you most maybe excited about, looking forward to in the coming year? Oh, good question. I I think these alarms will get even better. I think they will get even better with multimodal, so being able to better understand images and videos. But at the end of the day, I think it's it we're we're in the new era where computers are starting to understand us when we're talking in our own language, which is the big difference with these new technologies is that we have a new interface that can understand what we're saying, that can, that can, do work, and and that is really, really efficient, that improves our way of working. And, but at the same time, it's, yeah, it's an it's a really it's really interesting to see where it's gonna go and and the type of things it's gonna be able to do. And I think for, different workers out there, you need to get on board of it and to use it to do your work to get better at it. And I think it's a it's it's tools that can make every employee even more efficient. Awesome. Thank you. Thank you all for listening. We are gonna close this webinar at this time. We will send you the recording. Thank you for your questions. Thank you for your time. And, Seth, thank you for your expertise. Thank you, everyone. Have a nice day. Take care.