Alright. Hello, everyone. Welcome to this month's learning series webinar. My name is Claudine Ting, and I work on the global marketing team here at Coveo. For those who are attending a learning series webinar for the first time, this monthly webinar is where we get more hands on with how to enable Coveo's features that can help you create more relevant experiences. So it's a new quarter once again, and we're kicking off another brand new learning series. So the first session in so today is the first session in our three part series on machine learning. Today, we'll focus on better optimizing and validating your machine learning models. Before we get started, just have a few housekeeping items to cover quickly. For this webinar, we encourage you to type your questions on the q and a box and the chat box as we go along. At the halfway point and at the end of the presentation, we'll try to answer as many of the questions as we could. Lastly, today's session is being recorded, and you'll receive the presentation in the next few days. Today, we're welcoming Jason Melnick, director of customer onboarding, and Lipika Brahma, customer success architect. They're pretty much going to be our machine learning best friends at least for this quarter. So I think we're ready to get started. Levica, Jason, take it away. Thanks a lot, Claudine. Great intro. Hi, everybody. I'm Jason Nolik, the director of customer onboarding here at Coveo. So my responsibility is to work with customers to make sure that, you know, they are set up for success. I've been with Coveo for a little over four years now. I started off as a CSM, so I work with a lot of customers. And, actually, I'm joined here by Lipika Abrahma, who was one of the people who actually trained me when I when I first joined the company. So, Lipika, I want to introduce yourself. Thank you, Jason. So and that proves I have been in Coveo longer than Jason, and, I I'm also called, like, you know, the grandmother of Coveo because I've been here for so long. But, yeah, I am Lipika. I'm a customer success architect. And my role really is to provide that additional layer of support for our customer success manager. I also started off as a CSM, And very quickly we learned, you know, CSMing is a team team sport. It's not an individual sport. So we created a team of people around our CSMs to make sure that they are successful in delivering the value that our customers expect from our product and also from our services. So, let's get started. In today's webinar, and if you haven't already because this this webinar is kind of a series and it's it's after the first learning series, which was the machine learning learning series that we did before. If you haven't already, I would highly suggest go and watch those videos whenever you can, because that will give you a bit you know, kind of like a base, a foundation to to, you know, understand everything we're gonna discuss today. But we will start off with discussing how does Coveo machine learning one work, which will give you a base for understanding everything that happens after. We will look at optimizing ML models, and this is really for customers who have been live for a while, who have who have their ML models already set up, but are really looking to okay. What else can I do now? What more can I add to this? We will look at filter context, filter fields, and ML sub models. And we'll we'll dive into, you know, how those sub models are built and how they're used for personalization. We will look at custom context, and Jason has some really great examples over here. Custom context is a very powerful tool if you're looking to, you know, personalize, your your, your site for your customers via machine learning. And then we will look at a couple of different configurations that you can do with your machine learning model that might not be available through the UI, but there are tons of options available for you if you want to, you know, optimize what you have already. So let's dive in. So when we say machine learning model, I always feel like we are generalizing way to watch because Coveo offers several different kinds of machine learning model. It's not just machine learning. It's the automatic relevance tuning model, the query suggestions model. You know, we have two types of recommendation models, event recommendations and product recommendations. And we also have what we call as a DNE or a dynamic navigation experience model. So to to kind of give a very brief intro of what these models do, the ART model is the model that really learns from user behavior and bubbles up, you know, documents or products that get a lot of clicks. So it'll learn from your searches that you do. It'll learn from the clicks that you do, and it'll bubble them up. Same with the query suggestions. It learns from user behavior and it's going to start suggesting, you know, queries that, that have similar keywords that have done by other users on your on your on your UIs. Event recommendation model, very simply put, it is people who viewed this also viewed this, but more in a setting of a, you know, like a document or a video or a training, because we have a different recommendation model when it comes to product where people who viewed this also viewed this or people who bought this also bought this. The dynamic navigation experience is what is is really directed to reduce your customer effort. You know, if your customers always have to come on your site and say, for example, they're searching for laptops and they always go and select the thirteen inch monitor. So what Coveo does is it understands this user behavior and it'll automatically move that thirteen in if that facet was on the bottom of your screen, it's gonna move it to the top of your screen. We can also auto select it so customers don't have to select it. And we can also, you know, give a very slight boost, very, very slight boost to all the thirteen inch monitors. So they they they're kind of closer to the top of your, of your UI. Now how do these machine learning models work and what do they learn from? So if you look at the ART and the query suggestion models, they, and the DNE model, they all learn from search events and click events. And really to back up the bus a little bit, search events are, you know, very simply put, searches that happen on your site, click events that clicks that happen on your documents or in your products, view events could be viewing a product or viewing an article detail page, and purchase events are if they're buying a product. Then comes event recommendation model, which really looks at click events and view events. So people who viewed this also viewed this. And then comes product recommendation model, which looks at clicks event, view event, and also purchase events because we have several different strategies when when it comes to the product recommendation model. We can add a ML model to your cart page. So people who, you know, added different things to the cart, we are continuously learning from that, and we are trying to increase your average order value, basically. So now that we kind of have a foundation on on how the ML, you know, machine learning models work or how what they learn from, let's look at a couple of tools that are there for you to optimize your ML models. So the things there are four different ways you can optimize your ML models, and this is at a very, very high level. We look at filter fields, which really make sure that we are recommending the right result to the right audience in the right place. Then comes context wheel which, you know, adjacent context and custom model parameters which Jason is gonna talk about. And these are really tools for you to personalize your your machine learning model because, you know, machine learning model learns from user behavior, but you can help it. You can make it, understand your business faster than it would require maybe a number of events for it to learn from. But you can make it learn faster by saying that, hey. This actually means this by maybe adding a thesaurus entry or, you know, maybe adding a custom context which which tells that this is my customer's entitlement. This is the product they bought. So make sure when you're recommending a product, you're keeping that in mind. And then last but not the least is the is the default configurations. And these are there when you're setting up the machine learning model. But as you, you know, as your organization matures and as you have more, you know, events in your bag, you can adjust these settings as well to further optimize your machine learning model. Now let's look at filter fields and submodels. So, this is just for your reference to kind of figure out where you can find these, things that we are talking about. So if you already have a machine learning model in place, and in this scenario, we are talking about the ART model. So if you have the model in place, you can go into the machine learning section and go into models and filter down to your ART model, and you will find this page here. This is Jason's art model here. And if you look at it, there are several different things on this site, but this is our model detail page. We will be focusing on what is on the second, like, the bottom half of the page. So let's look at it. So on the bottom of the of the page, we can see the different kinds of filter fields or filter context, and JSON is also gonna elaborate this, you know, further. But there are three different kinds of filter context. One is your language, one is your one is your search hub, and one is your tab. So this is really making sure that your customers, you know, if you have a multilingual site that the recommendations that you're getting, you're getting it in the language, that the customer is, you know, has has chosen to look at the results of. So let's look at how these submodels are formed. So, really, our submodels are formed by looking at these filter fields. So, say, for example, you have, and you've been a Coveo customer for a while or even if you're a new customer, you might have different, use cases that use Coveo. So you might have a customer community. You might have a tech hub, where, you know, you provide, you know, search results, provide, you know, personalization to your technical users where you show them videos, documentation, more technical things than what a customer community will have. So you have two UIs with us. And then in these UIs, you might have different tabs. So in this example, we are saying that we have two UIs. And within those two UIs, we can have up to four tabs. And you don't have a multilingual site and you only have English. So if we look at if we do the math here, really, the machine learning model is going to build four, eight submodels. So two into four into one. So we're gonna be build eight sub models. And this is what they will look like. So for every origin one, it's going to take a combination of a tab and a combination of a language. So really go back to your school days and remember try to remember permutation and combinations. That's exactly what's happening here. We are creating all of these submodels that have information about every hub, everything we every event we learn from every hub, every tab, every language. So if you had maybe four languages, you do the math. I'm not that good at it. That many sub models you will have, and so on. So so before I go ahead, with this, let me see if there are any questions because I see the orange light, pinging here. But if there are any questions, let me know. Okay. Doesn't look like. Okay. All good. So moving on to the next, next phase, which is you know, these are questions that we get from customers all the time. And I know, like, in today's audience, we have customers who have been with us for a while and then customers who are getting started so basically at in all stages of your life cycle. But this question comes up so often is when is your site ready for machine learning? When am I ready for machine learning? Should I wait? Should I wait thirty days, ten days? How long should I wait to to switch on machine learning? And I think Jason and I, like, whole handedly, every time the customer has asked us, we say day one, like, switch it on. There is it has never ever had a bad effect. It only learns. It learns from user behavior, and everything depends on search activity. We have customers who can reach, you know, hundred thousand queries in a day. So why not turn it on day one and, you know, have machine learning, learn all those amazing user behaviors and start recommending results right away. So I'm just pause a little bit and see if Jason has something to add here. I see him nodding his head, so and then maybe I'm just I am happy that you're you're you're communicating this. Because like you said, I I I agree as well. For a long time, it's been sort of a black art that's we wait until we've got ten thousand queries or whatever, but, you can't really go wrong by turning on machine learning out of the gates. Yeah. And and that's that's really it. Like, you cannot go wrong with turning it on right off the gates. Then it the the question is about when does it get fully optimized? Obviously, it's not gonna run at its hundred percent amazing capabilities unless it has all the search events and all the user behaviors are knocked out. Right? Unless it has those sub models build which has rich, rich data. And that rich data comes from, you know, search activity, which can happen for some customers in a day, for some customers in a month. But, really, there is there's there's nothing wrong in switching it on in day one. But I still know some of some of our customers are not comfortable doing this day one, but so we do have options. So what do you do if you don't want to turn it on? Day one is you can still do an AB test. So you've gone live. You can do an a b test. You can do you know, you can send fifty percent of your traffic to, to a pipeline that has machine learning and fifty percent to one that doesn't have machine learning, and you can compare the metrics. And, you know, hundred percent of the time customers see better metrics with the pipeline that has machine learning. And having machine learning also reduces the manual effort on you because every time otherwise you're going in you're saying oh there's a content gap now I do you know I need to do something about it. So if if it is Coveo and, you know, there is a gap in there's a search that doesn't return results, but the user moved on and they maybe changed their query and they clicked on a document, Kubernetes is gonna learn from that. And very soon, it's gonna start recommending results. And if you don't turn on machine learning that is an activity that is solely on you now. It's it's on you to to do that task and make sure that your customers are seeing a search result. So those are some topics I had for you before, I pass on to Jason who's gonna talk about, you know, everything that I spoke about briefly, filter fields, custom context, and all the other, amazing things that you can do to optimize your machine learning models. Thanks. So, Jason, I can stop sharing, and you can take over if you want. Sure. Okay. Yeah. And thanks. And I'll and I'll say that the AB test is actually a great way if you want to, you know, demonstrate ROI to maybe your, executive sponsors or something like that. You can kinda show them proof positive that Coveo machine learning is having, you know, the desired impact on your results. So Ludhika started talking to us a little bit already about the types of of context. So I wanna break this down a little bit more for you. So she mentioned filter context, you know, where we've got search hub origin one, tab origin two, or language. So these kinda act act as those hard barriers between your model subdivisions just like she described. It creates those, you know, different buckets and combinations. It actually isolates the learning between those buckets. There's a little bit of a differentiator here because in addition to the filter context, which comes out of the box with every Coveo machine learning model when you when you go live with it, you have the option to add custom context as well. And this is where things get really interesting. So this allows you to boost content based on those custom values. So unlike the filter context where it kinda subdivides and filters learnings out, This is more like boosting certain content in certain situations. So it doesn't really isolate that learning. The dimensions, however, are not provided out of the box by Coveo. They're really easy to implement. I'm gonna show you a couple ways to do that right now. But when you add this context in, it's really gonna have a major impact. So when we talk about custom context values, again, I I like to use the analogy of, one of those old school bike locks or combination locks. So think about all the different ways you can break down, like like, context. Maybe you've got, like, for me, the, like, the country that I live in or the department that I work for and so on and so forth. These all act like different combinations. So somebody who works in one department and lives in one country can search for a certain keyword. And if those contextual elements are being promoted to the machine learning model, the results that machine learning brings back may be a little bit different from user to user. And the more combinations you have, the more personalized your results can be. Now a couple of different ways that you can actually deliver this custom context. So the first one, which I think is probably the the easiest way to to to to to to show is with record fields. And this is gonna be an example, like, maybe a Salesforce or ServiceNow insight panel or any or case creation page, anytime where you've got input other than someone typing in a query. And I'm gonna show you an example of that in a second. And another option here is user context. So this is information about a user. So it works great when you're logged in. If I'm a logged in to no one user, if there's certain attributes about myself on my profile, Coveo can scrape this information in and bring it in as context. So I'm gonna get out of this PowerPoint presentation here for a moment and get us into, a live demo so I can kinda show you a little bit about what what this looks like. So I just have to change my screen. So it looks like there might be a question in the chat, or maybe there's not. Yeah. Maybe I can, say the question out loud. So when I run an AB test, is there any way to just view specific time frames? I had an instance where traffic was irregular, and I can't adjust the base metrics. So, Lithica, maybe you wanna jump in on this one. I know that we've recently updated the way that we promote, the way that we conduct AB testing. Back in the day, I know you could always set the parameter value the period value in there. I think the new method doesn't really have that option, though. Yes. So, it it it does have an option, so you should be able to adjust the parameters still. So you could go in and edit it. I don't know, Chris, if you've tried the new, you know, the new AB test UI, but it's a little bit different than what we had before where we, you know, we had to duplicate a query pipeline and then it was more work basically. So now it's lesser work where the test pipeline is just created for you and you can go and adjust it. You can still go and adjust the settings. What I would say is typically, what I have seen is it's, if you have irregular traffic, what I would suggest is going up to, you know, two weeks, three weeks to kind of settle it down. And also it depends on the use case. Right? So if it is a commerce use case and you're doing an AB test during back to school season, you'll have a ton of traffic. Whereas, you know, if you did the last week of back to school and then compare it to, when children are back to school, we don't care about lunch boxes anymore. It's gonna be a totally different experience. Right? So I would say like think about time periods where your traffic is fairly regular to do that a b test, for machine learning or for for any other purpose, for that matter. Yeah. And and in your in the in your dashboards, so you should be you could actually probably create just any old report. You should have a dimension that's called split test version. And that dimension is specifically designed to show you which version of an AB test you wanna look at. So you can use any old dashboard, set your your time period in there, and then add a filter and, therefore, split test version, and you could toggle back and forth. I think this would be a a good way to do that too. Okay. Great question. So if you guys can see my screen now, where where I'm at right now is actually a a Salesforce, sales console. And this is actually leveraging Coveo for Salesforce. So the Coveo component here is this insight panel over here on the right side of the page. And you can look at some of the fields that I have. So, basically, what I'm looking at here is a lead record. And the lead record has a few, context or record fields that we're using actually to provide context to Coveo. So if you wanna take a closer look at some of this, you can see things like there's an industry field that says I'm in health care industry. The solution interest field says it's Coveo for ServiceNow. I've got some product information. I've got a description about what this, you know, lead is like. A use case for website, and a use case description, maybe it's an online community. If you look at my results over here, you can see Coveo for ServiceNow is automatically the top result. Other ServiceNow related information is up in here. So without having to do anything, this helps the end user because they're able to just get information. What I wanted to show here, though, is what way we're doing this is by sending contextual values from these record fields. And so if I were to just send a query really quick and take a look in Chrome developer tools I'm gonna get a little technical here, guys. Check out your v two event. This is the Coveo search API call in which all this information is sent. Usually, if you I was gonna send in a manual query, for example, that would be sent along with a search API call. In the form data section of your v two event, all the way at the bottom, you're gonna see this section or parameter called context. And you can see all the context that's being sent here. So what you're seeing here is actually the API name for the different fields in Salesforce. These are the record fields that are being sent to Coveo as context. So it comes right along with the search API call. So when you think about the way traditionally, like, an automatic relevance tuning model would work, if I search for the word cat and I land on a bunch of results and I click on and I click on one of those results, there becomes a, you know, a connection between that original query of cat and the result that I clicked on, which maybe was a a tabby or a tiger or whatever I've got in my cat index. Right? In this case, we're sending the context along. There's no manual query here, but we are sending along this contextual information, which is being processed individually as different buckets within that machine learning model. So this can be accomplished pretty easily in a, insight panel like this. If you're if you're a Salesforce, admin, you can go and edit the page, and you can simply add in the API names of all the fields you wanna send to con to Coveo, and that act alone will create this this this contextual flow. Now user context is a little bit different. So what I'm showing you here is our corporate intranet, Coveo Caveo. And in this case, it knows a little bit more about who I am. So there's a different way. Here we're scraping that information from my active directory profile. So I can do a search. Maybe I'll look look for Lipica, my friend here, and there's her result. Now if I look at the same thing, the the query that I'd sent, I'm gonna take a look at my my v two event, go into my headers, go into the form data section, and actually just scroll all the way down to the bottom, and you're gonna find that context, parameter again. Now you can see what it's got about me. So my department is customer success. My manager is Elaine Cobb. It's got my title in here. It even tells you where I live, Rochester, New York. I'm a I'm a at home user here in the United States. So that all that information is getting sent to Coveo as part of this API call along with the query. So you can actually see the query text is in here as well. That's a different parameter I searched for Lipica. But not only did I search for Lipica, I searched for Lipica. I'm Jason Milnick. I'm a director of customer success. I report to Elaine Cobb. I live in Rochester, New York. You have all this information about me that's coming along with that query. This is all context that our machine learning models can use and process to personalize these results to me. And so this is where all of those attributes actually continue to add on to just how rich, we, you know, we can process information here. So I'm gonna go back to my presentation. And I just wanted to add one more thing is sending custom context is never a bad idea. You can send as much context as possible. And the the good thing about context is that, and Jason is gonna talk about it, is that you can use it for reporting as well. So if you're trying to, you know, create a report to see what's the kind of or what's the adoption of Coveo by department. That's where you get it from. From your custom context, you can create a report and you can easily look at adoption by department by just passing that context on. Yeah. That's a really great point, Lipika. It adds so much to your reporting. And not only reporting, but you can create manual rules too. So you can actually create a condition that says at contacts equal, you know, location equals Rochester, boost all this content for them. So so all those contextual values, not only do you have you can rely on machine learning, but you can actually manually control that information as well. So it gives you so much power at your fingertips. So what I'm looking at here, guys, this is like if I were to dig back into the model view, the detailed view that I showed earlier in the Coveo Cloud organization, you can actually kind of break down and look at what context your machine learning models are learning from. And so in this example, there is a lot of different custom context that this customer is sending. It's for an internal HR, like, help support, portal. And with this, I can see a few things like region. You can see that there's different breakdowns of APAC, Canada, EMEA, US, so forth. There's department. So, you know, there's different names of departments, so these are kind of jargon. But for the customer, when they look at this, they know exactly what department all these people look at. And then you can see the number of candidates. So this is learned results for that particular combination. And so imagine all of these. There's one, two, three, four, five, six, seven different layers in here. That's like seven different spinners on one of those combination locks. People can have all different combinations for, and they've got a really personalized experience because of all this contextual information that they're sending. So the way you can review this is this little table here. You can look at your context key, for example, department. Underneath the context key, there are the context values. So within department, these are all the departments that we have information for. And then within each department, you have the number of candidates. So these are the learned results that were learned, and boosted by machine learning, per department. So that gives you kind of an example, again, of all the different, ways you can leverage this user context. And Do you wanna jump in? Yes. One more thing to add, is, you know, when you're thinking about custom context, I know sometimes it can be, you know, for for us to think about, okay, what do I send? What do I send that'll make sure that it is actually helping machine learning? I would say we've made it we've made that task easier for you. Coveo's machine learning, you know, will only learn from context that that has a good cardinality. And what do I mean by that is it has it has a number of, you know, just like you see here, there is a country and there are maybe possible like six or seven values. It's a good one. It's a good context to send. And then same for here, department. It has, you know, one, two, three, six possible values. Again, a good context to send. But if you send something like case ID, every case is gonna have a different ID. So, you can still send it. So don't worry about it. You can still send it. What Coveo machine learning will do is it'll automatically remove it from its learning. It's gonna say, hey. I'm not gonna learn from this model because there is way too many values and I cannot create these buckets of rich information if I have so many different values. So it's only going to consider the ones that have very rich information and also good cardinality. So automatically making that task easier for you. And, also, you can pick and choose which one has more weight, which one has less weight, and, Jason will also talk about that when he's talking about the custom model parameters. But when we are looking at the context keys, remember that not every context is a good context, but you can still pass it because you can report on it, and machine learning will make sure that it is not taking it into consideration. Yeah. Absolutely. If I could give any advice to anyone implementing Coveo is send as much context as possible. You could only help. It's it's not gonna hurt. But Lipica brings up a really good point because, those contexts are filtered out automatically if they're not valuable enough. The cardinality is not there. But with custom model parameters, you can even manually control this. So you can decide which context you want to include in a model and which context you want to exclude. So there are different ways you can go about this. So now we're gonna get deeper into this. It it actually involves editing the JSON of an a model itself. So a lot of you know, you know, you hear about ML out there and AI, and sometimes it's a black box, and we just say we've got, you know, machine learning. But at Coveo, you have access to the actual, you know, JSON that you can go into and edit every parameter of a model and really fine tune that if you want to go into that level of detail. So we're not gonna go into all the different parameters. There are many parameters you can edit. But I'm gonna give you some examples today of, like, some of the cool use cases that you can achieve, when you are going into this JSON and actually editing some of the the values that are in a machine learning model. So, you know, like I was saying, you know, I guess the first bullet covers it. Our models are not just limited to the out of the box default configurations. And so some examples of things you could do, you can change, like, the number of results that are boosted remove certain context, like filter context, for example, search hubs for cross training, like filter context, for example, search hubs for cross training models across different experiences and stitching data together. That's the way you do it. You can also, like, have help a model learn faster. So I'm gonna do a quick exploration into each of these use cases. And later on, we're gonna share with you, after this call, like, some, appendix that has a lot of documentation of all the different parameters that you have the ability to influence in the model, and you'll find that it's, really quite robust. So thinking about the increasing number of results. So this is a screenshot from, Pet Supplies Plus or one of our customers. And, they, they're a commerce customer. And when you navigate to a category listing page on Pet Supplies Plus website, there are many different products. So in this case, I went to dry dog food, and I've got, you know, hundreds of products potentially that I could look through. But I didn't put in a query, and all I did was navigate to the page. And the the question that the customer came to us with was, you know, your automatic relevance tuning model is boosting five results, which is okay. That gives us, like, the first top row and maybe one more. But we have all these products in this category, and we wanna make we would like to provide more of a dynamic flow of the way these categories, load. So that was pretty easy. All we did is we went into the machine learning model, and we adjusted the max results, parameters. So by default, it's set to five. We moved it up to twenty four. So in that case, twenty four results are automatically all boosted. And and so what happens here is not just the the top five results alone don't just change their position over time. It's basically the entire first results. The page of results will will adapt and dynamically change. So it's a better way for them to, keep track of, you know, what products are leading their categories and trending downward and whatsoever. And they don't really have to control that or manually support it. They they turn machine learning on and let it do its thing. Another use case here is if you wanna unify learning across your hub. So you'll hear us at Coveo talk a lot about unifying relevance and stitching data together across your end user journey at whether or not you're in, like, a commerce type environment where you're prospecting, you're looking for a product, you become a customer, and you go on to support site. And even if you're, you know, on the other side of the fence, maybe you're an employee who's receiving a case that came from support. How do we, like, stitch all that information together? Well, Lipica told you earlier on that, like, the search hub itself, the interface, is a filter context out of the box, which means that if I'm on a commerce site, what I'm searching for models learning from my behavior is and it's bucketing. Everything thing that happens on this site is this is only learned on the commerce side, and it's completely segregated from what's happening in, like, the community or the support side. This is the default nature because you're gonna have different search hubs, and those search hubs filter context. But with this little command line that's in here, where you can go in and edit a parameter, there's this option of filter fields. And if you leave that filter fields option blank as it is in this example, it actually removes, like, the search hub and tabs and everything from those filter context, and it opens up your model to learn across the entire experience. So that may be a little bit liberal, but that's an example of what you can do. Now to what she was saying earlier, you can actually add different fields in here. You can tell it what fields you wanna have as filters. So maybe you wanna learn by a specific product context. You could add that as a filter field. So that creates this hard barrier between product and product. That's that could be a custom context. So you have a a world of opportunity here, lots of different options for how you want to control the experience. You can make it as liberal and open as you want, or you can control it by adding additional filter field context in here and really subdividing your model more specifically. And another interesting use case is just helping machine learning models learn quickly. Now in my role, I support new onboarding customers, which means that most of the customers I'm talking to are in UAT at the moment. And and and as we just talked about, like, models learn over time. Right? So it takes actual data. It takes people in there querying and clicking on things, which is great when you're on a public, you know, production site. There's all kinds of good stuff happening. But proof positive is in that UAT when we're building a Coveo project. So how do we show the stakeholders that machine learning is actually doing its thing and learning quickly? Well, by default, it takes up to ten clicks, before actually a a result is learned by the automatic relevance tuning model and so forth. But we have this cool parameter, the minimum clicks query, as you see here in red. If you set that to one, basically, it means all it takes is one search, one click, boom, the models learn this. So for UAT testing, this is great. You can put a bunch of UAT testers at a site, have them go through, run their paces. And at the end of the day, you're gonna have a whole model full of interesting learning material. So you can run this on reports. You can see how quickly machine learning has had an impact, what it's had an impact on, and so forth. And so this is obviously an integer value. You can grow it. You can you can make it as as small, as large as you were as you want to. So you can actually control, you know, the rate at which a machine learning model learns. At a higher level is, you know, the the the frequency, you know, for refresh and the frequency periods that you can control in the UI. But this just gives you a little bit more ability to really fine tune this and, help your model learn quickly. And I think Ellipa is gonna talk with you about some of those default configurations next. But I'll pause here before I turn it back over to her to see if you guys have any more questions. Looks like maybe not. Okay. Alright. I can go ahead. Let me let me go ahead and share my screen. Okay. So along with all the configurations that Jason was just talking about, which are, you know, you going into the JSON and updating the model, there are some default configurations that also can be tweaked, that will also help you optimize your machine learning model. So really if you look about it like go back to the very basics of when you created your machine learning model. So right when you're creating machine learning model, any machine learning model, be it ART, query suggestions, DNE, any machine learning model, you have to specify the data period that it learns from and how frequently it learns, you know, how frequently it refreshes. So by default we make sure that, you know, we learn from the freshest of data. So, there is also a theory about data decay which means that we will we will learn more quickly from data that is fresh and is, you know, that's happening again and again. Whereas, you know, a search and a click that happened maybe six months ago, not that important anymore. Right? Because we we wanna move with the trend. We wanna be be, you know, adaptive to the user experience. So really quickly, if you look at the data period and the frequency, these are some best practices that you can follow when thinking about what to do with your you know, it's a big question. Right? Just like when you're selecting, you know, what custom context should I send? So what data period and frequency? The good news is the data period and frequency is also very fluid. You can change it anytime you feel your model may not be learning enough. An example is, you know, you have five different sites, and there is one site, that that probably doesn't get as much traffic model tell the model to learn from a wider range of data, so maybe three months of data, and then it has more information to learn from. And then also make it refresh maybe one week, every week, or every month depending on how much traffic you get. But really, when you're considering data period and frequency changes after your machine learning model has gone live, really think about how you're going to optimize that richness of the data. So if you have a site that, you know, that has new content every day, if you have a commerce site where, where, you know, your your site has seasonal products. So make sure that your machine learning model is is respecting that and you change the data period and frequency to be able to handed handle the change of user behavior in what they're searching. You know, in summer your customers might be searching for sandals, whereas in winter, they will, you know, fall, they will start searching for boots and, you know, warmer clothing. So if you have a seasonal, you know, seasonal requests on your site, please make sure you consider, the, that you learn from the freshest data. So keep your data frequency as short as possible so we are learning as quickly as possible. You can still learn from maybe one month of data, but make sure you're learning fast enough. So these are some considerations and some things you can change after your machine learning has gone live for a while, and these are also very, very easy to change. As soon as you change, you know, your new machine learning model gets created within a matter of twenty to thirty minutes, and you will start to see the change right away. The other two options that I wanted to really talk about is and these are very simple options, but can be very, very powerful. So these are configurations. I wouldn't, they are there in your UI directly for you to, you know, change them whenever you want to. Whenever you're creating and this is specific to the ART model. Whenever you're creating an ART model, which is the automatic real relevance tuning model, it it gives a score of two hundred and fifty points to, documents or products that it has learned are getting a lot of clicks or are being bought a lot of times, it's gonna give two hundred and fifty points to those documents or those products. So when if if needed and if you feel that machine learning is promoting, you know, more results, and you would probably want, you know, a healthy mix of, what I would say is the Coveo's out of the box algorithm plus what machine learning is doing, then you can lower this ranking modifier as well to not be two fifty or maybe be it be lower. But if you have a site that has a lot of manual settings, you know, in a query pipeline that have really high scores assigned to to products and documents, then you can even increase that, that ranking modifier to more than two fifty. And you might when we if you go back to one of the custom parameters I talked about where you could increase the number of results that are boosted by ML, if you wanna boost a lot of results, you might wanna tick that up a little bit too to spread out the the different allocation of points across all of your boosted results. Exactly. And and the other thing, I I think we spoke about this, but I'll just like to mention it again is that when we say machine learning is you know, you can choose more results to boot or you can boost five results. It is up to five results. It is up to fifty results. It doesn't mean that for every query, you will get fifty results that are boosted by machine learning. It really depends on how much machine learning has learned. Right? So how much information it has gathered about that specific query, and that's how it's that's how it's it's able to recommend products or documents. The other option, which I think is more common with our commerce customers, is matching the query. But I have also seen customers who have, you know, very specific, scenarios, where what we are saying is in your if your query is matching, we will only recommend products that products or documents that have the keyword that you have entered in this in the in the text box in the in the search box. So what we are saying here is machine learning by default, it learns from user behavior. So it's gonna learn from the first query that you did, and it's gonna look at the last click that you did, and it's gonna form a connection there. A lot of times, the first query that you did and the last click that you did, that that last document might not have the keyword that you were searching for in the first place, but we still make the connection. An example is and it it has proven very, very helpful to reduce content gap because not always do your customers know exactly what words or what keywords to put in for you for them to get the exact result. They might be calling something something else, but your documentation might not be, might not have that keyword. But we will still return it because we will learn from their behavior. But when it comes to commerce and like Jason was saying, if if I'm searching for dog food, and I have a cat and a dog, in my session, I might look for dog food, but I end up buying cat food. In such situations, I might not want to inject queries with cat food. It really depends on on customer to customer. A lot of our customers, feel like, you know, when you when my customer is searching for a hammer, I'm not gonna show a planter to them because that's not what they were searching for. And if if we have to suggest results that say, you know, people who bought this also bought this, we always have our recommendation component that can that can do that for them. So I see, Jason, did you wanna add something? Yeah. So the match to clear one is really interesting. Another use case that we had for this was the company, Lee Valley Tool that so they're, you know, a tool, a woodworking company, gardening, and all that kind of great stuff out of Canada. And they, they have a situation where a lot of customers would look for a very specific item number or internal users would look for, like, a product ID. But we had automatic relevance tuning in, and match the query was unselected like this. So they would get, like, the one result that they were looking for, but then they'd get, like, five other results that were all getting boosted by ML. They didn't have to be related because they were just all like, hey. In that one session where someone bought this, they also, you know, clicked on all these other products, and it was kinda weird. And so, actually, what we did for them is we just created a regex expression, which, kinda match it was it was would look for the query, and it would match it against the the type the format of the item ID. And then during if someone searched for an item ID, we would just, you know, basically bypass the automatic relevance tuning model because they didn't really want that experience. So that's an interesting configuration and interesting use case, but one that probably a lot of people can relate to. If you've got people who are looking at least for specific items and you only wanna see that one item, then, you know, matching the query could actually create an unwanted experience there. That's right. Okay. Moving on. For some reason, my slide does not want to move on even though I want to. Let me do this again. There's actually, a bonus content here. So what I would call it as the, the recommendation badge. So if you go on our website, what you would see is, every result that is returned, it has a badge on it, especially the ones that are returned by machine learning. We have a badge on on top of those results that tells that, okay, these are the results that have been recommended by machine learning. It it's it's kind of a way of telling that, you know, a lot of a lot of customers like you have on these results, so maybe take a peek. So this is super, super easy to implement, and I just wanted to share that very, very quickly with you. So if you wanna implement the promoted results badge, you can do that for either your featured results or for, you know, items that are recommended by machine learning. It is a very short snippet that you can add to your page. If you're using your JS UI framework, it's easy even easier to do it. So you just have to there's the component, and you use that component. It's a out of the box component. You can you can add that piece of code that I'm showing here, and we will share the details of the, you know, you know, of the component and the code, but it's basically what I'm showing on the screen. That's it. So once you add that, what happens is any any any results that that are recommended by machine learning or machine learning has learned from them, they will get that badge called recommended, and you can call it anything you want. You can call it recommended by machine learning. You can call it promoted, featured. You can make it orange. You can make it rainbow. It's up to you. It's all about, you know, to to match your brand, brand standards. And you can add that recommended badge component on on onto your result list. But that was the bonus bonus component or the bonus content for today. I I would highly recommend if if you can go ahead and add it because it also helps users feel like, okay, these are content that other users are also clicking on. So it's kind of like, you know, promoting the, you know, other content that you might have that are getting a lot of clicks. So do a to do a very quick recap before we go into questions. So today we learned about all the different machine learning models that we have and how they work. We learned all about, you know, the different types of optimizations that we can do. You know, Jason showed us how we can look at different custom model parameters. And we're gonna share the, you know, the, the documentation that actually tells you about all the different custom parameters that we have. We have different ones for ART, different ones for query suggestions. So, there's a number of different options for you to play around with there, and also, like, any of the default configurations that you can have for optimizing your machine learning model. But I think I had a lot of fun presenting today, but, with Jason. So we can get into questions now, Claudine. Okay. Maybe we can get started with these questions. Can we set custom fields as filter fields? I can take it, Jason. So this one, yes. You can. So I think Jason kind of hinted on, discussed a little bit. But when you have, by default, the filter fields are your search hub, your origin one, which is your search hub, your origin two, which is your tab, and then your language. But you can have different fields that you can consider as filter fields. So you can add, you know, we have another dimension called, origin context, which basically tells you what type of use case you are on. Are you on a search page? Are you on a community page? Or are you on a inside panel? It tells you that. You can add that as a custom well, but you would do it through the custom model parameters that Jason was showing. Yeah. It's on slide twenty one, and we'll send this out afterward. So you'll see that that particular, configuration has that that parameter blank, but you any fields you put in there would actually append the existing filter fields and create additional filter fields for you. K. Awesome. Thank you. Another question we have here. How many machine learning models can I have in an org? Jason, do you wanna take that? Yeah. I well, in my opinion, you can have a lot of machine learning models in an org. It really depends on your configuration. So, I would say the most important one is how many do you have per query pipeline. So you should in any given query pipeline, you can have, you know, an automatic relevance tuning model, a query suggestions model, dynamic navigation experience. Recommendations models will need their own unique query pipeline for. So it's just a matter of looking at your use case and, like, how, you know, how much segregation do you wanna have between your different experiences? If, you know, there's completely different experiences, you may wanna have a lot of different models. But bear in mind, what we just talked about, you have the ability to use a single model and add a lot of filter context in there. And so that filter context alone can basically give one machine learning model the flexibility to support all of your use cases. So this may be actually the leading practice. What what are your thoughts on it, Lipika? Yeah. Yeah. No. I completely agree. And like you said, you can have as many machine learning models as you want. It really depends on your use case. So if you are a commerce customer, you might be, you know, you have a commerce application. You might be using product recommendation model, event recommendation model, like, all of it. We have customers that use our all kinds of our machine learning models. And within models, also, we have different strategies. Especially for our product recommendation model, there are different ways you can utilize it. You can have one model, but you can have different strategies that look at, you know, the most, you know, trending products or the most bought products or, you know, products that were added to cart together. So there are different there are different, you know, machine learning models you can have. And if you're maybe this question was triggered by thinking, will it have any performance issues? There are absolutely no performance issues. You can have as many machine learning models as you want. Okay. Any other questions before we wrap up? I think that's pretty much it. Thank you so much, Jason Lipica, for kicking off the first session of our learning series for this quarter. I mean, I there's there's a lot of information packed within, almost an hour, and this is such, great content that was requested by our customers. And I'm leaning to that. I'm promoting, please take the survey after this webinar. The more you give us feedback, the more we know what type of webinars, what type of content to produce, what type of training, to share with you. Thank you so much, everybody. We hope to see you next month. Have a great rest of your day. Bye for now. Thank you. Thank you. Bye.