Okay. Hello, everybody. Welcome to this year's first learning series webinar. We're so happy you could join us and made learning, made being better with Coveo part of your New Year's resolution. We have a whole quarter for you to actually make sure that you keep up with your New Year's resolution. My name is Claudine Ting, and I work on the marketing team here at Coveo. For those of you who are attending a learning series webinar for the first time, this monthly webinar is where we get more hands on with how to enable Kubeo's features to help you create more relevant experiences. Again, this is the first session of our three part series on optimizing relevance tuning. So today, our focus is on relevance tuning for advanced enterprise solutions. We'll be building ranking expressions, filter ex and filter expressions, within your query pipeline in order to optimize the search experience for your users. So just before we get started, quick housing rules. We encourage you to type your questions in the q and a box or the chat box, and then our experts will be answering them either halfway through the webinar or at the end of the session. As per usual, today's session is gonna be recorded, and you will be receiving the presentation in the next few days. Today, we're welcome back our speakers, Lipika Brahma, customer success architect, and Jason Molnick, director for customer onboarding. I'm sure you're probably familiar with them by now. If not, they're amazing. I love them. You're gonna have a great time learning from them. So let's get started. Lipika, you wanna take it away? Yes. Thank you, Claudine, and really welcome, everyone. I know it's too late to say happy New Year, but I will say it anyways. So happy new year and welcome everyone. Like Claudine said, today's focus will be on relevance tuning for advanced enterprise solutions. Today's learning objectives will be, you know, we'll look at the art of relevance tuning. I call it an art. It's also a science and an art, but we will look at how we can tune relevance to make sure your customers and your employees are getting the best experience when they, you know, when they're trying to look for, things to, you know, make their work easier. We look at different tools that Coveo provides, for doing, a relevance tuning and how you can use them, you know, what kind of tools are available, what is best to use in what scenario. I'll go through a couple of examples, adjacent to where we look at, you know, how you can create those, the how you can use those different tools. We'll do a relevance tuning one zero one where we'll go through a couple couple of basic things as to how do you start with something like this, what you need to know, and we'll also talk about, you know, manual tuning, how you can manually tune relevance. We'll also talk about our machine learning model, which is the automatic relevance tuning model. Jason is gonna talk to you about, you know, how does all of this happen. You know, you're tuning relevance, but how do you know it's working? How do you figure out if it's doing the thing that you want it to do? All of that is what Jason is gonna talk about. We'll talk about ranking factors, you know, out of the box, things that are available with Coveo and how Coveo actually does the search and how they affect when you tune relevance, how they get affected when you tune relevance. So let's get started. So what is relevance tuning? So relevance tuning is really an exercise, that we do with our customers and customers continuously do with, you know, with our Coveo platform is to continuously improve the user experience. So there are two types of relevance tuning that can happen. One is manual tuning, and the other one is the one that, that that our machine learning algorithm provides, which is the ART model, the machine learning powered relevance tuning, where you don't have to do anything. It is automatic. It learns from user behavior, and it tunes automatically. But we also have the manual tuning available for you to influence, you know, certain types of results, to for for for your users. Now what are the tools that are available for relevance tuning? There are different kinds of tools available in the Coveo platform for you to manually tune your relevance. So, and I will go through each one of them. But for today's, webinar, we really focus on ranking expressions and filter expressions and how you can build those. What we have seen is for, you know, for our advanced enterprise customers using these ranking expressions or filter expressions, really something, you know, that helps them a lot when they are trying to influence a lot of results at once. So which is why we're gonna focus on that today. But let's talk about each one of them, very briefly. So thesaurus entries and all of these before I go into that, all of these, you will actually if you are in your career org right now, if you have it open, go into your query pipeline and you will find all of these options over there. And that's what we are talking about. All of these tools are available in your query pipeline. So the first thing is thesaurus entries. Thesaurus entries or synonyms, are are really there for you to help machine learning give, you know, help machine learning give context of your business. So, a lot a lot of our customers, you know, come from different come from using different tools where, you know, they're used to entering, you know, singular, plural, different forms of the same word. Coveo does a very good job at that out of the box. So we do something called as stemming, which means that we will look at different variations of the same word. We will go back to the root of the word, and we will look at different variations of the word. So if you if you choose the word, given, we will go back to give, and we will look at giving. Given, all of the different variations that that the word can have. So you don't need to build asaurus entries for any of those. So what do you need them for? So you really need them for, you know, like I said, providing your business context. So an example would be, you know, if if you have a customer community, a lot of times customers are coming to your community if they have a question or they have a problem with one of your products. So they might look for, you know, how do I troubleshoot this? Or how do I solve this problem? Or or I have this issue? So in in terms of that troubleshooting problem issue, they're very, you know, they're words that are used in the same context. So you can create a synonym where whereas if a customer is you searching for the word troubleshooting, we also search for problems, issues, and so on and so forth. There are different types of synonyms you can build. You can even replace. Say, for example, your organization has, deprecated a particular product or renamed a particular product, you would want to use a replace entry, which means that customers who are searching for the older product name will now automatically look at the new product name. So they don't have to remember the new product name or they don't have to be informed about the new product name. Then comes ranking expressions, which is what we will go through in-depth today. So I'm gonna leave that out for now. Stop words. Stop words are words that you don't want Kovir to pay attention to. So these are commonly used words. What I would say is stop words is really your last resort if you have trouble with relevance. Typically, Kovir does a very good job with ignoring words that that occur too many times your index. And Jason is gonna show you how these, you know, index, works with, you know, different kinds of words. And if there are too many times, how much importance it pay pays, to those words and how they're ranked, how the results are ranked based on that. We already have a list of stop words, that are there so you don't have to add them. But remember when you're adding stop words, if at all you're adding stop words, not to add more than a dozen. Then comes featured results. So featured results is something that you would use if you wanted to, you know, put a particular result right on the top for a particular query or for a particular situation. So say, for example, in your customer community, one of the top questions that comes is how do I change my password? So what I would do is I I know there is a knowledge article that solves this problem. I would put that as a featured result when the customer searches for, you know, that query. And we look at all of these examples live, in a bit. Now there are more tools for relevance tuning. Filters is what we're gonna see in-depth today. So filters really are used to scope the results that are displayed to your users. Query parameters, it they have multiple different purposes. But one of the most popular use of query parameters and query parameters are wave ways for you to control how Coveo, you know, searches for different keywords or the really control the algorithm. The query parameter that is most popular is called partial match. Now partial match is something that we we put for all of our customers because without partial match, we try to do a hundred percent match, which means if your customer entered four keywords, we will look for all of those four keywords in a document and only those documents will be returned. Now we all know that customers, you know, usually wouldn't know every keyword that is there in your document. So we always use partial match for our customers so that there are chances that they will find their document even if they didn't enter the right keyword. And they will find more results without them leading to see a blank page with no results at all. Then comes ranking weights. And, again, this is something that Jason will explain as well, so I'm gonna leave that out as well. Triggers is something, these are special actions. This is mostly used for, say for example, you have an intranet for a workplace and your users are searching for, you know, how do I apply for a vacation or paternity leave or things like those. And the system that is used to apply for any of these vacations is in your work day or or wherever else you're using. You can click, you can create a notification that tells them that, hey. You know what? This is the link to go there. Go there and create your vacation request. There are several other applications of trigger. This is the most simplest application and the most used application as well. But there are many other applications of triggers as well. Now before we go into the details of ranking expressions because we're focusing on that today, let's learn about query pipeline language. So the query pipeline language is the language of tuning relevance. Now don't worry too much. You don't have to learn another language to actually write these ranking expressions. We have a lot of things in the UI that'll help you with that. But underneath all of those things is what we have is called the query pipeline language. Now the query pipeline language is a very simple language that is used in your query pipeline rules. So if I just jump out of here really quickly, if I went into my query pipeline and I went into the thesaurus entries, so any of the rules that you create over here, they're basically created by using query pipeline language. And the query pipeline language, you know, some is very, very simple. So, for example, this one is a thesaurus entry I created to to show that when users are searching for experience, search for support as well, or when users are searching for support, search for the word experience as well. So, this is what it looks like in the UI. But in the in if I wanted to look at it over here, this is called the query pipeline language. So this is how it looks like. Again, you don't have to remember any of this. You don't have to write code at all, but we wanna know we want you to know that this is what it looks like inside. More advanced users who understand these expressions can actually create these expressions in our advanced mode over here. So if I do and I want to create a new, add a rule, I can either add or add a rule from here or I can go into the QPL and change things over there if I'm comfortable. I might add one other thing on this. Let me come up. You're gonna talk about it. This occurred to me that if you're an advanced user and you ever wanted to, like, export your query pipeline, it exports into a dot CSV file, and you can, like, edit these rules in there and then reimport them if you wanna, like, you know, migrate a pipeline from one org to another. In that mode, it will actually show it in in sort of, like, the QPL version versus how pretty it looks here in the UI. Yes. Definitely. Thank you, Jason. So I'm gonna proceed over to our relevant student one zero one. I'm trying I'm I'll try to be as hands on as possible without going through, you know, too much of the deck, but I'll still keep looking back to make sure that I am, going through all the points that I wanted to make today. So let's look at ranking expressions. Right? Why would we use ranking expressions? So ranking expressions are really, you know, they are your way to influence a lot of different documents or content at once. Every document or every result that shows up on a Coveo page so this is, you know, one of my, demo pages and every result that shows up on this page is is in that position because it has a score. Now what ranking expressions help you do is manipulate that score and say that, you know, if it is a knowledge article, it has more importance than documents. So I'm gonna give it a boost of a certain number of points. So that's how you create, ranking expressions. Ranking expressions are very, very useful especially for our, you know, enterprise customers and for most customers who want to influence many documents or many types of documents at once. There are different ways you can create rank expressions. Same, like I said, either you can if you're comfortable creating these expressions on your own using a QPL, you can do it. But otherwise, you know, we have a fairly user friendly UI that will kind of help you create these expressions so you don't have to figure out the syntax in your head. Before we go there, these ranking expressions, the one thing that I wanted to go through is the core to creating any of these ranking expressions is finding your field. So the way we create ranking expressions is so if you go into the content browser, which will typically is your browser of your entire index, so everything that you have indexed will show up over here. I'm just gonna change it to my pipeline. So everything, all of the sources that you have in your Coveo org, you can view them all, view all the content in in the content browser. And that's that's the purpose of it. Now, for example, I want my Coveo, how do I say? Coveo answers to be given more importance than any other type of document. What I want to know is how do I know a particular content is coming from Coveo answers? Like, how do I uniquely identify it? The way to do that is, first of all, filter it down to that source because you know the conveyor answers is your source. If you don't know what the source is, what I would suggest is take a search result, like take a title from your search result page, which you know is a answer, put it over here and search for it, and you will see the result pop up over here. So once you see the result, go into the properties, and you will see all the metadata that is coming through with it. And these are all the fields that are available for you to create ranking expressions. Any field that is present here, can be used to create a ranking expressions. Obviously, not all fields can be used because some you know, they they don't matter that much. For example, something like row ID, you know, size, all all of these things, that have a meaning can be used for ranking expressions. So for example, let's go back to what we were trying to do. We were trying to say that anything that comes from Coveo answers should be on the top. So now I know that there is a source. There's a source field. So anytime we refer to a source anytime we refer to a field, we always put the at sign in front of it, and I'm gonna show you how to create these ranking expressions. So first, I need to look at my field. So the source is the Coveo answers, and the value is the field name is source. The value is Coveo answers. It can have other values as well. So the way to look at it is you will always see field and value pairs. So your field name is your in this example, it's source. Your value can be conveyor answers, conveyor blog, conveyor website. These are your values that can be present in that field. So use that field value pair to figure out what you want to be shown on the top of your search results. So for example, here in my demo art, I am indexing all of these different sites, but I really want Kovia answers to be on the top. So what I'm going to do is I'm gonna just copy this from here. So that's my field value that I need to remember. I'm gonna go into my query pipeline, go into result ranking, and I wanna say add a rule. I'm gonna select add a ranking expression. I can assign it a name. I can say boost answers. And I can give it a condition. So one thing that you need to remember is that if you create a ranking expression and say that boost conveyor answers, it will boost those that particular source no matter what the query is. Even if a even if a customer comes and looks for YouTube, we will find relevant results, but that scoring will still apply. So as much as possible, use a condition. Cal, if the query is this, do this. If the tab is this, do this. Because that kind of limits your scope, but don't limit it too much as well and Jason will go through some of the best practices by the end of this presentation. So now let's come to your QPL. Now we're not asking you to write an expression, but if you want to write an expression, go ahead and you can write it over here. If you don't know, you click on that edit button. You'll be on the basic mode. And in the basic mode, you can select the source name. You can select is equal to. You can select, you know, what's the competitor. Is it equal, not equal, defined, not defined, Is equal to and then typically, you will see all the value pairs here too. I copied it so I wanna make sure that I'm using the right value pair. So for me, it was conveyor answers. So what I'm going to do is I'm gonna select Coveo answers. I'm gonna create a very simple expression. But if you wanted to create, you know, a little bit more complex expression, you can create, you know, sources this and, maybe show article or show show answers that were, you know, that happened in the last week or something like that. On the right hand side, you're already seeing a preview of the results and you click done. Oh, one more thing. Before you go away, you have to give a score to it. You have to say boost it by a hundred points, a fifty points, or two hundred points. As a best practice, try to go with a lower number at first and then move on to a higher number. So I will go with the lower number. You can add a user note. By default, you should make it a practice to add a user note so you know what you're adding the rule for. So I'm gonna add it over there and add the rule. So now what I should see is I should see my answers come up on the list. So right now, there are a couple of answers that are coming up on the list, but not all of them because I give it a boost of hundred. So now if I go back and give give it a boost of, I don't know, two hundred points or one fifty points, I would see them come up right to the top of the list. So to me, it it always makes sense to give a good mix of results. So make sure that when you're boosting a particular source type, you don't boost it too much that it becomes almost overwhelming. Boost it a little bit. So, yes, it it kind of bubbles up to the top, but don't boost it as much that it it kind of overpowers everything else. So now I'm going back to the deck. I'm making sure that I covered all the points over here. So here are some of the examples that that are there for different situations where you would want to use, you know, query ranking expression. Now let's try another one. Let's try, for example, I have a YouTube source over here. And what I want to do is boost, all of the YouTube content that has views more than a hundred maybe or or a hundred fifty. Because right now you see it is boosting content, but it's it's not really boosting it by the views. It is boosting it by, you know, what was what is the latest content that we have. But I wanna see content that is based on the view. So what I would do is go back into my query pipeline, and I would add a rule. Again, I would choose ranking expression, add a rule. And this time, I would say edit. And remember I'm using it for the YouTube source. But I come here and there's a ton of fields here. I don't know what to use. So what I'm going to do is I'm gonna cancel out And the first step I'm gonna do is go back into my content browser and boil it down to the YouTube source. And now try to find the field that gives me the view count. So I'm gonna go to my pipeline first and then go to the YouTube and look at the properties. And I'm just there is a filter on top over here where you can enter some keywords. And there. That's my field. So YouTube view count is the field that I'm supposed to use. So I'm gonna go back into I'm just gonna duplicate this. And this is how I get a hundred tabs throughout my day. So I'm gonna go back into my query pipeline, And I'm going to create that expression now. So I'm going to go to result ranking, gonna say add a rule, select ranking expression, add a rule, and I'm gonna say click edit. And I'm gonna just paste it because that that way I can find it easily. And I'm gonna say is greater than or equal to, say, one fifty. So on the right hand side, now you see all the content filtered out. So it's kind of showing me a preview of how much content I'm affecting. So I'm affecting almost three hundred and forty five YouTube videos. Now if you're creating an expression and you don't see anything over here in the preview, which means your rule is not affecting anything at all. So the number that you see here is the amount of content that your rule is actually, you know, influencing, let's say. So I'm gonna say done and I'm gonna change the setting to maybe two hundred. The other thing I wanna do here is I wanna create a conditions because I don't want it to apply to every situation. I only want it in the YouTube tab. I can create a condition over here. I can say when the tab is YouTube. Okay. So you see that error that came in because I was testing earlier and says there's a condition already. You don't need to create it create it again. So I'm gonna cross out and I'm just gonna select the condition over here. I'm gonna search for tab. Tab is YouTube. There you go. Add a rule. And that's the condition I have over here. So my view count is more than one fifty and my tab is YouTube. Now the moment of truth. And if all the demo gods are with me, it is gonna work. So there you go. So everything that you see here, the view count, you see they are all two ninety five, two hundred and two, two sixty seven. So everything above one fifty is getting boosted by at least two hundred points. And, I'm not going into and showing how that happened. That's what Jason is gonna do, and he's gonna show you how to find how it's like, how to verify if it's working or not. So, the next thing that I wanted to show really quickly is creating filter expressions. It's very similar to creating ranking expressions, but filter expressions are mostly used to create, filters or to reduce the scope of the results. Now say, for example, on my page, over here in the all content, I am seeing so many different sources. Say, for example, I don't wanna see the Coveo blog source anymore. So what I would do over here is I would go into my query pipeline again And I would go into the filter section, which is under advanced. You see filters. And you see I already have a filter here, which is basically telling my UI what sources should to be displayed. What I'm going to do is I'm gonna edit this and I'm going to remove, the Covio blog source and done and save. And now I'm gonna go back into my page, reload it, and you see the Coveo blog source is gone. So that's how we create filter expressions and ranking expressions. Our UI does a great job at, you know, showing how these filter expressions you know, you can create them. There is a editor for you so you don't have to write it. But we do have advanced editors for every user to be able to write one if they need to write one. So that brings to that brings me to the end of, you know, the series that I was presenting over here. But next, you know, Jason is gonna show you, you know, you did all of that. You created a ranking expression filter. Now how you're gonna look at those and how do they actually how does the inner working work? So I'm gonna stop sharing, Jason, and you can start sharing. Alright. Thanks a lot. Great. Before you transition, I just wanted to, get this question in the q and a box out of the way as well. So what are some good practices for relevance tuning if you have a small amount of content, like, maybe less than five hundred articles and only one type of content, only Salesforce articles, for example. It feels like a lot of the tuning is overkill when most queries return, around ten or so results. I do you want me to take questions? Yeah. I mean, that that's probably exactly right. I I think it might be overkill. Like, there's and we'll get into best practices, but it's it's it's important that you have a real legitimate business use case to leverage a ranking expression like this. And a lot of times, it is a way to kick start machine learning. Or one thing that we often would do is look at, like, your content gaps report or look at, like, your queries with low relevance report in your analytics dashboard. So if you can see people are searching for a certain thing, and they're often not landing on the right kind of content, but you know that content exists. It just happens to not maybe match the keywords you're looking for and maybe ML hasn't picked it up yet because it's still fresh, then you can find that document and boost it. But you might wanna do this only one or two documents at a time and and really don't go for, a ton of, of boosting. As you'll find out in a minute, when you boost a lot of content, it kinda limits the scope of what some of the other ranking factors are gonna do to help fine tune your results down the road. Did you wanna add any Thanks. No. I'm good. Okay. Cool. Alright. Well, thanks, everybody. Thank you, Lipica. Really great setup because, there's a lot of stuff going on in the chat. So the chat was live, and we're gonna have, a lot of answers to your questions in the in the upcoming slides here. So I'm gonna talk with you guys a bit about ranking factors as a jumping off point. So whenever you send a query to Coveo, it's gonna go through this funnel, basically. And there are different phases, four phases, in fact, of ranking. That first phase is called term weighting. So what we're looking at here is things like term location. So is the keyword in the title of a document? Is it in the summary of a document someplace else? Is the keyword, like, capitalized, even casing, and and is it is it stemmed and so forth? This is also where ranking expressions come into place. So Lipika just talked to us all about QREs. This all happens in that front phase one. This is also where machine learning takes place. The next phase is, document weighting, and this is gonna apply to the the best fifty thousand items or less. And it's gonna look at mostly things like the dates of a document and so forth. Then we're gonna look at keyword frequency. So how many times does a certain term show up in the document? How many times does it show up in the index, and how relevant is that keyword? And then finally, the last phase, phase four, is looking at keyword adjacency. Or, like, you know, if you have two words, how close to them to each other are they in the results? So a result that has two words back to back just like you type them is gonna have a lot more ranking than one that has one word here and the next word three paragraphs down on the same on the same document. A lot of these factors can be controlled within Coveo. It is actually what makes up the Coveo core out of the box algorithm. And when I think about ranking and and relevance tuning, I wanna make sure everyone sees this as like a stratified, like, a lasagna of of of relevance. So the bottom layer is gonna be the out of the box Coveo algorithm, which we're gonna talk about. That's what these ranking factors are. You wanna get that tuned as good as possible right out of the gates. And generally speaking, Coveo is pretty good at this. So, usually, you'll find if you've come in from another search application, just without having to add any rules or do anything, pretty much the algorithm is gonna get you where you need to be. But sometimes you do have to nudge it along a little bit, and that's where ranking expressions come in. It's, like, the next layer on top. And so you could use some ranking expressions to, like, maybe format like like, put certain types of content higher than others and and and kinda tune it to your taste. And then finally, the icing on our I guess I'm talking about cakes now. But the the sauce on top of the lasagna here is gonna be the machine learning. And that is what's gonna help automate all of this process for you. And so it's gonna help fine tune things and and grow dynamically and learn as your users are interacting with it. So you don't have to go in and continuously manage these manual rules. So let's talk about the weighting factors. So when you go into your query pipeline, you can actually control this. This is sort of the Coveo secret sauce. When you go into a query pipeline, go into the advanced tab, and then from advanced, you'll see ranking weights. And when you open that up and hit add a rule, you're gonna see this screen right here. These are the out of the box Coveo ranking weights. And just like Lipica was showing you what the ranking, result ranking, you can, like, kinda take that slider and move it up and down. You can do that with these weights as well. They have a much lower impact, however, than, like, the actual ranking rules. So what are those these factors? So the first is last item modification. This comes into phase two. If you would look at that funnel on the previous slide, it's a document it's it's document ranking based on the date. How fresh is the content? If you feel like you want your freshest content to be up at the top, you can take that slider and boost it up, and that's gonna have more of a relevance actually on date. If, however, that doesn't really matter, you've got a lot of really good evergreen content in there and you don't really wanna bias your newest content, you can turn that all the way off and remove that as a as a factor in the ranking equation. The next couple of them are related to that first phase. It's the the keyword term, location. So this keyword I'm sorry. Keyword frequency is the next one, and that's actually phase three. That's how often does the keyword show up. You can change this. The next three, keyword and concept, keyword and title, and keyword and summary are all about where those keywords are. So if you want the keywords in the title to have a more more of a ranking weight than others, you can boost up that particular thing. If there's a summary field or a description field that you've got indexed, you might wanna boost up, you know, the relevance related to those keywords existing in this in the summary. And finally, there's this concept field. So the concept field actually is a field that Coveo renders at indexing time. When we do the scan of your document, it'll look at the other keywords that are already in the index, and it'll find keywords from each document that seem to be relevant, that seem to have a lot of bearing. It's sort of like a way of, like, using AI to tag your content. And so that field is generated. I'm gonna show that to you in a minute. The last ranking factor on here is keyword proximity, and that's the last phase of the funnel. It's the adjacency. So in Coveo world, like, people search for the words machine learning a lot. Right? Those are two keywords that are very closely related to each other, and we'd expect to see a lot of content with machine learning right there at the top. However, we might have on our index, you know, a a document where the word machine is here and the word learning is, like, three pages deep into the the PDF or whatever. That document will still have a keyword match for both of those terms, but because they're not close to each other like this, it's gonna be further down in the result list. You can control that with this factor. Again, out of the box, this is a pretty good formula. There are some reasons why you might wanna adjust some of these factors and your CSM or onboarding manager will work with you, in in helping identify those scenarios. So let's move on to quickly showing you, again, keyword and title and summary and then concepts. If you look in your content browser, you can see these these concepts in action. So this is a content browser. I have an item opened up here, and you can see I've highlighted the title. Obviously, if the keyword I searched for is in that title, that's gonna be where the ranking comes from. Here's a picture of that concept field. So Lipika showed it to you live a little bit earlier when she double clicked on one of the documents in the content browser. You can see the field name of those concepts, and then these were, like, the keywords that were automatically tagged by Coveo and pushed into that field. And then, also, when you look at the content browser, just on the main page, you can see summary. It's in the form of a description that shows up there. It's sort of like a excerpt almost in your in your result template. Alright. So how will you view result ranking and actually see what's happening and and know why a result is where it is? Well, there's a couple ways to do this. I'm gonna show them both to you. I'm gonna talk about them here, and then I'll show them to you live really quick. So the first one is query debug. So, basically, this one, if you're a Mac user, you just hold down the option key and double click and somewhere in the white space or nonactive space on a result, and it opens up a debug window that you can look at and interpret. This only works with Coveo JavaScript UI framework. So for those folks who may be developing using headless, there is a way to do this, but you have to actually use, the Chrome developer tools and look in the network tab, and you'll have to have the debug equals true parameter enabled in your query pipeline. If you got questions about that, see me after class. We can work it out. The other one here, which is a little bit easier on the eyes, is called the relevance inspector. This is another tool that only works with UI framework UI, and it also works only with HTML based search pages. So it doesn't work, unfortunately, like, in a Salesforce console, for example. But if you've got a web page, all you do is write in the end of your URL string and debug equals true and then rerender your query, and now you're gonna have the relevance inspector option. So let me just show you quickly what that looks like, and I'll use Lipica's page because she's been doing some work in here already. I've already gone ahead and added and debug equals true up here in the URL string. And you can see what's happened was it adds this relevance inspector option here. And what you do is actually see it, like, in this format right here. If, like, if I didn't have that, for example, let me just take it off of this page and just show you what it would look like without it. The results just look like this. It's a standard search results page. I'm not enhancing it in any way. With the relevance inspector on, it is enhanced. I have some options right here. Let me show you the debug thing. This one first, I'm on the white space of a result. I'm gonna hold my option key, double click right here, and then I'm opening up this debug window. You wanna make sure you've got enable query debug selected. And as I look through here, I can see some things. I haven't actually executed a query, so let's let's try doing that. Let's do, machine learning because I like to talk about that one. It's it's nice. If I do my debug here, I can see the term weights. So look at and it breaks it down one keyword at a time. So you can see the word machine. It's gotten a four hundred points because the word machine is in the title of this document. You can actually see the title of the document, you know, in in the metadata here as well. It's got the keyword in the concept, keyword in the summary. So you can see the points are starting to tally up for this one right here. Same thing with the word learning. If there's stemming, it'll get stemmed as well. So you get a pretty good example, and then this is gonna start to contribute to the overall relevant score. It is this score right here that determines the order of your results. So the higher that score is, the higher up it's gonna be in your results. So if you have a result that's down five, six in the results and you wonder why it's so low and it should be higher, just do look at look look at the scores, and you'll see there might be some other factors that are pushing other things ahead of it. So let's look at how that works in the relevance inspector. So a little again, easier on the eyes. With the relevance inspector, at the top of each result, you can see the total score. You can look at the combined, ranking factors up here above including the QRE, which we'll get to in a moment. If you go and look toggle term relevancy breakdown, you can see each keyword, and you can see this one does include the stemming. It's got learning and the word learn in here as well, so you can see what are those stems to. And then you get a breakdown of the keywords and how they relate to each one of those ranking factors. So that that is this really quick. Let me go back to my my deck, and we'll continue on. So, we talked about this already. I just did a quick, demo of this in real time, so I'm not gonna spend too much time on these slides. But, again, the slides that we'll send afterward are just kinda showing you this is what it looks like if I did a query for about smart snippets. These are the keywords. You can see how they were they are ranked based on their keyword ranking factors. And then here's how it looks when using a relevance inspector. So the next thing is QRE. So we'd spent some time already playing around with some QREs. Now and and I don't know. Like, one thing that's important to note, Lipika mentioned it. I'll mention it again. When you are playing around in your, in your cloud platform, when you, boost something, say you boost it by a hundred points, you're gonna have to multiply that by ten. So what looks like a hundred in the Coveo cloud platform shows up as a thousand in here. And so this is an example from a different search page, but you can see I added an expression at source is equal to Coveo docs. And I and I gave it a boost of a hundred points, which shows up as a thousand points here in the debug, and you can see that showing up right there. I'll I'll just toggle back to the search page because has done some of this stuff already. So if we look at some of the things that she's done, I'll look at this toggle QRE breakdown, and I can see she had a a expression in here that's boosting Covea website content, and that's being attributed here in the QRE. If I take a look at the query pipeline itself, this is that the how it looks on the back end. Source at, Covey website, boost by one hundred. In the front end, we see it right here. If you were to look at this in the debug window, you would see that right here, and that score is being attributed into the QRE. So that is how that part works. If I looked over at our YouTube tab, she spent some time building a YouTube, you know, view count based QRE. So I can let me just remove my query here so I got a blank slate. But you can see the expression YouTube view count is greater than a hundred and fifty. She boosted it by two hundred. It's getting a boost of two thousand right here. You can see that being applied in the QRE, and it gets added to this total score. So that's why these are now gonna be up a little bit higher. It's simply adding points onto that score. Alright. Let's move on. So, let's talk about machine learning. So the automatic relevance tuning model is the remember, this is like, I guess, the cheese and the and the sauce on top of the lasagna. This is gonna help really keep things moving along and and dynamic as you go. This also is something you can control in the query pipeline. So when you go into the pipeline, go into machine learning, find your automatic relevance tuning model, and you go into it, and you'll see there's a ranking modifier piece right in here. And just like everything else, it has a slider. And by default, that slider is set to two hundred and fifty. In my example here, I boosted it up to five hundred just for kicks, so I could see a better growth. Now what this will do is it will add an incremental score of learned items where the highest and most favorite item is gonna have whatever your maximum value is here. So in this case, the the most relevant result that's been clicked on the most given a query is gonna get boosted by five thousand in my UI. Then the second most boosted item is gonna be maybe boosted by four thousand, and the third will be boosted by three thousand and so forth. By default, the automatic web instrument model will learn up to five items for any given query or or or contact set. That is also configurable. So if you want your model to learn, like, twenty items, like, I have one client who's in, like, an ecommerce use case, and they want their entire first page of results to be all ML learned results. Great. We can kick it up to twenty or however many results you have in there. So that's definitely possible. And this is gonna be visible as well in the QRE section. So if I were to show you what that would look like, using this example, what you wanna look for to tell if it is a machine learned boost result, there's a couple ways. If you're using the debug panel alone, you can look at the expression area and look at any expression that begins with at permanent ID. This is how the onetwo minuteimum tuning model attributes, like, basically a value to a specific item. We use permanent ID because if we use title, sometimes the title of a document might change. Sometimes if we even if we use a URI hash, if you were able to, like, rebuild your source or rebuild, you know, your maybe migrate from one, source to another, you might lose some of that data that resets it, and it loses all that machine learning. But by using a permanent ID, we kinda future proof your implementation that's just built right into the code. Now if you are on the UI, not in debug mode, but if you're using a relevance inspector, again, you can look at the toggle QRE breakdown, and you can see it right in here that that was boosted by automatic levels tuning by the app permanent ID. That's the giveaway. And, of course, if you wanted to go the extra level and, like, with a visual cue to your users, there is this cool, like, badge you can put on your result templates that's recommended by Coveo ML That takes all the guesswork out of it. When I've got that badge, implemented, I know that this particular result was boosted by machine learning. So to recap on these phases, let's look at them again. Phase one, it's all about term weighting. It's about term location. Is that gonna be a a keyword that's in the title? Is it in the summary slash description? Is it in the concepts field? We also are gonna look at case of, you know, the lowercase, uppercase, and stemming of those keywords and, most importantly, ranking expressions. That means any manual boosting rule you create is gonna be applied right here at the beginning of the phase. And then machine learning gets popped in there as well. So so your keyword matching, and your machine learning and your ranking expressions are gonna bring back a giant swap of results. From there, we're gonna look at document weighting. So this is gonna be, like like, things like the dates and so forth. And there's not a lot that happens in phase two that and a lot of you most of the time, you'll see that these factors kinda remain static. So it's mostly related to the date of content. A lot more happens in phase three. This is gonna be related to keyword frequency within the document as well as relative keyword frequency within the index. That's gonna narrow down, it's gonna impact the top of hundred results. And, also, the top one hundred results are gonna be impacted by keyword adjacency. So we would definitely expect that it'd be a string of keywords in your query that the the one word that has the most keyword matches that are closest to one another is gonna be closest to the top of the result list. And that is pretty much how that all works. So some best practices to consider. Definitely number one is use ranking rules for legitimate reasons. We talked about that a little bit before the segment starts. If you don't have a huge index, you may not have a big need for this. But if you do have a larger index and you have a lot of different scenarios and and and so forth, you might have good business, rules to do this. And, generally, what we wanna do is use ranking expressions to help maybe give machine learning something to learn from. So you can test really, like like, train a model a little bit, using something like this. So that's definitely possible, and there's lots of different use cases for it. In our next couple of webinars that we have this quarter, we're gonna dig deeper into some of those more specific use case based scenarios where you might wanna use a ranking expression to achieve a specific result. Use them sparingly. So, you know, maintaining a large list of ranking rules can be very cumbersome for you. And the whole point of Coveo, for many customers of ours is that we wanna take some of the administrative burden out of managing your search experience, through the use of things like AI, which is what we're really known for. Obviously, ranking expressions exist. We there's definitely a strong purpose and a business value for for for for all of our applications of them, but you don't need to go crazy. In fact, keep it keep it sim simple. And I put this a little bit lower down, but try to keep the the boosting rules within, like, fifty points plus or minus at a time. In our examples, we use more extremes. We use a hundred points, sometimes two hundred points. I bumped my, I I bumped my art model up to five hundred. But that's just to give you a a sense of what it can do in the demo environment in the real world. You know, try to keep them definitely within double digits, in terms of, your your movement one way or another. And, you know, negative boosting can be a really good friend. So we had a little chat about this here as well, the negative boosting is a thing. We didn't really talk about it yet. But if you got, like, say say a lot of file types and you know, like, there are some like, you have a lot of Excel docs in your in your in your index, but that's not gonna add a lot of value. And you may have some keyword matches in there that's gonna bring those Excel docs to the top, but it's probably not the content that you wanna front. You can use a negative boosting rule to push, like, anything that's file type XLS down to the bottom as a quick example. It's better to do deboosting where you can because it's gonna have less influence on machine learning and all the other good results. But if you add too many positive boosting results in there, you might actually start to, like, impose a bit too much upon machine learning, and it becomes a bit too, manufactured, I guess. And finally well, not finally, a couple more here. So make your expression rules just specific enough. So, yeah, so try to apply a condition to each rule. Don't be too specific with them. Otherwise, it's not gonna have much of a wide spread impact. You just wanna have enough impact that it is helping to train machine learning models and, and and impact positively the the large amount of people. The more people you can impact positively with this with with, less, rules, the better. And that's something that your CSM can work with you on, finding the right balance there. And finally, test your rules. So we have some options available for this. You can use AB test within the query pipeline. That will allow you to get a concept, put that concept into use, run it, see how it see how it plays. If it doesn't if you don't like the results, then you can always just revert back to normal. If it works, you can push it forward to full production. We also have a feature called groups and campaigns, which we're gonna talk about, in our third session of this series, which allows you to create a whole book of rules and give it, like, a time bound application. So you can apply them and say, I want all these rules to apply, you know, only today. Today only. Great. And then you don't have to touch it after the time comes to the end. It'll demote all those rules and put them back into standby until the next time you wanna use that campaign. So it's another way that you can kinda time it and and and test things out. And with that said, we're gonna send this along later. We'll have the recording of this of this session. There is some documentation to share. A lot of the content that Lipica and I shared with you today is actually in LevelUp, which is our online elearning, website. It's a fantastic place to go. If you're a new Coveo customer or even if you've been with Coveo for a long time, but you still wanna brush up on some things, go over to level up dot coveo dot com. You will find a plethora of good things. And, actually, we do have a road map in there that's called new customer onboarding. It is designed for any new customer to go through things like analytics, for the basics, and, of course, relevance tuning, which has a lot of this content and even, more, more depth, in there as well. So with that, I think that was all that I had, to share today. Alright. Thanks, Jason. I know we went over time, but, I hope you don't mind if we answer some questions that popped up in the q and a box. First one that we have, can you exclude content from the results from for example, if the source has release notes from twenty seventeen, but I don't want twenty seventeen release notes to appear in the results, can I exclude release notes from the search results? Oh, absolutely. All you need to do that is the right metadata. And so I'll I'll do one really quick, here in in the org. So we talked about a lot of ranking expressions today, but we didn't talk about filter expressions. But let's just say, oh, Whims whiteboard, Wednesday. This is great. If you guys haven't seen that yet, it's cool. Maybe I don't wanna see Whims whiteboard, Wednesday. So I could find something in here as long as I find a field value pair that matches the content that I want to exclude. So I'll just do one real simple here. Let's just say it's I know this the the title of this document, or even I I can use a category. Let's just use science and technology, and I can see that the category is YouTube y t category. If I wanna just exclude all of those from my, my experience, I can go into my pipeline, go into advanced, and then you're gonna use this filter expression. You can hit add a rule, and we can talk about I I'm not gonna go into all the detail, but I'm gonna pick a constant query, which is basically what you wanna use if you're doing a filter and say something like not at, YouTube category, I think is what it was, is equal to science and technology. And when I apply that, you can conditionally apply it as well. But when I do apply this, it will remove all of that content from my search interface. So it's using QPL just in a in a different way from ranking, but it's the the format of how you write that expression remains the same. Were there other questions? Yes. Another one. Our case deflection rate is only a few percent, and we have ART enabled, various ranking expressions applied. Is there anything else we could do? Yeah. There's, there's definitely a lot of things you we can do on this. I without opening up a whole, can of worms on this call, I would say talk to your onboarding manager if you're still in onboarding. If you're an if you're an existing customer, talk to your CSM. Reach out to your CSM at once. We've got some great tactics on this. Usually and and one thing I'll say about this, a lot of it has to do with the query parameters settings in your query pipelines. So when you look in your query parameters, you should be able to see what is driving relevance on your case deflection page. Is it, like, you'd when they type in the case subject, are there other values, like, the they select the product category and so forth? Or is it just, like, kinda like grouping everything together and and and and and and and, you know, seeing what comes up? So there's ways that you can tweak the query parameters there that will actually improve the results. That's gonna be what we cover in our next session is actually gonna be around, that particular use case and how we can use context to to drive better results in insight panels and case to question pages. But and you'll have to wait until then. Talk to your CSM, and we'll we'll get right on it. That's awesome, Jason. We there's a comment here. It sounds like everyone has a book of go to expressions. Is there a cheat sheet that can be leveraged that, can be used by everyone? The Coveo expression is somewhat of a learning curve. Yeah. There is and I'll I'll probably include that, actually, as one of these resources in here. The the relevance tuning, solution document has some of that in there. There is a document on Connect that is all about the different, like, I guess, fields, parameters, relevance, you know, expressions, and so forth. And there are some that are use case specific too. And one thing I didn't get into today, but I will definitely will get into in our next session is, like, using more of, like, the dollar query or dollar keywords slash query kinda thing where it's a variable. So no matter what they type in, you can, like, apply that to a certain field. So that is, like, sort of the secret sauce to a lot of the things that we do in, like, commerce, like, for example. So maybe you wanna boost the your categories. If you've got, like, listing pages and someone searches for, you know, sunglasses, you know, that might be a category, not just a result. So you could boost everything in that category by using a variable based expression. So there's a lot of things like this. We'll try to do our best to provide you some documentation around this and, again, a good topic for your CSM because there is a lot of tribal knowledge out there that we do the best we can document, but it's you know, every use case is a little bit different. Okay. Jason, last question for today. Can the name of the badge be changed instead of boosted by cover ML, maybe boosted by organization name? Yeah. Absolutely. You can, completely do that. So, basically and maybe I'm still sharing my screen, am I? You may even be able to see You are. Yeah. You may even be able to see it in here. So there is this, badge component when you are working in the the code view of your, of your result template in here, and I might not be able to find it right away. But, certainly, there is the the coveo I think it's probably on the whole page. But, basically, you can give that a name. So the I just used, like, the default out of the box naming convention for it. You do have to kinda get into the code. It's a single line of code. It's I'll follow-up exactly with what it is. There's there's a document all about that, but, certainly, you you can absolutely change that. Okay. Is that is that all, Jason? We're good to go? I think so. Thanks, everybody. Oh my god. Thank you thank you so much for the questions. Like, we really appreciate you asking those questions on the fly. It helps us understand what you need what you need to learn more, and we can tweak our content with the questions that you're sending us. So please keep at it. Keep engaging with us. We appreciate it. As a reminder, we will be sending a recording of this webinar, so don't worry about it. You're not gonna miss any, any part of this webinar once, once we send out the follow-up email. K? So that's it for today. Thank you so much, everybody. We'll see you in the next learning series webinar on March third. Have a great day, everyone. Bye for now.