AI. Welcome to Tech on Deck, a podcast series focusing on the critical business challenges facing b two b tech firms with an emphasis on enabling technology. I'm your host, George Humphrey from TSIA, and I'll be joined by Angelo Saniford, a senior director of global services operations from Coveo. For this discussion, we're gonna be talking about what great looks like real world lessons from scaling AI powered support. So, Angelo, welcome to the podcast. Thank you for having me, George. I'm really excited about our conversation and looking forward to it. Yeah. Yeah. Me as well. I'm I'm super looking forward to it. These conversations are contained to a short period of time. I know with this topic, you and I could probably sit here for about two hours and talk through it. So we're gonna try to keep it really, really focused for this session. And for for me at TSIA, being one of the lead researchers on AI and leading the research organization for technology services, this there's really never been a more exciting time, yet it's really never been more challenging for AI and technology services. It's kind of overwhelming for a lot of organizations. We see a mad scramble. The data tells us that over eighty percent of technology services organizations are investing in AI right now. So there's a lot of investment happening. Unfortunately, there's a lack of clear corporate strategies, and we really see more art projects than you can shake a stick at. And so as a result, we are on a journey. We're looking for great success stories that are gonna inspire the community, inspire great tech companies to be better, and to lean into some of these next generation technologies that we think are really gonna change the landscape. And so we know at Coveo, you have your own implementation of AI in support, and we think it's one of those great success stories that we wanna talk about. And so I wanna jump right in with a series of questions for you. Fair enough? Fair enough. Alright. Cool. So so the first question I have for you is you've seen a wide range of customer implementations. Right? So Covea has probably seen the good, the bad, and the ugly. What really separates the companies that succeed from those that really stall out? Really, are there patterns in both patterns of failure and patterns of success? You know? And so maybe we just start there with with that broad question. Yeah. Yeah. It's it's it's a great question. Like, I mean, if I think about if I think back to, you know, what I've experienced myself, what our company's experience, what our customers experience, you know, one of the things that I would say from a success standpoint that really stands out is, you know, to your point, the the organizations that really, create a strategy around it. Right? Really focus in on the outcomes that they wanna achieve, rather than it focus on, you know, oh, what's that nice shiny new feature that's available with Gen AI or with AI in general. Right? They really have a strategy that's focused on, you know, process people and and and the technology behind it. Right? And and this is important. But the other thing that's really important important is is the foundation. And, you know, I can't stress more, the importance of starting off with a healthy foundation and implementation. If you're, you know, if your data that you're pulling from is messy, it's not, you know, curated in a way and indexed in a way where you can access it from various channels. Right? Because in the end, enterprises, you know, we know that we have so many sources of data that we pull from, often unstructured and structured data that, really can't be accessed easily unless you're you're operating off of a a AI search, something that brings it all together and and make sure that it's grounded, grounded and and and and precise and personalized. So this is the thing that really sets, you know, the sick those that are successful apart from those that are struggling and and faced with a lot of challenges. It's that initial strategy and having a solid content management and content strategy behind that is is really crucial. Yeah. Content management, content structure. So these are like, if you ask most organizations, what's the state of your data today? What's the state of your knowledge? You're probably gonna get an answer that it's pretty chaotic. I think most organizations struggle with that. So I think that's a really key trade a key focus. I love that you started on the word strategy. Do they have a formal focus strategy on how they're going to deploy AI. But but what about the flip AI, the maybe less technical side? What are the cultural differences between the companies that are succeeding here and the ones that are struggling? Well, certainly, I think, you know, one thing that's important is having the backing of an executive sponsorship. You know, having that executive sponsorship to really drive that top down strategy, and get people to buy in into knowledge management as a an essential. It's not a nice to have. Right? You gotta have that knowledge management practice. You know, we leverage KCS as an example. Right? This is is is key to to really driving that, that culture and fostering that culture within the organization. If people understand the importance of knowledge, how we capture it, how we create it, how we curate it, then they'll understand the benefits that they can draw from it. And and this is to me crucial and and and, you you know, you can't do without. You know, I had to laugh a little bit to myself while AI I hear you say that. I I spent time in my previous position years ago really obsessing over our knowledge management database. And I kept trying to say to the leadership team, do you have any idea how important this is? And if you think back, you know, ten years ago, we had no idea. If we had only thought about this knowledge structure and the discipline behind it back then, we probably would be way further ahead than many organizations are now. Absolutely. Absolutely. Yeah. So so speaking of, you know, companies that have these challenges, these obstacles, and the ones that are successful, can you share a story from the Coveo background? Maybe one of your customers, you can remove the names to protect the innocent or the guilty, but maybe a success story from your customer base. Yeah. Absolutely. I mean, we've had and seen AI a few of our customers, really benefit from, you know, having that strategy and and tackling, you know, this challenge of how do you bring, how do you bring AI and and and generative AI into your mix, into your into your, ecosystem. And, you know, if we think, you know, I think back to Forcepoint, it's it's a great example. One of our one of our our our our our customers within the, the industry, they they struggled with their, digital experience. Right? Both from an internal and external perspective. So not just the the the customer experience was was suffering, but they also had that, you know, employee experience that was that was that was suffering as well. And, I mean, when I think about how they they took, you know, they they leveraged, our CRGA, our our Cabelo relevance, answering and brought it together to unify. They had over, like, nine nine different, knowledge sources that they had to pull from. It was they had trouble crawling through, you know, the the different repositories to get information, get accurate information. So adoption was low. You know, the success in terms of self-service success wasn't where they wanted it to be. They encountered a lot of challenges along the way, but when they brought in our solution for, you know, to help them with the the generative AI, piece and and and and search optimizing search, they really saw some great success. I mean, you're talking about, you know, a fourteen percent increase, that they saw within their self-service success. You know, twenty five percent decrease in their, AI to resolve, ninety seven percent, CSAT for self-service, and two hundred percent increase in case deflection. And that was within a very short period of of of time. We're not talking about years. We're talking about, you know, weeks and months. So I think, you know, when when you think about starting off correctly, AI we said earlier, this is exactly what we did with Forcepoint. Right? They set off they start off with a smaller use case. They built on it and then, you know, slowly iterated and optimized in order to get the optimal, the that those optimal results that they were looking at that they were getting in the end. That's that's fascinating. So we've been on this journey as well to document these real world case studies, and I expected to see some Agentic, performance when we started to go through some of these case studies. And, like, those numbers you just gave me blew me away. And I just it's unbelievable how many of these really compelling case studies are emerging. We're not just talking about two, three, four, five, six percent increase in efficiency, ten percent increase in efficiency, but a two hundred percent improvement in case deflection. That's absolutely staggering. So very, very cool. I'm I'm sure you have a number of those, and I'm sure the audience would love to hear more of those case studies. Is that something they can see maybe on Coveo's website? A repository of all the case studies that, that we love to share with our with our, with our customers and and anyone who wants to to see how they can how they can, replicate something similar. That's great. That's great. We we have another podcast at TSIA. It's called the Tectonic Podcast with Thomas Law, and we had Thomas and Jeremy Delitezi, our own SVP of AI, software development and analytics, to talk about the journey that that we're on within AI. And so we ourselves at TSIA, we've been creating this AI technology to improve our members' experience. But I wanna latch on to something that you just said in your answer there and then also your last one, which is it's not just an improvement in the metrics for the customer's experience, but it's these internal metrics that we're seeing as well. We're seeing a dramatic improvement in the efficiency of TSIA's employees. So maybe you could share a story about how your own support team has evolved with AI over the past few years. You know, what would say your biggest challenges along your AI journey, and what do you think your greatest successes were? Oh, yeah. That that that that I love to tell this story, to be honest. I mean, you know, we've come a long way. We've come a long way in the last couple years to your point. You know, one of the first things that we started off with is, you know, how to become more efficient. Right? And and I think this is this is an ongoing theme, right, as we we see even even today. But certainly, one of the things that, you know, we wanted to achieve was, you know, better productivity, but also, an improved customer satisfaction, result. And and, you know, one of the things that we did is being a a one tier, support, organization. We needed to be able to bring in, information, solutions, recommendations, all in one flow and make it easy for our product specialists to be able to, you know, deal with situations which they may not have encountered in the in the in the past where, you know, we're, we're a small volume shop in in comparison to many organizations out there, just the nature of what what we offer. But, the the problems are are there and they can be quite complex. So, you know, in terms of looking at, the agent experience, what we needed to do is to bring in this knowledge in a much more effective way and also look at optimizing. So what we did was we used AI, from a, auto case assignment to remove that manual intervention that was required by either the product specialists themselves or, from our managers. And this in itself helped to, you know, increase the efficiency, get cases to the right people because, with this intelligent case routing that we we we, implemented, we are now, routing our cases to the most proficient agent that's available at any given time in a day. So this is where we were able to really start digging and and and scraping at improving what was already a fairly good customer satisfaction, you know, performance at ninety ninety percent plus, but still go and get up to that ninety seven, ninety eight percent, in terms of CSAT on a consistent basis, you know, with a high customer effort score of, you know, close to ninety percent as well. So it we were able to take some of that AI, capability within within our organization and serve serve up, that, information internally. And then the other thing that we did, moving forward was then we incorporated, CRGA into the mix as well, which brought in a, a more effective and efficient way of also surfacing up information with responses that were already, that were already formulated for this product into the, case comments. And this way, they didn't have to spend time worrying about, okay, is this formatted correctly? Am I am I looking this right? All they have to do is validate that the information was there and needed the needs, as best as possible to the customer, and then, you know, and present that. So we saw, you know, an improvement in in mean time resolution, median time to resolve, up by forty percent, improvement. We saw, you know, again, the improvement to the CSAT there. It it it just had a trickle effect to so many different areas of our of our of our organization as a support, organization that, we're still, you know, reaping benefits from it because we were able to, continue to manage an increasing volume case volume, and and remain, flat with our headcount. Wow. I I heard I heard so many powerful things in that answer there, and I'm gonna try to maybe not oversimplify AI, but really make sure that the listeners understand. You know, one thing is is that you don't have to choose whether you focus exclusively on the customer experience or on your own employees' operational experience. You were able, with this approach, to meet the needs of both. Not only did you improve the customer experience, but you improved your your team's operating experience as well. Absolutely. Absolutely. That's a very powerful story. I I think the other thing that you said there that really everyone needs to take away is we we have this framework called the black box of AI. What that means is we see a lot of companies running around, like, where do we apply? Where should we could we apply AI? And it's just kinda chaotic. And we always just say, like, slow down. Take something that is already well understood, a process, a function, but happens to be very manual in nature, very laborious, or very prone to error, and focus on that, a simplification, digit digitizing, and then having maybe an AI agent perform that particular function. Would you agree? I totally agree, and and I would go a step further and and and top up on that. Like, we also incorporated, AI into our intelligence warming. So, you know, from an intelligence warming perspective, as I said earlier, like, you know, our our cases can become quite complex in nature just because of of of our our our product and and, you know, the the the complexity with with which we deal with in terms of the data from our customers. And so, one of the things that that helped as well was the intelligence warning. So bring bring giving specific line of business or product area into a conversation where they can share knowledge and ask questions for things that they may not have found an answer on because it's something new, no new an, a new issue, or it's something that steps slightly out of their their immediate area of expertise because they've never they've never experienced this before and bring that conversation into a, a workable flow for the for the product specialist and and leveraging AI to understand who are the subject matter experts that should be included in that and and really be able to, you know, effectively get answers out quicker than, than we did historically. Wow. I I love it. And so I'm gonna, you know, stay on this thread a little bit here, because we're we're talking effectively about, to to some extent, some AI agent capability here. And there's like, it's AI agent mania right there, and everything's gotta be an AI agent. And so can you share your perspective on that? And, you know, what's an AI agent? What is it not? What's the difference between something like an agent, a copilot, maybe even a teammate? I've heard people use these interchangeably, and I'm not really sure they're they really should be synonyms. I think they're really different functions. What's your perspective? Yeah. Yeah. I would agree. I think I think there is a difference because when we talk about the agentic framework itself, right, the the main difference I'd wanna call out there or I'd call out to anybody is the fact that with agentic the agentic framework, it we're giving the AI models the permission, if you will, to take action based on the information that they that it pulls out and that it's able to reason with. Right? And that's the difference between whether you're looking at a AI or or or anything else. Like, the action piece is missing from from the equation there. Whereas when we're talking about whether it's Copilot or even agent and and so on and so forth, there's still a there's there's still an element of action that's missing that typically has to be taken by someone. So with the agent the agentic framework, this is where, you know, we we see that potential use case and benefit where, you know, to to the point you made it and alluded to earlier is removing that need to focus on the mundane, the the the, you know, error prone things that a machine can do really well. Right? And leave the more human centric stuff to the humans. Right? And this is where you I think we can gain the most benefit from, and this is the distinction that I I would really call out. Love it. Love it. I what I say is AI is gonna make humans be more human. I think we all have to work on things like empathy and relatability, and helping our customers. You know, helping will sell, and selling doesn't necessarily help. So, you know, couple more quick questions for you here in the time we have left. How is Coveo how are you helping your customers help their customers take practical steps forward with AI and autonomous experiences? Yeah. Absolutely. You know, the way the way I see it is that, Coveo, we we we see ourselves as as trusted AI. Right? And and we, we offer, the platform that allows, organizations, companies, enterprises out there to be able to really make the most of what they already have. They already possess this information. They already have the content. They have this data in many different areas. And through our, you know, our our suite of products, right, it allows them to either, you know, have a turnkey solution through the platform, through, you know, a full full service. They can bring their own NLM's. You know, we have the p r, our, passage retrieval API. We have, you know, different ways that we can equip them and help them to be able to, tackle their challenges, you know, achieve the outcomes that they're they're seeking to achieve in whatever way they they desire. Right? Whatever whatever we see will fit best as a solution. We're not there to your point, you know, to sell for selling sake. Like, I mean, that's not gonna help anybody. And when we, you know, advise them in the best way possible, we're doing so in a way that they'll benefit through the objectives that they're trying to achieve whether it's reducing their, median time to resolve, their first contact resolution, and the list goes on. That'll automatically translate to an improvement and enhancement of the customer experience because if they're getting more accurate, relevant, and precise personalized information to their their customers, their customers are gonna be satisfied in in in in return. So I think it's just it's just us helping, through an extended arm their customers as well. That's the way I would I would kind of phrase it. Absolutely love it. And I really love I I think it was Sir Lord Kelvin that said one cannot improve what one cannot measure. And so you you really talked about very measurable things, very tactical approaches in place that organizations can take. I think the whole industry is starving for that kind of information. Final question for you, Angela. So I wanna turn it to maybe a little bit more personal. We've been very technical, very focused on company improvement. But right now, there are a lot of leaders that are trying to find their path forward in AI. So if you had one piece of advice to give to a support leader that's trying to make AI real in their organization, what would what would your one piece of advice be? AI would say, ensure that you have a very disciplined, approach and strategy to content and how you manage it. You know. I think it it is, crucial as I I mentioned earlier to the success of your implementations of AI and generative AI as a whole. Like, you know, you could have the best tool. You could be driving a Ferrari if we let's use that as an illustration. Right? But if you don't know how to drive, if you don't know how to shift gears, you're gonna lose the race. So it's the same principle to me as if you don't have the right, tools, the right, environment that is set up to set you up for success, you're gonna you're gonna fail, unfortunately, and that that's a a reality. So for me, it's being really disciplined around content, having a content content management strategy, and and and focusing on how you will, you know, curate that throughout time in order to ensure that that information is always, the healthiest it can possibly be. That data is always healthy, and and that'll give you the outcomes and the results you're looking for, when you're when you're looking to leverage, AI. That's that's great. Be be very intentful, and have a tactical strategy so that you don't wrap your AI Ferrari around the tree, and you can feel AI you're wasting the first place. Excellent. Okay. Angelo, this has been I can't believe we're already through twenty five minutes of the conversation here. It's been great talking to you. I hope we get to talk a lot more about your journey, about your customer's journey, and we hope everyone gets a little bit of value out of this conversation here today. So thank you very much again for your time. Thank you so much, George. Appreciate it.