Hello, everybody, and welcome to another episode of In Conversation with CCW Europe, the podcast series bringing together the leading and brightest minds in the worlds of customer experience and customer management. I'm Simon Hall, your host and industry analyst at CCW Europe. And joining me on the sofa today, I have Jeff Branch, a Northern Europe sales lead at Coveo, and Remi Preganella, the director of product documentation and digital adoption at Sophos. Gentlemen, welcome. Thank you. Hi. How are you both? Good. Good. Good stuff. Well, to kick off these podcasts, I always like to begin by asking our guests to introduce themselves a little bit and provide some insights into their professional backgrounds and the work that they're doing in their current roles. So, Jeff, perhaps you can kick us off. Sure. Thank you, Simon. So, yeah. I let as you say, head up Northern Europe for Coveo. Supposed to be specifically focused on some of the regions around UK and I, Nordics, Benelux, and France, as I saw some of the stronger regions that we focus on. Been at Coveo now for eighteen months, and I suppose my history at Coveo of those eighteen months has predominantly been focused around customer and employee experience, and then more recently on the commerce side of things as well. But before I joined Coveo, I suppose I spent the last fifteen, twenty years now, I suppose, in and around customer experience, employee experience, and helping organizations improve that sort of element around CSAT, you know, NPS scores and things like that. And then also around improving contact center performance and efficiencies as well. I suppose the reason I've always stuck at that is we're all consumers and customers. We know what happens when we get positive customer experience and how that drives our loyalty to that brand. And then probably what happens more often than not is around that negative experience around customer sort of, experience as well and, obviously, the impact that can have as well. So I think there's, thankfully, a a world of improvement that could be had for organizations out there. And, yeah, I enjoy trying to help organizations move the needle and help them improve. Great stuff. Remi? Yeah. So, I've been working at Sophos for about fifteen years, and at different roles within Sophos really. I started a long, long time ago, just doing some localization. And then I Coveo to kind of working on the website, manage the documentation. And now I'm looking kind of more kind of broad kind of role looking at, so the knowledge part of it. So it's product documentation, but it's also kind of knowledge base as well of digital adoption. And digital adoption is helping our customers basically to use our products. Mhmm. And as Jeff said, really, I mean, it's really interesting because you always need to look at the customer experience and making sure people can find the right content to be able to self serve themselves. Fantastic stuff. So picking up on a lot of what you've just discussed there. The reason that we're here today is to unpack the Sophos Coveo partnership and the work you're doing together with artificial intelligence. And I guess to frame our discussion, Remi, can you break down the challenges that Sophos were looking to solve by bringing Coveo into the picture? You know, where were you falling behind the curve with regards to your customer experience? Yeah. So one of the challenge we've got is that we've got a massive amount of, knowledge around disseminated through the company. So we've got our knowledge base or the income of Salesforce. We've got kind of documentation, which is kind of private provided through different static websites. We've got content within, Confluence. We've got content in Jira. So we've got a lot of different products that interacts with each other, and it was very difficult for us to surface all that knowledge internally and externally. So, what we wanted to do was to find a solution that's going to help us in kind of, aggregating everything and having a simple search solution. And that's where Coveo came in. But he's just kind of helping us in kind of making sure we've got different sources that are kind of coming to a single point, for customers. Mhmm. And of course, introducing AI into a business in any capacity actually requires a degree organizational restructuring, the scale of which will depend on the level of anticipated transformation. Right? So can you share any best practices around how you went about securing executive sponsorship and buying from all the relevant stakeholders? Yeah. So so I mean, we we've we've done different things. And the first thing was kind of really kind of look at different products. Because Coveo was not the only one on the market and we we wanted to make sure that basically we we we tested different solutions. So so COVID was kind of really kind of one of the great great, solution because they've got a vast amount of, kind of good case studies really. That was impressive. Lots of kind of different partnerships with kind of similar, companies as so forth, especially towards kind of technical, side of things. And then to through through the testing, what we've done is calculating what was kind of the return on investment. Mhmm. And this is how we can really decide which product you're going to to go for. And we looked at kind of basically the time to resolution for our agents, which how much we can decrease it. But also we had to look at how much content we would surface to our customers so that they can just kind of have solutions very, very easily. Yeah. And and so in terms of kind of the executive sponsorship, I mean, the numbers spoke by themselves. It was just showing, okay, this is kind of what we want to implement. This is going to be the experience for customers and that's how much kind of, we're going to be able to deflect. And he was just just kind of straight to the point and say, yes. This is the solution we need to implement. Okay. That's great. And just piggybacking off the data and content challenge, that sounds like it's very important for you guys in terms of underpinning the success of this AI transformation. The interconnectivity, the quantity, the quality, the cleanliness of data or critical considerations for you. So what work have you done or are you doing to ensure that you have everything in order on that front? Yeah. So so first of all, I mean, it's it's an iterative process, and this is something that's never going to stop. We always have to improve our data so that we can we can have a best kind of search, results as possible. So one of the thing we've done is that we looked at kind of all the content we add. And first of all, we wanted to remove all duplication. Duplication is kind of means that potentially we're going to to give conflicting information to the customer. So this was kind of one big exercise. So we've gone through kind of legacy kind of documents. We've kind of removed, and really look at kind of consolidating between kind of our documentation and our knowledge base, for instance. The second part is that we slightly changed the the way we write, the content. Because one of the things is that we wanted to make sure that the machine learning and the kind of the AI especially with Chennai coming up is that we write kind of in a more conversational way. So that it's a lot more friendly to the users when kind of the machine learning and will pick up, the content. And to do this, we've just kind of removed kind of quality realism. We've been trying to, write content which is kind of straight to the point. So we avoid kind of marketing fluff, because because this is impacting basically the the results we've got. But not only that, but we've tried to keep things simple, small, and small chunks so that the machine learning and the AI can just reuse these components. The longer the document you have, the more difficult it's going to be for the machine to identify the right content. Yeah. So this is what what we've done. And I said, we we still look at kind of data so that we improve this this kind of on a constant, basis. Mhmm. Yeah. I was gonna add to that. I think, as you say, it is an interesting experience. It's not a case of one and done. Certainly, we see that a lot of our clients is saying that it's, you know, the first time you roll this out is the first step in that journey, and then using the insights to try and sort of change that sort of, continuous learning behavior. And I think the other thing we also see is around, as you said, the data. Yes. It's obviously getting it into the right sort of format. But I think a lot of clients ask us, especially in the early days, about how do we get the data clean? Do we need to do a lot of work? And actually, is you know, the easy answer is no. You can use the AI as well to give you a certain amount of assistance in that respect because it will naturally elevate the good content. And it will actually start to sort of hide and degrade some of the, the poor content as well. So naturally Coveo time that will evolve. Yeah. I think the the other thing as well is looking at the metrics. We can identify kind of gaps in our knowledge. So we look at what customers are searching and we identify gaps and this is something that Coveo provides us. Is there some insights that just allows us to kind of bridge these gaps for customers by either kind of changing the terms, you know, with documentation or just creating brand new content. I think this is this is key. Okay. Great. And just pivoting on now to implementation, there's still a great deal of unease throughout the industry around implementing customer facing AI for obvious reasons. So so the first thing is we had extensive testing. We So so the first thing is we had extensive testing. We wanted to validate the content internally. So we had kind of a team of of peoples, from kind of agents, from documentation, the knowledge best team, just kind of testing the kind of gen AI component. We've all asked kind of different questions and we rated the accuracy of the answers. So so it was kind of rating the kind of linguistic quality, the quality of the answer itself just to make sure that, the content provided was correct. And because because, you know, we've especially with Gen AI, we've got the kind of concept of hallucinations where the AI can just start creating content out of thin air. And this is something we want to avoid, especially we're working in so forth. It's kind of cybersecurity company. We want to make sure we provide the right information to our customers. So this was this was very, very important is we we wanted that the accuracy was correct and that we could give them for customers the right information so that we had simple steps to implement their processes. So, yeah, from the proof of concept, we had, the the feedback, was was pretty good. The accuracy was rather impressive. I mean, there's always kind of things that don't work as as expected. But, we've asked kind of this team of tester if they were happy to use the Gen AI internally and externally, and we've got seventy percent of people that said yes, because the quality was was good enough. So so this was kind of a buy in we wanted. It's just kind of validation from the field. People that are getting people that are going to use it day in day out. I'm making sure they were comfortable with it. This is interesting because we always have also the concept of kind of gen AI kind of potentially kind of frightening some some some jobs. And I don't think people were were too scared about it. I think they they realize that it's a good tool, moving forward. And, yeah, it's it means that they don't kind of do the same thing all over again. They can work on more exciting things, and I think that that's an exciting challenge for people. Yeah. Good. Jeff, any implementation best practices? Definitely. I think, really just Coveo touching base on what you were saying, I think, Remy, is that a lot of customers are going to test this internally first, maybe looking with their agents because they want to make sure that they get the accuracy and they're confident and comfortable before they externalize that in front of customers. So I think it's a very sort of a typical way that customers have done the initial sort of phase around implementation because they can control that And obviously, learn from that before they sort of go external. And then even then sometimes when they have then start to look at this as an external sort of use case, then they start to do things like AB testing as well. So they can really be sure about the impact it's having. So I think that's very typical. And then also, I think, in regards to the agent experience, you're right. I think people see this as sometimes a bit of a threat is like, do we need to are we gonna lose jobs? Are we gonna lose people? And I think very much actually, it starts to improve the agent experience because I think as AI and now generative AI comes into it, you can start to do much more case deflection, dry self-service, which obviously has a massive impact to customer satisfaction because I I think people naturally gravitate to self serve first. I know I certainly do if I go to a website. But then if they do go to sort of raise a case, it can then start to help the agents to actually respond to that customer quicker. And it does help them to actually have that more meaningful conversation. So I always think that, the self-service should should really do that high volume, low value sort of interaction, that sort of very repetitive tasks. And therefore, it's freeing up agents to have that really high impact conversation with customers and driving that value. And we certainly see customers having a an impact on their agent retention. And I think it's because they're having a more enjoyable sort of experience. So definitely something we see in the marketplace. Yeah. Great. So we've talked there about stakeholder buying, we've talked about data, we've talked about implementation. Can you take us on a deep dive now about what your AI solutions are are doing for you today? How have you improved the customer experience? And can you put some numbers around it? Yeah. So numbers are always difficult to to go out to to give. But but I I think in general, what what we've seen is that since we've implemented the solution, we've got we surface a lot more content. So so we had kind of a look at what happened in especially last last year. And I think for the first first six months, we had kind of about eight hundred kind of case deflection, explicit case deflection. So so it's really kind of we we can measure this and this has kind of a big impact on customer experience, but also kind of cost for us. Because every time you contact an agent, then you take time from the business. So I think for this this part was kind of critical really. Is we see that kind of self self serving mechanism is is working. And the more, we surface documents, the more people are going to use it, and the system is going to get better and better. So I think overall, the the experience has has been great, and we want to do more. I said, we want to bridge more content gap. We want to kind of index more content, and one of the things we don't have to talk about kind of the documentation and knowledge, but we started to index as well some of the videos we've got, some of the, community, post we've got as well. So so we start kind of bringing a lot more content to other customers, which means that down the line, people will find exactly the content they want. So yeah, it's it's been very, very positive and say it's an iterative work and we continue working with Coveo and we've kind of our CSMs just to make things better. We analyze data, we iterate, we change, we do AB testing from time to time as well, just making sure that we're on the right track. But yeah, the results are kind of very, very positive and we cannot wait. You really can properly use Gen AI moving forward because it's going to be even better. Great. Jeff? Just to add to that, I think, yeah, I think, looking at things like the standards of AI capabilities, I mean, typically as you say, I think it all starts with around the the impact in the business, you know, goals and objectives that you're looking for. From an across industries, we typically see sort of increase in self-service around this or fifty five percent using standard AI. And again, that has a huge impact on reducing operational expenditure because clearly, if they're self serving rather than raising cases, it has a massive impact on cost and reducing those costs. And And then also, I think, you know, when cases are raised to be what we see is a reduction in case resolution time by about twenty five percent because the agents have access to the the relevant information to answer that customer's inquiry quicker. Again, impacts the customer experience, but also helps agents go through more calls per day as well. And I think when we look at sort of case deflection, typically, we look at a case deflection rate of about eighty percent increase of case deflection. And then when we specifically look at things like generative answering as well, which is also the new sort of buzzword in the marketplace. Then we've already seen customers in fact, Xero, who is one of the first customers to go live with our capability, have gone public and they see an increase in case deflection of twenty one percent above and beyond what they saw before. So significant impact there. And I think if we look at other clients, we're starting to see very similar sort of low to mid-20s sort of additional case deflection rates. So again, it's already starting to increase that sort of performance our clients are seeing. So I'm sure we'll see more of that sort of go through. Yeah. That's interesting because in terms of kind of the agents, time, I mean, we've seen the kind of a reduction about fifty percent. So so it's above the industry standard of twenty five percent Mhmm. Which means that they've got more time to deal with more cases, which is which is brilliant. Mhmm. And I think in the more complex environments, I think that's just gonna be accelerated, as you say. I think, obviously, a complex environment like yourselves, I think, you know, it helps the agents sort of compile that response in a very quick and seamless fashion that they can impose that knowledge on to customers quick. And, yeah, I think that's gonna be something we see, certainly in the complex environments that, we typically work in. But I think moving forward with Gen AI, I mean, this is something we can do directly to the customer rather than having to go through an agent Is that we generally will provide the right answer formatted properly with the steps directly to to the customer rather than having to go through an agent. Mhmm. Well, that's kind of a perfect segue actually onto my next question, which is around sort of the human piece of the AI adoption puzzle. Naturally, the long term success of any AI or digital transformation initiative comes back to talent. So can you share any best practices around preparing the staff and the teams who are working closest with this technology? Yeah. There's there's not much to do really. I mean, we we had to look at it. And and because because the tool, Gen AI, is just a conversational tool, you don't need to do much. You don't need a massive amount of training. I think the only thing is we need to convince people to use it. Mhmm. Especially internally. Just making sure they go to kind of the, the right search section of our work on website or kind of the the internal tool so that we can get the most of, the the AI, parts. But in terms of training, there's there's there's not a lot. I mean, we we know from from kind of different different sources. We always talk about kind of setting the stage for the AI and making sure we've got the right parameters. I think I think it's great if you if you want to have complex answers and if you want maybe to to have some code being given to you in a certain format. But for simple questions on how to set up a system, you don't need a massive amount of of training. I think it's just enter your query in natural language, and then that's it. I don't know what, Jeff, you've seen, but Yeah. Very similar. I think, you know, as you say, I think, you know, because the the solution is embedded into the existing UI, then the user interface that customers are used to. It could be the same customer portal. They're not going anywhere different. It's a single sort of search box. So for their that adoption is usually pretty pretty quick because it's just continuing with the same journey they had. And very similar for the agents as well because they're in exactly the same sort of application whether, you know, know, it's the agent portal. And again, it's the single sort of user interface. They don't have to then do that typical swivel chairing that most agents have done with, you know, eight to a dozen tabs open to try and find that answer. They just stay in one user interface. So actually what we see is that user adoption is very quick because they can see the the value in it. And I suppose that's really what's important because if agents and users don't see value in it, they won't use it. And I think because it makes their day a lot easier and they can get information to answer a customer inquiry quicker, then typically, it's a very quick adoption rate. Existing application, so training time is pretty much, you know, negligible. I think the only thing we do see is, I think you touched on this a second ago, actually, you and these around sort of our customer success team within where with our clients to then say, actually, well, point one, how do we help customers get to that sort of initial sort of objective? And there's and the measures of success. And then over time, how do we accelerate that? What's the next step? How do we improve and accelerate those returns that we're we're providing? So, yeah, it's usually pretty quick. Yeah. I think of the there's one thing maybe that in terms of training. And it's it's how do you search? Because the way you're going to search is different. You don't use keywords anymore. I mean, we were all used to to have key specific keywords, you find the content. Now you don't need to do that. You just ask your question. And I think that's that's the big change is that the way you're going to enter your kind of search query is going to be slightly different as it is to natural language. Mhmm. And that's the beauty of LLMs. Right? Well, looking more broadly at the search landscape then and the changing dynamics of consumer behavior, how is AI and more recently generative AI impacting the way that consumers are searching for and finding information? Jeff, process one for you. Yeah. Happy to do. I think, as you say, I think customer behavior has been changing. I mean, they always been around for, you know, a dozen years or so now. So I think, you know, customers and users and even employees have got used to sort of answering or asking questions in a slightly different way, being a bit more verbose, as opposed to sort of keywords as in when you look at sort of semantic sort of keyword searches back in the day. And I think, you know, generative has just accelerated that. Now people in their sort of, you know, normal sort of, work and home life are using that sort of capability to ask more comprehensive and complex questions or multiple questions in one sort of request. And because they're expecting things like chat GPD out there and that generative capability to provide a more personalized response is definitely something we see in changing behavior. And, obviously, customers we're working with are seeing similar sort of changes as well in customer behavior because once people realize there's a capability and a technology out there, they embrace it and they expect other organizations to provide that same sort of level of response as well. Yeah. Yeah. That's interesting. I mean, just just add, I think now we we see Genoa is everywhere. I mean, Microsoft is kind of deploying kind of copilot nearly nearly everywhere. You've got things on phones now using So this is going to become the norm. And I think that's it. It's it's that we need to evolve and we need to make sure we use that technology because otherwise, we stay behind and people are not going to be able to search and find the content they want. So so this is becoming the norm. And that's one of the big, big change, I think, in the Yeah. Industry. Absolutely. And I think one of the things that I think you touched on as well actually, Rem, is around the host nations. I think when we come to sort of our enterprise organizations, I think, as you say, the expectation there is that generative is now an experience that people expect. But looking from an organizational point of view is people then wanna make sure how can we make sure that the information we provided is validated, is correct. So again, from all sort of side of things, we try to make sure that everything is grounded within that organization's data and content. It's not gonna go externally to that organization's content and data sources, so we can make sure it's grounded in their own their own sort of content and therefore avoids that sort of hallucinations. Because that's usually people's biggest concerns is what happens if it starts to sort of fill in the blanks. Some things are outside of the organization, you can't control it. It's not validated, and that's when some challenges can sort of, you know, manifest as well. So it's making sure that we can be grounded within the organization's content is typically sort of paramount importance. Yeah. That's why really we want to have a solution that's going to enable us to use the kind of large language model. We've also our work on content because it means that we can control what is kind of shown to customers. The large language model will be kind of taking part of a conversation, but our own content will be kind of the technical information that is given. And this is very important. Not only we can control what's served to our customers, but we can ensure that it's of quality and it's updated on a regular basis. Yeah. Absolutely. Well, diving into some key takeaways before we wrap with a look to the future. And Remi, with all the work that you've done with AI so far, what would you say were your biggest learnings? What aspects of the transformation would you have done differently? Or on the flip side to that, what aspects worked particularly well? I think we've all these learnings and see how good it is and how can the technology is progressing. I think I think maybe we should have started to implement it early, because because we would have had kind of much more experience, but but it's fairly easy to, to learn. Nowadays, I mean, you've got plenty of kind of information on GenAI. Yeah. I think it's just you need to kind of stay in touch of what what are the best practices. I think if, yeah, if if we are to do something differently, probably do it a bit earlier, to be able to afford. But but, I mean, we we've been using AI at so forth for quite a while. I mean Yeah. It's it's not as kind of a a system to kind of return search, but we've used it for machine learning, especially, which is kind of the the ancestor of the grandparent of Yeah. Of JNI really, to to kind of just do kind of automatic rules for detections for kind of, threat in the threat landscape. So this is not something new. We we've been using it, but kind of for kind of customer service. Yeah. That's kind of something we should have done a bit earlier. Yeah. Jeff, any advice for brands looking to implement? Yeah. I was really just gonna add on to that. I think as Remy said, I think, you know, once we saw the emergence of chat DBT and generative in the marketplace, which is probably what eighteen months ago, it's not that long ago, really to sort of change the landscape. It drove an awful lot of excitement, but it also drove a lot of nervousness. And I think that's what we do see is some sometimes people have been maybe a little overcautious, you know, where they could maybe adopt it internally, as you say, test it out internally, sees all the power and the value that can drive, and then I'll see you go from there. But I think, yeah, it's it's driving a lot of excitement, but also some caution. There are some sort of horror stories out there. Typically, the the ones you see and you read in the marketplace are where the the the content is not grounded because, again, it's making sure that people only see content they have the access rights to see. So making sure you have that rigor and, that process in place. But, yeah, as you say, just just maybe sort of do some testing, make sure you can sort of get comfortable and confident with it, and then start to roll it out, accordingly from there. Yeah. I think you need to find the right balance between the kind of speed of execution and just launching Gen AI because because it's fashionable and everybody's got it. We've kind of making sure you've got the right data. Yeah. We've heard of oral stories and things that kind of you can't break systems if you don't give the right information. So I think you need to be very cautious, not too much, but you need to still be to be sure that your content is safe. Yeah. I don't think we we certainly sort of, have a sort of a phrase internally. It's almost better to not provide a right answer than it is to provide a wrong answer because the actual impact to the organization, the brand and everything else is is very, you know, disproportionate in that respect. So, yeah, it's you know, the impact of giving a wrong answer is is pretty impactful in the wrong way. Yep. So, yeah, it's just making sure you got the right controls and balances in place to make sure that you're giving the right information. Maybe one thing is that when we use Gen AI and when we're going to use Gen AI, especially with Korea, we always have a disclaimer and say that Mhmm. To explicitly say that the answer has been provided by Gen AI. So Yeah. So one of the great thing is that we can also do citations. So we the AI will cite the resources. So people can always verify that the content is correct. And I think it's important. You need to know where your content is coming from. Yeah. Certainly. So you for kind of a user perspective, you just reassure them about the content given by January is is correct. Yeah. It's just that transparency element. Yeah. Yeah. Right. Bring this home then, gents. You just touched my there. AI is developing at brilliant speed and really disrupting the way that brands are communicating and engaging with their customers. But, Jeff, come to you first. What are some of the innovations that excite you most? Yeah. I think as you say, I think the level of innovation now on sort of generative side is is pretty sort of fast paced. I think most people got a lot of weight and sort of focus on that to drive things forward. And of course, there's a lot of sort of benefits that can be delivered by that. But I think the things that we see probably the next stage at the moment, I suppose you've got that generative answer being compiled, is then going to the next stages of being more conversational. So again, if you've got a bit of information that's provided, that might draft another question in your mind is then how can I use that as context to refine that question as well? So, yeah, probably being more conversation is probably where we see the next part of that, moving forward. Yeah. Remi final words? Yeah. I mean, it's everything is really exciting. We don't know what's going to happen in six months. It goes so fast really. And people develop so different solutions. But I think for me, what I want to have a look at and especially coming from kind of a localization background, I want to make sure that this works in all the languages. We mainly concentrate on English at the moment. I know that COVID has got other solutions in French, German, Italian. Mhmm. But but I want to see see kind of more more languages coming into into the picture. And one of the things that potentially kind of having a way to kind of automatically translate a question, find the answer, let's say, in English, and then return it in kind of localized. Which means that even if you don't have some particular content available in one language, you will surface it anyway. And we this could change kind of really the way we create content moving forward. Gentlemen, thank you. Thank you for giving us a whirlwind tour through your partnership in the world of AI today. Thank you for coming on. Thank you for inviting us. It's been a pleasure. Thank you. So that brings to a close this episode of in conversation with CCW Europe. But please do stick around on the CCW Europe website and social media channels where you can find so much great content related to customer management and the customer experience. Thank you once again to Jeff and Remi for joining me. Thank you so much for tuning in. Until next time. Goodbye.