Hello. Welcome, and thank you for joining us today for our first AI master class. My name is Juanita Olguin. I am Senior Director of Product Marketing here at Coveo. And today, I'm so excited to be with our special guest, Zach Kass, who is an AI Futurist and Former Go To Market Lead at OpenAI. I can confirm Zach was one of our most popular speakers at our Relevance 360 event just a few weeks ago. So we're really excited to hear a bit more from Zach to dive a little bit deeper on what he's hearing, not only from enterprise business leaders, but also board members who are actively having these AI and generative AI discussions. We do wanna make sure we get to as many questions as possible. So, Zach, why don't we just go go ahead and jump right in? Sounds good. Thanks so much, Juanita. Okay. We're gonna talk about applied AI for thinkers today. And, I'll start by saying that there is so much happening on the science side right now that it is it it can and often does feel impossible to know where to pin the tail on this donkey. And my position has been and continues to be that the sooner you've adopted, the better. But assume that the market will change just as fast as you make decisions. And so as much as as much as I can today, I will help you not with tactical next steps, but with, thoughtful strategy so that you can start to think about ways to bring this technology in your business without feeling like everything you do is gonna be obsolete tomorrow. So first, let's just let's just start by agenda. I think we need to talk about how we got here because it's the best way that I will help you understand where we're going and how these things should be applied to your business. And so first, and I think, crucially, we should talk about the four things that led us to this moment in this acceleration of AI. And the first and and the most important is the invention of the modern transformer. In twenty seventeen, eight Google researchers, now known as The Transformer Eight, wrote a paper which is called Attention Is All You Need. And I talk about this a lot because I think it's actually lost on people what got us to this moment. More and more people understand this now, and that's great because it will help us better understand where we're going next. The debate you're gonna start to hear more and more is, is the transformer architecture the architecture of the future? That has actually started to creep up in the zeitgeist, which is, does the transformer get us to AGI, artificial general intelligence, or is it a new architecture that maybe is more cost efficient, less GPU intensive, etcetera? The second thing that happened was an explosion in the amount of available compute. So between NVIDIA's breakthroughs on the GPU, but also this zero interest rate environment, we had companies like OpenAI able to raise what was previously considered sovereign nation amounts of money and put it all towards compute power. And we saw this sort of happen across the board where, what we used to consider would require an Oppenheimer style project, now actually is becoming much more tenable for private companies because of these factors. The third is this explosion in the amount of available compute. Between twenty ten twenty ten and twenty twenty four, the Internet grew four orders of magnitude. That's a lot. It also got a lot cleaner. So projects like Common Crawl, which organized the Internet's information and make it accessible to researchers, became really rich treasure troves. Now, if you combine these three factors with a crucial element, which is publishing, and then you see companies like this share their best practices, what you have is this perfect storm of events that leads us to these outcomes. On the left is large language model performance measured by precision, which is just a science, measurement, and LLM size measured by billions of parameters. And between twenty fifteen and twenty twenty one, these models got a lot better by by the measure of precision, but really by any measure, certainly our eyes test. They also got a lot bigger. So they went from, you know, a couple hundred a couple hundred million parameters to a billion a hundred billion parameters pretty fast. Well, then they did it again. The super linear curve persisted. In twenty twenty one to twenty twenty three, precision grew again almost, by an order of magnitude, a little less. And LLM size also jumped from, from a hundred billion parameters to almost one well, now we know over one trillion parameters. And this is incredible. What's particularly fascinating about all this, right now, what we are observing is an LN performance is not asymptoting anytime soon. This certainly seems like a world that we are going to continue to observe as a hockey stick. But LLM size may. And the reason we think this is because of this. And this is the slide that gets me most excited as I talk about the application of this technology. And this is the thing that you all should be paying attention to, which is that as we see the new frontier models come out, we also see the older models get much smaller and less expensive. GPT three was ten dollars per million tokens when it arrived, Today, it's less than ten cents per million tokens. GPT three point five, twenty dollars, a little less. Today, it's less than thirty five cents per million tokens. More was sixty dollars. Today, it's less than thirty two. This is an incredible trend. And if there's one thing that you take away from this, it should be that the models are getting a lot better, the frontier models are getting a lot better, and the legacy models are getting a lot less expensive than this when we when we when we see these trends happening in an economic environment where a critical resource declines in price precipitously, it sort of marks this new era. This is true for things like water, obviously electricity. And when you talk about AI as sort of this critical resource, everyone in the world wants to see it decline in price to to near free. The thing I think needs that's gonna need to happen, and the reason I bring this up with all of you, those of you who are building systems now, is I think that people and companies are going to need to be to distinguish between technological thresholds and societal thresholds. A technological threshold is "what can machines do?" That's the simply the question that technological threshold asks, "what can a machine do?" And the societal threshold is, "what do we want a machine to do?" And this is really interesting and important because it's gonna be something that you have to ask yourself and your customers a lot. In particular because the future is about to unfold much faster than the past. And here's how I think this is going to play out. I basically think that we are going to see three phases of adoption. And the first phase that we are in right now is what we call the AI enhanced application phase. And the AI enhanced application phase is a really nice phase for us to be in because it comports to the world that we know really well. If you wanna use AI, you download an app like Chat GPT, etcetera, on your phone. You explore the app and you close it or you open up Gmail and use use the autocomplete features. AI sort of exists in the apps that you know, and you don't have to do anything different. This is good because it doesn't actually require societal update. Phase two is autonomous AI agents, and this will require some update because this is going to basically require that we feel comfortable surrendering the work that we do inside of the apps or on the Internet to AI. Autonomous agents are coming. Make no mistake. They are proven technically possible at this point. It's just a matter of time. And the easy way to wonder when we will meet the societal thresholds for autonomous agents is actually to consider something like autonomous vehicles. Fifty thousand people die on the US roadways every year, and yet autonomous vehicles pulls at, like, fifteen percent approval. Why is this? Well, there I think there are two reasons. The first is we love cars. Americans just love cars. But the second is actually slightly more nuanced, which is society has exceptional threshold, exceptional tolerance for human failure. We don't mind that much when humans do things that are wrong, which is why we're so willing to let cars crash all the time on our on our roadways. But we have no tolerance for mechanical failure. And this is a good thing on large part. It's why planes don't fall in the sky. It's why buildings don't fall over. It's why, you know, if your Internet goes down for a second, you're on the phone with your provider. What's going on? I need I need consistent performance. So it's a good thing. The problem is we're gonna need to quickly understand what are we willing to start to surrender to machines because autonomous agents are going to let us not actually use our phones the way we use them today. For example, if I I'm in Portland, Maine right now. Now. I want to fly home tomorrow. And there's a lot of work that needs to happen in order to do that. I need to go into United and go into Hotels.com and rearrange an Uber and all sorts of things. Autonomous agents are just gonna let me say to my phone, hey, I'd like to do this thing. And the the rest will happen inside of the apps that that that already exist on my device. Now, I think this is going to happen really quickly because I think the utility is going to outweigh the uncomfortable cost, but there's going to be a threshold or place at which we say, "you know what? We don't want agents to do that". Which leads me to the third phase, which is going to be the most interesting to observe and witness over the next probably ten, fifteen years, which is the AI powered operating system. And if you really want a future plan, I mean, really want a future plan, you could think about this. Although, I would caution this seems pretty far out right now. I was having a debate with a with a leader from a former leader from Apple the other day, and we were we were debating, what is the last iPhone we will use? And when, you know, there will be a final iPhone. And the over under seems to be iPhone twenty three right now. That seems to be a healthy over under, which is to say iPhone will be made for the next eight years. I'll take the under on that. I think there's a very high likelihood that we start to see the devices move from physical screens with the CPU and the screen is embedded into a more bifurcated world where the screen is more of a wearable and the CPU maybe fits in a small credit card style in our pocket. And the reason that I talk about this is because the AI powered operating system is pretty obviously that future where the it's where the AI exists maybe on the CPU and powers everything like the electrical grid or the Internet, which leads me to, I think, my favorite sort of outlook on AI, which is that I think it becomes a utility. And if you really wanna plan for AI, think about the ways in which not how you build these enhanced applications necessarily, but how you start to actually embed it into all of the functions that you're doing. And in the same way that today, no one would ever say, hey, I have an, I have a refrigerator that's electrical or I have a computer that connects to the Internet. These are sort of staples. I think in the future, most of the things that we build will actually be built on an AI powered grid. There are a bunch of products in the market today. And the reason that I always showed I don't always show this slide. The reason that I usually show this slide is to say that the consumer market is basically exploding. There are thousands now of products and and "a16z" recently published their one hundred hot, AI powered consumer products. Of course, these are in the top ten. What what is particularly interesting about this is that this market still has very low adoption actually among, for example, the traditional Google or Internet user market. A lot of people still have not used these products. And I remind everyone out there that there is still a lot of we are still very, very early, and there's incredible amounts of room in the market to build really exceptional products. One of the things that Chat GPT taught me, and it taught me a couple of things, but the most important thing it taught me was that the application layer matters. Chat GPT was built six months after GPT three point five was released in OpenAI's API. It was built on GPT three point five. There was no major research breakthrough. The breakthrough was in the application. And this is such an important reminder to builders and consumers alike, which is that the application layer matters. These companies are increasingly not differentiated necessarily on the research. The research may actually commoditize. What you get to do is differentiate on the actual buyer experience, the the user experience. I got to do a study recently with a with a major consulting firm. The average improvement in profit margins by companies adopting AI. So there has been so much talk about how this might be a false dawn. And first of all, I'm here to say as a small business owner myself, it is definitely not a false dawn. I get to do things today that I would not have imagined doing previously, because of AI. But simply, we already are observing incredible gains in the actual underlying performances of businesses who are adopting this technology. And this we can observe in a really simple stat, which is the gain in the S&P five hundred market cap since the release of GBT-4 is six trillion dollars. That's incredible. Five hundred billion of that obviously goes towards NVIDIA. That's super impressive. But there is a ton of money left for the rest of the S&P five hundred, and I looked into it. These are fundamental improvements fundamental improvements to these companies. And so what we're seeing, it's a marker. It's just a proxy. Not so much that the S&P five hundred is going to rip roar and not everyone else will suffer. It's that the early adopters of this technology are already gaining massively in their revenue improvements and their and their profit improvements. And when but we we are going to see incredible efficiency gains going forward by companies who adopt this technology either through the prosumer apps like Chat GPT, where an entire company gets to use it, or through the APIs themselves. And let's talk about some cases of companies that are building with the APIs and interesting things they're doing. The first is Klarna. Klarna has done some incredible things with the with AI APIs and they've published them. And it kind of went under the radar, but it's been pretty fascinating to see what Klarna has been able to do. And there are two things that they've done that are really important. The first is they've automated all of their customer support using AI. And this is impressive considering that Klarna actually has somewhat complex customer support. "Buy now, pay later" involves a fair number of financial, conversations and requires a a quite a bit of understanding of your product suite. That alone is interesting and massively improves the the the performance of this of the stock and of the interest in the company. Other than Klarna's data is they figured out the way that they should be testing their customers' ability to to return yields on their loans wasn't exclusively to check their credit score. It was actually to start to explore the purchases that they were making and whether or not they were what they consider luxurious or frivolous. What's really cool is that Klarna is using AI now to to build more robust credit score style information around their users that is dramatically improving the yield. So we're not just seeing a decrease in traditional cost structures. We're actually seeing a net new way to run and evaluate a business. Things like this are going to happen all over and just build much, much more interesting companies. Second is Coca Cola. So Coca Cola is now doing a number of really interesting things. One, and and the one that they sort of made themselves famous for was they started automating a lot of their marketing by taking a lot of the work that agencies were doing and putting it in house, giving their designers access to products like DALL-E and Midjourney to do a lot of the work that their agencies were previously doing. And if you know anything about Coca Cola, you know it's at the core a marketing company. They spend about four billion dollars a year on it, and about half of that goes outside into the agency world. And so if you can start to, one, empower your own team with the tools that your agencies normally reserved, you obviously will save a lot of money. But two, something else interesting happens, which is actually start to take more control over your assets. And so they've talked about that. The The other thing Coca Cola is doing is they're starting to automate the work in the field. So it's one of these consumer package goods companies that's doing a whole lot of work to improve the relationship between the salesperson and their buyer. And what people forget often is so much of this is a very human experience. Someone delivers a a rack of of product to a merchant, that merchant investigates it an hour later and realizes that a bunch of bottles are broken. So they just call their rep back, and then their rep has to call the warehouse, and the warehouse has to figure out how to get new product to them. And it's very arduous. And by the way, these salespeople aren't traditionally good at this exact task. So automating that work that exists between the, salesperson of Coca Cola and their, store owner is actually incredibly valuable to both parties. And so we're starting to see processes like this start to automate using using chatbots that the store owner can actually just get immediate insight into into the going through their their sales rep. Morgan Stanley has now sort of made their business case famous, which is really cool. They have, Andy Saperstein came to OpenAI in in, twenty twenty one, or late late twenty twenty. And with a very specific vision, he said, I want my, wealth managers to love their jobs more and to be able to serve more customers. That was his vision. And what he was imagining was a world where Morgan Stanley wealth managers didn't do the busy work that actually they weren't inclined to and that didn't they weren't very good at, didn't serve them, and instead focused on the work that they that made them successful as wealth managers, the relationship management. And what they built was a product that that allowed Morgan Stanley wealth managers to basically automate most of the note taking and follow-up and next step work that, you know, didn't accrue value to them and felt more administrative than anything else and spend most of their time on the high value work. In the end, Morgan Stanley wealth managers are now able to run books forty percent bigger than they used to. And this is really cool because it means a couple of things. One, they can actually start to charge less and provide financial goods and services to bigger swaths in the market. That's a very good thing. The second thing that can happen is they can pay their people more or even collect more margins. The best part about this is you actually are just gonna start to see worlds where the cost of goods and services decline and every constituent can win. And then this last example is is sort of one of these incredible ones if you work in a big company. Because Salesforce, Marc Benioff basically said to Clara Shih, wait. You need to help us reinvent Salesforce. And so Clara has gone out and built Salesforce AI and sort of reimagined what Salesforce would look like if it was, a a startup again, renewed by this sense of magic in AI. And they have built an incredible suite of new products powered by their traditional Salesforce ecosystem that feel really magical. And it I bring this case up not because I love Salesforce, I thankfully don't have use it anymore at this stage of my life. But it is a fascinating moment because it actually Salesforce, Mark, and Clara can finally deliver the magic that they have been promising the market for twenty years. That Salesforce would make salespeople so much more happy and so much more successful. We're finally arriving at a moment where that can be true, where all of these interesting products that they've sort of been taunting, you know, or flaunting to the market can actually now become really, really magical because of AI. This is what I tell every company. This is this is my single message as as sort of a download from these examples, which is that your you now have three major competitive advantage. The first is your data. And I don't think your data's competitive advantage insofar as it will help you fine tune models. It may. You may need a fine tune model. But I think your data is far more interesting as a way to build hyper personalized experiences. The future is very much vector search and retrieval powered by RAG, wherein you will do a whole lot of work to fine tune a precise experience and then call a database with all of your data to build these really hyper personalized experiences. The second is distribution. Distribution will remain one of the hardest things that any company does. And the more things change, the more they stay the same. Sales and marketing are just critical. You can build the world's best product. Doesn't matter if no one buys it. Focus, focus, focus on figuring out how you can streamline your distribution, ideally powered by AI. The third is brand trust. In a world where everyone starts to provide exceptional experiences, what will set some brands apart from others? I am basically certain is going to be brand trust. And this is going to be true at an individual level where all of our skills and commodities will eventually all of our skills and knowledge will eventually commoditize. And the likelihood is we will be differentiated on a personal trust basis. What does our brand say to other people? And at a corporate level, I think offers that companies provide will start to look equally amazing because of AI. And why will one person pick one, particular vendor over another? I think it's going to be because of of the quality of the brand. I'm so glad that we were able to spend this time together. I, won't keep us much longer, but and I'm really excited to open up the floor to to you all for Q&A. Thank you, Zach. That was incredible. Lots of great insights. We do have few questions coming in, so I'll get to those in just a moment. Do wanna say really love your three phases of thinking. I also love you proposing that machines can do a lot. So what do we want them to do? With that, let me start getting to some of these questions. The first has to do with, the ethics of AI. And the question is, are the leaders that you are meeting with asking about the balance between ethics and innovation? Yeah. I think a lot of people ask about ethics. I I'm pretty, I wanna separate this idea of ethics from, responsibility. There's a there are a lot of laws in this world, especially in in the western world, in United States, that protect consumers and businesses from each other, and we should enforce those laws. The thing that we should be really concerned about right now with respect to responsible adoption of AI and responsible manufacturing of AI are three things. One is alignment. Is the model concerned with the human experience? Is it imbued with human values? That's what the alignment problem asks. And that's the thing that should be policied at an international level by an international body, in my opinion. The second is explainability. Can a model explain how it made the decision? Models, we should assume, will have bias. Everything does. But what separates AI from humans is that it's exceptionally capable or should be at explaining itself. How does it arrive at a decision so we can at least account for it? Policies will be really critical. The third is bad actors using AI. We should make it untenable, terrifying for bad actors to use AI in bad ways. And from an ethical standpoint, these are the things that I focus on. Because most everything else in this world is protected in a in an appropriate way already by law. There are things that companies should and shouldn't do, etcetera, that already have sort of been adjudicated for, you know, hundreds of years since the post industrial society. And I don't know that companies need to sort of relitigate or reexplore these things. Amazing. Thank you for that. Moving over to your conversation around AI becoming a bit of a utility, we have a question here asking how do you think the UX of enterprise search might change, given the utility? And you also mentioned application layer is becoming important. Yeah. I mean, Coveo knows this well. The future of search is not the one that we're used to. I think Google has sort of exhausted all of our patients with I maybe I shouldn't say this. Well, I think it has. Google has exhausted our patience with with traditional keyword search. Don't no one wants it anymore. It's it's just not how the world works. We also don't wanna sell our personal information to the highest bidder anymore. These are these are sort of relics of a bygone sort of web one point five era. The question is how do we how you know, what replaces it? I think enterprise search is the first thing to sort of watershed to mark this new this new moment. And not very high likelihood is everything becomes natural language, that we will build vector databases that store all of our information, and we will retrieve that information through very natural language queries like, "can you show me all the contracts expiring tomorrow?" "Can you tell me how our performance was last month?" You know, "what does EBITDA look like if we if we took out taxes?" You know, interesting questions that normally require exceptional amounts of heavy lifting and and analysis. That, I think, will also then trickle into how the consumer world changes. And the reason that consumer search is gonna take a while to change is because Google has a lot of vest and interest in making sure that it doesn't, or at least it doesn't quickly. And so I think enterprise search, and Coveo get to sort of signal this new era of natural language living. Absolutely. Along those same lines, Zach, there's a question here around how important it is to customize your GenAI implementation with your specific enterprise wide knowledge. Can you comment on how important that might be? And Yeah. So I I think I think we should separate I think we should separate, well, customizing an experience versus fine tuning a model. Increasingly, there are going to be reasons to fine tune models. The customization of an experience, I think, that the effort that should be put into custom building a custom experience should be around the actual application layer itself because that's probably where the differentiation is going to come. The big thing, the the major consideration that every business needs to ask themselves before they start building their own experience is should our vendors be doing this? And if you are designing a system or a solution and you find yourself wondering if a vendor should be doing it, you are almost certainly right. There is very little that most enterprises should actually build on their own because so much of their tech stack is actually gonna be solved by vendors. And a lot of this work is just gonna feel obsolete in three or six months anyways. So my strong recommendation to most companies is put pressure on your vendors to build future proof applications so that you aren't sort of left trying to piecemeal a bunch of systems together. And the things that you choose to build should fall under the Venn diagram of, is critical, cannot be solved by a vendor, and we are in a unique position to solve it. And if you can sort of build for that, then you or or is is solvable by AI, I suppose, is is is the easier way to put it. If you can run that Venn diagram, then you will figure out exactly what you should be designing. That makes a lot of sense. Wanted to ask, you know, you mentioned you you you meet with and you talk to a lot of enterprise leaders and even board members. Do you think or are you hearing them talk more LLMs and building, or are they also talking about, you know, partnering to build those best in class user experiences? Yeah. I think everyone is trying to figure out what should they build versus what should they buy. And my recommendation is constantly to buy first or try to buy first. Building has, building has just a lot of costs, a lot of hidden costs. And I think it's one of these fools errands where you can, you know, throw as much as you want at a problem. But if it's a if it's a general problem in the market, it'll probably get solved more effectively than you can solve it because someone's gonna focus exclusively on it. So, you know, most companies are are considering building a very limited set of things, and those things are very critical to their business, and they don't think it's things that their vendors can can support them with. Awesome. Thank you. Much has been talked about or said, obviously, with Gen AI and the technology side. Are you hearing about the talent side, upskilling workers, anything you can share there? Yeah. Well, I have my own opinions on it, and I'm hearing some things. So I'm I get to do a fair amount of research, and I, am an executive in residence at UVA, which is, you know, a a fancy title for someone who gets to teach a class and hang out on campus with their with the McIntyre undergrads. And so I get to see a little bit about how AI is changing their jobs and how it's changing their learning. And I have two observations. The first is that pinning again, this is a pin the tail in the donkey situation. Trying to identify the skills that someone will need to upskill to in the future is going to be a fool's errand. It's just impossible. What is possible is basically recognizing and admitting that the only thing we know is that adaptability is going to be critical. Teaching children how to teaching young people how to learn for the sake of learning is going to become more important than any skill that they actually acquire. And we, you know, con kind of already observed this, which is, like, a lot of lawyers who are graduating today can probably tell that they're not gonna practice law forever. And but that doesn't mean they shouldn't have studied law. Studying law is interesting insofar as it taught them how to become really capable and and smart on a subject, in the same way that someone who learned the cello at a master level, you know, may not have played the cello forever, and and quite honestly, maybe burnt out by it. But in fact, learn how to do something at an expert level and knows what that feels like. So increasingly, my message to students and and young people is study things rigorously, not because that thing is going to gain you some economic competitive advantage. It probably won't, insofar as a lot of our skills and knowledge will just commoditize. But it will teach you how to learn and ensure that you are sort of future proofed, robust to things that change. And by the same token, I tell employers, think about hiring for soft skills and people who learn how to learn. I don't think that the future is made up of, geniuses who struggle in teams. You know, Reed Hastings had that had that fun quote. Netflix had the no assholes policy, I think, in, like, two thousand seven or eight. And what was interesting about that quote is not that they had a no asshole policy. It's that he had to say it. It's that so many people were hiring brilliant jerks at the time. And I think that that era is just definitely over. We we we are going to observe a world where AI ends up doing a lot of the computationally intensive work, and what we are left with is things like teamwork. How how how courageous are you? How curious are you? How empathetic are you? How much you know, how well do you collaborate and communicate? And and so my strong encouragement to employers is, again, to hire for these soft skills and hire for people who who have learned how to learn. I think that's great. I love your perspective. We call those those soft skills or emotional intelligence as well, so totally agree with you on that one. We have time for just one more question, so I'll I'll throw that to you, Zach. The question really is you talked a bit about brand trust. You talked a little bit about, you know, LLMs being commoditized and, you know, AI becoming a utility. How can companies tell these vendors apart or how can they tell quality? You know, how can you see quality companies over those that are maybe just stitching together a solution? Any advice you can give them? Well, this is this is gonna be really hard. I think a lot of people are gonna buy snake oil. I think a lot of people already have and a lot of people will. I don't think that companies that sell snake oil are gonna sell it for very long. So this is this is sort of why this is why I talk about brand trust. I think that we are going to discover really quickly that, vaporware in this world doesn't work. People are gonna realize that there are there's a big gap between a vendor that actually is embedded, you know, or GPT four into their workflow versus an end vendor that, you know, is just still sort of dabbling with it. I think I think it will I think world will get out in the marketplace really quickly. I think people will discover that that information is becoming far more perfect and is shared more frequently. I think there's gonna be huge consolidation in the private markets, especially in technology between companies that adopt this really quickly and companies that are just saying they're adopting it. And so my advice to to every, you know, buyer out there is put, you know, put pressure in your procurement process. Actually, trial products if you can. Demos are all gonna look super flashy now. Everyone's gonna have a flashy demo. Make sure that you're trialing products. Make sure you're putting pressure on vendors to actually build these magical experiences because there's basically no longer an excuse, for why they can't. Makes sense. Thanks so much. Alright. We are at time. Really appreciate all of your insight, Zach. Thank you for spending this time with us today. For those of you whose questions we could not get to, don't worry. We'll be following up with you offline. We hope you enjoyed this time with us today. We have more sessions like this to come with AI experts like Zach, who you can learn from and hear from over time. Stay tuned for the recording. We'll get you full access. Zach, wanna thank you for the time again. Have a nice day. Juanita. Yes. You too. Bye.