It’s hard to ignore the buzz around generative artificial intelligence (generative AI for short). Every few months, another viral tool or large language model (LLM) seems to capture the world’s imagination. Yet for many enterprises, the initial excitement of rolling out AI-powered chatbots and workflows has often clashed with harsh realities: hallucinations, data security pitfalls, and the endless complexity of real-world use cases.
In our work with clients across industries, we here at Coveo have seen the emotional rollercoaster firsthand — leaders excited to discover “the next big thing,” only to run into long lists of technical and governance roadblocks.
The good news is, none of this means you should give up on generative AI capabilities. We sat down with Eric Immerman, Practice Director for Search and Content, and 2024 Coveo MVP Zachary Fisher, Senior Solutions Architect, of Perficient, Inc, a long-time Coveo collaborator that has worked to implement multiple generative AI deployments, to examine how enterprises have navigated generative AI initiatives this past year — from the early hype bubble to the sobering reality checks.
We’ll then outline key aspects of a successful generative AI strategy that will help your enterprise adopt secure, scalable AI solutions that drive CX and EX success.
Top 2024 Generative AI Trends
Hype Bubble Pops: From “Off-the-Shelf Magic” to Reality
Not too long ago, generative AI was on everyone’s lips — what one might call a “kitchen table conversation.” Enterprises, swayed by the remarkable potential, raced to plug LLMs into every business process they could think of. The assumption? These models could “think like a person” right out of the box.
According to Fisher, the hype was both exciting and dangerous:
“There was a lot of very, very high level of expectation that intelligent artificial intelligence … is here, and we can just kind of start using off-the-shelf AI models to think like a person. … Over the next couple of years, it became clear … there’s a lot [they] can do, but they’re not foolproof.”
Enterprises quickly encountered “hallucinations,” where the AI confidently generated wrong answers. They discovered that controlling the AI’s outputs wasn’t as simple as flipping a switch. And in some cases, not a readily available capability. As Fisher explains, many customers “dialed back when they realized they had a little bit less control … without either building a model themselves or having better ways of controlling what those outputs would be.”
Despite these sobering lessons, generative AI capabilities are far from a lost cause. Organizations are maturing in their approach — seeking more scalable, governable ways of leveraging AI. Fisher highlights that an offering like Coveo can bring structure, security, and reliability to AI use cases.
Relevant reading: Chunking Information: Best Practices for Generative Experiences
Stuck in Pilot Mode: 2024 as the “Year of the POC”
Alongside the hype vs. reality tension, many enterprises struggled to fully move from pilot to production. As Immermann noted, 2023 and 2024 have been defined by a flurry of proofs of concept (POCs):
“You could very clearly build something that looked really slick and fast … for a demo … 90% of the way there for 10% of the effort.”
The challenge? That final 10% of work — compliance checks, security vetting, and handling edge cases — often sucks up 90% of the overall effort. Immermann shared some infamous news headlines:
“Everybody’s heard the horror stories of selling a car for a dollar or … telling someone to eat rocks for their digestive health.”
These lurid examples underscore the risk of letting a generative AI pilot loose before it’s fully tested. Enterprises soon learn that real-world deployment requires a robust infrastructure to prevent embarrassing or even harmful mistakes. From passing rigorous security approvals to building user trust, delivering a production-ready AI solution is a marathon, not a sprint.
Rise of Contextualization
By far, the most prominent lesson from the past year is the critical importance of context. While a human can tailor an answer based on who is asking, for what purpose, and in which environment, a large language model lacks that innate contextual sense.
Immermann explains:
“We as humans do a really good job … who are we talking to, what their motivations may be, [and] what this person should know about. LLMs … don’t have a lot of that contextualization.”
Without the right guardrails, an AI system might reveal confidential salary data, or offer inaccurate instructions (e.g., resetting passwords) across dozens of unrelated products. This problem, says Immermann, is not just about personalization; it’s also about security. An AI model needs clarity on who is asking and which content sets are allowed for that user.
When done right, contextualization is a game-changer. It keeps the AI from making disastrous mistakes, enforces security boundaries, and provides experiences that are tailored for an individual’s role or location. GenAI “for the sake of GenAI,” as Immermann puts it, isn’t the end goal; it’s about using AI to deliver meaningful, trustworthy, and relevant experiences — both to customers and employees.
Relevant reading: Evolution of Generative AI: How To Keep Up In A Rapidly Changing World
Key Considerations for Building a Successful Generative AI Strategy
Successfully adopting a generative AI model involves more than “turning on” an AI capability. True success hinges on your data quality, technology stack, user governance, and experience design.
Below are four key considerations that will help you outline a plan for elevating both customer and employee experiences with your generative AI initiative, while also driving real business outcomes.
Evaluate Content Health
Garbage in, garbage out — this old saying rings doubly true for generative AI. If your underlying knowledge base is poorly maintained, incomplete, or outdated, you’re effectively training your AI to produce mediocre or outright incorrect answers.
However, you don’t need to “boil the ocean.” A modern search platform that offers robust usage analytics can help you identify which documents or topics users engage with most — pinpointing the high-value content that your AI should lean on. By focusing on your most critical information, you can build a more efficient AI strategy.
Key action points include auditing frequently used content, updating or retiring outdated articles, and ensuring that access controls and classification are accurate so AI doesn’t mix sensitive and public data.
Relevant reading: 6 Data Cleaning Best Practices for Enterprise AI Success
The relationship between content and generative AI inspired the creation of Coveo’s Knowledge Hub, now in open beta for Relevance Generative Answering users. Designed to simplify the auditing process, this interface provides the transparency and tools you need to refine your content strategies, address gaps, and ensure reliable, accurate AI outputs in the generative AI world.
Start with a Strong Foundation
No single technology can solve all generative AI challenges. For instance, vector search (where queries and documents are converted into numerical embeddings) can excel at recall — finding thematically similar content — even if the user’s keywords don’t match exactly. Yet vector search alone can struggle with precise retrieval for nuanced questions.
Keyword-based search still matters for precision. And behavioral search provides feedback loops — if 1,000 users search for a phrase and all click the same document, the system learns that this result is likely the best match.
By stacking these technologies into a unified search platform, you gain the flexibility to handle a wide variety of user queries. Plus, future-thinking search vendors are incorporating retrieval augmented generation, or RAG, which uses a secure, tuned search backend to pull the correct data for AI to synthesize information from multiple documents.
Coveo offers a robust platform that incorporates these methods — keyword, semantic, behavioral — and techniques like RAG to provide one cohesive solution.
Relevant viewing: The Best Retrieval Method for Your RAG & LLM Applications
Change Management
Securing and configuring the technology is only half the story. Your internal teams, from leadership to frontline staff, must understand and trust the AI experience. Change management involves promoting A/B testing, controlling the scope of who sees AI-driven features, and iteratively improving the system’s accuracy based on feedback.
Tools like Coveo allow you to segment tests without major code changes, letting you learn what works — whether it’s surfacing the generative summary at the top of the page or limiting it until a certain confidence threshold is reached.
Designing the Experience
Microsoft’s infamous Clippy had good intentions but a poor sense of timing. It often popped up uninvited and overshadowed the core functionality of the application.
In designing a generative AI experience, remember to serve the user, not distract them. A good approach is to show the AI answer only when there’s strong confidence it helps, give an option to disable the feature, and keep the interface clean. Done well, AI can feel like an invisible ally that quietly saves your users time rather than a loud pop-up demanding attention.
How Perficient and Coveo Are Helping Enterprises Differentiate
Putting these strategies into action requires not only robust technology but also expert guidance. Perficient and Coveo have partnered to provide exactly this kind of support — combining deep enterprise consulting with modern AI search tools.
How SAP Concur Reduced Case Volumes by 31% with GenAI
SAP, a global leader in enterprise software, faced a critical challenge in simplifying their self-service support process, which was burdened by a vast, aging knowledge base and complex search experiences. Customers struggled to find relevant answers quickly, leading to frustration and increased case submissions.
By adopting Coveo’s Relevance Generative Answering, SAP Concur leveraged AI to streamline knowledge retrieval, resulting in a 30% reduction in case submissions and a 91% decrease in customer search efforts.
“We’ve dramatically improved customers’ ability to find what they’re looking for,” said Michelle Lewis-Miller, VP, Head of Strategy and Transformation at SAP. This transformation not only enhanced customer satisfaction but also significantly reduced operational costs.
Through AI-driven innovation, SAP Concur redefined their customer support approach, achieving measurable results and long-term efficiency. The integration of AI allowed them to free up resources, avoid extensive hiring, and reinvest in strategic growth initiatives.
Reflecting on the process, Michelle remarked, “No two days are the same,” emphasizing the value of experimentation and agility in deploying cutting-edge technology. SAP Concur’s journey underscores the transformative power of AI in addressing enterprise challenges and delivering superior customer experiences.
Relevant reading: Driving Transformation with AI: How SAP is Leveraging GenAI for Support Excellence
Bringing It All Together
Generative AI isn’t a magic wand, but with careful planning, the right technology stack, and a strong governance framework, it can become a powerful enabler of innovation — transforming both customer and employee experiences for the better.
Evaluating content health, starting with a strong foundation that combines multiple search technologies, embracing a structured change management approach, and designing the experience so that AI is genuinely helpful rather than disruptive are all key pieces of a successful GenAI strategy.
When these elements come together, you can deliver AI-driven CX and EX that feels polished, secure, and genuinely valuable.
If you’d like to continue your journey, contact Coveo to discuss success strategies for your business. Our experts can help you design, build, and refine a GenAI experience that meets the highest standards of security, usability, and ROI.
Dig Deeper
Or listen to the full discussion, featuring Patricia Petit Liang, Eric Immermann, and Zach Fisher for deeper technical insights and real-world success stories.