Generative Search: Rethinking Financial Services in the AI Era

Every financial services executive has heard the mandate: modernize with AI. 

But between the boardroom directive and actual implementation lies a minefield of failed pilots, wasted budgets, and vendor promises that don’t scale. Your team has probably already tried a proof of concept—maybe two or three. Some worked in isolation. None transformed your customer experience.

The problem isn’t AI itself. It’s that organizations are building skyscrapers on quicksand.

We sat down in a recent webinar to discuss how financial services have had to scramble to deliver AI-powered experiences when ChatGPT raised customer expectations overnight. Karine Hamel, SVP of Finance at Coveo, observed:

The rush to implement generative AI and agentic systems has obscured a fundamental truth: without the right foundation, even the most sophisticated AI will fail. That foundation is search and retrieval—and most organizations are getting it wrong.

The following is a summary of our discussion; to listen to the full conversation, check out the on-demand video here.

The Challenges Financial Services Organizations Face

The investment is real: 80% of banks have increased their AI spending. Yet financial institutions are grappling with unique obstacles as they navigate this landscape. 

The most significant challenge? Decision fatigue. Every vendor claims to be an AI company, making it difficult to separate signal from noise. Many organizations have already invested in AI projects that haven’t yielded expected results, leading leadership teams to reassess priorities and question where to allocate limited budgets.

But beyond the market noise, financial services organizations face operational challenges that make AI implementation particularly complex:

Siloed Systems and Disconnected Content

Customer information is trapped across disparate systems—core banking platforms, wealth management applications, insurance databases, compliance repositories, CRM systems, knowledge bases, and document management systems. The customer journey spans multiple touchpoints: websites, authenticated portals, mobile apps, chatbots, ticketing systems, and call centers. Yet these systems rarely communicate with each other.

The result? Customers are forced to restart their journey at each touchpoint, repeating information they’ve already provided. If you silo your inputs, you’re siloing your outputs.

Relevant reading: How The Silo Problem Is Killing Your Customer Service Experience

The Self-Service Expectation Gap

Today’s customers expect ChatGPT-level experiences for financial education and support. They want immediate answers to complex questions:

• “How do I set up a 529 college savings plan?”
• “What’s the difference between a traditional and Roth IRA?”
• “What are the tax implications of early retirement account withdrawals?”
• “How do I update my beneficiary information?”

The content to answer these questions often exists—but it’s buried or disconnected, and generic search experiences can’t surface it effectively. This creates a trust problem. 

Financial services are rooted in trust, and when customers can’t find accurate information quickly, they question whether you truly understand their needs.

Why Search & Retrieval Is the Foundation for AI Success

It’s an understatement to say that ChatGPT has raised the bar for what modern digital experiences can and should look like. But rushing to implement generative AI without the right foundation is a recipe for failure.

Search and retrieval represent a strategic, low-risk entry point into the AI landscape. Your website—whether your main site, support portal, or customer portals—is your digital front door. More importantly, search and retrieval form the foundation upon which all other AI features must be built. If your basic search experience delivers poor or irrelevant results, stacking generative or agentic capabilities on top won’t solve the underlying problem.

Think of your search box as the voice of the customer. When customers search, they’re telling you in their own words what they need. This data is invaluable: it reveals how customer needs evolve over time, identifies content gaps, and surfaces opportunities you might not be aware of. For financial services organizations with complex data ecosystems, diverse personas, industry-specific terminology, and stringent compliance requirements, this visibility is essential.

Understanding Generative Search: The Evolution of Information Retrieval

Generative search represents the evolution of traditional search. While organizations might start with search improvement projects, generative capabilities are almost always part of the vendor selection criteria—because everyone recognizes this is where the market is heading.

But here’s what many organizations miss: It’s easy to generate an answer. It’s hard to NOT generate an answer. Any large language model can produce text that sounds authoritative. The challenge is generating an answer only when there’s high certainty—and refusing to generate when there isn’t. This requires grounding in your enterprise truth and sophisticated retrieval that identifies the RIGHT content for THAT specific user.

Relevant reading: AI Hallucinations: When No Answer Is the Best Answer

Effective generative search must understand multiple dimensions simultaneously:

• Who is the user? (What products do they have? What’s their location? What have they searched for previously? What are similar users doing?)
• Where are they in their journey? (What page? What section? Where have they been?)
• What do they have access to? (Account permissions, customer tier, geographic restrictions)

This contextual understanding is what separates effective generative search from simple chatbots that provide the same generic answer to everyone.

What to Look For in a Search & Retrieval Solution

When evaluating solutions, look beyond surface-level features. Here are the critical capabilities that separate enterprise-grade platforms from basic implementations:

Unified Indexing with Permission-Aware Retrieval

The solution must connect to and index all content types—CRM systems, knowledge bases, document repositories, core banking platforms—without requiring migration or duplication. But here’s the critical component that many organizations overlook: document-level permissions must be indexed simultaneously at scale.

This means users never see content they don’t have access to, and generated answers never use unauthorized content. Permission-aware unified search isn’t just a nice-to-have feature—it’s the foundation of secure, compliant AI deployment. 

This is where we’ve seen a lot of customers struggle. Coveo solves this by not just indexing content, but also indexing document-level permissions at scale. 

High-Quality Retrieval at Scale

The platform must excel at hybrid retrieval—combining keyword precision with semantic understanding. More importantly, it must rapidly narrow focus from millions of documents to the most relevant paragraphs. Going from 10 million documents to three highly relevant paragraphs quickly? That’s what matters.

Intelligent Answer Generation Controls

Enterprise-grade RAG (Retrieval Augmented Generation) keeps your LLM grounded in approved information. 

Critically, the system must be able to prevent answer generation when confidence is low—safeguarding customers from bad information and protecting your brand. If a vendor claims 100% response rates, that’s a red flag.

Analytics That Drive Continuous Improvement

Out-of-the-box analytics should reveal not just what customers find, but what they search for and don’t find. These content gaps provide clear direction for your content team and reveal product insights—what functionalities are customers looking for that you don’t yet offer?

Relevant reading: How Coveo’s Knowledge Hub Demystifies Generative AI Outputs

The bottom line: successful AI deployment requires control and visibility. Without both, you’re throwing darts blindfolded.

How Generative Search Benefits Financial Services

When implemented correctly, generative search transforms multiple aspects of your customer experience and operations:

Case Deflection Across Multiple Touchpoints

Smart organizations intercept customers at multiple points in their journey. At the website level, faster troubleshooting gets customers back to their goals. 

At the ticket submission form—where many customers go directly, bypassing self-service research—contextual help can improve case deflection.

For support agents, providing access to all enterprise content in the flow of work while showing the customer’s complete journey enables white-glove service at scale.

Maximizing Content Value

Financial institutions don’t need to create new content for every possible customer question. Traditional content management approaches require organizations to constantly produce new articles, FAQs, and guides to address every potential query—a process that’s both time-consuming and inefficient.

With Coveo’s Relevance Generative Answering, financial services organizations can maximize the value of their existing content by:

• Synthesizing dynamic answers from multiple content sources, reducing the need for redundant content creation while delivering personalized responses to complex financial questions
• Ensuring that older content remains useful by intelligently retrieving relevant information, even from archived knowledge base articles and historical documentation
• Automatically adapting content recommendations based on user behavior, customer profile, and journey stage to increase personalization and engagement
• Scaling content while respecting permissions with early binding security that accounts for document-level permissions at indexing time—critical for financial services compliance

Instead of constantly creating new content for every potential question, financial institutions can leverage AI to make existing content work harder. One comprehensive article about retirement planning can answer dozens of distinct customer questions. The answer might not exist in a single document but can be synthesized from multiple sources—maximizing your content ROI while maintaining security and compliance standards.

Product and Content Insights

Search analytics are a gold mine. They reveal what customers want, what’s causing frustration, and how to address it. Content gaps become a north star for your knowledge team. Even more valuable: queries for products or features you don’t currently offer provide direct input for your product roadmap.

How to Get Started: A Practical Framework

Success starts with clarity about your objectives and a methodical approach:

1. Define Your Business Objective – Start here, always. What specific business problem are you solving? Is this a true business objective or a “bell and whistle”? What ROI are you targeting?

2. Identify Your Use Case – Find the simplest path to high ROI quickly. Prove value with one clear use case, then expand.

3. Benchmark Your Current State – This cannot be emphasized enough. Without baseline metrics, you can’t prove value. BENCHMARK, BENCHMARK, BENCHMARK. Do not go live with customers unless you’ve established a current state.

4. Establish Security Requirements Early – Understand your organization’s security and governance committees upfront. Don’t get blocked after you’ve made progress.

5. Start Small with Flexibility – De-risk your project with a limited proof of concept. Demonstrate ROI before expanding.

Selling AI Internally

Getting buy-in from finance, procurement, IT, and security teams requires speaking their language. Focus on cost efficiencies and revenue improvement. Show what the organization loses by not implementing. Be realistic about resource constraints—not just budget, but the people who will implement and maintain the solution.

As Hamel advised: “Don’t let your admin team, your finance team stop you from trying to achieve excellent success. Make sure you’re ready to convince them about your project.” Demonstrate that you’ve de-risked the initiative with clear success metrics, established benchmarks, and a path to scale if successful.

Relevant reading: The Buyer’s Guide to Relevance Generative Answering

Build vs. Buy: Understanding What Fails

Some organizations attempt to build custom retrieval solutions with small teams, thinking they can create enterprise-grade, secure, grounded retrieval that scales. The common result? “FrankenSearch” deployments that work for one use case but can’t expand. Organizations waste millions of dollars and 6-12 months before realizing they need a different approach.

• AI projects typically fail when organizations:
• Implement AI for its own sake rather than solving real business problems
• Skip the foundational infrastructure in favor of advanced features
• Launch without administration capabilities or analytics
• Underestimate the complexity of enterprise-grade retrieval

Core differentiators should be built in-house. Infrastructure and plumbing? Consider buying from specialists who focus exclusively on solving these challenges. Factor in time to value, required expertise, ongoing maintenance, and your ability to keep pace with rapid market changes.

Relevant reading: Build vs Buy Search? Choosing the Best Path for Your Enterprise

Conclusion: Building the Right Foundation

Customer expectations have permanently shifted. Financial services organizations that establish the right search and retrieval foundation now will be positioned to lead as technology continues to evolve. Those that skip steps or chase disconnected POCs will struggle to scale and risk falling behind competitors who took the methodical approach.

The path forward is clear: Start with clear business objectives. Benchmark everything—you can’t prove value without baseline data. De-risk projects by starting small while planning for scale. Given the complexity and trust requirements inherent to financial services, organizations in this sector are uniquely positioned to benefit from AI-powered search and retrieval.

Most importantly, establish a foundation that enables you to adapt as new technologies emerge rather than constantly chasing the market. The organizations that get search and retrieval right today will be ready for whatever comes next—whether that’s more sophisticated generative answering, agentic capabilities, or innovations we haven’t yet imagined.

Watch the full conversation here:

Or check out our on-demand demo for financial services: