You read the Gartner Magic Quadrant for Insight Engines1 every year to wrap your brain around what’s best when it comes to enterprise search. You might have even started implementation. But what exactly is an insight engine, and what do you need to leverage one effectively?

We cut through the jargon and give you the skinny on the 12 capabilities insight engines must have for scalable enterprise deployments.

What an Insight Engine Does

An insight engine combines keyword search, data connectors, UI components, and machine learning to provide customers and employees alike with more relevant information and personalized experiences. Where a database tries to structure data to answer a few select questions, an insight engine uses structured and unstructured data to answer an almost infinite number of questions.

An insight engine is not just “search” but search, data, and intelligence. It’s efficient, intelligent search. The difference between a search engine and an insight engine is the difference between a wheelbarrow and a self-driving electric pickup truck.

And now that you know what an insight engine does, let’s talk about what the best insight engine should have.

What an Enterprise Insight Engine Must Have

#1: Connectors and text-preprocessing

It’s 10 p.m. — do you know where your data source is? Most organizations have data everywhere, in many different systems and many different formats. Loading and transforming data might not be the most attractive feature of an insight engine, but it’s a critical foundation. If you can’t get data from your systems, you can’t get the answers you need from your insight engine.

Coveo connects to hundreds of different types of systems: learn more>>

#2: One Unified Index

Think about the last time you spoke with a large healthcare or service provider. Frequently, they put you on hold while they look something up or transfer you to a different person. Often, their websites aren’t navigable or searchable because the information you need is in multiple places.

Better, more relevant answers require a unified index — that is, centralizing and unifying data in one index. Entropy is the nature of data. Data disperses and becomes less coherent. While most search is contextual, supporting cognitive search requires looking in multiple places, including answers from data in multiple formats.

See Humana's TransformationRelevance 360 Presentation: Digital is Redefining Your Competition

#3: Instant Results: Customers and Employees Can’t Wait!

When was the last time that you went to a website, used the search bar or really any interface component, saw it spin, and you just waited? Oh, you left? Customers don’t wait either. Moreover, while challenging to measure, the cost of employee wait time continually drains dollars and cents out of a company. In a modern business, answers should be both instant and correct.

An insight engine combines search results into a single index and uses machine learning to suggest queries to users as they type. It does not require going back to the source system to provide instant gratification.

#4: Facets and Filters

Facets and filters may not be as new or shiny as artificial intelligence, but any insight engine should allow searching, sorting, and filtering on categorical data called “facets.”

Filtering on a facet is how digital workers limit their search to a particular domain (i.e., Human Resources Documents), or a customer searches on a specific product category (i.e., Shoes) or customer support topic (i.e., Coveo Headless Tutorial). Most searches are pretty straightforward, but your most dedicated employees and customers likely dig deeper, and these tools make their life simpler by offering them relevant results.

#5: ‘Know what I mean?’ NLP and Intent Detection

For a person or an insight engine to answer a question, they need to know what the question is and what the person means by the query they type into a box. Natural language processing (NLP) goes beyond mere keyword matching and derives actual meaning from the question. NLP finds synonyms and matches phrases.

People do not always say what they mean or ask complete questions. According to Coveo customers, most searches are two words or less. A modern insight engine needs to not only understand the language or just match keywords but understand user intent based on user behavior.

#6: Automatic Relevance Tuning: ART of Relevance

Perfect results may not come from matching what users asked for with the data at hand. An insight engine is smart enough to notice when users aren’t getting the best search result. Tuning a query isn’t even a one time fix — the meaning of a search may change over time. For example, consider a general ecommerce retailer. In October 2019, the search query “mask” should return a different result than it did by March 2020 at the onset of the COVID-19 pandemic.

Modeling, interpreting, and tuning to user behavior must be automatic in the modern world. The days of poring over logs and getting user feedback to tune results are over. In most cases, it would take too many users looking at too much data on too massive of a scale to make this possible. An insight engine should see this behavior and modify itself.

Learn more about Automatic Relevance Tuning (ART)>>

#7: Rules

“No one knows how to spell HIPAA,” said Eric Immerman, practice director for Perficient, while working with Humana. While an insight engine might eventually notice this and tune itself appropriately, sometimes users are really creative with spelling.

Moreover, even the best AI algorithm requires a lot of big data before it learns. Sometimes you just need a simple rule to tell the insight engine, “when they say ‘HIPPA’ or ‘HIPPO,’ they mean ‘HIPAA.'” Rules have a lot of usages beyond spelling.

Sometimes you want a higher margin product or this year’s new employee benefits manual near the top even though something else had more clicks.

Learn more about Rules and Ranking Expressions>>

#8: Question Answering and Chatbots

In the early days of the net, every question was answered with a search engine result page (or SERP). These days, if you ask Google, Siri, or Alexa any number of questions, you get a direct answer.

If you type “What is the weather” or “Who was Milton Freedman,” you get a direct answer. Insight engines use a combination of NLP and other techniques to not just provide search results but actual answers.

Question answering technology is useful for everything from giving employees answers in an enterprise search application (like smart snippets) to handling support requests.

However, not all questions are just a question and an answer. These days chatbots can answer questions that have “branches” or need more context.

For instance, if you ask, “Where can I download a driver for my webcam?” it might ask your model number and which computer you have. If you have questions about what your health insurance covers, maybe it asks which policy you have or where you live. Chatbots guide a user to answer beyond what search can provide.

Learn more about chatbots>>

#9: UX Components and Tools

Nearly every search application has a search bar, query suggestions (sometimes erroneously referred to as type-ahead, which is different), and a slew of user interface and user experience traits. Redeveloping them for every application is absurdly wasteful, which is why an insight engine provides prewritten components that can be composed using JavaScript or TypeScript.

Learn more about the Coveo Javascript Search Framework>>

#10: Deployment Flexibility Across Clouds

The world is moving to the public cloud quickly, but not everything lives there yet, and some information requires more careful handling. An insight engine is only truly useful if it can handle data from both public and private clouds.

A cloud-based insight engine that adheres to robust enterprise security protocols, provides the flexibility to connect and index content both within and beyond the organization’s internal data sources. Needing to index and unify content that lives on an arcane internal file server, SharePoint, and Google Drive, is typical for most enterprises.

Learn more about how Coveo handles private cloud content>>

#11: Security

Most businesses have a team of infosec professionals trying to keep employee as well as customer data safe and systems secure. However, your security is only as good as your weakest link. An insight engine must preserve ownership and permission information. It must encrypt data in transit and at rest.

To truly maximize the value from an insight engine, you need the confidence that comes from knowing a third party has audited and assessed vendor practices and procedures as well as the software itself.

Learn more about security at Coveo>>

#12: Contextual personalization

Being responsive to user requests and giving them what they want, is the base level of search. Insight engines go much further: suggesting what other content will be helpful to them – despite not being asked. We call those “recommendations” — and anticipate what visitors will need next.

Understanding the context of what people are searching for, what they have already done, and what others like them have already found useful, provides the basis for personalizing relevant results and recommendations that meet their immediate needs, and guide them to what they are going to need next — all delivered automatically using specific machine learning models for each purpose.

You also need analytics that provide actionable insight into what your users are searching for. This tells you what they’re finding, as well as what’s working and what’s not — which queries are resulting in content gaps? Which turns them into content opportunities for you.

Beyond the Best Insight Engine

However, those are really the minimum for a modern insight engine. You can get somewhere on a bus, but if you can take a high-speed electric train and get there faster and cheaper, why would you go the more challenging route?

Modern software is necessarily complex. There are multiple options, topologies, connection points, and more. However, the configuration and administration should not be complex. Ease of configuration is not something your technology users will notice. Simple configuration might not even be something employees see daily, but it is more than added “chrome.”

Ease of configuration and administration allows you to better use, tune, and get better results from your insight engine. It also saves time and money. If DevOps professionals have to jump hoops, there is a cost.

However, simple configuration and administration is also a security feature. While bugs can happen in any software, a quick perusal of the news reveals that many if not most major security breaches have had something to do with poor configuration and how hard it is to administer the software.

Simple configuration is a great feature, and all of these insight engine tools are helpful, but life, search, and artificial intelligence are not standing still. Ideally, the team developing your insight engine is always thinking about making you and your customers’ life easier. Ideally, there is ever more R&D going into AI, chatbots, and implementation practices.

Learn more about the latest developments from Coveo’s AI Labs>>

While full-text search is at the core, as you can tell, an insight engine contains a lot more than just a search engine. While these 12 features are essential aspects of any insight engine, Coveo goes much further.

For Coveo, developing an insight engine is about doing everything necessary to make people’s lives easier, whether competing in the marketplace, searching for employee benefits, elevating the customer experience or the security or DevOps professionals responsible for tuning and keeping it all working. No matter what your interests, Coveo is here to help make your digital experience a great one.

Learn more about how intelligent search and enterprise search are finding the answers customers and employees need.

1-Gartner, Magic Quadrant for Insight Engines, Stephen Emmott, Anthony Mullen, 17 March 2021.

All statements in this report attributable to Gartner represent Coveo’s interpretation of data, research opinion or viewpoints published as part of a syndicated subscription service by Gartner, Inc., and have not been reviewed by Gartner. Each Gartner publication speaks as of its original publication date (and not as of the date of this report). The opinions expressed in Gartner publications are not representations of fact and are subject to change without notice.

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. or internationally, and is used herein with permission. All rights reserved.

Share this story:

About Andrew Oliver

Andrew C. Oliver is a developer, technologist, and writer. He's worked on data, search, and software architecture for some of the world's largest companies. His regular column in InfoWorld covers AI, cloud, and data architecture along with other developer topics.

Read more from this author