Metadata search filters and Coveo
Hello, all. I'm looking for ideas.
We're using the rest API Coveo Enteprise Search, on-prem, as a text-search engine for a few document stores, up to about 60,000 documents apiece. It's a custom .NET application that runs two searches -- a straight text search in CES, and then a metadata search in another database -- then attempts to combine them. We've had reports of false negatives: a search using the metadata but no text keywords might return, say, 8 documents. But adding to that a text search containing a word common to all 8 documents might return only 3.
I've investigated, and it appears that the culprit is the "numberOfResults" parameter. If Coveo is asked to return the first thousand hits, Coveo obviously have no way of knowing whether those thousand will all be included in the metadata filter. If I put the "numberOfResults" parameter up too high, however, it slows substantially and sometimes times out.
I've been pursuing a possibility of adding my metadata to Coveo, but I'm not sure that's going to go anywhere. Does anyone have any suggestions?
Is there a way I can send in, perhaps, a list of file locations to coveo and say, "only search on these files"? I would doubt it, as this list would have to be dynamic.
Thanks for your help.
You can and should index your documents with this metadata in CES. If you don't, you miss out the major advantage of using Coveo.
In the Coveo world, we call this metadata "Fields". Adding those fields will make them available when searching. You can explicitly add an expression like
@myfield=="myMetadataValue" to get all the documents that match this condition. Have a look at the query syntax, this will give you a good overview of what is possible with fields in Coveo.
The fields can be individually configured to be, for example, "Free-Text Searchable". This way, a simple query will also search for this field. For instance, with the same field as the example before, a simple search for "myMetadataValue" would return this same document.
As for your last question, if you manage to get your files in Coveo with their metadata, they will be searchable. If you add some fields that contain, for example, the file's path, you could search for
@path *= "myPath/* and it will give you every file matching this path. It all depends on the fields set on your documents, and there are many, many possibilities about what you can put there.
Now, to index them, you need to use a connector. They take data and transform them into documents that Coveo can index.
You should really have a look at this documentation section to see what is possible with Coveo and the possible connectors to see if one fits your document type. There is also the possibility to push your own documents and metadata using custom scripts, if this is what you need.
Moreover, you should not ask for a thousand hits. Coveo is not a database, it is a search engine. You should expect to get the best results within the first page, you never need more than a couple of results.
Don't hesitate to ask if you need more help!