Gravatar for koradav@deshaw.com

Question by vrkorada, Jul 23, 2014 10:15 AM

How to merge search results from external source?

Hi, We have a home grown document management system which uses Lucene for indexing/search. When user search for a keyword in Coveo interface, is it possible to fire a parallel query to our document management system, get the results, merge them with Coveo search results and render in Coveo UI? We want to make Coveo interface as single place for search, but not sure how to merger search results from other sources (federated search). If the search results from our document management system need to follow a specific schema, we can do that.

Regards, Venkat

Gravatar for olamothe@coveo.com

Comment by olamothe, Jul 24, 2014 10:02 AM

Are you using the .NET search interface of the JsSearch interface ?

Gravatar for koradav@deshaw.com

Comment by vrkorada, Jul 24, 2014 10:03 AM

we are using .Net interface.

Gravatar for slangevin@coveo.com

Comment by Simon, Jul 25, 2014 9:15 AM

Do you currently index this document management system? I mean, do you have a connector that fetch the documents and add them to the Coveo Index? Are these documents part of your index at the moment?

Gravatar for koradav@deshaw.com

Comment by vrkorada, Jul 25, 2014 10:20 AM

No. The document management system index them separately using Lucene and it has separate UI for searching within the application. Now we want to include the content to the global search using Coveo. We have two other home-grown systems uses Lucene. We can't index them directly using Coveo because Coveo do not support real-time indexing which is one of the requirement for these applications.

Gravatar for yrivardmulrooney@coveo.com

Comment by Yannis Rivard-Mulrooney, Jul 25, 2014 10:51 AM

I just wanted to drop in to add that Coveo now supports near real-time indexing starting with the June monthly release. You may want to take a look at that feature if it helps simplify your setup.

http://onlinehelp.coveo.com/en/ces/7.0/Administrator/minimizingthesearchabilitydelaywithnearreal-time_indexing.htm

Gravatar for koradav@deshaw.com

Comment by vrkorada, Jul 28, 2014 3:01 AM

How will Coveo knows when a new content/document is added/modified? Does it supports on-demand indexing? Something like external apps can make an api call to Coveo and pass details to be indexed?

Gravatar for yrivardmulrooney@coveo.com

Comment by Yannis Rivard-Mulrooney, Jul 28, 2014 9:36 AM

This depends on the connector you are using. Setting up a very frequent incremental refresh schedule is probably your best solution, except if you are using a custom connector and can send the indexing commands directly to Coveo. If that is the case, you can send the commands directly to the index without waiting for a refresh to pick it up.

Gravatar for koradav@deshaw.com

Comment by vrkorada, Jul 28, 2014 9:51 AM

We discussed internally about using high frequency refresh sometime back, but discarded the option because this will increase the load on our application servers by firing queries very frequently even if there are no changes and we want to avoid this. Instead if there is a way (API) that we can use to index individual entries/items,that will be a good option.(Coveo can have some kind of index queue which will be processed continuously and external applications can keep adding items to the queue). May be this can be considered as an enhancement. I think SharePoint/FAST has similar option.

Gravatar for ldblanchet@coveo.com

Comment by ldblanchet, Jul 28, 2014 11:24 AM

One crawler we have that does something like that is the Queue crawler. It monitors a RabbitMQ Queue for documents. So if you push your documents to this RabbitMQ queue, they will be immediately crawled. (The format is xml-based and is specified in an xsd file). I won't go into too much detail, but that could be a solution.

1 Reply
Gravatar for slemieux@coveo.com

Answer by slemieux, Jul 25, 2014 9:46 AM

That could probably be achieved by implementing a custom "search provider". A search provider is basically a few classes that implement a series of interfaces: ICESSearchProvider, ICESQuery, ICESResult, etc. Built-in, we have 4 search providers: CMF, CMF465, Demo and Xml. The idea would be to develop a search provider that would internally aggregate results from our built-in CMF search provider with results coming from your other source (Lucene). Somewhere in your custom code, you would have to execute a statement like the following: SearchObject.Provider = new YourCustomSearchProvider(SearchObject.Provider, other, necessary, arguments, here); That would make the search control responsible of rendering the search page using your custom search provider that would aggregate the Lucene results to the CMF (Coveo) ones.

Gravatar for koradav@deshaw.com

Comment by vrkorada, Jul 25, 2014 10:25 AM

Thanks for the update. Looks logical. Our life would have been simplified if Coveo support federated search and merge the results :) Are there any samples to start with?

Gravatar for mdrolet@coveo.com

Comment by Mathieu Drolet, Jul 25, 2014 10:38 AM

One thing to keep in mind and the reason why this is not something easily achieved is ranking. How do you rank "merged" results coming from two separate engines? This doesn't help you accomplish your goal but I wanted to mention it as relevancy is important and hard to achieve when federating results from 2 different engines.

Ask a question