Gravatar for rdhuyvetters@reference.be

Question by Rik, Apr 24, 2017 3:01 PM

Coveo for Sitecore: Indexing stopping when indexing media library

Hi, we have a problem when indexing (from Sitecore).

When reindexing, Sitecore first takes content items and they are being handled and sent to Coveo at a fast pace.

At a certain moment, the unacknowledged items in RabbitMQ start to go up at the time SItecore starts indexing the media items.
Looking at the index.log we see the following warnings:

212358922613683387|WARNING|2017-04-24|11:37:21|0|Exception while processing an item: -> The request failed with HTTP status 403: Forbidden.|||||Sitecore Search Provider||||Coveo_Web|SUCCESS|||18496|17550||

Aparently Rabbit MQ keeps those items in the queue until it reaches the max

Socket descriptors and then stops accepting requests, so sitecore stops halfway the reindexing.

We're investigating why the media items return a 403, but the question is:

Is there a way for coveo to handle this and get the items from the queue as deliverd.
Or is this intended behaviour?

Best regards

Gravatar for slangevin@coveo.com

Comment by Simon, Apr 24, 2017 3:11 PM

What is the version of Coveo for Sitecore?

1 Reply
Gravatar for slangevin@coveo.com

Answer by Simon, Apr 24, 2017 3:14 PM

In version 4, the messages in RabbitMQ are only links to the document in Sitecore.

So in your case the "ping back" is able to retrieve the document from the tree, but not the media library.
This means that:

- The user identity pushed to Coveo does not have the right to crawl the media library;
- The 'serverurl' element is pointing to an instance not fronting the media library (like a CD instance).

Have you set 'serverurl' in your CoveoSearchProvider Custom Configuration file?

Gravatar for rdhuyvetters@reference.be

Comment by Rik, Apr 25, 2017 8:26 AM

Thank you for the feedback, Simon.

Like I said, we're looking into why we get the 403's but my main concern is that RabbitMQ doesn't accept any more requests, so no indexing is possible.

Does Coveo leaves those messages on the queue to try later?

Is the 403 a blocking issue for the indexer and should we resolve them first?

How do we let Coveo pick them up again?

Socket descriptors are at maximum level and red

File descriptors are at near max and in yellow.

Erlang, memory and diskspace are all green.

Gravatar for rdhuyvetters@reference.be

Comment by Rik, Apr 25, 2017 3:29 PM

Update, we've fixed the 403 issue, but indexing was still blocked.

The RabbitMQ queue remained blocked and inoperable.

I ended up deleting the index in coveo admin and deleted the queue.

Reindexing recreated both and now the indexer ran to the finish.

I'm not really confident in the stability of the indexer and the queue, but I think a full index like this is not something we'll do on a daily base.

Ask a question