Gravatar for josh.vohs@vml.com

Question by jvohs5, May 20, 2016 3:03 PM

Exception while processing an item

We are using Coveo for Sitecore and CES version 7.0 x64 Build 7914.0. Everything has been running fine for months. We recently had a prod deployment where we removed some crawlers from the Sitecore config and various other things. After the deployment we've been getting errors similar to the following repeated 5 times in the CES index logs. They come in batches of in batches of about 100 every hour. I can't identify what we touched that would have caused this. Searching for the IDs referenced in the errors yields no results in the CMS (master or prod DBs). I've rebuilt the master index through the Sitecore control panel, but the errors persist.

Exception while processing an item: -> Coveo.Connectors.Sitecore2.SitecoreWebService.Exceptions.CoveoSoapException: The Sitecore item identified by "sitecore://master/{F08D42C6-D89B-4EC1-8401-CD1C9828CAAB}?lang=en&ver=1" couldn't be found or was denied with the current credentials. at Coveo.Connectors.Sitecore2.SitecoreWebService.SitecoreWebService.TryCatchWrapper[T](Func`1 p_Action, String p_MethodName) at Coveo.Connectors.Sitecore2.SitecoreWebService.SitecoreWebService.GetBinaryDataAtUri(String p_ItemUri)

Any idea how I can troubleshoot this?

Gravatar for jflheureux@coveo.com

Comment by Jean-François L'Heureux, May 20, 2016 3:24 PM

That is very strange. The code throwing this error is executed because the Queue crawler received a message to index the item with this ID. Then, it contacted Sitecore to download the binary of the item and Sitecore said the item cannot be found.

I see 2 possibilities:

  1. A hundred RabbitMQ messages are stuck on your RabbitMQ queues and reprocessed every hour.
  2. There's a process in your Sitecore instance that indexes those 100 non-existing items every hour. It may even be an import process that creates and the deletes the items.

I have a couple of questions:

  • Is this the same IDs in every batch or errors?
  • Are you aware of any process in your Sitecore that imports/indexes/publishes items on a hourly basis?
Gravatar for josh.vohs@vml.com

Comment by jvohs5, May 20, 2016 4:25 PM

They do appear to be the same 20 IDs reported for each batch of 100. I do have a process that runs every hour and creates items, but the timestamps do not line up and it's never created more than a couple items.

Not knowing my way around RabbitMQ, is there a way I can view/clear the queue? Or do you recommend a different approach?

Gravatar for jflheureux@coveo.com

Comment by Jean-François L'Heureux, May 20, 2016 5:00 PM

For RabbitMQ, you can check at http://localhost:15672 on the machine where CES is installed.

Work with caution here. If you purge the queue or delete it, you will lose any messages in it. Can you first log in and just report me the number of messages for each queue for each message state (ready, unacked, total). The default username/password is "guest".

Since you have a process running every hour to create items, I would also investigate whether this process is temporarily creating 20 items with the IDs you see in the errors and then delete them immediately.

Gravatar for josh.vohs@vml.com

Comment by jvohs5, May 20, 2016 5:13 PM

Sure thing. The report is probably what you expected.

Master: Status - Idle Ready - 0 Unacked - 96 Total - 96

Web: Status - Idle Ready - 0 Unacked - 0 Total - 0

1 Reply
Gravatar for jflheureux@coveo.com

Answer by Jean-François L'Heureux, May 20, 2016 5:33 PM

Is the number of unacked messages seems to fluctuate if you wait one minute with this page opened or it is staying at 96 ?

If it is stuck at 96 for a long time, those messages are probably the ones causing the errors. I guess these items were added to the content tree and deleted very quickly before the queue crawler had time to index them. In this case, I recommend you to purge the queue (Click on the queue name > Delete / purge > Purge).

Then, a rebuild of your master and web Sitecore indexes is recommended as the purge may have deleted valid messages that were added to the queue just before you purged it. The rebuild will ensure the indexes are up to date.

Gravatar for josh.vohs@vml.com

Comment by jvohs5, May 20, 2016 5:58 PM

The number of unacked has remained at 96.

I can click the purge button and I get a "Queue purged" message, but the unacked number stays at 96. Even after a page refresh.

Edit: I was able to purge the queue by moving the rdq files from the persistent store and restarted the service. I generated traffic by kicking off the reindex in Sitecore and saw the queue reappear in the rabbitMQ dashboard. I'll continue to monitor, but I think this may solve our problem.

Thank you for all your help!

Gravatar for jflheureux@coveo.com

Comment by Jean-François L'Heureux, May 23, 2016 9:19 AM

Hi,

You could also have deleted the queue from the RabbitMQ UI. Coveo for Sitecore always recreates the queues in RabbitMQ if they are not present.

I think this will solve your issue as well.

Ask a question