HTML indexing with HTML content fetching in Sitecore kills the IIS worker process
In our Sitecore solution, we have included the HtmlContentInBodyWithRequestsProcessor in the configuration to have the rendered version of our pages indexed.
It works fine if we publish one item for example, but when we launch a full rebuild (or reindex part of the tree), the indexing process starts for a while then stops (in error). After some investigation, we found out that the IIS process of our Sitecore server crashed, that's why the indexing stoped.
My hypothesis is that the fetching of the HTML is in cause since if I remove the processor from the config file, the indexing goes on to the end. I tried to use the new ExecuteGetBinaryDataPipeline, but we get the same problem.
Another hypothesis is that this problem is new since we upgraded our Coveo for Sitecore modules (we did not launch a full rebuild since because we did not need it, until now).
I looked over and over in the logs but found nothing interesting. In the Sitecore server event log, I do see the message about the faulting w3wp appliation, but that's about it.
I was wondering if anyone has ever had that kind of issue ?
I must specify that the problem we are investigating right now is in one of our dev environment, configured as a standalone Sitecore instance (Coveo is installed on a separate server though).
Here are the different product versions we are using :
Current Coveo for Sitecore version: 4.1.286.9
Current Sitecore version: 8.2.170728
CES : Coveo Enterprise Search 7.0 x64 (8996)
API : Coveo Search API 8.0.1135
Well, after some more investigation we found that AppDynamics seems to be the source of our problems. Once deactivated, the indexation process finished normally.
I still have not found why AppDynamics creates this problem, but for now we will leave it deactived on our Sitecore server.