Gravatar for paldrich@ascedia.com

Question by Paul Aldrich, Mar 14, 2018 9:44 PM

I see a "(503) Server Unavailable" when custom code tries to query the Coveo web index

I am working on a client's website that recently had Coveo for Sitecore v4.1 (Hive) with Coveo Cloud V2 implemented. One of their custom components queries the content index for the site (which is the Coveo_web_index) for data. Until today, we did not know that the component was failing. We are seeing an issue on all environments. This specific component is simply not displaying any data. In the console a 500 error can be seen when the API endpoint for this component is hit. In the Sitecore logs, we see this error:

21524 12:25:15 ERROR An exception was thrown by the ItemListSearchService.<br>Exception: Coveo.Framework.Utils.Rest.HttpClientException<br>Message: Failed to obtain resource located at 'http://qa.aopa.beta05.ascedia.com/coveo/rest?indexName=Coveo_web_index'.<br>[BEGIN RESPONSE BODY]The service is unavailable.[END RESPONSE BODY]<br>Source: Coveo.Framework<br>  at Coveo.Framework.Utils.Rest.HttpClient.HandleResponseException(WebException p_Exception, String p_Url, Int64 p_CallId)<br>  at Coveo.Framework.Utils.Rest.HttpClient.ExecuteRequest(String p_Url, Func`2 p_CreateRequest, Int64 p_CallId)<br>  at Coveo.Framework.Utils.Rest.HttpClient.PostRaw(String p_Url, String p_Body)<br>  at Coveo.Framework.Utils.Rest.HttpClient.Post(String p_Url, String p_Body)<br>  at Coveo.SearchServiceProvider.Rest.ClientSessionWrapper.ExecuteQuery(QueryParams p_QueryParams)<br>  at Coveo.SearchProvider.LinqToCoveoIndex`1.ExecuteQuery[TElement](QueryParams p_Query, IEnumerable`1 p_QueriedTypes)<br>  at Coveo.SearchProvider.LinqToCoveoIndex`1.Execute[TResult](CoveoCompositeQuery p_Query)<br>  at Sitecore.ContentSearch.Linq.QueryableExtensions.GetResults[TSource](IQueryable`1 source)<br>  at AOPA.Library.Services.Search.Impl.ItemListSearchService.RunQuery(ISearchIndex index, ItemListQueryItem queryItem, Int32& totalResultCount) in D:\projects\aopa\web2.0\src\AOPA.Library\Services\Search\Impl\ItemListSearchService.cs:line 123<br>  at Castle.Proxies.Invocations.IItemListSearchService_RunQuery.InvokeMethodOnTarget()<br>  at Castle.DynamicProxy.AbstractInvocation.Proceed()<br>  at Jabberwocky.Autofac.Extras.MiniProfiler.Interceptors.AsyncProfilingInterceptor.Intercept(IInvocation invocation)<br>  at Castle.DynamicProxy.AbstractInvocation.Proceed()<br>  at Castle.Proxies.IItemListSearchServiceProxy.RunQuery(ISearchIndex index, ItemListQueryItem queryItem, Int32& totalResultCount)<br>  at AOPA.Website.Api.ItemListController.GetItemList(ItemListRequestModel itemListRequestModel) in D:\projects\aopa\web2.0\web\Website\Api\ItemListController.cs:line 134<br>  at AOPA.Website.Api.ItemListController.Results(ItemListRequestModel model) in D:\projects\aopa\web2.0\web\Website\Api\ItemListController.cs:line 57<br><br>Nested Exception<br><br>Exception: System.Net.WebException<br>Message: The remote server returned an error: (503) Server Unavailable.<br>Source: System<br>  at System.Net.HttpWebRequest.GetResponse()<br>  at Coveo.Framework.Utils.Rest.HttpClient.ExecuteRequest(String p_Url, Func`2 p_CreateRequest, Int64 p_CallId)

I checked the diagnostic page immediately and was surprised to see all green lights. When I go to the /Coveo/rest page, I am not seeing any errors and it is not blank. It is displaying JSON. Our search page displays fine on a site where we are seeing this issue, so I don't understand this error.

1 Reply
Gravatar for hsantos@coveo.com

Answer by Hugo Santos, May 29, 2018 1:36 AM

Hi @Paul Aldrich,

So as far as I can see your client have a few custom classes which are querying using Linq.

On this specific case this is the class which is triggering the search:

  1. AOPA.Website.Api.ItemListController.Results(ItemListRequestModel model), which calls
  2. AOPA.Website.Api.ItemListController.GetItemList(ItemListRequestModel itemListRequestModel), which calls
  3. AOPA.Library.Services.Search.Impl.ItemListSearchService.RunQuery(ISearchIndex index, ItemListQueryItem queryItem, Int32& totalResultCount), which calls
  4. Sitecore.ContentSearch.Linq.QueryableExtensions.GetResults[TSource](IQueryable`1 source)

And since Coveo is the default Linq search provider in your website, this line of code will trigger:

  1. Coveo.SearchProvider.LinqToCoveoIndex1.ExecuteQuery[TElement](QueryParams p_Query, IEnumerable1 pQueriedTypes), which calls
  2. Coveo.SearchServiceProvider.Rest.ClientSessionWrapper.ExecuteQuery(QueryParams pQueryParams

There is anything special about this stack trace, it's all good here.

So my suspicion is that the issue is somehow related to the communication between the server and the endpoint http://qa.aopa.beta05.ascedia.com/coveo/rest.

I'd have a few questions for you to try to isolate the issue:

  • What is the setup of this website? Is it failing on CM, CDs or both?
  • Are you sure that the machine which is initiating this stack has access to the http://qa.aopa.beta05.ascedia.com/coveo/rest endpoint?
  • Can you tell me more about this specific component? What does it do? In which context it is used?
  • Some useful detail about the ItemListSearchService.RunQuery implementation? How are you making sure it is always fetching from the web database even if it is running on a CM instance?

Gravatar for paldrich@ascedia.com

Comment by Paul Aldrich, May 29, 2018 8:10 PM

Hi Hugo,

Thank you for your answer. For this particular issue the client decided to change the web content index referenced in their custom component from the Coveo one to the Solr one they were using before; when I first started installing Coveo I had changed the index used by most components because I thought they weren't going to use the Solr one anymore. When they switched back to the Solr index the component started working, and that's how they've been running since they made that change back in March.

Unfortunately, this doesn't solve the core problem we were seeing and this issue is already resolved; I can try to answer some of your questions but it's pretty null and void at this point:

  • Pretty sure the component was failing on both CM and CD (for live at least); for QA it was just failing on the one server.
  • I didn't verify that from within the machine so that could be something.
  • It was a news listing component used on the news and media page, it pulled latest articles from the content index.
  • I don't have an answer to this question; seems null and void seeing how the issue was present on QA as well, where there is no separate CM and CD.
Ask a question