Gravatar for dipsindol@gmail.com

Question by DEEPTHI KATTA, Feb 13, 2018 12:42 AM

Best Practices when using PUSH API

Hi There!

So, I know this is a very generic question, but, starting to build a solid basis.

We have an implementation where a push source is being loaded by reading Sitecore data that is needed from a very old Sitecore instance. We then will use it as an external source on our Coveo Search Component.

In this scenario (I wish we had other option, but, have to proceed in this direction) what do you all think we will loose and what can we do to ensure the search works seamlessly. I did few quick experiments by having a search term that exists on random fields on the index and it does seem to work fine.

What I do not know or doubtful of -

1. Metadata -> Is it a good idea to pass in meta data when we are pushing items to Coveo cloud push source?

2. Wont we loose on Free text Search on HTML as obviously this is manual construction of Push source? Or is there a way we can make this happen regardless

Anything else that you all think off as a loss due to this decision. I confirmed with Coveo that on this architecture we can still turn on ML, so, we should be safe there.

1 Reply
Gravatar for jflheureux@coveo.com

Answer by Jean-François L'Heureux, Feb 13, 2018 11:10 AM

Hi,

A Coveo for Sitecore source is a push source under the cover. Thus, with a custom push source, you should be able to achieve a lot.

I do not think you will lost anything by indexing an old Sitecore instance through a custom push source. Some points to consider:

  1. If you want to see these results in the search interface facets or sort them, you will need to use the same fields for the old Sitecore items and the ones from the new Sitecore site (I think there is a new one with Coveo for Sitecore installed in it). Since Coveo for Sitecore creates one field per database with different hash, I recommend you to set the fields you are using for facets/sorts/query filters as external first. This way, Coveo for Sitecore will only create one field without an hash. you will be able to set this field value in your custom push source mapping using the pushed items metadata.
  2. Have you read the Push API documentation? It contains a best practice document.

For your questions:

  1. Metadata: It is really important to include metadata in your pushed items. Metadata is used in the source mapping to set the index field values. Make sure to send useful metadata that you want to filter on, sort on, or display in your search results.
  2. A pushed item have a DocumentBody parameter. This is the parameter used by Coveo for Sitecore to send the HTML of an indexed Sitecore page item. When processing your old Sitecore instance items, you could detect which ones have a layout and execute similar code to fetch the HTML page from the Sitecore server. Then set DocumenBody parameter to the response of the web server. If you do not want to fetch HTML pages, you should at least decide for each item template, which Sitecore field should be free-text searchable. Send the value of those fields as metadata. Make sure to map those metadata to free-text searchable index fields in the source mapping. Note that rich-text fields will be indexed with the HTML elements and the text in those elements (br, div, span, a, table…) will be searchable unless you clean-up the HTML yourself before setting the metadata values.

I hope this helps,

Jeff

Gravatar for dipsindol@gmail.com

Comment by DEEPTHI KATTA, Feb 15, 2018 6:42 PM

Hi @Jean-François L'Heureux, so good to hear from you like always.

It is a huge relief to know push source will not cause us to loose any goodness from Coveo perspective. :)

Now, on the points you mentioned to consider. #1. Yes, I am aware that we will be using external fields as facets and yes we will be installing Coveo for sitecore on newer version of Sitecore that will then use External push source( That has sitecore data from old sitecore site). I think you guys have good documentation on how to use External fields for Facets and avoid hashing on field names, so, good there.

On #2, I did see the best practices document and figured is more towards Push Source and status related, what I was looking for -" End user experience of search and results should be as good or close to what Coveo would do if potentially if all the old sitecore content was for say in the same instance where we are installing Coveo for Sitecore". So, to be able to achieve this is the initial questions I had posted on the question.

I have few follow up's on the answers for my questions you have in here.

1. Do you have some links/documentation out there to be able to push meta data through Push API accepted Json(data), I see some here, but, it looks more like HTML than Json.

https://developers.coveo.com/display/public/CloudPlatform/Adding+or+Updating+a+Single+Basic+Item+in+a+Shared+Push+Source

Currently, we are reading fields from Sitecore item(everything available as a field on Item and massaging it to ensure they are of correct coveo supported format for field name), serializing in to Json and pushing it Coveo Push source through Push API by passing in json on "request.content". Would this suffice or should we ensure that Json has a metadata object defined as well? If yes, could you provide some direction or link to format of Json and what should ideally be there in that.

on #2. Thank you, will potentially look in to this, but, I think would like to tackle step 1 of ensuring metadata makes in to Coveo Push source.

Gravatar for fbeaudoin@coveo.com

Comment by Frederic Beaudoin, Feb 19, 2018 1:08 PM

Hi @DEEPTHI KATTA,

When you add or update an item using the Push API, you can include any arbitrary metadata key-value pairs in the body of your call, as long as those keys do not collide with reserved keywords (see Reserved Item Body Keys). The sample request in Adding or Updating a Single Item in a Push Source includes metadata (the `date`, `documenttype`, `filename`, `isfrequentlyupdated`, etc. key-values are all metadata). Do not confuse item metadata (information about the item) with item data (the content of the item itself).

In a Push source, an incoming metadata automatically populates the corresponding Coveo field in the index, if such a field exists (do not confuse Coveo fields with Sitecore fields), unless there are custom mapping rules overriding or complementing this default behavior (see Understanding Push Source Item Metadata).

I suggest you take a look at the Push API Tutorials; the third one deals with metadata, Coveo fields, and mappings.

Gravatar for fbeaudoin@coveo.com

Comment by Frederic Beaudoin, Feb 19, 2018 1:08 PM

Hi @DEEPTHI KATTA,

When you add or update an item using the Push API, you can include any arbitrary metadata key-value pairs in the body of your call, as long as those keys do not collide with reserved keywords (see Reserved Item Body Keys). The sample request in Adding or Updating a Single Item in a Push Source includes metadata (the `date`, `documenttype`, `filename`, `isfrequentlyupdated`, etc. key-values are all metadata). Do not confuse item metadata (information about the item) with item data (the content of the item itself).

In a Push source, an incoming metadata automatically populates the corresponding Coveo field in the index, if such a field exists (do not confuse Coveo fields with Sitecore fields), unless there are custom mapping rules overriding or complementing this default behavior (see Understanding Push Source Item Metadata).

I suggest you take a look at the Push API Tutorials; the third one deals with metadata, Coveo fields, and mappings.

Ask a question