Gravatar for christian.rikong@agencyq.com

Question by crikong, Oct 27, 2016 10:50 AM

Adding Item To Push Data Source Using the CompressedBinaryData Parameter

Hey There,

My name is Christian and I am new to COVEO and I am a .NET Developer. I am following the push API documentation (https://developers.coveo.com/display/public/CloudPlatform/Pushing+Items+to+a+Source ) regarding adding documents to push data source.

I was able to add content using small plain text. However, when using compressed encoded content, I am getting a 400 error. Below is my code, I have included the namespaces at the top of ease of use. I am using Zlib to do the compression. Any help/feedback that you guys can provide would be greatly appreciated. Thank you again. Take Care

using zlib;

public static bool AddBinaryDocumentToPushDataSource(string OrderingId, string SourceId, string DocumentUri, string DocumentFileExtension, byte[] DocumentBytes, string DocumentAuthor, double DocumentSizeInBytes)
{
    bool Added = false;

    try
    {
        string APIUrl = ApplicationSettings.CoveoPushAPIBaseurl + "organizations/" +
            ApplicationSettings.OrganizationId.ToLower().Trim() + "/sources/" + SourceId +
            "/documents?orderingId=" + OrderingId + "&documentId=" + HttpUtility.UrlEncode(DocumentUri) + "/";

        //First we compress the document bytes and then encode them into Base 64
        byte[] CompressedDocumentBytes = null;
        DataCompressorUtility.CompressDataWithZLib(DocumentBytes, out CompressedDocumentBytes);
        string EncodedDocumentBytes = System.Convert.ToBase64String(CompressedDocumentBytes);

        string DocumentBody =
                "{" +
                    "\"FileExtension\": " + "\"" + "." + DocumentFileExtension + "\"" + "," +
                    "\"CompressedBinaryData\": " + EncodedDocumentBytes + "," +
                    "\"author\": " + "\"" + DocumentAuthor + "\"" + "," +
                    "\"size\": " + DocumentSizeInBytes + "," +
                    "\"date\": " + "\"" + DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + "\"" + "," +
                    "\"indexeddate\": " + "\"" + DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + "\"" + "" +
                "}";

        Added = ExecutePushAPIDocumentAdditionRestCall(APIUrl, DocumentBody, "PUT", "application/json");
    }
}

private static bool ExecutePushAPIDocumentAdditionRestCall(string APIUrl, string DocumentBody, string RequestMethod, string RequestContentType)
{
    bool SuccessfullyExecuted = false;

    try
    {
        byte[] Data = Encoding.ASCII.GetBytes(DocumentBody);
        WebRequest CurrentRequest = WebRequest.Create(APIUrl);

        CurrentRequest.Method = RequestMethod;
        CurrentRequest.ContentType = RequestContentType;
        CurrentRequest.ContentLength = DocumentBody.Length;
        CurrentRequest.Headers.Add("Authorization", "Bearer " + ApplicationSettings.CoveoAccessToken);

        using (Stream stream = CurrentRequest.GetRequestStream())
        {
            stream.Write(Data, 0, Data.Length);
        }

        using (WebResponse CurrentResponse = CurrentRequest.GetResponse())
        {
            HttpWebResponse CurrentHttpResponse = (HttpWebResponse)(CurrentResponse);

            if (CurrentHttpResponse.StatusCode == HttpStatusCode.Accepted)
                SuccessfullyExecuted = true;
        }
    }
    catch(Exception exception)
    {
        ExceptionHandler.HandleException(exception);
    }

    return (SuccessfullyExecuted);
}
1 Reply
Gravatar for jflheureux@coveo.com

Answer by Jean-François L'Heureux, Oct 27, 2016 11:26 AM

HI Christian,

Welcome to Coveo development!

Your issue seems to be a known issue with the zlib compression method. See the answer on this other question. It links to the known issue. https://answers.coveo.com/questions/7635/cloud-v2-push-api-large-file

Gravatar for christian.rikong@agencyq.com

Comment by crikong, Oct 27, 2016 11:39 AM

Hi jflheureux,

Thank you for your input and taking the time to answer my question. My file is not large at all and even when I do not use zlib for compression I still get a 400 error. Here is my updated code without using zlib.

I am really confused about this. Even when trying it in swagger ( https://platform.cloud.coveo.com/docs#/ ) it does not work. Although in Swagger it returns a 202 but no document is added to my push data source. Any additional insights you can provide me, I would greatly appreciate it.

public static bool AddBinaryDocumentToPushDataSource(string OrderingId, string SourceId, string DocumentUri,
  string DocumentFileExtension, byte[] DocumentBytes, string DocumentAuthor, double DocumentSizeInBytes)
{
    bool Added = false;

    try
    {
        string APIUrl = ApplicationSettings.CoveoPushAPIBaseurl + "organizations/" +
            ApplicationSettings.OrganizationId.ToLower().Trim() + "/sources/" + SourceId +
            "/documents?orderingId=" + OrderingId + "&documentId=" + HttpUtility.UrlEncode(DocumentUri) + "/";


        byte[] CompressedDocumentBytes = CompressStream(new MemoryStream(DocumentBytes));
        string EncodedDocumentBytes = System.Convert.ToBase64String(CompressedDocumentBytes);


        string DocumentBody =

                "{" +
                    "\"FileExtension\": " + "\"" + "." + DocumentFileExtension + "\"" + "," +
                    "\"CompressedBinaryData\": " + EncodedDocumentBytes + "," +
                    "\"author\": " + "\"" + DocumentAuthor + "\"" + "," +
                    "\"size\": " + DocumentSizeInBytes + "," +
                    "\"date\": " + "\"" + DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + "\"" + "," +
                    "\"indexeddate\": " + "\"" + DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + "\"" + "" +
                "}";


        Added = ExecutePushAPIDocumentAdditionRestCall(APIUrl, DocumentBody, "PUT", "application/json");
    }
    catch(Exception exception)
    {
        ExceptionHandler.HandleException(exception);
    }

    return (Added);
}


private static byte[] CompressStream(Stream Input)
{
    using (var NewCompressStream = new MemoryStream())
    using (var NewCompressor = new DeflateStream(NewCompressStream, CompressionMode.Compress))
    {
        Input.CopyTo(NewCompressor);
        NewCompressor.Close();
        return NewCompressStream.ToArray();
    }
}


private static bool ExecutePushAPIDocumentAdditionRestCall(string APIUrl, string DocumentBody, string    
RequestMethod, 
    string RequestContentType)
{
    bool SuccessfullyExecuted = false;

    try
    {
        byte[] Data = Encoding.ASCII.GetBytes(DocumentBody);
        WebRequest CurrentRequest = WebRequest.Create(APIUrl);


        CurrentRequest.Method = RequestMethod;
        CurrentRequest.ContentType = RequestContentType;
        CurrentRequest.ContentLength = DocumentBody.Length;
        CurrentRequest.Headers.Add("Authorization", "Bearer " + ApplicationSettings.CoveoAccessToken);



        using (Stream stream = CurrentRequest.GetRequestStream())
        {
            stream.Write(Data, 0, Data.Length);
        }

        using (WebResponse CurrentResponse = CurrentRequest.GetResponse())
        {
            HttpWebResponse CurrentHttpResponse = (HttpWebResponse)(CurrentResponse);

            if (CurrentHttpResponse.StatusCode == HttpStatusCode.Accepted)
                SuccessfullyExecuted = true;
        }
    }
    catch(Exception exception)
    {
        ExceptionHandler.HandleException(exception);
    }

    return (SuccessfullyExecuted);
}
Gravatar for dberube@coveo.com

Comment by Dominic Berube, Oct 27, 2016 1:09 PM

Hi Christian,

Have you try adding double quotes around EncodedDocumentBytes when creating your json. This attribute is a string.

Also, what is the exact error message you are getting with the 400 error?

Dom

Gravatar for christian.rikong@agencyq.com

Comment by crikong, Oct 27, 2016 2:07 PM

Hi dberube,

Thank you for your feedback. I appreciate it. I actually tried that in swagger and through my code. In swagger, I have a 202 response code and through my code I have a 400 response code. Basically the 400 response simply says "Bad Request" and no item is added to my source. I have read over and over the documentation and followed its instructions and I can't figure out why this is not working (even in swagger). Have you encountered a problem similar to this before ? Thank you.

Christian

Gravatar for christian.rikong@agencyq.com

Comment by crikong, Oct 27, 2016 2:30 PM

Actually Dom,

You were right, The missing quotes were the problem as far as the 400 error. Now, my request goes through meaning it is accepted. However, my item is not added to my push source.

Gravatar for dberube@coveo.com

Comment by Dominic Berube, Oct 27, 2016 4:40 PM

Ok, so your json document was accepted by the PushAPI, but the document is probably rejected by the index on the platform because the compressed data is missing the 2 bytes header.

The .Net Framework has a known issue when compressing with Deflate, you need to add 0x78 and 0x9C at the beginning of the stream. With that you should be Ok.

See http://stackoverflow.com/a/21544269 and http://stackoverflow.com/a/17176881

Dom

Ask a question