Gravatar for loebrandy@gmail.com

Question by rloeb, Jun 26, 2016 3:31 PM

Paging failing...why?

I'm trying to page using rest api. It dies after 1000 records when I do 100 per page, i.e. 10th page.

Here are the relevant logs for "first result, requested, returned"

999 100 0 No Relevance Erroneous TOOMANYREQUESTED_RESULTS
899 100 22892 No Relevance Successful SUCCESS 799 100 22892 No Relevance Successful SUCCESS 699 100 22892 No Relevance Successful SUCCESS 599 100 22892 No Relevance Successful SUCCESS 499 100 22892 No Relevance Successful SUCCESS 399 100 22892 No Relevance Successful SUCCESS 299 100 22892 No Relevance Successful SUCCESS 199 100 22892 No Relevance Successful SUCCESS 99 100 22892 No Relevance Successful SUCCESS 0 100 22892 No Relevance Successful SUCCESS

Why? Thanks

Gravatar for loebrandy@gmail.com

Comment by rloeb, Jun 26, 2016 3:38 PM

Also, btw it fails at 1000 no matter page size I use, I tried 100 and 10.

1 Reply
Gravatar for mlaporte@coveo.com

Answer by Martin Laporte, Jun 27, 2016 2:54 AM

By default the index won't allow a user to retrieve more than the first 1000 results of a query. Retrieving items "farther down" the result sets is a gradually more expensive operation for the index; this limit exists to prevent someone from hogging all the index resources.

If you want to stream all documents matching a particular query, have a look at the @sysrowid field. The usual trick is to query again and again sorting on that field and filtering for all values below the last one that you retrieved.

Gravatar for loebrandy@gmail.com

Comment by rloeb, Jun 27, 2016 4:48 PM

Thanks, this worked.

Ask a question