Gravatar for jshin@coveo.com

Question by jashin, Sep 13, 2018 5:08 PM

How are A/B tests split?

How does Coveo determine which side of the A/B test a visitor is taken to?

My research suggests that Coveo uses a mix of cookies and IP addresses. E.g. User searches at midnight, leaves, searches again at 10am, then again at 5pm, then not again until 1am the next day. Coveo counts this as 4 visits, but 3 of them were on one split and 1 visit on another split at 1am the next day. The second split would be Coveo "rolling the dice" to determine which random split to send the user.

Can someone confirm if a) this is correct, b) expand how IPs play a role, and c) how Coveo splits users who reject cookies.

Thank you!

1 Reply
Gravatar for dreisch@coveo.com

Answer by David, Sep 13, 2018 5:39 PM

Hi,

Pipeline A/B tests set a cookie on incoming searches, which randomly assigns a pipeline to that user. There is no IP address checking involved.

While the cookie expires after 30 minutes, we reset the cookie on each query so a user doing queries regularly will remain on the same split. However as you observed, if a user makes no queries for 30 minutes their next search will be reassigned a random split.

A user who rejects or otherwise does not send us back the cookie will end up on a random pipeline with each search.

Hope that helps!

Ask a question