Recommendations when A/B testing FindMine

Things to watch out for:

Issues experienced with Timing of calls:

• Optimizely / Adobe don’t record FindMine page loads, so the results and interactions attributed to the FindMine branch are significantly under-reported
• Optimizely / Adobe record FindMine as if it has loaded when in fact FindMine never received an API call
• These issues are due to timing of the API call relative to the time window when Optimizely/Adobe are looking for the call
• In particular these issues are exacerbated on mobile with variable / slower connections

Multiple modules loading and one visually suppressed:

• We do not recommend this strategy
• Using a data layer to suppress FindMine visually for half the traffic causes both modules (control and variant) to load at the same time

Speed of calls:

• If the experience loads slowly for the customer or “flickers” there is another variable introduced into the test

Testing vs. another element on the page that is not outfitting:

• Outfitting naturally requires some constraints: product category variety, the outfit has to look good and respect the brand’s guidelines, personalization, etc.
• FindMine also optimizes for KPIs like margin etc. which may be in direct conflict with conversion or some other metric
• Any control group that features a solution pushing top performers instead of showing an outfit will perform better on short term revenue and conversion, but at the expense of the long tail of product and at the long term benefit of customer loyalty (by showing something that’s not an outfit)
• These kinds of tests also typically show the control group cannibalizing another element on the page run with the same strategy

A/B Test Best Practices