Guide · Experiments
Test a ranking change before everyone sees it.
A synonym set or a new boost can lift one query and quietly hurt fifty. An experiment splits live traffic between two saved profiles and lets the numbers decide.
Set it up right
Two profiles, one change
Control is your current profile, treatment is a copy with exactly one change. Two changes at once tell you nothing.
Sticky per visitor
Assignment is deterministic on the visitor id, so a person doesn't bounce between variants and the comparison measures ranking, not noise.
Check offline first
Run the golden set against the treatment profile. If nDCG drops, there is no point in spending traffic.
Read the result
Primary metric
Search conversion or clicks per search, decided before you start. Choosing the metric afterwards finds a winner every time.
Run full weeks
Weekday and weekend traffic behave differently; stop on a weekly boundary, not when the chart looks good.
Promote or roll back
Promoting makes the treatment the default profile; rolling back ends the experiment with no change.
How to start
- 1
Create the experiment
Pick the index, the control and treatment profiles and the traffic share.
POST /api/v1/experiments?index=products { "name": "Boost new arrivals", "controlProfileId": "prof_default", "treatmentProfileId": "prof_boost_new", "treatmentWeight": 50, "status": "running" } - 2
Start it and wait
Searches without an explicit profile are assigned automatically; the report compares clicks and conversions per variant.