Impact Estimation

Pre-Experiment

Things to consider:

1. How much time will it cost? a. Min 2 weeks for the exp b. 2 weeks - 2 months dev work - training data collection - model creation - experiment infrastructure - model hosting - post experiment analysis
2. How much headcount will it take to implement ?
3. how much will it benefit the company/customer.
4. Pretend you build a perfect model, what will be the lift
5. What is the cost of the experimentation
6. Decide point of diminishing return
7. Identify risks to the business a. Loss of conversions, ATC, sessions b. time for customer to get familair with experience c. customer trust d. data leakage e. outages f. how quickly can we stop experiment if it is going bad
8. If the experiment might be drastic, can run a shadow test a. send logs to other service but don't act on it b. use to perform sanity check c. measure differnces d. look for errors/faults e. look for cpu/memory/disk latency

Post Experiment

Things to consider:

1. Is the experiment result valid ?
   • Bias Correction
   • Extrapolation appropriate
   • Stastically valid
   • experiment collisions : was another experiment running or is there carryover effect
2. Is the new experiment worth launching
   • variants and invariants
   • did customers service metrics change
   • did cannibalization occur: push one product over another