I would encourage everyone, if you can, look at some of the experiments that you thought were your biggest winners. Look at the downstream metrics for a year, two years on that experiment. And I'll bet you'd be surprised how many times the metric is different than what you thought it would be after a year.
Short-term wins often disappear long-term
Growth → Experimentation & Metrics
Archie AbramsHow Shopify Grows: Long-term Experiments, Absolute Metrics, and 100-Year Vision
I think there's probably two things that have been very common. And I would say in quite a few cases, you get a lift on a metric up front, a more short-term metric... And then you look a year later, and there's actually no incremental lift on GMV from that cohort.
Archie AbramsHow Shopify Grows: Long-term Experiments, Absolute Metrics, and 100-Year Vision
It's in the 30 to 40% range.
Archie AbramsHow Shopify Grows: Long-term Experiments, Absolute Metrics, and 100-Year Vision
Most of growth loops spin out their ability to produce meaningful results for you within the first five to six to seven years.
Elena Verna10 growth tactics that never work
To me, the key word is lifetime value, which is you have to define the OEC such that it is causally predictive of the lifetime value of the user.
Ronny KohaviThe ultimate guide to A/B testing