Description of Experimentation

Experimentation is a fundamental characteristic of nature and, indeed, of evolution. All organizations need to change in order to thrive and grow. Change, further, happens only via continuous adaptation … and that adaptation occurs only through Experimentation. 

The classic modern example of Experimentation is A/B testing a website: that is, putting out two versions of a page to two different viewing audiences to see which version performs better. Other types of testing approaches include randomization, causal inference, and counterfactual testing. Since our 2014 book, the power of Experimentation has increased tremendously due to the availability of data, the automation of testing models, and a plethora of tools to help track experiments.

As Jeff Holden described his Experimentation process to Peter, we extracted several key themes:

1. Passion. “To be experimental, you must first be deeply passionate about the subject, to the point that it consumes you, that you are willing to invest yourself completely in your search for the best solution. There have been times in my career when people call me first at my office at 11 PM because they assume I’ll be there. And they’re right.” 

2. Culture. History has shown repeatedly that great products and inventions come from companies that have a culture of innovation and Experimentation. 

3. Intelligent failure. Innovative companies aren’t forgiving of failure; they are forgiving of the right kind of failure: the failure that comes after an incredible, intelligent effort. In these companies, bad failures are punished as a waste of time, people, and resources. Innovative companies, on the other hand, embrace “failure [that] is always advancing our understanding.”

4. Ideas. A seemingly great idea may prove impractical, while an apparent clunker may prove the most viable once it’s been tweaked. It’s crucial to keep an open mind about ideas: to avoid the tendency to fall in love and stick too long with an unworkable idea or, conversely, to give up on an ugly idea too early. Often the problem with a seemingly “bad” idea is not intrinsic but a prejudice—stemming, perhaps, from envy that it was another person’s idea. Cultivate intense honesty and patience when it comes to generating and refining ideas.

5. First principles. The very best ideas are grounded in first principles: foundational assumptions or propositions that cannot be broken down or deduced from other assumptions or propositions. First principles thinking requires tremendous amounts of research and investigation, and ideas built upon first principles are often unstoppable. “Once you are convinced that something is possible—even if it is assumed to be impossible—you can’t be knocked off your perch; your conclusions are unassailable,” says Holden. First-principles thinking is how seemingly crazy ideas—like rebuilding the economy from molecules and competing with physical stores with virtual alternatives—get funded and attract the best talent. That’s because facts built on first principles can’t be dismissed. Holden points to Elon Musk as the master of using first principles to come up with ideas, like the Hyperloop, that others have dismissed out of hand.

6. Volume. Great experimenters come up with dozens, even hundreds, of related ideas and then narrow them down. Amazon and Uber, says Holden, ran thousands of experiments each year. “Every company should have that number.” What’s more, even rejected experiments may hold viable and valuable kernels; amalgamated, they may result in the best idea yet.

Iteration. Before he finally created the incandescent light bulb, Thomas Edison was asked if he was doomed to failure. The great inventor replied that he hadn’t failed—rather, he’d found 10,000 ways the lightbulb didn’t work. Testing new ideas requires an experimental model that uses previous results to modify the original idea, improving each subsequent iteration. Repeat until you run out of incremental improvements—and you likely have found your product or service.