Case Study - Astro Teller and Managing Experimentation

Astro Teller, the chief executive of Google X and a celebrated speaker on the company “Moonshots,” is known for his innovative approach to Experimentation. Moonshots, the ultimate form of Experimentation, involve ExOs converting successful experiments into large, funded initiatives within the guidelines of their MTP. These projects carry significant expense and risk but also offer tremendous potential payoffs and propel the company closer towards the realization of its MTP.

The following three principles, Astro explains, describe a good experiment:

1.  Any experiment where you already know the outcome is a BAD experiment.

2.  Any experiment when the outcome will not change what you are doing is also a BAD experiment.

3.  Everything else (especially where the input and output are quantifiable) is a GOOD experiment.

Seems simple enough, right? You must ask questions to which you don’t currently know the answers and whose answers, if you discovered them, would change the way you operate.

If you already know the answers, or if you’re testing an insignificant detail that doesn’t matter, you’ll just be wasting time and money. To get good questions and therefore good experiments, you must create a culture that incentivizes asking good questions and designing good experiments.

Astro describes a very unique approach to doing just this: “At X, we set up a ‘Get Weirder Award.’ The whole point of the Get Weirder Award was to focus the team on experiments and to drive home the need to think in terms of experiments.” Teams would be challenged to ask “weird” questions: to put forth crazy ideas around framing problems differently, and to design experiments that push the limits. Critically, Astro gives out the Get Weirder Award only after the experiments are run. “If you give out the award after they’ve run the experiment, independent of the results, then people start to really feel that you don’t actually care about the outcome. You care about the quality of the question.”

In this way, X constantly (and viscerally) reinforced the asking of good questions. Accordingly, they’ve built a culture around celebrating the questions themselves.

Recently, when Teller was addressing Peter’s Abundance360 Summit, he explained further how he helps his teams at X conduct more effective and meaningful experiments in a three-step process:

1. He asks his team to bring him a list of proposed experiments in order of their perceived importance. Astro reviews that list and says, “Humor me and reorder your list so the items at the top are the ones that would teach us the most in this project.”

2. When the team returns with their reordered list of experiments/action items, Astro instructs them to “Go do the top two things on your Learning List.”

3. Once the team returns, he asks them to create a new list in order of importance. Chances are the new list won’t look anything like the original list. This is why Teller emphasizes doing things in learning order.