Case Study - Jeff Holden and Amazon

Peter’s friend Jeff Holden is a renowned innovator and past XPRIZE board member. His illustrious career is marked by key roles at Amazon, Uber, and his current venture, Atomic Machines. As chief product officer at both Amazon and Uber, he spearheaded the creation of Amazon Prime and popular Uber services like Uber Eats and Uber Pool (now UberX Share). He has a wealth of experience in disruptive innovations, which he distilled down to four key lessons when Peter sought his advice.

Jeff’s first directive is to design your company as a potent experimental engine from the beginning. In an era where change is the only constant, continuous innovation and Experimentation are crucial for survival and hyper-growth. Jeff incorporated this ideology at Amazon, Groupon, and Uber.

Integrating this experimental engine isn’t an effortless task, and it often requires a cultural transformation in established companies. It also requires a commitment to ceaselessly test bold ideas, new business models, products, and processes. This principle was evident in Amazon’s early days when a comprehensive experimental platform was made available to almost everyone.

This openness led to an onslaught of experiments, many of which proved futile. As Jeff explains, Amazon ran countless experiments out of sheer curiosity, despite their costs and inefficiencies.  Eventually, Amazon formed an Experiments Group. Before executing an experiment, its hypothesis and potential value to the company had to be convincingly presented.

Proper interpretation of experimental results is Jeff’s second key lesson. Like Uber, which runs thousands of monthly experiments and bases decisions on statistical significance, knowing the difference between significant and insignificant results is crucial. At Uber, only about 20 to 30% of experiments are successful, but all provide valuable insights. Jeff’s advice? Ignore the external noise, stay focused, and continue to build.

In his third directive, Jeff provides two key pointers about constructing a team and a company: nurture an experimental ethos within the organization and hire individuals familiar with Experimentation and the data-informed mindset. 

Finally, Jeff emphasizes the importance of comfort with being misunderstood. Strong experimental cultures often invite misunderstanding from external observers. Jeff cites Amazon Prime, which was initially met with skepticism but now has over 50 million members, as an example: “Amazon Prime could have been one of those catastrophic failures,” says Jeff. “We tried auctions, and that failed, and we tried zShops and that failed. But we just kept going, and we finally cracked it. Then, when we launched it to the world, the response was: ‘You guys are insane! This is like, super risky. You’re going to blow up with all this margin from shipping.’ Bezos, characteristically, replied, ‘Yeah, I kind of figured this would be misunderstood.’”