A Collaboration with Productify
Product people are both lovers and haters of experiments. On the one hand, they are great at helping measure the result of changes.
On the other hand, launching anything as an experiment takes more analytics and engineering work, and typically also slows down forward progress. If you already have a vision of the future, does it make sense to test your way there in twice the time?
For Netflix, the answer has been ‘yes’ for nearly its entire quarter-century history. Since its days as a DVD mailer, through the transition into a streamer, and now content behemoth, the company has used experimentation to generate sustained levels of growth seldom seen.
Investors have richly rewarded Netflix for its culture and practice of experimentation. There’s a reason the ‘N’ in FAANG is for Netflix. The company has absolutely trounced the Nasdaq 100 and S&P500 over the past 20 years. Do you see those grey and blue lines hugging the x-axis? Yes, those are the indices:
Netflix (NFLX) has returned 487x over the last twenty years. The Nasdaq 100 (QQQ) has returned 9.5x. The S&P500 has returned 3x. Chart via Zack’s.
Netflix’s product development process is a big contributor to this outperformance. As a result, for today’s piece, we set out to answer: How does Netflix manage all the tradeoffs of experiments? Why does it believe so strongly in them?
Netflix was too difficult a subject to tackle as a lone product analyst. This is a company that takes up 15% of the global internet bandwidth, and 31% of internet television time (versus 21% for YouTube). There is a lot of surface area and nuance to cover.
So, for today’s piece, I am excited to present a collaboration with my friend Bandan, a product leader at Booking.com ($100B market cap), and before that Gojek (valued at $30B). We have been doing quite a bit of research into Netflix’s experimentation.
Since the earliest days of Netflix, before it even began as a DVD mailing company, it has had a culture of experimentation.
By 1997, Reed Hastings was already worth hundreds of millions of dollars. He had sold his company Pure Software for $700M and was CEO of the new combined entity Pure Atria. One day on a commute to work with his marketing executive Marc Randolph, the two began discussing the opportunity to mail movies.
Reed saw the DVD format proliferating in Japan but had never used one himself. So he drove to the store and got himself one. He then mailed it. To his great surprise, it came back in great shape. It worked.
This little “validation experiment,” would be enough of a seed for Reed to help get Netflix started. He provided $2.5M to Marc to build out a team, left Pure Atria, and went off to pursue graduate studies at Stanford.
Marc and the team went about testing out different types of packaging. The key, the team realized, was creating a low-cost envelope that protected the discs. The longer the team could extend the life of their discs, the better the unit economics.
Many of the employees’ families helped test. Since DVD technology was so early at the time – 2% of American households had a DVD player – many did not even play the movies. They just shipped them back to test disk durability in the mail.