How can growth teams win by borrowing a bit from their engineering counterparts?
Find out in this week’s Growth Snack: The Breakout Growth Podcast Short. Here, Sean Ellis and Ethan Garr discuss the idea that growth teams win when they “Write the Report Before they Run the Experiment”. Ethan says growth teams that do this run better experiments, more frequently, with more predictable outcomes.
Sean likes the idea because he has seen just how difficult it can be to get high-tempo testing to take flight. Because this technique forces teams to start with the end in mind as they run experiments, he believes it is a good habit to get in to if you are looking to improve your growth trajectory.
If you want to learn more about applying this concept after you listen, read more here.
We discussed:
- Writing the report first is a good idea if you want to test more effectively (28.81)
- Why faster testing shouldn’t come at the expense of quality learnings (01:18)
- Documenting what you are going to do; a lesson from GoPractice! (03:48)
- Begin with the end in mind to think ahead and drive test/learn cycles (04:26)
- Who should build this habit (05:34)
And much, much, more . . .
Transcript:
00:00:00 Sean Ellis
If you are leading growth building a startup or looking to ladder up your skills then you’re probably really busy. So every other week tune in to grow snack the breakout growth podcast short where ethangar and I share one key learning to help you on your journey to break out growth. Success. It’s food for thought for. Anyone hungry for growth all right? Ethan let’s talk about the concept of write the report before you run the experiment. What is it and why is it important.
00:28.81 Ethan Garr
That’s a really simple concept. My hypothesis is that if you’re leading growth and you want to run better experiments more frequently with more predictable outcomes then you should work to build a habit around writing the report before you run the experiment and why is that important. Well I think we’re all driven to run high velocity testing. We want to run more and more tests so we can out test our competitors and and get those learnings but it’s only valuable if it actually produces quality learnings that are going to power that flywheel where you take a learning and you apply it to what you’re going to do next. Um, So if you write that report before you run the experiment I think you’re going to improve the odds that you’re going to get those actionable insights that are going to matter and help you grow. Yeah, the danger is that instead of your experiments turning guesses into facts you’re going to turn guesses into either more guesses or more unanswered questions.
01:06.31 Sean Ellis
Right? Okay, so what happens if you don’t do this.
01:18.50 Ethan Garr
And actually think it could be really a culturekilling mistake if you do that? Um, while we again, we’re all trying to run these high velocity experiments when your growth meetings turn into these unanswered questions where everyone’s pointing at each other saying did we learn anything. Do we do did we get the data does anyone understand what we learned. It really it sets you back and you might think you’re achieving something because you’re running a lot of tests but they’re not really experiments if you don’t get anything out of them.
01:44.60 Sean Ellis
Right? Yeah I’ve I’ve definitely seen teams particularly teams that that go from not doing any testing. They suddenly start testing. They follow my advice to hold yourself accountable on how much testing you’re doing I really pushed high tempo testing for a long time. But.
01:52.69 Ethan Garr
Um, yeah.
02:02.82 Sean Ellis
What what happens is that they start to just go through the motions of testing they’re like okay this is what we’re supposed to do and almost become robotic around it and I think when you do that You’re just you’re not learning what you should be trying to learn from that testing.
02:07.94 Ethan Garr
Um, yeah.
02:15.41 Ethan Garr
Yeah I mean I definitely blame you. This is definitely all your fault but but no I mean truthfully you know you got me super excited about this idea of high tepo testing I heard it from you. Um, you know early on I think maybe while you’re still writing your your book with Morgan Brown and right away I took it I went back to my my team and I was like we’re going to do this and I got everyone excited but we got ahead of our skis and I take full credit and blame for it. Um, we ah, we kept making these unforced errors you know because. We would run what we were calling an experiment but we didn’t have a flushed out hypothesis or we didn’t have the data or even if we had the data. We didn’t have the skillset to actually understand that data and it really just it was you know I think we’re in danger of really setting ourselves back instead of moving ourselves forward and I remember I was chatting with our director of engineering at the time. And he was telling me about test-driven development which is this idea and honestly I’ve stole a lot of this from from this idea of test-driven development where you write your test case before you write your code and you do that because it has these forcing functions. It forces you to make sure that you’re just writing code. That meets the requirements instead of adding stuff on top of it. That’s irrelevant or writing stuff. That’s just irrelevant because you got excited about something else. Um, and it also has this neat forcing function of getting teams talking to each other people you know talking why are we doing this. Why does it matter. And you know so I just sort of stole that and turned this into this thing thought on report driven testing where you write the experiment the report before you write the experiment and I’ve as I piloted it it with teams I’ve really seen it make an impact.
03:48.83 Sean Ellis
Yeah, you know this is really consistent with something we teach in the go practice simulator which is when you’re creating an experiment and you’re building an experiment document One of the key components of that document is what are we going to do.
03:54.30 Ethan Garr
Um, me.
04:06.63 Sean Ellis
Based on the results of this experiment and it’s and it’s often 1 of the most overlooked areas of an experiment document and it’s something I really learned from Oleg and and have put into practice and seen the value of that. But it yeah it actually reminds me of one of my favorite books over the years seven habits of highly effective people from.
04:07.19 Ethan Garr
Um, yeah.
04:26.47 Sean Ellis
Stephen Kovey where he talks about begin with the end in mind and so I think if you’re really intentional about what you’re going to do based on the learning from a test you’re much more likely to run that test in a way that you’re going to get that learning and. Learning really comes from the report so this fits really nicely with that concept of of you know what? what is the outcome that we’re shooting for what we’renna? What are we gonna do based on that outcome and let’s make sure that we know exactly.
04:44.43 Ethan Garr
Right.
04:58.58 Sean Ellis
What we’re going to use in terms of information to make those decisions.
05:00.32 Ethan Garr
Yeah, and I think Oleg is really good about sort of running that sort of yeah, helping you understand how a systematic approach to this can really drive that end goal more efficiently and more effectively. It’s one of the things I really loved about go practice. But I you know I think with that too is when you. When you start with that end and in in mind and you know where you’re going it just makes the whole process easier I mean if you think about it like you and I are very anti- bureau bureaucracy. We’re not so we don’t I don’t want people to write extra reports and extra documentation. You’re going to have to do this anyway.
05:27.23 Sean Ellis
Yes.
05:32.86 Sean Ellis
Right. 05:34.54
Ethan Garr
But if you move this earlier on in the process. It improves the odds that you’re going to succeed and and look I think you know you pointed out but and I mentioned you know early on for me when I was leading a growth team. It was really important at the beginning and I think for most teams this is going to be just good growth. Hygiene. It’ll help you? Um. Just run more effective experimentation. But obviously if you have a really highly tuned experimentation process. I mean if you’re Karen Flanagan at Hubspot or Morgan Brown your co coauthor of hacking growth at shopify. Yeah, we’ve talked to them. They have incredible tools.
06:08.86 Sean Ellis
Yeah.
06:10.93 Ethan Garr
Available to them to do just awesome experimentation if they don’t do this ahead of time they might be okay because they have those tools they have all the the infrastructure to support their experiments but especially if you’re getting going in this I You know I think this is ah just one of those simple things you can do to really improve your odds of success.
06:19.47 Sean Ellis
Yeah.
06:26.99 Sean Ellis
Yeah, so this is this is really a quick introduction to this topic and and as we as we talk about with growthsnack. We’re really trying to. Have this be food for thought that that gets the wheels turning so that you you start to look for for ways to improve how you and your company are approaching growth. But if somebody wants to go deeper and really dig deeper into this topic and and.
06:39.46 Ethan Garr
Um, yeah.
06:52.14 Sean Ellis
Think about specifically how how do I apply this where can they go for that information.
06:53.38 Ethan G
arr
Yeah, yeah I mean I’ve put a lot of thought around this and I actually wrote a short article on our shared website breakoutgrowth dot Net It’s called write the write the report before you run the experiment So easy to find if you go to our blog. But yeah, just in general I Just want to let our audience know that. Yeah, we created breakoutgrowth dot net for them. It’s a place where you and I share any thoughts we have that we think will help teams accelerate their growth Trajectories. So Certainly check it out there. There’s a lot more.. It’s a short article but there’s a lot more depth to this and if you have any questions don’t hesitate to reach out to me.
07:28.40 Sean Ellis
Absolutely so that’s our first grows snack. This is something that is is something we’re hoping to do on a regular basis as part of the breakout growth podcast but just ah, a short version of it I consider it. Ah.
07:44.47 Ethan Garr
Um, yeah. 07:45.66
Sean Ellis
Pilot so we really want people’s feedback on that and if if it’s positive. That’s great. We’ll keep doing it if if you don’t find it valuable I think that’s important as well. So you can reach out to us through Twitter or Linkedin send us a message or even through the breakoutgrowth net site. There’s a. Link on there to to contact us. And yeah, if the feedback’s good then we’ll we’ll start doing this every other week in between our our longer episodes and ah. And yeah, even as you as you reach out. There might be some specific areas that you’d like us to cover. So so let us know that in the messages as well. But thank you everybody for tuning in hey see yeah.
08:20.11