Hello all, and welcome back to Teach Me How 2 Product! I won’t help you dance, but I will help you make better products. I’m glad you could join us again!
We are continuing with the question from my very sharp brother-in-law at Uber:
How should you prioritize which experiments to run? And how should you choose KPIs affected?
And we have arrived at our middle section: picking metrics that ultimately determine success or failure of your business. If you were following along from the last post, that is this part of the flow diagram:
We discussed building a hypothesis with observations for your experiment and key results for that experiment. That felt super practical and applicable, so this might feel pedantic. Thus you might be asking yourself “Why is this important Ajay?!”
I’m glad you asked imaginary friend. The reason that I want to cover this is because you need to know the key metrics that capture your business engine in order to know how size the impact of stories against your business engine. Knowing this definitively helps you understand how the things that you are working on either drive the business forward or don’t. It helps you make those critical trade-offs in your backlog.
Now that we all know where we are, let’s talk about the key metrics that drive your business.
(I lied. One last disclaimer: this post is summarizing years of learning on this topic. It’s not capturing every nuance related to setting these key metrics. Think of it more like a primer so you can get started with thinking about this.
Okay, here we go!)
What about your product drives value for your users?
A place that is always, always, ALWAYS a solid default place to start with any product-related exercise is understanding what value you are driving for your users with your product. If you can understand that as fully as you can in the moment, you can drive the best possible outcomes with your product.
The business landscape is littered with dead companies that didn’t wholeheartedly focus on their customers and truly drive value for their customers. Your key metrics that represent the value you drive for your users are no different.
What is the value that drives your business today?
Next, you are looking for what is the value that drives your business. So, out of what customers really value in your marketplace, what also makes your business run today? What’s going to drive the things that your business cares about today?
There’s two key dimensions to this really: what does your business care about right now, and what will it likely care about in the medium and long term.
What is value for the business right now?
Here are some examples of things you might care about as a business right now:
- Conversions. This category of business drivers include things like product purchases, bookings, and successful purchase of services.
- Users. This is generally pretty self-explanatory. It’s about getting as many users as possible. You might define this different ways though, depending on what you care about. It could be things like visits, visitors, visits per visitor, etc.
- Engagement. Generally about how users are using your product, how often, etc. This could be things like visits per day per user, conversations/user in a chat app, retention of users over time, LTV (long term value of a user)…ya know? Real simple stuff like that.
What is the business value that will drive your business in the future – medium and long-term?
This is a more strategic set of questions, but directly related to how you might consider measuring the success of your business immediately. You really what to think about what matters to your business in the medium term and longer term. This gets to be really important for one key reason: you don’t want to pick metrics in the short term that box you in from your medium and longer term vision.
Some examples of things you might consider for the medium and longer term:
- Growth. At least in the medium term, you might need to focus on growing your business. If the key metric that you are going after hinders that growth, you’re going to have problems.
- Marketplace value. If you are running a marketplace, the value of new business operators getting their first transaction might be more valuable over the surer bet of sending that transaction over to an established player. If you opt for the latter, you could be eroding the potential value of your marketplace over time. It might be important to think about how marketplace value would force changes or adjustments to your chosen key metric(s).
- Increasing margins. You might be happy with your growth trajectory, and might now need to find ways to increase the margin on the products you are selling by providing additional value. If you anchor on a key metrics that disables your ability to do that, could be a problem longer term.
What is the intersection of user value and business value?
Okay, so now that we’ve figure out what our users value and what drives value for the business, what is the intersection of those things? The intersection of user value and business value is what I would call your business engine. It is what will ultimately drive your business into the future, whatever your future looks like.
Now, how can you capture the intersection of those things in your key metrics? What kind of metrics best represent the ultimate action or set of actions that generate user value AND capture the engine that drives the business?
If you can look at this objectively, you’ll be able to get a set of metrics that should best capture that intent and allow you and your teams to pursue the right improvements for your product.
How does user behavior ladder up to the chosen key metrics?
So let’s say you select a key metric or set of key metrics that best represents your business engine. Is that it? Can you declare victory now?
Well, dear reader, not completely. See, while you do have a key set of metrics, those metrics won’t tell you the full story. They don’t tell you about key behaviors in your user journey that would affect those key metrics but aren’t singularly captured in these few numbers.
So, surprise! You have to be diligent and do more homework. Did you think you could get away with this that easily?! Come on, this job is way harder than that!
We’ll get into some examples to better crystalize what the heck I’m talking about here. But, before that, one final piece of the framework…
Don’t be afraid to change!
Your business changes quickly, and your metrics should too. You are going to learn so many new things as you move forward, iterate on your product, and talk to your customers. You don’t want to be shifting your key metrics all the time, but I think it’s perfectly reasonable to understand the different pressures and changes on your business and changing course appropriately.
You should find a cadence of change that’s appropriate for you. That could be once a quarter, once a year, or once a month depending on what’s required and what you and your team are comfortable with.
I realize that all of this is a more complicated concept, so I’ll try to capture it with a couple of examples. Here is one out of my own professional life: Vrbo.
Example #1: Vrbo
Vrbo is a vacation rental marketplace where travelers can book a property to stay at from over 2 million properties that partners list on our marketplace around the world.
A short story before we dive deeper into the specifics of this example. Back in the day (like 10 years ago, which is really not that long ago but these days it feels like an eternity), Vrbo was primarily run as a vacation rental subscription business. A partner who owned a house could list on Vrbo by paying a subscription fee. You’d pay a certain amount of money, and list your property on Vrbo however you wanted. Simple and easy.
That subscription fee is what drove revenue for the business. And so, the entire business optimized around driving as much revenue as possible from subscription fees. Not a bad idea, right? Well, it turns out that this didn’t drive the best traveler experience. Listings didn’t always have the best quality, the traveler experience was under-invested in to make way for more owner experience investment, and thus product investment was out of balance.
The market also shifted. Travelers started expecting a better vacation rental experience, and the bar was set higher by other competition in the space. Thus, what really used to matter to VRBO was now not aligned with what the other side of the marketplace cared about.
And finally, growth in subscriptions was capped naturally by how many partners were available to list their properties on VRBO and the possible price offered. You could raise prices, but ultimately you could only squeeze a certain amount of juice out of that orange (or whatever fruit/vegetable you prefer.)
We at Vrbo went through a similar exercise that we’re about to go through, where we took a step back and thought about what our business engine ultimately needed to be, now and in the future. Here’s generally what that looked like.
Since we are a two-sided marketplace with two key user cohorts, we have to think about each cohort separately. Let’s do that here as well.
Traveler value: Everyone reading this post has likely been a traveler before. As a traveler, what would you assume you ultimately care about from a vacation rental marketplace that drives value for you? If you said booking and staying at a great property, that’s a great answer! That’s ultimately what all travelers care about and why you would come to a marketplace like Vrbo: to book and stay at a great property.
Why is the stay part important? Because it gets at a really important point: If you try to book a property and don’t get to stay at it because the partner on the other end doesn’t accept your booking or cancels it before you get to stay at the property, the value disappears. Critical that you have both then.
Partner value: On the partner side of the marketplace, what could you assume that partners value? In the world of subscriptions, a partner would likely want the most amount of value for the amount of money they are paying to list on the site. That value really came in the form of bookings for their properties. The more bookings you had for properties in question, the more money the partner makes.
Business Value: Immediate
In the immediate term, the business valued subscriptions and subscription renewals. That’s how Vrbo ultimately had money in its proverbial pockets to invest in the business.
As mentioned above, this was fine until you saw where the investments ultimately ended up: the partner side of the experience. Everyone in the company was over-invested in the partner side of the business, and the value travelers were getting out of the business was lower than expected, which was a vicious cycle in the medium and long term.
Business Value: Medium and Long Term
Why was this a vicious cycle? Well, if travelers stop coming to your marketplace because the experience isn’t as good, then traffic wanes, and partners don’t get as many bookings as they used to, so the value declines for partners as well, partners delist from the marketplace, which then erodes traveler value, and on and on it goes.
In the medium and long term, we at Vrbo knew that we needed to meet traveler expectations and provide value to travelers as well as partners to keep the marketplace in balance and healthy. This meant we needed to invest in the traveler experience in a way that didn’t throw things out of balance for partners. Fortunately, per the above traveler and partner value assessments, the answer was right under our nose.
Business Engine: Net Booking Conversion
The business ultimately aligned on a metric of net booking conversion that better captured the value driven in the marketplace writ large. Bookings were ultimately what drove traveler value, partner value, and value in the traveler service fee to Vrbo. Specifically, bookings and stays that actually happened, which encapsulated accepted bookings by the partner and cancellations by either party.
By re-aligning on these key metrics instead, we were able to rebalance the marketplace and invest in the traveler experience the right way. See, we changed! And the world didn’t come to an end! Cool right?
So now that we decided on net booking conversion as our key metric, what are the kinds of things that are important to that number that also matter? I mentioned some examples above:
- Booking acceptance
You can imagine other behaviors matter as well that capture how travelers submit bookings, like:
- Travelers coming to the site/app
- Travelers bouncing from the experience
- Travelers engaging with the experience by performing a search
- Travelers filtering by the aspects of the property
- Travelers viewing a property after they have viewed search results
All of the above metrics matter for traveler behavior, along with others that are best indicators of traveler behavior that ultimately matters to booking conversion over time.
On the partner side of the equation, you can assume there are other metrics that ladder up to accepted bookings and cancellations, like:
- Partners responding to traveler messages
- Partners accepting a booking (this should go up)
- Partners declining a booking (this should go down)
- Sending payment requests for a booking
That does it for Vrbo, an intriguing example. Let’s take a look at another example that’s not actually marketplace or e-commerce related: WhatsApp.
Example #2: WhatsApp
Let’s take a look at another example, WhatsApp. Beloved app, used all the time to communicate with your family and friends around the world.
Okay, so what is the value of WhatsApp to you as a user? Why would you want to use WhatsApp?
Well, to me the value of the product to a user ultimately is that you want to be able to communicate with your family and friends. You want to be able to chat or, per the above, send useless (but fun!) stickers, GIFs, and just generally keep in touch.
Business Value: Immediate
From a business perspective, WhatsApp is only as valuable as the people that are using it and how often they are using it. In the immediate term, it does seem pretty important that WhatsApp remain a staple in people’s lives. Looking at the company’s strategy, it seems pretty apparent that everything relies on the number of users using the app, and how engaged those users are.
So, short-term, we’re probably looking at making sure that WhatsApp can acquire and retain as many users as possible.
Business Value: Medium and Long Term
Where is all of that ultimately heading for WhatsApp? Doing some research, it does seem like monetization is on the horizon. A more obvious answer in that realm is ads, but WhatsApp could have pulled the trigger on straight banner ads a loooooooong time ago.
What seems more likely is that they will pursue a monetization strategy that involves their own version of mobile eCommerce. It looks like WhatsApp is already taking steps in that direction with business-supported customer support. We can expect to see simple things like stickers and more complicated commerce funnels like the aforementioned in the future. So what does WhatsApp need in order to make a more mobile eCommerce-based strategy successful?
If you guessed users and retention, you’d be correct!
Okay, now what’s at the intersection of the business value that the company is looking to drive, and the user value that users, well, value from WhatsApp?
In order for WhatsApp to pursue a strategy involving mobile eCommerce, it needs users that will stick around. Users want to use the app to communicate with people they care about. Thus, it makes sense to measure WhatsApp’s success in the following way:
- Daily Active Users. Pretty simple: how many users are coming to the app on a daily basis?
- Monthly Active Users. Again, very simple: how many users are coming to the app on a monthly basis?
- DAU/MAU ratio. Divide the two above numbers. This gives you some sense of how regularly users are coming back to the app.
- D1, D7, D30 retention. This number basically tells you how many users, after they first use the app, are still around after 1 day (D1), after 7 days (D7), and after 30 days (D30).
And what a coincidence! This is how WhatsApp reports the health of their business to Wall Street.
Now, is this the only way you would want to understand your users? Nope! That would be dumb. But this is likely how you guide the ship. If those numbers go up, pop the bubbly! If those numbers go down, sound the alarm.
If you were running WhatsApp, you’d want to know the above numbers every day, week, and month. You’d obsess about them. But, there are other things you’d probably want to obsess over to drive the above numbers.
There’s not enough time to go through all of them, but here are some examples:
- App downloads. If people aren’t downloading the app, hard to activate them yeah?
- Sign up funnel completion. Pretty simple: did a user make it through the sign up flow after they opened the app? If they can’t sign up, they’re going to have a hard time using the app duders.
- Number of friends per user. This gets at the core user value of the app. If your friends aren’t on the app, what’s the point right? Driving this number gives you a shot at getting a user to stick around.
- Number of conversations per user. Again, a good health indicator for the core use case. If users are starting conversations, likely they’re getting use out of the application. If they’re not, warning sign.
Wrap it up Ajay!
Alright, we could spend hours on this topic, let’s call it for today. Great job sticking with me on part two! That’s how you think about setting key metrics that represent your business engine.
Next time we will round out this conversation with the final topic: how to use the experimentation framework and key metrics that drive your business engine to synthesize other dimensions that help you prioritize your backlog and roadmap. See you in the next post!
- This is the second post, feedback welcome!