Naive metrics 📈 😶 ☁️
The maze of reasonable metrics, getting the timing right with your next release, and being more intentional about an early and common startup killer.
Welcome to the 66th edition of The SaaS Baton. A fortnightly newsletter that brings you three, hand-curated pieces of advice drawn from the thoughtful founder-to-founder exchanges and interviews taking place on Relay (curated with 💛 at Chargebee) and the interwebz. So, stay tuned!
In this edition, you’ll find the following instructive and inspiring pickings:
#1: eesel’s co-founder, Amogh Sarda, details how well-meaning, “obviously important” metrics can, without a system, muddle what’s happening, what’s intended, and what’s required.
#2: Correlated’s co-founder and Head of Product, Diana Hsieh, shows how product efforts draped in the finest frameworks of roadmapping and prioritization amount to little if one doesn’t understand timing.
#3: Referral Rock’s founder and CEO, Josh Ho, urges founders to approach the matters of aligning with co-founders/partners with great deliberation and lists questions that can be pondered together.
🗞 Recently on Relay:
Heuristics and Hunches — Starting with the Price, Making (Hard-Fought) Peace with Platform Risk, and Giving Up Adderall Superpowers with Rodeo’s Co-Founder, Ben Fisher
Nonetheless, the reason why I decided to still build Rodeo on top of Shopify was because I really understood the platform. I get, probably better than most, what Shopify cares about and how they make decisions. To add to that, I had built a deep network of merchants.
#1: Naive metrics
(From: eesel’s Amogh Sarda) (Source: The Paperclip)
I'd wager that most folks start naively. You care about user or revenue growth, and perhaps knowing if you are default alive or dead. That's a good start, but these are output metrics. They aren't actionable on their own and you need more granularity to investigate why growth is the way it is.
So you start looking at things that are interesting or feel obviously important - daily active, daily uninstalls, monthly active, retention curves, power user curves and so on. Not too far in, things get messy.
1. Premature focus creates blindspots
It's dangerous to bring tight focus to a small number of metrics, before you have full view of the business. You could be stuck with implicit assumptions, but have no way of challenging them. There could be a grave problem or big opportunity outside your observed view.
A cautionary tale about this is covered by Casey Winters in this post. Both GrubHub and Seamless do food delivery. Seamless equated revenue for Gross Merchandise Volume (order size and number of orders), while GrubHub viewed revenue as GMV multiplied by the average commission.
A subtle difference to include "commission" in the system drastically changed how the two companies optimised for revenue. For example, while Seamless search results sorted restaurants in alphabetical order, GrubHub sorted them in order of commission earned.
When the two businesses eventually merged, Casey recalls that "this was one of the first changes that was made to the Seamless model as it would drastically increase revenue...even though it didn’t affect GMV".
Similarly, say you read this classic Superhuman post and decide to track retention for a tight cohort of users. Even if you decide to ignore all other users, you need a system in place that acknowledges these ignored users and spells out your assumptions explicitly.
Your assumptions to bring focus could be wrong, there could be some hidden problem (e.g. you need to drastically improve the quality of acquisition), things could change with time (e.g. your ignored users become a good fit) and so on.
2. It's confusing to have lots of metrics
If you just come up with the most intuitive set of metrics for each particular context, things can get confusing. You can end up with many similar things measured slightly differently, and no clear sense of how things relate. For example, you could be looking at churn rate and Month on Month engagement for one feature, but Week 4 retention and Week on Week engagement for another.
Without the guidance of a global system, local definitions will keep piling and complexity will compound. For example, fundraising will introduce concepts like burn and cash balance, which could be tracked differently to revenue.
3. Non obvious definitions introduce mental burden
While some arbitrary definitions are unavoidable, each time you do this, you introduce some recall for your team and future you, and this can be quite disruptive. Things can make total sense in context, but not when you zoom out or come back later.
For instance, in eesel, "disabled" users were users not seen for 30 days. Even if we knew this by heart, we would still have some friction when looking at a visualisation of "daily disabled" - "ah this is a laggy metric", "ah it's 30 days not 28 days".
[You need to first make a map - a system of metrics - before you navigate your data.]
#2: READY and ABLE customers
(From: Correlated’s Diana Hsieh) (Source: Startup Monologues)
So when thinking about product management, it’s not just about the tactical “jobs to be done”, it’s also whether or not your customers are ready and able to do the job.
The problem is the art of turning a “problem” into a “solution” skips the question of whether or not the problem is ready to be solved in the first place! So before going ahead with building a larger product line, make sure you ask yourself the following…
Are your customers READY and ABLE to use your product?
What factors go into READY and ABLE?
- Market dynamics - is what you’re working on becoming a trend? Don’t underestimate the power trends have in providing rocket fuel for your product
- Customer “ability” - do your customers have the level of training they need to get the most out of your product? A great example of this is something like Webflow. That product, despite being “no-code”, is pretty complex to use. Does your target persona have the ability to use it?
- Customer sentiment - do your customers think what you’re working on is critical? Do they need a solution now, either to compete with their competitors or to fuel their business?
- Customer “state of the world” - does your customer have all the requirements needed to be successful with your product? For example, if you’re building an analytics product, do they have the right data warehouse setup?
It’s possible to try to create the above on your own. Maybe you can create a market dynamic via brand marketing. Maybe you can train customers so that they know how to use your product. Maybe you can convince customers that they need what you need now with stellar positioning and sales people. Maybe you can build the “state of the world” you need to get customers to success.
But with all startups, it’s about time, money, and talent. Does you team have the time, money, and talent to get there?
…
Now an important thing to note about “Ready and Able” is that different customer segments are ready at different times. Early adopters will be ready to adopt something when lagging enterprises are not. What is important is whether or not you believe that the cohort of “Ready and Able” customers is large enough to go after.
So next time, when thinking of whether or not to build a feature, don’t forget the importance of timing! Your customers need to be both ready and able to use your product, and if they aren’t, it’s an uphill climb to get them there.
#3: Magical alignment
(From: Referral Rock’s Josh Ho) (Source: Founder Mojo)
Bad partnerships often kill startups, but great partnerships help the business go further than you would alone.
…
It’s hard to get to the root of alignment without a conversation.
These questions could help keep partners on the same page up front and help establish some agreeable ground rules + acknowledge the potential imbalances where partnerships go badly.
Questions to ask early in a partnership (ideally before officially signing as a partner)
- What types of tasks either partner never want to do (customer service, sales, manage people)
- What are expectations on exit scenarios? (timeline and amount)
- How will we fund the business? (raise funds, bootstrap, combination of both)
- What happens if one partner contributes cash to help the business?
- What happens if one partner needs to take a salary and the other partner does not?
- Do both partners value their time equally?
- What happens if one partner is committing "materially more time" to the company?
- What happens when a partner wants to leave?
- Do you enjoy the subject matter space?
- How do you view your standard cost of living expectations and increases over time? (i.e. what’s the minimum you need to get by)
- What will each partner be responsible for? Key responsibilities and expectations (day to day)
- Who manages accounting/money management?
- Who manages service/front line employees (regardless of if you think you’ll need them)?
- Are there specific quantifiable assets/IP either partner is bringing to the company?
- How do you best take feedback? (in person, written)
- When is the best time to give you feedback?
- What happens if there is a deadlock on a specific key decision (product direction, taking an enterprise client…)
It’s easy for these questions to feel very personal, it may be best to answer them independently in a written format so emotions can be less of a factor.
…
Don’t rush into a partnership and try not be blinded by shiny things or “go to the grocery store hungry” (i.e. making a rash decision because you are desperate for someone else to bear the weight).
- It’s easy to be infatuated by someone’s experience, connections, great story, personal charm, or successful track record.
- It’s easy to "default” to thinking you have shared values (where as much as we try, we truly only have one point of view of the world).
- Be aware if you have this feeling of “could I be this lucky” or “I don’t want to spook them by asking personal questions”.
Ask them! Get aligned and get to that magical feeling!
🤝 Founder social:
Until next time,