Posted on
July 1, 2022
Expert Panel: Tips & Tricks To Measure Product Success
You Can’t Improve What You Don’t Measure
You can’t improve what you don’t measure. And measuring product success starts with having a deeper understanding of “How are my customers using the product?” Without a complete picture of the customer journey, you’re operating in the dark regarding improving value for your customers and, ultimately, metrics like engagement, conversions, and retention that help grow your business.
The following is a summary of a panel discussion to give you tips & tricks on using analytics to measure product success. You can watch the entire panel discussion here.
Meet The Panel
Steven Fitzsimmons
Freshpaint, CEO and Co-founder
Freshpaint is a HIPAA-compliant data infrastructure. Freshpaint helps you implement analytics tracking, and integrate growth and marketing tools on your site, or in your app, without code.
Mapolo Buessing
Osmind, Director of Product
Osmind is a mental health EHR focused on measurement-based care.
Milan Thakor
Cold Start, Director of Product
Cold Start is a venture studio focused on digital health businesses and marketplace platforms
Questions About How To Measure Product Success
Q: What are some obvious questions that people should be thinking about as they start to build out their tracking infrastructure?
Mapolo
Yeah. First off, product management, I think most of the time, you're kind of flying by the seat of your pants. You don't really know what you're doing so the more information you have, the better. I always like to treat it, when I'm trying to roll out analytics at a place, treat it like a product. Ask people what they want to know. "Do you use your research within the organization? What questions do you have that you don't have today?"
It seems obvious, but a lot of people just jump in, and they're like, "Yeah, we need to track everything." I think you don't have to track everything, and you should figure out what people care about, what open questions are, and be pointed in how you start attacking building an analytics platform. That, to me, is the most obvious question. Yeah, in product management, again, you're making all these assumptions, and you want to use analytics to eliminate those assumptions. Those are the obvious questions for me.
Milan
I think when I actually start, certainly from a product lens before I think about analytics, we think about how do you serve the users and the customers of your product and that everything should try to boil up to that. As you create unique value for your users, you want to make sure that you understand whether you are creating unique value for those users, and that's basically would be how I would think about prioritizing the analytics that you care to implement.
Then as it comes to your stack, you want to think a little bit ahead of like, "Hey, what are all the things that I really think I want to impact to my users? What am I going to need to measure to get there?" It'll help you prioritize. I'm basically a minimum viable analytics stack, which is not so different from prioritizing a minimum viable tech stack, and not so different from prioritizing a minimum viable product strategy. I think if you start from that lens, you'll not overbuild early and you can kind of build as you need.
Q: What does setting up the infrastructure look like?
Mapolo
I think it's important to just understand what you want. I would even start with like something simple like Hotjar if you can implement it. Just get a baseline about what's important, what are users doing? Then you can answer deeper questions. I think analytics requires a certain amount of volume for it to be useful. If you're a brand new startup and you have three customers, analytics are kind of pointless, you just need to talk to those users, you need to see what they're doing. I would start there and then start building your mental model about what's important. If you're an e-commerce business, you probably care a lot about funnels and you're going to want something that's more product, data analytics focused.
You could go very database heavy where you're just interested in a certain few actions and just tracking that growth of that over time. Yeah. I think I would try to start small. One thing that I've discovered is the quick and dirty tools are the best tools to start with. Now there's so many tools out there where you just drop in a snippet of code and you can immediately get some insights and start getting some direction. I would always start there and then start building out what you think your product needs to be over time. Eventually, you might have big data warehouses and data science teams and all of that, but that's how I would start, is something very quick and dirty.
Milan
Yeah, I can chime in here. One thing that I would think about a little bit on this front. Obviously it depends on the stage of product. If you're running a spoke test, doesn't matter as much you can can talk to users as you're building a more fully fledged software product. I actually like to start a little bit more with what you'd consider in a larger setup or company, a data engineering team, and likely in a smaller setup, just whoever the core development team is then really think about how data's structured up front. If we go through that first prioritization of who are your users? What do we really care about impacting for them? You start to build your understanding of what that MVP level of what you need to track is, and think about making sure the engineering team understands that really well.
I think if your data's not structured well at the way that it's actually written and organized, nothing's really going to matter after that. I think it helps align the team really well. It also makes it really nice really early on whether you're thinking about a CDP, whether you're thinking about a data warehouse. If your table structures are accurate or the way that you're tracking things on the web and have a funnel set up, you'll be in a much better place. I think step one, I would really think about engineering team development structure, both your back end and your front end and think about what are the tables I'm going to really care about?
If you can be a PM that can ask even that question up front, it will let your engineering team understand that you can speak together about the same thing and then lets you pick downstream tools, which I would think. Probably the things that I think about, I think some of them Mapolo mentioned, is really great. I love the idea of the Hotjar, FullStory from a web perspective, and then there's great tools for basic funnel analysis that are free and great, and the Amplitudes of the world have done a great job there. Once you go to that, then you can start thinking about your data warehouse, your CDPs, your deeper tool set. But I think if you start at the top, it'll be really, really helpful, no matter what tools you choose and plug and play in and out of.
Mapolo
Yeah. Plus one on even just something simple, like setting up a taxonomy, makes a huge difference. I know I have my colleague Dan in the room and we've been rolling our eyes at some of our date formats in our database and how hard that makes stuff.
Steven
Yeah. I see at least with founder friends that I know, and I'm an small angel investor in some healthcare companies as well. The best thing that I've seen early, early days is when you have a handful of users, just like what Mapolo said, is use a session replay tool. Something like Hotjar, FullStory, something like that, even just talk to your users, and then as you start to scale, then the quantitative analytics becomes a little bit more useful, but even then, don't stress about the decision of, "What tool should I use? What's the differences between Mixpanel and Amplitude?" It doesn't. The more time you stress about that, the less time you have to build a kick-ass product and pretty much any popular tool these days will most likely solve your needs, especially in the beginning. Yeah. Spend your effort and resources and brain power on your product and not your picking a specific vendor is probably the best.
Mapolo
I will say that the screen capture gets a little tricky in healthcare. I know you're all nodding your heads. Yeah, I've used it in other roles and it's really nice, but sometimes it's hard in healthcare.
Steven
Yes. That is one thing that you should spend brain cycles on when selecting a tool is making sure that you're keeping patient data, patient information, sensitive data safe and protected. I think that's everybody's duty here.
Q: How do you think about understanding what your users are doing in the product?
Milan
Yeah. Maybe I'll start with a general comment and then we can break apart and would love to have Mapolo jump in. Again, if anybody has thoughts and things that are working or questions that'll be more helpful, let's try to make a discussion there. I think if we're talking about at the earliest stages, we're talking about either a brand new company or still a young company, we're talking about the company level. I think almost regardless of what we're talking about, we're going to talk about things that are focused on growth. We're going to talk about things that are focused on engagement, and we're going to think about things that are focused on monetization. Obviously, there's a huge amount of asterisks depending on your business. If you're a pure content business, maybe you don't care about some of these things.
I think on the consumer side of framework that I really like is the pirate metrics set up. If you don't know that, I think it's Dave McClure from back in the day, but it gets used a lot, and basically, lets you see everything from the top of your acquisition funnel through activation, retention, referral, basically this entire pathway. You can look it up. It's called the pirate metrics, AARRR, something like that, and it's really nice. What I try to think about early on are two things. I want to make sure that my customers get value. That the users will get value out of your product is the most important. Usually that's going to be something downstream that's engagement related, and it's a little bit more of a downstream metric to track, but you want to know that's worth tracking.
I think almost always I think about that as some version of an engagement metric in healthcare, especially with again, asterisks. I have, relatively recently, built a remote patient monitoring business in the oncology space that was done with Redesign Health. Of course in that kind of setup, what you're looking for for any remote patient monitoring is, are people monitoring in whatever way you've set up your remote patient monitoring setup. That is really important to know what your downstream impact metric is. Whether it's an outcome, whether it's an engagement metric, I would think about that, but it's often also harder to get a read on early, which is why you want to then move up the funnel. Don't ignore that deep downstream thing. It is important. You should get set up to be ready to understand that in whatever way you can.
Then I would usually move to making sure I get earlier signals. You're looking for things that are going to be quicker to get a directional read on. Usually that's things that are little more early funnel, whether it's an onboarding sequence that you have, whether it's patient profile setup, whether it's a partner integration, again, a little more BDC focused right now, but those are the kind of things I would start to think about that help you get an early signal, whether your product is even meeting the interest, if not the engagement, of your user base. Maybe I'll pause and, Mapolo, you want to talk about anything more there or the B2B side?
Mapolo
Yeah. I guess I could just talk a little bit about when I joined Osmind, because I feel like this was the question that I was trying to answer for myself, was why? It seemed like we already had some product market fit when I joined and I was trying to understand what our customers are doing in the product. EHRs tend to end up being this massive expansive features. I had this hypothesis that people only really do like two or three things 90% of the time. I wanted to validate that. That's where I started was engagement. Then I wanted to validate what I was seeing from sitting in on sales calls and what I thought folks cared about with how they were actually engaging in our product.
Yeah. Just quick and dirty, figuring out, set up a few events manually through Freshpaint to figure out, "Hey, they're writing notes, their scheduling appointments," just very surface, high level, and to really sign as a signal of what's important, where should I focus the product development team. I tend to double down on the things that they do the most, that they care about the most, the reasons why they're purchasing their product as opposed to keep building new shiny features, the long list of EHR features.
Q: What are specific tools in your stack that you think of as when you want to take action to engage customers?
Milan
I think as you were talking about it in your recap, we probably haven't gotten into specifics and it's going to be on a case by case basis, but talking about which exact metrics you want to be tracking is really important, especially as a PM, going into every product brief and every product plan, understanding both downstream metrics as well as leading indicators, I think will help you, one, separate which those two things are and then make sure that you are actually building with your team the proper, whether it's an event, whether it's the right data that needs to be tracked for that to be able to be able to measure it. I think it's not exactly what you meant, but the word "action" made me start thinking about a dirty and forgotten part of that analytics stack, we might call it, which is actually things like CRM notifications, messaging.
Basically, can you think about the ways in which you would want your product to improve as an experience based off of what a user does or what data is collected? I think that'll help actually bring this viscous list of analytics requirements into something that feels more product heavy. For all of you folks that are very product heavy, it'll make it feel a lot more obvious. It's like, "Great. When a user does this action, I want a notification to be sent. When a user has entered enough data, this is where I want the product to start to show it's power, because we now have a sense of, in the case of a remote patient monitoring company, we have a sense of some indicator of your health that we are now able to have enough signal on."
If you can do that, you're basically building, using your product roadmap to identify, why do we need analytics? Because we need to track the right things to make the right informed decisions for users, not just for you to look at on the outside as an analyst and tell somebody to make better decisions in the future. That's something that really came to me in terms of action orientedness, is that literally playing a product strategy based off of the data that you collect from your users and they can literally be triggering actions within your product and that'll help you. I think that'd be very, very fine tuned about what data you're collecting, how you're collecting it, and how do you utilize it in the future. CDPs are a great, great tool to support that, obviously, if this is a promo video.
Mapolo
Yeah. I think similar to me, when I read that question, it's messaging. Often you have, when you look at your analytics, you just have more questions and often I'm just using the analytics to identify who I want to talk to. I don't know, an Intercom or some way of messaging with your customers using that data to reach out to them I think is a must have. Yeah, maybe it's an in-product notification to drive more engagement or more adoption in something. Sometimes it's just for user research being like, "This is really weird. Why is someone doing this?" Or, "This person's not using it. This person is. What's the difference between the two?" Just being able to email them or send them a chat or whatever.
Q: How would the analytics needs differ between a feature focused product manager versus a growth product manager?
Milan
I'm happy to take the first. This one is a start and the core experience first growth. Once you're at this stage, the general differential usually, if we are like defining growth, PM and growth teams, it obviously varies in the industry. I think I really like a definition where it focuses on either acquisition or engagement, meaning you're not focused on the core experience and delivering core user value. You feel like your company has some version of product market fit and there are teams that are definitively focused on improving that. Just to separate my understanding of that definition and I'm happy if people have added or better definitions, but it's really, you're focusing on purely one of those two sides, acquisition and engagement as primary growth levers. From that standpoint, I think at scale, the most definitively different thing that comes from that, is that you are running tests to improve your growth.
When I say tests, hopefully at scale means AB testing and experimentation, and thus, that adds the most interesting tool that you don't need early in your product stack or your data stack, but is an experimentation tooling kit and tooling setup. Ideally, you want this to be integrated with your analytics product, but likely you'll be using a separate analytics platform at scale. There's a lot of things that are, everything's kind of melding, but that's, I think, the biggest skillset difference for a growth PM to start thinking about, is how do you set up tests that give you very clear and measurable impact to your acquisition or your engagement efforts. There's lots of ways to do that, but generally you want to be very clear about a very specific metric that you're moving, and then focus on an experimentation product strategy, and experimentation tech stack, and a experimentation tooling set up that is often separate, but related to your data stack.
Mapolo
Sure. Yeah. I touched upon the growth side, maybe a little bit more. I think on the, I don't know, feature PM or... I think it's more around engagement. I would say it's more focused on retention. You're trying to identify the needs of your customers and maybe being a little bit more thoughtful about building maybe larger features. Sure, you should be doing some AB testing, but maybe it's a lot more about doing user research, trying to identify the needs of the customers and build stuff that's going to keep those customers happy and retain them for a long time. A lot of times I would say you have maybe more established metrics that you're trying to drive. Yeah.
Q: Question from another product leader. Every time she rolls out a new feature, she asks engineering to make sure that all the things that she's going to care about in the future are tracked properly. Is there an easy way to go about this or is there a service to put a pixel in place to capture all data without creating custom data layers?
Steven
I promise everybody, Sarah's not a placement in the audience, but this is partly what the company that I started, that I run, Freshpaint, is what we do. You just install a single JavaScript snippet on web, for example, and we will automatically instrument your site or your app for data collection. We'll collect all your user and all your user behavior kind of up front and then allow you to access it later on, so you never are in a situation where you're like, "Oh shoot, I forgot to track that thing and I want to send that to my warehouse, or Intercom, or Mixpanel, or whatever."
Mapolo
I think the automatic tracking is awesome, but if you have no things that you really care about, call them out when you're in, as part of your engineering flow. You want to know that data. There's limitations to auto tracking. Especially if you're trying to drive engagement, make sure you call those things out, and have your engineering team implement them it should be part of the product development process.
Milan
Yeah. I think a manual version of it that I really like, I'm just getting ERDs for new parts of the product that are being built is really helpful. It's usually pretty static and there are tools that help automate it and you can look at it, but, again, looking at your database in a cleaner way. I just open up the chat for the first time. There's a lot of chat on just try learning SQL and be better at mocking around, writing queries that don't crash your database and then implement something so that you can't crash it, when you're querying something like Mode. I don't know. I think that's a great way to do it that's at every level. We're not talking about scaled large, large scale companies. I really love the discussion here. I think everybody should take the SQL class if you don't have SQL experience, and even if you do, probably can be a lot better at it. I certainly can be, and everybody can be.
Mapolo
Yeah. One thing I would add is make sure you work with your engineering team on what is your definition of done? For me in past roles, what I've done is said, "Hey, we can't define this feature as done. It's not shipped unless I have... If I know what the tracking that I care about, what questions I'm going to want to answer from this, how many users engage with this thing, we can't call it done or shipped without the tracking set up for that thing." Just make sure you're very clear with your engineering team on what that stuff is, what the requirements are there. I think, like what Milan was talking about, in ERD, a requirement stock is a very easy way to just force that. It's like, "Hey, these are the requirements. It's written down, you can't call it done until all these things are done."
Q: Thinking about early stage startups, what’s the one thing you can do early on related to personalization that can have a big impact?
Milan
I'm happy to start here. Especially for early stage folks, one framework, and then one specific, I guess, tool and product function. That framework, I really like Nir Eyal. He's a great writer on growth and product strategy, and the broad consumer world. He has a model that's called the hooked model. It's basically a model that's focused on triggering a user, asking them to create an investment, which, of course, asks them to take an action, that's an investment from them, and then you get a reward. That's this four part model that basically flows into itself where you do something, but you ask the user to do something, and for that investment, they better get a reward. I think that's just a really great model to think about for product building in general.
It touches on this idea of personalization, because if you can build a product that shows value anytime somebody does something or pretty soon after, and if it doesn't do it on the first time, then you better be telling them, "Hey, two more things, and then we provide a lot of value." You meet a user's expectations and you drive this growth loop, then the easiest way early.
I kind of touched on this in my last answer. I have a background in engagement, so I'm a little bit biased here, but would be the notifications universe. I think it's very straightforward, simple to develop, does not require a lot of development resources or generally great third party tools that plug into other great third party tools that manage data. Whether you're thinking about the BRazas, the HubSpots, the Intercoms, there's lots of ways to trigger messaging to users that shows like very clear personalization, and it takes very little to do that. There's lots of ways to get personalization variables and attributes into those systems. There are some great tools that do that effectively, automatically for you and other ways that you can set that up. That's the tool and function I would advocate for, think really early about that, and it shows personalization value without having to spend a lot of depth resources and thinking power.
Mapolo
Yeah. I love that. I think the other thing, when I was thinking about with personalization is kind of on the marketing side, or you're upfront, you could have the same product and just position it different ways and change the expectation for the user coming in or based on where they come in. Yeah, you could use Intercom or something to give them, ask them to do a couple... Basically, onboard them differently and set them up for success. That's the other area that I've seen really successful personalization, and you can identify. Yeah. You can optimize those inbound funnels and acquisition funnels based on your buyer persona.
Q: Has anybody benefited from, or any courses you can recommend for leveling up on analysis or just general data skills?
Mapolo
Yeah. This is tough. I did a lot of data stuff in college, so I feel like that's where it came in, but I think there's good stuff out there. I saw Reforge thrown out there. They have really great material. Yeah, they're pricey, but it's... I don't know. They have great material, so I would check them out. You could take some basic stats class at a community college, if you really want to get into the ins and outs. My other suggestions is, I know there's this, Amplitude has this big workbook that they put out early about mastering retention. I think it's free. It just costs you your email. I learned a lot just reading through that book and kind of going deep in there. Thinking about cohorts, and retention analysis, and day one versus day seven, and yeah. Super useful for establishing a baseline.
Milan
What is it that trying to do that you're not able to get right now?
Attendee
I also work in a consumer product. We have a mobile app and there are lots of different items that users see and interact with every day. We launched data labeling contests with different medical data. There are so many factors in a contest. How long is it running for? When did it start running? What is the content? What is the setup? How does the leaderboard pay out? There's so many different variables that it's hard to get signal from the noise with just regular regressions and just pulling SQL information out. It's the kind of thing that you would ask a data science or a product analyst to look into, but we don't have one.
Mapolo
In the absence of a data scientist, you're going to learn that, I think... Yeah, that Amplitude: Mastering Retention, I think sounds like it would be pretty useful.
Milan
No. I think that's a great challenge and it's good to have a very clear use case and problem. I was just curious if it was personal learning and yeah, I think there's... I actually think Mapolo's suggestion around statistics is a great standpoint. Data science is a title that's like five years old, doesn't really mean anything except you know stats and write in a Jupyter Notebook.
That's fine. I think that's fine. It's really good, to me, here that you have a very clear use case. You can decide on what the tool and skillset you want. It doesn't matter if you're doing it in SQL or in Python or whatever, it matters that you know what you're trying to answer. I think if you can be clear about that, then you can go and ask people that are really good at what they're doing. Probably go ask data scientists on forums and they'll help you. You don't need to have them hired and then you can learn it yourself.
Q: In an early stage startup, how can you use quantitative metrics instead of qualitative interviews to really figure out what users want when you do not have a huge user base just yet?
Milan
I'm happy to start here. I've been with Redesign Health and Cold Start, which are two different venture studios. We spent a lot of time spending probably too much effort researching a concept before building, which is, honestly, not what my recommendation would be, but it is a great learning process. One of the things that I just would call out really early is quantitative user research exists and it's generally based if you simplify it to some version of surveying really well. I'm not an expert on user research, but it's been great to see user researchers who will talk to thousands of people or hundreds of people. That can be clinicians, it can be patients, it can be folks in industry, and you can get good signal and learning about what matters and that is quantitative and it's without literally having a product.
My point would be that can just start really early for talking about quantitative measurement. It is a very real and statistically valid and academically validated way to build perspective on what is important for people. It is also just talking to them, but it's talking to them and measuring their answers with data.
Mapolo
Yeah. If you can find forums or groups that you think are your target customer and offer some incentive and survey them or talk to them. Totally worth it. Yeah. That's good data. I think if you have five people coming in your funnel, you should track those five people, but I don't know if you're going to get anything.
Milan
I will put a plugin in the PMF side and in the talking to user side for what's called the PMF survey. I think it's the new alt NPS, especially for very early stage companies. It's a survey question methodology that tries to get at the question of, do you actually have product market fit or not? Which is a very esoteric thing in the webs to say to investors and to say to yourself, and it's a great question that is data backed the way that NPS is data backed. It has a methodology. It's called a PMF survey. Product market fit survey question. You should, you can Google it and find it. It's a good framework. I think it's from whatever that Superhuman email tool, those guys, I think it's their framework.
Steven
I just dropped a link here to another session that's very similar to this and is basically a how to guide on Mapolo's journey. He's now director of product at Osmind, but before he was Osmind's first PM. Mapolo basically just dropped a bunch of knowledge bombs on a Zoom and it's recorded, and it's basically how he went from not having any analytics or any telemetry, zero measurement set up to how he basically like earned some small wins, convinced the team that, "Hey, look what I can do with these small wins," got the team to plow more resources into setting up his tracking and analytics, and then I think that's the reason you got promoted, right Mapolo?
Start With The Customer
Measuring product success starts with having the goal of adding value for your customers. So what questions do you need to answer to improve your customer experience? Use that as the focal point in building out your analytics infrastructure.
If you’re trying to build a more unified view of the customer journey but pulling engineering off customer-facing features to do it isn’t an option, then Freshpaint can help. Create a free account to see how our no-code visual editor enables product and business teams to configure customer events and send them to the tools they use to analyze and engage customers.
.jpeg)
