Data has become a central part of product strategy thinking and most product managers struggle with it. In this recent AMA with 280 Group, I got together with Marilyn McDonald and Ryan Cantwell to answer some of the most common questions on product analytics, metrics, and the craft of combining qualitative and quantitative data to make informed product decisions.

How do you get started with even defining KPIs? As an institution we know we need to do it, but we never seem to make that first step for change.

Establish metrics based on

1. User journey - All the way from discovery to conversion to usage and churn. Collect data on each customer touchpoint to understand the value delivered to the customer.
2. Business Metrics - Being able to map user journey data points to business metrics will help you ultimately optimize business outcomes.

We all know it's extremely difficult to get KPIs right, how do we ensure we're collecting the right data?

Use a framework for establishing metrics to give you a more holistic view of the data:

1. Combination of Leading and lagging metrics
2. Evaluate metrics as being Input or output metrics to make sure you evaluate levers and impact independently.
3. Avoid game-able metrics
4. Counter metrics to avoid blind spots"

If we spent time and effort collecting the data for a period of time but then found out the data is not as good as we expected, what can we do given we have limited time and resources and may not have the luxury of running it again before making product decisions?

Use a combination of qualitative and quantitative data to handle scenarios where you don't have enough data or trust in data. Even if you get some signal, no matter how minor, to help you justify continuing investment in the product, you can buy more time to get the right data instrumented.

What to do when data is not supporting the strategy which is frozen?

A number of companies struggle to be agile and sometimes it's also a challenge to remove bias in decision-making. Data is only as powerful as it is used to make impactful decisions. As product managers, even if we can't change the strategy, it is still our responsibility to point out risks and set expectations.

Is the smart (hard) bit, knowing what to measure/tag or the actual interposition of the analysis.

The smart bit is to be able to evaluate what the data shows and be able to poke holes in it to validate or invalidate various interpretations of the data. Building a narrative with data by looking at it from various angles that support the overall narrative is the smart bit.

Intrigued about what tools are used to gather, analyze and visualize the data?

Tooling varies from company to company, but lately, I have seen Looker be really popular. Tableau is also very popular but not with large-scale data. I have also tried Mode Analytics with good experience. You can consider using journey metrics, combined with biz metrics, and then API and predictive metrics to get holistic views - Amplitude, Mixpanel, Qlik for API info, PowerBI, Tableau, but tell a holistic story.

Any recommendations on a system or methodology for deciding what data to be collected?

Outcomes, not outputs! As Marilyn says, OKRs are a great way to focus on outcomes. Big fan of OKRs but you need to be cognizant of the metrics you need to "fully run your biz" - usually a set of metrics that define your biz and customer journeys (e.g. - Attract / Aquire, Convert, Monetize, Engage & Retain, Reactivate)

What data can be gleaned from Win / Loss analysis to help inform the roadmap?

Win/Loss stats give good insights into what is working and what is not in terms of

1. Competitive insight: Are we constantly losing to the same players, if yes, on what specific points.
2. Positioning Insight: Is our product positioning not resonating with the audience we are trying to target and could we change our positioning?
3. Pricing insight: Are the customers interested but not convinced with pricing? Pricing can be an important decision-maker for customers.

Are there guidelines for product analytics regarding what percentage of participation, or the size of sampling participating population to draw meaningful conclusions around user behavior?

A sample size of as little as 30 is considered statistically significant, In the consumer space asking 30 people may not seem like a lot, but in B2B settings talking to 30 companies is a huge undertaking. Even in the consumer space doing 30 user interviews can be significant. If you reach 100 people and 30 people reply, it's important that you get 30 data points, the conversion of a 30% response rate is not as important in this context.

With an extremely large portfolio (a very high-level number of SKUs), what are the typical criteria to get to a portfolio decision?

In the case of SaaS products, 3-4 SKUs are the maximum we have the luxury of presenting to customers, present any more and customers are overwhelmed. In a business like Shein, they invest in AI to generate near infinite SKUs. SaaS products try to make customers make decisions easily, Shein is trying to optimize for conversion as much as for browsing time because browsing is making customers feel they have choices and that brings the joy of shopping for them. Depending on the nature of the industry and the customer psyche in terms of need-based products and want-based products you can determine the optimum level of SKUs.

If we only have a very limited amount of data because the customers are not quite engaging or prefer opt-out of the data collection, how can we make good use of the limited data we have and avoid being skewed?

In scenarios where you don't have data or have limited data, it is important to build iteratively evaluating every decision for future scaling or pivot. Incentivized user research methods come in handy in these scenarios so that you can have a more nuanced understanding of the customer even if you don't have quantitative data. Combine this with market research and competitive analysis to establish a strategy that can reasonably scale your product to the next phase.

Have more questions? Leave a comment and I'd be happy to answer.