AI Product Manager Interview Question

Prepare for AI PM Interviews

Prepare for AI PM Interviews

We will keep on discussing AI PM Interview Questions asked recently → If you want stay Updated, you can consider Subscribing

Question—Meta recently launched a Gen-AI based chatbot, and you need to measure its success.

Yes, this is an interview question recently asked in a Meta PM Interview.

But this question is super important for any PM interview which are happening nowadays, as the Gen AI based Chatbots are increasingly getting built day by day.

So, how would you go about solving this?

Crack PM Interview like Top 1% — First Principle Thinking

Understanding the Product

First, we need to understand about this Chatbot, because until and unless you understand the value Proposition of this Chatbot, how would you go and identify the success metric correct?

So the first thing that we need to clarify with the interviewer is that,

1. What is the Goal of the chatbot? Is it a Support chatbot for Creators or advertisers? Is it a discovery chatbot which is helping users discover different things like posts, people, videos, and communities on the platform? Or is it a shopping or transaction assistant, like Instagram has Instagram shops, so these shops help in purchasing things from Instagram?

See the success, or the metrics will depend on what this chatbot does? I hope you are able to get this.

Let’s assume that this chatbot helps in discoverability on the Instagram platform, like Posts, people, Shorts, etc.

2. The second question we will ask — Is this Chatbot integrated across all the Meta applications, or is it there on a very specific platform like Instagram, Facebook or WhatsApp? Let’s assume that the Meta has first launched it for Instagram, and soon they will expand to the other applications as well.

High-Level Strategic View

Now that we have an understanding of the core value of the Chatbot, let’s think about why Instagram built this in the first place.

So if the chatbot is helping people discover relevant content on the platform very easily, discovering new groups, interests, and people, they may end up spending more time on the platform and strengthen the vision of connecting people or the social graph. This is beneficial for Instagram overall.

You should always talk about this in an interview. If you want to learn how to crack a PM interview like the top 1%, You can join our Next COHORT. This is the only cohort you will require to become an expert in answering PM interview questions.

So now that we have understood the strategic view, it’s very easy to define success. The success here will be that People are successfully discovering content, communities, short, etc, correct?

So, what do you think the Northstar Metric would look like?

North Star Metric

Is it active Chatbot Users?

If your answer is NO — that you are in the right direction.

Why Active Chatbot is not a Northstar metric because it doesn’t capture the success here. You can gamify this metric by putting it in an overexposed position. But it can happen that people just getting into chatbot but not engaging with it just drop off.

Is it NPS?

NPS is a very important Metric, but it doesn’t tell you the adoption and depth of engagement of the chatbot. So this also shouldn’t be the North Star metric.

What should it be?

Next # Weekly Users finding value from Chatbot — Yes, this can be the Northstar Metric, but here we have to define the Value.

This metric is capturing both Adoption and Success.

How, let’s think about the user journey. A successful User journey will look something like this

Users open the chatbot → They start engaging with it or chatting → They get some response, and if they find it relevant → they will click on Post, Video, Community, that’s the real success.

Crack PM Interview like Top 1% — First Principle Thinking

Ask yourself, if you are chatting with a chatbot, asking something and getting a response, but it is of poor quality, what will you do? You will not click on the response that the chatbot gave in terms of Community Link, post link, Shorts link, etc. So the success would be when you find the response from the Chatbot Relevant and you further click on the response, etc, etc. That’s the success or the value.

So I hope you are able to get this -

Caveats of North Star Metric

Should it be # Weekly Users with Successful Discovery via Chatbot

Or

Should it be # Successful Discovery per week via Chatbot

Both the potential North Star Metric -

Let’s understand the caveats for both of these metrics -

# Weekly Users with Successful Discovery via Chatbot

This metric can be less sensitive to the incremental improvement you have made in the discovery. Imagine you have made the discovery more seamless with some feature, but these metrics will be less sensitive as the # of users who are already doing successful discovery will continue to make successful discovery. So the incremental aspect may not be captured here.

Talking about another one -

# successful Discovery per week via Chatbot

This doesn’t tell you unique users who are finding value from this; a small set of highly engaged users can inflate this metric, correct?

I would go ahead with # Weekly Users with Successful Discovery via Chatbot, and complement it with # successful Discovery per week via Chatbot

Because it depends on the broad goal, since Meta has recently launched this Chatbot, they would want to improve discovery across a large and diverse user base, that’s why I am going ahead with the first one.

I am telling you because you will get asked these kinds of questions in the PM interviews.

So you need to prepare holistically and always think from the first Principle. Gone are those days when you can get away with the Frameworks.

We have recently created an entire course, primarily focused on First Principle thinking. This will help you in developing your thinking Process.
After this, you would be able to solve any question in a PM interview.

Moving ahead, it’s time to break down the North Star metric so fully define success.

Crack PM Interview like Top 1% — First Principle Thinking

Breakdown of North Star Metric

Let’s break it down into different L0 metrics -

# Weekly Users with Successful Discovery via Chatbot = Weekly Active users having access to Chatbot * % Weekly Active users engage with Chatbot X * Weekly active users having atleast 1 Successful Discovery

→ The first term tells you the Reach or Rollout of the Feature,
→ The second term tells you the adoption of Chatbot and
→ The final term tells you the success of the chatbot.

If you define it like this, that would be a real game-changer; people often recite the framework. Nowadays, companies are not looking for any framework, they are looking for your raw thinking.

So now let’s move on to L1 metrics or supporting metrics.

Crack PM Interview like Top 1% — First Principle Thinking

Counter Metrics

1. Users Reporting negative Experience, or

2. % users who are doing a downvote to the response.

3. We can also measure the % detractors.

Resource

Crack PM Interview Like top 1% ( 10 Videos Lectures ) + ( Exclusive Videos of Detailed Solution to PM Interview Question) — Download Here

About Me

Hey, I’m Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking.
For more, check out my Live cohort course, PM Interview Mastery Course, Cracking Strategy, and other Resources