AI Product Manager Interview Question 3
Prepare for AI PM Interviews
We will keep on discussing AI PM Interview Questions asked recently → If you want to stay updated, you can consider Subscribing
For More Such Questions → Download this
Question — Downvote on Chatgpt Response has increased, how would you go about this?
This is one of the most important PM interview questions asked recently in so many companies.
One of the most crucial Product Manager interview questions asked frequently in recent times revolves around analysing a dip in a key metric. Imagine this scenario: “Downvotes on ChatGPT responses have increased. How would you go about solving this?” This isn’t just a technical puzzle; it’s a test of your structured thinking, product sense, and ability to drill down to the root cause.
This article, drawing insights from an expert perspective on tackling this specific challenge, will walk you through a systematic approach to breaking down and solving this problem in a PM interview setting.
Step 1: Clarify the Problem Scope
Before diving into solutions or root causes, the first and most critical step is to ask clarifying questions to ensure you and the interviewer are aligned on the problem.
Define the Metric: What exactly does “downvote rate” mean? In this context, it’s defined as downvotes per ChatGPT session, where a session begins as soon as a user lands on the ChatGPT interface. Understanding this definition is fundamental to identifying root causes.
Identify Response Types Affected: Is the increase specific to text responses, image responses, code snippets, or other types? This helps narrow the focus. The interviewer in this scenario indicates the issue is primarily seen in text-based responses, as image responses haven’t reached the same scale of adoption yet.
Understand Timeframe and Nature: When did this increase begin, and what is its pattern? The interviewer states the rise is from the last month and is gradual, affecting users across all device types (mobile and desktop).
Analyze User and Language Segments: Does the problem affect specific user types or languages? It’s important to check if the downvotes are concentrated among, say, new users, expert users, or users prompting in a particular language. In this hypothetical interview, the issue is described as being across all user types and languages used for prompting.
Examine Traffic Sources: Is the increased downvote rate linked to a specific traffic source, such as organic or inorganic traffic? The quality of traffic can impact user behavior. However, the interviewer confirms that the traffic source has not changed and the share of organic traffic hasn’t decreased.
These broad questions help establish the context and eliminate several potential factors from the outset. To summarize the initial findings: downvotes on ChatGPT have increased over the last month, gradually, across all categories, users, geographies, languages, and device versions.
Step 2: Understand the Strategic Importance
Why is an increase in downvotes a significant problem? Downvotes directly indicate user dissatisfaction. User dissatisfaction, in turn, impacts the real engagement rate of ChatGPT. In a competitive market with alternatives like Claude and Gemini, and considering ChatGPT is still in an early stage, sustained user dissatisfaction could lead to users switching to other platforms. This highlights the strategic necessity of addressing the downvote issue.
Step 3: Structure the Problem — Focusing Inward
Problems like this can often be categorised into System Related Issues, External Issues, and Internal Issues. In this interview scenario, the interviewer directs the focus specifically to internal issues, stating that system and external issues have already been checked and ruled out. This means the root cause is likely within the product’s user experience or performance.
Step 4: Analyze the User Funnel and Key Metrics
To identify internal issues, it’s crucial to understand the user’s journey on ChatGPT and track relevant metrics at each stage. The user funnel relevant to this problem looks something like this:
Landing on the ChatGPT screen.
Writing or selecting a prompt (potentially using autosuggestions)….
Submitting the prompt.
Getting a response.
Interacting with the response (copying, sharing, asking follow-up questions, downvoting).
We need to examine metrics associated with these steps to pinpoint where things might be going wrong. Several metrics were explored:
Traffic Split: Already discussed; ruled out as a cause.
New vs Repeat Users & Landing Page Bounce Rate: An increase in bounce rate alongside downvotes could suggest new users are confused by the UI or prompt guidance. The interviewer indicated no jump in bounce rate, ruling this out.
% Autosuggest Usage: If autosuggest usage and downvotes linked to autosuggest prompts increased, it might point to poor quality suggestions. While autosuggest usage has increased, the downvotes are happening for both autosuggested and manually typed prompts, so this isn’t the core issue.
% Sessions with 0 Searches or Time to First Query: If these increased with downvotes, it might indicate onboarding issues where users don’t know how to prompt effectively. The interviewer stated this is not the case.
% Image Search, Voice Search, Queries per Session: Increased adoption of image/voice prompts paired with downvotes specific to these types could indicate issues there. Adoption of voice and image prompts has increased, but the downvotes are seen across all prompt types (voice, image, and text), so this isn’t the sole cause.
Response Latency, Prompt Length, and Regenerate Rate: These metrics relate directly to the response quality and user interaction after receiving a response. The interviewer highlighted a key finding here: the regenerate rate has increased, while response latency and prompt length are fine.
Step 5: Identify Potential Root Causes (Hypotheses)
The crucial finding is the correlation between the increased downvote rate and the increased regenerate rate. When users regenerate a response, it means the initial response wasn’t satisfactory. If they then still downvote after regenerating (or perhaps downvote because regeneration didn’t help), it points towards underlying issues with the model’s output itself.
Based on this, two primary hypotheses emerge:
Model Performance Issues: The most recently released model might have decreased in quality. This could manifest as increased hallucinations (generating false information) or a drop in the coherency of the responses. If the model is consistently providing lower quality or factually incorrect responses, users will be dissatisfied and downvote.
Lack of Diversity in Regenerated Responses: Even if users regenerate, there might be little significant change or diversity compared to the original response. If regeneration doesn’t produce a better or different outcome, users might get frustrated and downvote the unsatisfactory response, especially if the regenerated one is similar to the first poor one.
These hypotheses directly link the observed increase in regenerate rate to the subsequent increase in downvotes….
Step 6: Propose Next Steps
As the Product Manager, the next steps would be to validate these hypotheses and pinpoint the exact root cause.
Solidify the Root Cause: Quantify the correlation by measuring the regenerate rate and downvote rate specifically before and after the release of the latest model. If there’s a clear jump corresponding to the model release, it strongly supports the model performance hypothesis.
Qualitative Analysis: Sample 100 random downvoted responses and their corresponding prompts. Analyze them manually to identify any recurring patterns. Are there specific types of queries where the model fails? Are there common errors (e.g., factual inaccuracies, nonsensical phrasing, failure to follow instructions)? This qualitative data provides crucial context that metrics alone cannot.
By combining quantitative analysis (measuring metrics pre/post-change) and qualitative insights (analysing specific examples), a PM can confidently identify the root cause of the increased downvotes and recommend targeted solutions, whether it’s rolling back a model update, refining the model’s training, or improving the regeneration mechanism.
For More Questions, download the Course Below
Resources
About Me
Hey, I’m Shailesh Sharma! I help PMs and business leaders excel in Product, Strategy, and AI using First Principles Thinking.
For more, check out my Live cohort course, PM Interview Mastery Course, Cracking Strategy, and other Resources