The transition from traditional product managers to AI product managers is a challenge from certainty to uncertainty. This article delves into the changes in requirements definitions, data roles, and launch standards in AI product development, pointing out that the core of AI product development lies in shifting from “logic-driven” to “data-driven”.
Make AI products, but you can’t find the steering wheel
“How much can the efficiency of resume retrieval be improved in the end?” “How accurate is resume matching?” “This is the biggest confusion in my heart in the past two months, not only asking my technical classmates, but also repeatedly asking myself, why don’t I know the answer?
In the past, when making products, the path and end point of each function and iteration were clear. It seems that no matter how good or bad the result is, it is under his control. Until I joined the company’s AI training camp and started doing intelligent recruitment projects, a “sense of loss of control” arose. It seems like all I can grasp is product interactions and prompts, and I can’t predict whether it will eventually meet business needs, and it feels like I have set up navigation, but the steering wheel is not in my hands. The discussion with development students is no longer about interaction and bugs, but about “model confidence”, “data distribution” and “F1 score”, but they don’t know how to set quantitative goals.
This experience from certainty to vagueness, from controllable to out of control, I don’t know if you PMs feel the same way. However, is the steering wheel really gone? I think it’s just a different form. To get it back, you must first understand what exactly has changed in this “car”.
Driver Transformation: Logic vs. Data
The essence of AI products is a paradigm shift from “logic-driven” to “data-driven”, which has led to the complete subversion of the “certainty” worldview we are used to. This fundamental difference reshapes the three core elements of product development.
Changes in the definition of requirements: from “achieving functions” to “achieving quantitative goals”
The requirements of traditional software products can be translated into clear rules, and user behavior will be limited to fixed rules. AI products are more like a weather forecaster, which tells you based on massive historical data that there is a 90% chance of rain tomorrow. Its behavior is “probabilistic” and is determined by the patterns learned in the data.
How can product managers do a good job in B-end digitalization?
All walks of life have taken advantage of the ride-hail of digital transformation and achieved the rapid development of the industry. Since B-end products are products that provide services for enterprises, how should enterprises ride the digital ride?
View details >
Taking intelligent recruitment as an example, the business demand is to improve the matching degree of people and jobs
- Traditional product manager: “On top of the existing support search matching fields, add structured fields such as institution type, communication ability, industry, etc., and support individual and combined searches.” It’s a black-and-white feature.
- AI product manager: “Support the use of large models to identify key information in recruitment needs, realize vectorized storage of key information, and match the key information of resumes in the database.” Under the premise of ensuring that the probability (Precision) is not less than 90%, the recall rate (Recall) is increased as much as possible. “There is a certain probability of how the matching result is.
This means that instead of targeting 100% correct filtering, we are ensuring that the candidates recommended to HR are reliable, with a reasonable probability. We accept the possibility of “making mistakes” from the beginning and try to quantify and manage it.
The Change of Data Role: From “Product” to “Raw Material”
- Traditional software: Data is “the product of value records”. The logs and transaction records generated by users in the process of using the product are the “echo” of the business behavior that has occurred. Their core value lies in traceability and analysis, which is used to generate reports, monitor business health, and review issues. Data quality determines the depth of post-analysis, but does not change the functional logic of the product itself.
- AI products: Data is the “raw material for value definition”. Data is the “source code” that drives the core functions of AI products, which directly determines the effectiveness of the model, the intelligence of the product, and the upper limit of value.
In intelligent recruitment systems, recruitment data from the past few years — which resumes were viewed, who made it to the interview, and who was finally hired — is the “digital gene” that trains the model. The quality, dimension and scale of data together define the “ceiling” of the product. As an AI product manager, you must examine the source, quality, bias, and hidden risks of data like a chef examines ingredients.
Changes in the go-live criteria: from “zero defects” to “above baseline”
The launch standard of traditional products is “complete functions and no serious bugs”. The launch standard of AI products is “model performance is better than baseline”
What is a baseline? It can be an existing manual processing efficiency or a simple rule model. For example, our intelligent recruitment system, as long as its recommendation accuracy exceeds the “accuracy of manual screening by junior HR” or exceeds the “simple keyword matching algorithm”, it has the value of launching.
The code of control: the triple cultivation of thinking, skills and methodology
Insight into the root cause of the loss of control is the beginning of regaining the steering wheel. This requires a triple cultivation of thinking, skill and methodology.
The following methods are combined with application examples in intelligent recruitment scenarios
Reshaping Thinking – From “Engineer” to “Scientist”
From pursuing the only correct answer, to proposing hypotheses, designing experiments, and verifying results.
- Embrace uncertainty: Habitual thinking and communication with “probability” and “confidence interval”. When a model makes a mistake, your first reaction should not be “This is a bug”, but “What category does this error belong to?” What is the probability that it will happen? How can we avoid its negative effects in product design? ”
For example, the HR feedback system recommends a good “JavaScript” front-end expert to the “Java” back-end position. The team reviewed and found that the model failed to adequately distinguish between the two terms when trained, leading to confusion. This is a probabilistic mistake, and the product manager should think about it: What is the probability of this kind of confusion? Can we reduce the confusion rate by adding specific labeled data? Or, clearly prompt on the product interface that “the candidate’s skills are partially similar to the job requirements, please pay attention to screening” to manage user expectations and avoid negative impacts.
- Build experimental thinking: Think of every go-live as a massive A/B test. Your core job is no longer to deliver functionality, but to experiment to find models and strategies that maximize your core metrics.
In order to verify whether the new algorithm can improve the efficiency of person-job matching, an A/B experiment can be designed. HR was randomly divided into two groups: group A used the old keyword-based filter, and group B used the new AI sorting model. Comparing the matching degree of people and jobs in groups A and B, the value of the new model is proved with real data, which provides a solid basis for the full launch.
Skill Upgrade – From “Functional Specialist” to “Full-Stack Thinker”
You need to build a new, cross-disciplinary body of knowledge.
Understand data: become the chief quality inspector of data
You need to be able to answer: What kind of data do you need to prepare? How do I clean and label data? How is data privacy and compliance guaranteed?
For example, in the intelligent recruitment project, cooperate with senior HR to collect real recruitment demand data and successful resumes of “final entry”, and also conduct in-depth interviews to mark the key information of these needs and the implicit success signals behind the resume. These high-quality, “expert knowledge” annotation data have become the “advanced nourishment” of the model, allowing them to learn to think like senior HR and greatly improve their ability to identify high-potential candidates.
Understand the Model: Master the language of evaluating AI
You don’t need to be an algorithm expert, but you must understand the core metrics of evaluating models and make choices in the right business scenarios.
Take intelligent recruitment as an example:
- Precision: “Better to be lacking than to be indiscriminate”. It answers “What is the percentage of candidates who are truly qualified among all candidates recommended by the model?” ”。 If the business goal is to save time for high-end, busy HR teams, high accuracy is the first choice, and we’d rather miss out on a few potential candidates than make sure the recommendations are high-quality.
- Recall: “Better to be indiscriminate than lacking”. It answers “How many of all truly qualified candidates are successfully recommended by the model?” ”。 If the recruitment is a scarce position and the goal is “to find you at the ends of the earth”, then high recall is more important, and we can tolerate some less matched recommendations and hand them over to manual secondary screening.
- F1 Score: The “harmonized average” of accuracy and recall. When both need to be taken into account, F1 scores provide a comprehensive evaluation criterion.
Understand interaction: designing a “fault-tolerant interface” of humanity
Since AI can make mistakes, we must provide users with “airbags” at the level of human-computer interaction.
For example, when an intelligent recruitment system recommends a candidate to HR, an excellent fault-tolerant interface will look like this: “High match (95%).” Main matching points: proficient in Python, 5 years of experience in distributed systems. Potential risk points: The project management experience is less described, and the model has low confidence in this assessment. This design not only expresses uncertainty but also clearly returns decision-making power to users and guides them to focus on key points that require human judgment.
Methodological iteration – from “waterfall delivery” to “closed-loop experimentation”
The development model of AI products has completely bid farewell to the linear and waterfall workflow and entered a closed loop of continuous iteration.
MVP’s new gameplay: “laying eggs” without “chickens”
AI projects face a classic “chicken lays egg” problem: without data, models cannot be trained, and without products, data cannot be collected. Two clever MVP approaches to break the ice:
- Wizard of Oz MVP: This is a “pretend automation” strategy. The front end gives users an intelligent interface, but the background is completely manual. Taking the intelligent recruitment system as an example, when recommending candidates more accurately in the future, after HR submits the requirements, the product team will manually screen the resumes and sort them back. This can be verified at the lowest cost: Are our recommendation results really valuable to HR? What dimensions do they value most?
- Human-in-the-Loop: The first round of processing is done with a very rudimentary model (even simple rules), and the results are reviewed and corrected by humans. These high-quality data that have been manually verified will become the best “nourishment” for the next generation of models. In a recruitment project, the model can be rounded by a rough screening and then the results can be marked as “pass” or “fail” by the junior recruiter, providing a steady stream of data for model iteration.
Restructuring of PRD: A new PRD structure
In the era of AI, traditional PRD must evolve. A valid AI PRD should include the following core modules:
Get behind the wheel
The transformation from traditional software product managers to AI product managers is a profound paradigm revolution in thinking. It requires us to bravely bid farewell to the logical world of certainty and embrace the world of data full of probability. And that “steering wheel” has changed from a tangible thing that controls “code logic” to an invisible rudder that controls “data flow, model iteration, and user feedback”. True control is not to put an end to every yaw, but to have the ability and method to get it back on the right track after every yaw. Welcome to the AI era and happy driving.
