When building a large-scale AI recommendation system, the core challenge is to push the ranking model from technology to the actual product environment. Product managers need to be deeply involved and lead multiple key links to ensure that technical capabilities are effectively translated into user value and business outcomes.
1. Definition of characteristic engineering requirements
When product managers define the characteristics and requirements of users and scenarios, they need to be guided by business goals, clearly describe the required data dimensions, and clarify their meaning and value in the product.
User personas usually need to cover multiple layers:
- Base Attributes:Includes relatively static information such as age, region, and device type. For example, clarifying the portrait label of “white-collar women aged 25-35 in first-tier cities” may be associated with specific category preferences (such as mother and baby, beauty) in e-commerce scenarios.
- Behavioral characteristics:It covers recent dynamic data of users, such as clicks, favorites, and add-on behavior sequences, as well as cross-terminal (APP/mini program/H5) usage habits. The frequency of collection of this data (e.g., real-time updates, daily aggregation) and storage granularity (e.g., per user, per session) need to be clarified.
- Psychological needs:By analyzing text data such as user reviews and customer service conversations (usually with the help of NLP technology), we identify potential deep needs (e.g., “pursuit of healthy living” and “desire for self-improvement”) and establish a mapping relationship between these needs and product/content categories (e.g., fitness equipment mapping to healthy living needs).
The design of contextual features should be closely centered around the specific environment in which the user is located and their behavior paths:
- Time dimension:Distinguish the characteristics of weekdays/weekends, morning and evening peaks, etc. For example, in a commuting scenario, the system may tend to recommend lightweight content (e.g., news summaries, short videos); On weekends, it may focus on in-depth content (e.g., long videos, knowledge courses).
- Spatial dimension:Use GPS and Wi-Fi positioning data to understand the typical location of users (such as home, business, business district). For example, when the user is in the business district, the system can recommend information about nearby places such as catering and entertainment.
- Interaction dimension:Capture subtle user interactions with the interface in real time, such as swipe speed and content dwell time. This data can be used to dynamically adjust recommendation strategies, such as reducing the complexity of recommended content when users swipe quickly.
2. Model goal setting
The goal of the sequencing model requires a systematic balance between short-term business goals and long-term user experience. It is common to employ a hierarchical strategy that combines primary goals with secondary objectives:
After 10 years of interaction design, why did I transfer to product manager?
After the real job transfer, I found that many jobs were still beyond my imagination. The work of a product manager is indeed more complicated. Theoretically, the work of a product manager includes all aspects of the product, from market research, user research, data analysis…
View details >
- Main objective:Choose metrics that are directly related to core business KPIs, such as GMV in e-commerce scenarios and video completion rates on content platforms. The calculation caliber of the metric (e.g., whether GMV includes refund orders) and the source of the data (e.g., from the payment system or the order system) must be clearly defined in the requirements.
- Secondary Objectives:Introduce metrics that impact long-term value, such as user satisfaction, diversity of recommendation results. Specific measures may include: controlling the recommended ratio of similar content/products (such as a single category accounting for no more than 30% to enhance exploratory activity); Set the User Activity Decay Coefficient to dynamically adjust the weight ratio of recent and historical behaviors in the model.
The allocation of multi-objective weights is a dynamic tuning process:
- Initial Setup:Set base weights based on business experience and historical data (e.g., click-through rate 60%, conversion rate 30%, dwell time 10%).
- Dynamic adjustments:Utilize online learning mechanisms to respond to user feedback data in real time. For example, if you see an increase in click-through rate but a significant decrease in conversion rate for a certain type of content, the system can automatically reduce the weight of the CTR target (e.g., by 5%-10%).
- Constraints:Set hard rules to prevent the model from biasing to extremes, such as limiting the maximum recommendation percentage of a specific category (e.g., ≤ 40%). At the same time, “fairness” considerations can be introduced to ensure that the coverage difference between different user groups (such as new and old users, users in different regions) does not exceed a certain threshold (e.g., < 15%).
3. Badcase analysis mechanism
Establishing a systematic closed loop of Badcase analysis is key to ensuring the quality of recommendations:
Problem identification and collection
Continuously monitor recommendation results and user feedback by burying logs.
Define the core Badcase type, for example:
- Low-quality content:Such as the title party information, repeated recommendations for the same product/content in a short period of time (such as ≥ 3 times).
- Interest mismatch:The content that the user clicked multiple times in a row (e.g., 5 times) did not appear at the top of the system’s recommendation list (e.g., the top 10).
- Experience Questions:The system still repeats recommendations after the user expresses that they are not interested, and the recommendation results take too long to load (for example, > 3 seconds).
Root cause positioning
- Recall layer check:Analyze whether the content library has insufficient coverage (e.g., content coverage under a category <80%).
- Sorting layer diagnosis:Check for deviations between the model’s predicted scores (e.g., estimated click-through rate, conversion rate) and actual user behavior (real clicks, conversions), and identify samples with significant estimate errors (e.g., CTR estimate error >20%).
- Policy layer validation:Investigate whether subsequent rearrangement rules, quantity preservation strategies, etc. have caused low-quality or low-relevance content to gain too high exposure.
Problem solving and iteration
- Quick Response:Implement regular solutions for stop-loss, such as short-term blocking of content with clear negative feedback from users (e.g., no longer recommended within 7 days).
- Long-Term Optimization:Typical Badcase samples are added to the model training data to drive feature engineering improvements (such as adding the “user negative feedback” feature) or upgrading the model structure (such as introducing a comparative learning mechanism to improve the ability to distinguish difficult samples).
Fourth, the core elements of productization
The key to translating algorithmic capabilities into user-perceived value lies in designing:
Interpretable recommendation presentation
Reason label system construction:
- Basic type:“Brands you have followed”, “Similar users also like it”.
- Scenario Type:“Popular content during commuting” and “Recommended weekend family activities”.
- Value:“User evaluation high-score selection”, “Recent price discount”.
Hashtag Display Strategy:
- Home:The “1+3” model can be used to highlight a primary recommendation reason, supplemented by a small number (e.g., 3) secondary reason labels.
- Details page:With progressive reveal, the initial display is concise, and after the user stays for a longer period of time (e.g., > 10 seconds), it can trigger a more detailed analysis (e.g., “This item is 85% style match with your collection of dresses”).
User value is explicit
Construct a “personalization index” that synthesizes indicators such as recommendation accuracy, diversity, and novelty to generate an intuitive “recommendation experience score” (such as 1-10 points) for users.
In scenarios such as the user’s personal center, the value brought by the recommendation system is visualized, such as “new preferences discovered through recommendations this month” and “estimated screening decision time saved for you”.
User feedback and engagement
Lightweight Feedback:Set a convenient “Like/Dislike” button next to the recommended results, and click to expand secondary options (such as “Not interested” and “Purchased”) to ensure that feedback can be reverted to update the user profile in real time.
Active Exploration:Provide a function similar to the “Recommendation Exploration Lab”, allowing users to actively adjust their recommendation preferences (such as “increase technology content” and “reduce ad recommendations”), and compare the differences in recommendation results under different preferences.
5. The key role of product managers
The key to successfully commercializing the ranking model in a large AI recommendation system lies in the effective role of product managers as “translators” and “bridges”:
- Requirements definition stage:Define technical requirements in clear, unambiguous business language to ensure that technical teams understand the business intent accurately (e.g., clearly define specific calculation rules for “user activity”: “3 logins in the last 7 ≥days and ≥ 10 content clicks”).
- Model development and tuning stage:Deeply understand how business goals (such as GMV) are composed of technical indicators (such as click-through rate, conversion rate, customer unit price) (GMV = click-through rate× conversion rate× customer unit price), and promote the alignment of model optimization direction with business goals.
- Product landing stage:Lead the design of interpretability functions and user value visualization schemes, so that users can understand the recommendation logic, thereby building trust in the system, and ultimately promoting the conversion of user value (such as clicks, conversions, and retention).
The productization of recommendation systems is a continuous iterative process that requires product managers to have a deep understanding of technology and business, and to seamlessly integrate algorithmic capabilities into user journeys and business processes to maximize their business value and enhance user experience.
