As the scale of the system exceeds the threshold of 100 million daily active users and 100 billion daily interaction data, its complexity has increased exponentially. At this point, the key challenge of system design is not only to improve the accuracy of algorithms, but also to build a comprehensive and sustainable technology-business ecosystem covering efficient data pipelines, accurate algorithm models, user experience optimization, content ecological incentives, commercialization strategies, and ethical risk control.
1. Systematic design of content supply, user value and commercialization
1. System engineering of content producer incentives
The thriving content ecosystem relies on sustainable creator incentives. Effective mechanism design needs to consider multi-dimensional value return:
Traffic Distribution Mechanism:Dynamically adjust the traffic distribution weight by combining content quality evaluation models (such as comprehensive scoring based on user interaction depth, completion rate, and negative feedback rate) and creator development stage (novice/waist/head). In terms of technical implementation, an exclusive channel based on creator ID or content category can be set up at the recall layer, and the creator growth stage factor can be introduced as a model feature in the sorting layer.
Multiple Income Model:Go beyond a single ad split and integrate:
- Performance-Based Incentive Funds:Bonuses are awarded based on key performance indicators of content, such as watch time, engagement rate.
- Subscription/Tip Share:Design clear revenue settlement rules and platform service fee structure.
- Brand Cooperation Matching Platform:Establish a standardized creator service capability tag library (such as audience portrait and historical cooperation effect data) and brand demand matching algorithms to reduce transaction costs.
Long-tail content support:At the algorithmic level, design traffic-weighted strategies or explore dedicated recall channels (e.g., diversity sampling based on content embedding vectors) for long-tail content that meets quality thresholds but is underexposed. At the operational level, special support plans can be set up to provide data insight tools to help creators optimize their content.
B-end product manager’s ability model and learning improvement
The first challenge faced by B-end product managers is how to correctly analyze and diagnose business problems. This is also the most difficult part, product design knowledge is basically not helpful for this part of the work, if you want to do a good job in business analysis and diagnosis, you must have a solid …
View details >
2. The balance mechanism between commercialization and user experience
To achieve sustainable commercialization, it is necessary to establish a refined regulatory system:
Advertising system design principles:
- Ad Load Rate Threshold Management:Through rigorous A/B testing and user satisfaction monitoring (e.g., NPS, retention rate changes), determine the maximum acceptable ad density for different user scenarios (feed, search, detail page) (e.g., feed ads do not exceed 15-20%).
- Ad Relevance Guarantee:Treat ads as special “content” and apply recommendation models similar to organic content (such as the DIN/DIEN model, which uses user behavior sequences to model ad interests) to ensure that ads are highly matched to user intent. Ad ranking should combine estimated click-through rate, estimated conversion rate, and ad quality scores (such as creative clarity and landing page experience).
- Optimization of bidding mechanism:Adopt conversion-focused smart bidding strategies such as oCPM/oCPC to balance advertiser ROI with platform revenue. Consider introducing a dynamic floor price mechanism to adjust the bidding threshold according to user value stratification or scenario value.
User Experience Protection Strategy: 1) Multi-objective optimization:In the model training and online inference stages, user satisfaction indicators (such as dwell time, negative feedback rate), ecological health indicators (such as content diversity) and business indicators (such as GMV, Ad Revenue) are explicitly optimized. Common techniques include:
- Loss function weighted fusion:Loss=a*Loss_User+b*Loss_Eco+c*Loss_Biz
- Pareto optimization method:For example, the evolutionary algorithm (NSGA-II) is used to find the optimal solution set for strategy selection.
2) Scenario-based strategy:Prioritize experience during high-active/high-value user periods (reduce ad density and improve content relevance); Moderately increase the commercialization weight at promotion nodes or specific user life cycle stages (such as churn warning periods).
2. Technical solutions to deal with information cocoon and algorithm fairness
1. Algorithmic strategies to improve recommendation diversity
Breaking the filter bubble requires active intervention at the algorithm level:
Explore and utilize the balanced framework: 1) Bandit algorithm application:For example, Thompson Sampling or LinUCB, dynamically allocate traffic to “leverage” (known high-click content) and “explore” (potential or diversity content).
2) Multi-channel recall and convergence:Design a dedicated “Explore Recall” channel that uses content embedding vector clustering, topic models (LDA), or graph neural networks (GNNs) to tap into potential user interests or discover diverse content that similar users like, and then fuse it with the results of the main recall channel.
3) Diversity control of rearrangement layers:
- Rule-Based Breakdown:Enforce minimum differences in categories, authors, topics for consecutive recommendations.
- Model-based diversity rearrangement:Models such as MMR (MaximalMarginalRelevance) or DPP (DeterminantalPointProcess) are used to maximize the overall diversity of the list while ensuring relevance.
Diversity Quantification and Monitoring:
- Content Coverage:Number of Categories Covered/Total Categories. Monitor whether long-tail categories are effectively reached.
- Gini coefficient:Calculate the balance of the distribution of content popularity (e.g., historical impressions/clicks) in the recommended list. The closer the value is to 0, the more equal it is, and the closer it is to 1, the more concentrated it is. Set the warning threshold (for example, >0.6).
- Proportion of long-tail content:Define the long tail (e.g., non-Top 20% popular content) and monitor its share of total impressions (target value, e.g., ≥30%).
2. Algorithmic fairness evaluation and guarantee system
Ensuring non-discriminatory recommendations requires measurable standards and monitoring:
Definition and measurement of fairness: 1) Group equity:Compare differences in key indicators between different protected groups (e.g., gender, geographical grouping):
- Exposure Variance:Calculate the standard deviation of exposure of the same quality content across different groups.
- Conversion Fairness:Compare the difference in conversion rates between different groups for the same recommended content.
- Population Coverage:Monitor the difference in the proportion of users in each group of users appearing at the top of the recommendation results (such as the Top 10).
2) Counter-factual fairness test:Construct virtual user pairs (which differ only in sensitive attributes such as gender, and other characteristics and behaviors are the same) to verify that their recommendations are consistent.
Technical Mitigation Strategies:
- Data Preprocessing:Identify and correct historical biases in your training data.
- Model training constraints:Add a fairness regular term (e.g., DemographicParityEqualizedOdds difference penalty) to the loss function.
- Post-processing correction:The sorting points of the model output are calibrated and adjusted according to the user group.
Real-Time Monitoring and Auditing:
- Build a fairness monitoring dashboard to track these core metrics in real time.
- Establish a regular algorithm audit process, including offline dataset testing and online A/B testing.
- Design bias feedback and intervention channels that allow users or internal auditors to flag potential cases of bias.
3. The core capability model and technology stack of AI product managers
1. Depth of technical understanding
To transform from functional PM to AI PM, you need to master the key recommended technology stacks:
Algorithm principle and application scenarios: 1) Collaborative filtering:Understand the cold start and data sparsity issues based on the similarity calculation of the user (User-CF) or item (Item-CF).
2) Deep learning models:
- Embedding&MLP：The foundation of Wide & DeepDeepFM.
- Sequence modeling:DIN (DeepInterestNetwork) How DIEN (DeepInterestEvolutionNetwork) captures users’ dynamic interests.
- Multitasking Learning:For example, ESMM (EntireSpaceMulti-taskModel) solves the sample selection bias of CVR estimation and optimizes CTCVR (Click-Through&conversionRate).
3) Vector Retrieval:Understand the central role of ANN (Approximate Nearest Neighbor) algorithms (e.g., HNSW, IVF) in the recall layer.
Data Processing and Analysis Capabilities:
- Proficient in using SQL for large-scale user behavior log analysis.
- Master the basics of Python and the commonly used data analysis library (PandasNumPy) for feature analysis and index calculation.
- Proficient in A/B testing experimental design (triage strategy, sample size calculation, statistical significance test) and platform (e.g., in-house platform or Optimizely).
System architecture cognition:
1) In-depth understanding of the core hierarchical architecture of the recommendation system and its collaboration:
- Recall:Quickly filter out 100/1000 level related items from a large number of candidate sets (Technique: CFEmbedding+ANNGraphEmbedding).
- Fine arrangement:Accurate scoring and sorting of recall results using complex models (e.g., deep learning) (technology: feature engineering, complex models such as DIN/DIENMTL).
- Reset:Apply business rules, diversity control, contextual adaptation, etc. to make final list adjustments (Technical: Rules Engine, MMR/DPP).
2) Understand the role of online services (low latency, high concurrency), offline/nearline training data flows, and feature storage platforms (Feature Store).
2. Cross-field collaboration and translation capabilities
AI PM is the hub of technology, business, operations, and compliance:
Collaborate with Algorithm Engineers:
- Translate vague business goals (“improve new user retention”) into quantifiable, modelable technical requirements (“New users’ click-through rate needs to increase by X% on their first day of recommendation lists, and increase 7-day retention by Y%”).
- Understand the business implications of model evaluation metrics (AUCGAUCRecall@KNDCG).
- Participate in feature engineering discussions and provide feature suggestions from a business perspective.
Collaborate with operations teams:
- Design interpretable, interventionable operational strategies: For example, establish a “manual curated content pool” mechanism that allows operations to inject high-quality content into the recommendation process (through traits or reflow rules) in specific scenarios (e.g., major events, cold starts).
- Provide algorithmically understandable data dashboards to help operations understand content distribution effects and user preferences.
Collaborate with legal/compliance teams:
- Lead the establishment of algorithm ethics review processes to ensure that the recommendation logic complies with data privacy regulations such as GDPR and CCPA, as well as emerging AI regulatory requirements (such as the EU AI Act).
- Participate in the design of user data authorization management and ExplainableAIXAI implementation (e.g., provide a simplified explanation of “why recommend this”).
3. System thinking and ecological planning ability
AI PMs need to have a vision to build and optimize ecosystems:
Content ecological planning:
- Category Strategy:Analyze the relationship between supply and demand, plan the content category structure, and identify potential categories that need support.
- Creator Lifecycle Management:Design a full-link support system from introduction (cold start traffic package), growth (skills training, data tools), maturity (business cooperation opportunities) to retention (exclusive rights).
User Lifecycle Management (LTV):
- Cold Start Strategy:Integrate content-based recommendations (Content-based), popular recommendations, guided interactions (interest surveys), and lightweight collaborative filtering (Session-based). Quickly establish user portrait prototypes.
- Maturity Strategy:Deepen personalized recommendations (sequence models) and combine scenario-based operations (push notifications, activity pages). Implement user stratification (RFM or value model) for refined operations.
- Loss Warning & Recall:Leverage predictive models to identify users at risk of churn and trigger intervention strategies (e.g., exclusive content/offers).
Business Ecological Design:
- Value Distribution Model:Clearly define the rules for the flow of value between platforms, creators, advertisers, and users (such as sharing ratio, bidding mechanism).
- Sustainable monetization model:Balance short-term revenue (advertising) with long-term user value (subscriptions, value-added services) to avoid drying up the lake.
4. Recommend system health assessment
Build a real-time monitoring system to comprehensively measure system health:
1. User value dimension
Core indicators:
- NPS (Net Promoter Score):Directly measure user satisfaction and loyalty.
- User Retention:Next-day/7-day/30-day retention rate, reflecting the long-term value of the system. Dismantle the difference between new and old user retention.
- User Activity:DAUAvg.Time, number of visits per person, and average depth of clicks.
- Interaction Quality:Like rate, comment rate, share rate, effective play rate (proportion of > X seconds played).
- Negative Feedback Rate:The frequency of actions such as “not interested”, “block author/content”, etc.
Optimize leverage:
- Sentiment Analysis:Apply NLP techniques to analyze emotional tendencies in user reviews and feedback.
- Real-time feedback closed-loop:The “Not interested” button triggers an instant model update or user profile adjustment.
- Satisfaction attribution analysis:Target specific modules (recalls/sorting/reordering) or content types that are causing satisfaction fluctuations.
2. Ecological health dimension
Core indicators:
- Content category coverage:Monitor the trend of the proportion of exposure in small and medium-sized categories outside the TopK category.
- Gini coefficient (content popularity distribution):Calculate regularly and set a cordon line.
- Long-tail content exposure/consumption ratio:Clearly define (e.g., non-Top 20% content) and monitor its proportion.
- Creator Distribution Health:The proportion of traffic of head/waist/tail creators, the growth in number, and the retention rate.
Optimize leverage:
- Diversity Algorithm Tuning:Adjust the intensity of the exploration strategy and rearrange the diversity parameters.
- Creator support strategy iteration:Optimize traffic tilt and incentive policies based on data feedback.
- Content Quality Evaluation Model Upgrade:More accurately identify high-quality long-tail content.
3. Business efficiency dimension
Core indicators:
- GMV (Total Turnover):E-commerce core indicators.
- Ad Revenue:Focus on eCPM (revenue per thousand impressions) and fill rate.
- ARPU/ARPPU (Average Revenue per User/Paying User):Measure user monetization efficiency.
- Advertiser ROI:Focus on advertisers’ cost of click (CPC), cost of conversion (CPA), and return on investment (ROAS).
- Platform gross profit margin/operating profit margin:Benefits after comprehensive costs (bandwidth, computing power, manpower).
Optimize leverage:
- User value stratification and refined operation:Identify high-value user groups and provide differentiated experiences and monetization strategies.
- Dynamic Pricing and Bidding Strategy Optimization:Adjust the advertising reserve price and bidding logic according to the relationship between supply and demand, user value, and scene value.
- Recommendation Relevance Boost:More accurate recommendations directly drive conversion rates and GMV improvements.
5. Build a recommended ecosystem for sustainable development
The ultimate goal of a large recommendation system is to build a self-growing, sustainable value network:
- Creator Side:Through transparent and fair traffic distribution algorithms and reasonable and diversified revenue sharing mechanisms, we ensure that creators at all levels (especially long-tail) receive positive incentives for continuous creation, and ensure the vitality and diversity of ecological content supply.
- User side:While enjoying the efficiency and pleasure brought by highly personalized experiences, through effective diversity mechanisms and transparency tools, reduce the risk of falling into an information cocoon, obtain a richer and more balanced information/content consumption experience, and improve long-term satisfaction and platform trust.
- Platform side:Achieve the intrinsic unity of business value (revenue, growth) and social responsibility (equity, privacy, well-being). A healthy business ecosystem is the foundation of sustainable development, and responsible algorithmic practices are key to earning long-term user trust.
The Role Evolution of AI Product Managers:From a “designer” who focuses on functional implementation to an “ecological architect” who designs complex adaptive systems. The core responsibilities are:
- Define and continuously monitor the health index as a dashboard for system operation.
- Navigate complex technology stacks (multi-objective optimization, federated learning, explainable AI, fair machine learning) to address core contradictions such as efficiency and equity, short-term gains and long-term value, personalization and diversity.
- Establish a cross-functional collaboration mechanism to ensure that technology, product, operation, and compliance goals are aligned.
When recommendation systems successfully evolve from a technology tool to a robust, balanced, and self-reinforcing ecosystem, their value will transcend mere information distribution efficiency and become the core infrastructure that drives long-term, healthy, and sustainable growth of digital businesses.
