The AI identification system shows positive technical potential in responding to the governance challenge of “more and more truth” of false information, and can be used as an important front-end fulcrum of the content governance chain. However, it is also necessary to face up to the fact that as a governance method that is still being explored, AI logos have obvious technical shortcomings, and need to coordinate with the existing content ecological governance system to focus on high-risk areas and improve governance efficiency.
With the rapid application of AI in the field of content, the use of AI to generate rumors and conduct false propaganda is having a negative impact on the public. According to incomplete statistics from the Nandu Big Data Research Institute, more than 1/5 of the 50 domestic AI risk-related public opinion cases with high search popularity in 2024 are related to AI rumors. [1] Entering 2025, AI-related rumors such as “a child was crushed in ruins by the earthquake in Dingri County, Tibet” have frequently appeared in the focus of public opinion. [2] In addition, illegal cases such as AI synthetic face swapping are frequently used for false propaganda, such as unscrupulous merchants impersonating celebrities such as Sun Li to bring goods[3], and fabricating “Miao Gu gold sticker intangible cultural heritage inheritors” [4], so as to gain traffic and revenue.
1. New AI technologies and old problems in governance
Compared with the past, the nature of illegal and harmful content generated by AI has not fundamentally changed. AI only further amplifies and accelerates the existing “old problems” of content governance, mainly focusing on three aspects:
The first is “easier”, that is, a lower threshold. Generating highly “realistic” content is no longer dependent on expertise or writing skills. According to the China Internet Joint Rumor Refuting Platform, in a village in a central province, villagers obtained traffic revenue by pasting hot Internet keywords into AI models to generate articles. [5] With the empowerment of technology, a large number of non-professional “grassroots” can also create false content that is close to the truth.
The second is “more”, that is, technology can make false information “mass produced”. For example, in the rumor that “medical giants died in a foreign land”, the suspect controlled AI to generate sensational rumors through algorithmic instructions, and achieved an average of 10,000 outputs per day through a matrix of 500 accounts, almost becoming an “industrial rumor”. The phenomenon of “AI pollution” of “garbage in and garbage out” is also a secondary negative manifestation caused by the mass production of false information.
How can product managers do a good job in B-end digitalization?
All walks of life have taken advantage of the ride-hail of digital transformation and achieved the rapid development of the industry. Since B-end products are products that provide services for enterprises, how should enterprises ride the digital ride?
View details >
The third is “more real”, that is, multi-modal and multi-detailed information is more confusing. AI-generated rumors are often mixed with camouflaged “news language” and visual elements, making them highly deceptive. The rumor that “hot water is coming out underground in Huyi District, Xi’an” contains a variety of details that make it difficult for ordinary people to distinguish between true and false. [6]
These three types of risks are not simply superimposed, but form a “multiplication” amplification effect in real communication scenarios, exacerbating the challenges of content governance. Synchronization also promotes the current new path of AI content governance, the AI logo system, to test its effectiveness and limitations in practice.
2. The governance value and limitations of AI logos
In response to the governance challenges of “low threshold”, “large volume” and “more realistic” brought about by AI-generated content, our country has gradually introduced a new path of technical identification. Based on the comprehensive normative system of “departmental regulations + normative documents + national mandatory technical standards” [7], a dual identification mechanism of explicit identification (user-perceptible prompt form) and implicit identification (technical identification that can be recognized by machines in document metadata) has been established. The responsible subjects of the logo cover the whole chain of entities such as generation synthesis service providers, content dissemination platforms, and users.
For content generation platforms, it is required to add metadata implicit identification to AI-generated content; For content that may cause confusion and misunderstanding, add explicit identification at designated locations according to the content type (text, picture, audio, video, etc.); For communication platforms, users are required to be reminded to actively declare AI-generated content and annotate it, and at the same time verify the metadata and other implicit identifiers added by the generation platform, and carry out corresponding identification according to the verification results.
1. Theoretically, AI logos can improve the efficiency and empowerment of content governance
The essence of the identification path is to “use technology to govern technology”, and in the face of the efficiency improvement of content generation, let technology also empower governance. On the one hand, through “implicit identification”, the governance threshold is moved forward to the content generation stage, and AI-generated content is identified earlier and more accurately through automatic logo generation. If the upstream model service provider can embed a stable and effective logo in the generated content, the downstream communication platform can quickly identify and judge the relevant content through technical detection without considering the circumvention or destruction of the logo. It enables it to intervene in content governance at an earlier stage in the face of “low-threshold” and “large-volume” AI-generated synthetic content, thereby improving identification efficiency and strengthening risk management.
On the other hand, explicit identification helps reduce the credibility of relevant content. A study published in the journal PNAS Nexus shows that explicit labels “AI-generated” tend to erode audience trust in content. The study asked 4,976 subjects to evaluate different types of news headlines and found that regardless of whether the news content was true or false, the headlines marked as “AI-generated” were considered more inaccurate and the participants’ willingness to share them was also lower. [8] Therefore, as an “intermediate” reminder mechanism, the “display logo” can play a minimum warning role in the case of immediate confirmation of authenticity, reducing the possibility of potential risk escalation and spread.
It is worth noting that because “display marks” have the characteristic of reducing the credibility of content, their scope of application will be limited. To avoid the disadvantages of overloading users with information caused by over-identification, and even failing to establish basic information trust. At present, AI is widely used in the content production industry, bringing positive effects such as improving quality and efficiency, stimulating creativity, and promoting the content industry such as advertising material production and education and training programs from “thousands of people” to “one person with a thousand faces”. In order to further promote the positive value brought by AI technology in the field of content production, our country currently limits the scope of application of display marks, focusing them on areas that are prone to negative effects such as “confusion and misidentification”, rather than applying them one-size-fits-all.
2. In practice, the effectiveness of the logo is still facing great uncertainty
As a governance method that mainly relies on technical means, AI logos inevitably have technical limitations. One is “easy to evade”, a Harvard University study pointed out that “under some clear assumptions, if attackers have simple and common capabilities, it is impossible to make slight modifications to the content and achieve strong watermark (logo) management.” [9] In addition, illegal users often do not operate through official APIs, but directly download open-source models and deploy training in local or anonymous environments, which bypasses compliance mechanisms such as watermark embedding and identity authentication from the beginning. [10] Extraterritorial open-source models such as Stable Diffusion can easily remove watermark components by attackers, resulting in unrestrained and watermark-free content. [11]
The second is “easy to forge”, that is, by imitating watermark (logo) embedding, creating fake watermarks (logos) under non-original models or unauthorized users, misleading traceability and attribution judgments, or marking human content as AI-generated [12].
The third is “easy to misjudge”. Taking text detection as an example, the study found that traditional methods (such as KGW algorithms) use vocabulary ratio to determine whether it is AI-generated, which is easy to cause misjudgment. [12] As reported by the media, well-known literary works such as “Preface to the Tengwang Pavilion” were also misjudged as “100% AI rate”. [13] In response, the professional said: “Due to the ever-changing nature of AI-generated content, AI detection can be misjudged. Although it is possible to reduce the false positive rate of AI detection through technological improvements, it is impossible to completely eliminate it.” [14]
In addition, the AI identification system also faces cost challenges. Introducing techniques such as nested watermarking can indeed improve the reliability of detection, but the computational resources required to decode layer by layer during the validation process may even exceed the generation itself. [15] A relevant study by Harvard University pointed out that in the context of imperfect tools, determining whether a piece of content is generated by AI may be a “costly, inefficient, and arbitrary process”. [16]
To sum up, at the current stage, the feasibility and effectiveness of logos are still full of uncertainty, let alone the realization of “once and for all” AI content governance, and we should avoid placing too high expectations on it. To give full play to the technical utility of the logo, it needs to be included in the governance system for overall consideration.
3. Clarify the strengths and weaknesses of AI logos and return to the fundamental logic of content governance
At present, the content problems caused by AI are still mainly concentrated in the fields of rumors and false propaganda, and the “more” and “truer” of such information pose real challenges to content governance. As a governance tool with “clear strengths and weaknesses”, AI technology logos should give full play to their “longboard” advantages, and at the same time rely on the existing content governance system to make up for the “shortcomings”, so as to maximize the overall governance efficiency. Specifically:
The first is to embed the identification tool into the existing content governance system and reasonably define the positioning and function of the identification scheme. Unlike Europe and the United States, which are limited by regulatory restrictions and lack of grasp, so they compromise and choose AI logos as a means of content governance, AI logos are only one of the tools in our country’s mature and sound content governance system. With the goal of creating a clear cyberspace, our country’s content ecology has established a sound system from users to platforms, from regulatory systems to community rules. As a part of the system, AI identification solutions still serve the fundamental goal of content governance. To this end, in the design of the system, our country is also focusing on the field of preventing “confusion and misidentification”, that is, to reduce the misidentification of highly realistic AI-generated content as much as possible, which in turn causes secondary risks such as rumor spread, fraud, and impersonation infringement.
The “long board” of the AI identification system is to improve identification efficiency, enhance user vigilance, and provide information verification buffers at the front end of governance, rather than making substantive judgments on the authenticity of content. At present, there are still a large number of illegal applications of AI technology that escape from the “logo” system, such as using extraterritorial models, choosing highly concealed communication channels, etc., and still rely on the original content governance measures such as complaints and reports, illegal identification, and account disposal where the labeling mechanism is “insufficient”, and laws and regulations such as the Civil Code, the Advertising Law, and the Consumer Rights Protection Law can also provide a clear law enforcement basis for such illegal acts.
The second is the “longboard” function of AI logos, which can focus on high-risk areas and respond to outstanding problems. Similar to the situation faced by rumor governance: “If the distortion of information content is used as the criterion, the huge amount of online rumor information obviously exceeds the existing social governance capacity”, so “it is necessary to set up different governance mechanisms in a step-by-step manner according to the degree of harm of online rumors”, and the fundamental purpose of governance is not to completely eliminate rumors, but to “minimize their social harm”. [17] Similarly, the focus of AI logos is not to cover all AI-generated content, but to identify and intervene in high-risk areas: for example, for rumors and false propaganda, we can focus on existing technology and regulatory resources to respond, and better coordinate with existing content governance measures (such as user reporting, notification deletion mechanisms, blacklist account management).
In contrast, for low-risk fields, such as data synthesis to meet model training needs, graphics rendering for processing and polishing purposes, and B-end applications in vertical industries, non-public communication fields with less risk may be explored. The EU’s Artificial Intelligence Act also adopts a variety of exemptions and exceptions for labeling, including: natural persons can clearly identify interactive objects, synthesize content for artistic expression, or have passed manual review, so that identification is not mandatory. The consensus principle reflected in this is that the implementation of the identification mechanism should match the degree of content risk, audience identification ability and actual dissemination scope, etc., to avoid counterproductive effects caused by excessive application of identification.
The third is to reasonably define the responsibilities of the generation platform and the communication platform under the existing conditions. Compared with the synchronous generation of logos by the generation platform in the content generation process, the detection and identification of the logo by the communication platform has significantly increased in terms of input cost and technical difficulty. Deal with the influx of multi-source content, which is prone to misjudgment, missed judgment or unrecognizability. Therefore, for communication platforms, governance needs to be inclusive and incentive, and more consideration should be given to whether the platform has achieved the goal of content governance as a whole, rather than pursuing “no omissions” in form. For this reason, neither California AB 730 and California SB 942 nor the EU’s Artificial Intelligence Act directly impose the responsibility of labeling on communication platforms. In the final analysis, the effectiveness of the communication platform in content governance is still in the exploration stage.
epilogue
With the rapid popularization and penetration of AI technology, AI-generated content will inevitably become the norm of information production, and the boundary between “artificial” and “intelligent” will become increasingly blurred, and the goal of content governance will still return to the nature of content itself. In addition to empowering governance in high-risk areas such as rumors and false propaganda, in the future where AI creation is ubiquitous, we should strengthen information literacy education and guide the public to establish an objective understanding of information media or more basic work.
References:
[1] Nandu Big Data Research Institute. “Generate rumors with one click! 50 domestic AI public opinion risk cases, AI rumors account for 20%.” Southern Metropolis Daily, 19 Feb. 2025, m.mp.oeeee.com/a/BAAFRD0000202502191052861.html. Accessed May 21, 2025.
[2] Ren Jing. “AI Rumor Public Opinion Characteristics and Risk Judgment.” Public Opinion Center of the Rule of Law Network, 12 May 2025, mp.weixin.qq.com/s/-1JtEBLOfYWYsWZs0Kcyog. Accessed May 21, 2025.
[3] Guangzhou Daily. “Deng Chao and Sun Li studio, issued a solemn statement.” 18 May 2025. https://mp.weixin.qq.com/s/ckJmhMYKqWBaKFX_LzAJnQ.
[4] “This Internet celebrity hot compress ordered by millions of people, even the spokesperson is fake!” People’s Daily, 28 Apr. 2025, https://mp.weixin.qq.com/s/m2BatFp6uXz-miaQFWpT0w.
[5] “Scenes are generated with one click, it is difficult to distinguish between true and false pictures and texts, and there is actually …… behind the AI batch rumors” China Internet Joint Rumor Refuting Platform, 11 July 2024, www.piyao.org.cn/20240711/0ad6f46ed21e480f8147c8b5bd4263e9/c.html. Accessed May 21, 2025.
[6] Cyber Security Bureau, Ministry of Public Security. “Using AI to wash manuscripts and spread rumors, Xi’an police punished many people in accordance with the law.” Cyber Security Bureau of the Ministry of Public Security, 27 Mar. 2024, mp.weixin.qq.com/s/lZjp_8HT_5eNJHNUFDCseQ. Accessed May 21, 2025.
[7] Departmental regulations: “Provisions on the Administration of Deep Synthesis of Internet Information Services”, “Interim Measures for the Management of Generative Artificial Intelligence Services”, “Regulations on the Administration of Internet Information Service Algorithm Recommendation”; Normative documents: “Measures for the Identification of Artificial Intelligence Generated Synthetic Content”; National standard: “Network Security Technology Artificial Intelligence Generated Synthetic Content Identification Method”
[8] Sacha Altay, Fabrizio Gilardi, People are skeptical of headlines labeled as AI-generated, even if true or human-made, because they assume full AI automation, PNAS Nexus, Volume 3, Issue 10, October 2024, pgae403, https://doi.org/10.1093/pnasnexus/pgae403
[9] Zhang, Hanlin, et al. Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models. Harvard University, 23 July 2024. arXiv, arxiv.org/abs/2311.04378.
[10] Burgess, Matt. “Criminals Have Created Their Own ChatGPT Clones.” WIRED, 7 Aug. 2023, https://www.wired.com/story/chatgpt-scams-fraudgpt-wormgpt-crime/. As early as 2023, the technology media outlet WIRED reported on the availability of such black language models, noting that they have taken a very different path from legitimate LLM services from the beginning: “Since the beginning of July, criminals have been peddling two large language models they claim to have developed on dark web forums and marketplaces. These systems …… Mimicking the features of ChatGPT and Google Bard…… But unlike LLMs developed by legitimate companies, these chatbots are marketed for illegal activities. …… These “black industry LLMs” remove any form of security protection or ethical restrictions. ”
[11] Hu, Yuepeng, et al. Stable Signature is Unstable: Removing Image Watermark from Diffusion Models. Duke University, 12 May 2024. arXiv:2405.07145. https://arxiv.org/abs/2405.07145.
[12] Dong, Ziping, et al. Imperceptible but Forgeable: Practical Invisible Watermark Forgery via Diffusion Models. The State Key Laboratory of Blockchain and Data Security, Zhejiang University, 28 Mar. 2025. arXiv:2503.22330.
[13] https://mp.weixin.qq.com/s/TeU3tNYPYSIp_FqCIvNQ3g
[14] “AI Detection Overturning Scene: 100% AI-Generated “Tengwang Pavilion Preface”? The measured results are here.” Yangtze Evening News, 10 May 2025, https://mp.weixin.qq.com/s/3sMO9U7lyGntot0qbQxBqA.
[15] Sowmya S., Sahana Karanth, and Sharath Kumar. “Protection of Data Using Image Watermarking Technique.” Global Transitions Proceedings, vol. 2, 2021, pp. 386–391. Elsevier, doi:10.1016/j.gltp.2021.08.035.
[16] Srinivasan, Siddarth. “Detecting AI Fingerprints: A Guide to Watermarking and Beyond.” Brookings Institution, 8 May 2024, https://www.brookings.edu/articles/detecting-ai-fingerprints-a-guide-to-watermarking-and-beyond/.
[17] Zhao Jingwu, Chen Yixuan. “Thought: A New Treatment of ‘Internet Rumors’.” Journal of Jurisprudence, 18 May 2025, https://mp.weixin.qq.com/s/SXl8YoM6JQIFI8663hnAfQ.
