California's Other Big AI Bill

On AB 3211, an under-discussed deepfake bill

Introduction

Across the US, states have proposed more than 600 AI-related bills in this year’s legislative sessions. The sheer number alone can be distracting, however. Most of those bills stand little chance of becoming law. Of those that do become law, many are anodyne, creating committees to study the impact of AI on some aspect of state governance or the state economy, clarifying that existing laws apply to AI, or tasking an agency with creating a report related to how AI effects that agency’s regulatory jurisdiction. The number of AI bills that contain meaningful new regulations and have a chance of passing is, as one might expect, much smaller.  

The most significant bill in that category is SB 1047, the bill from California State Senator Scott Wiener. Indeed, I believe that the debate over SB 1047 is the most important AI policy debate in the country, which is why I’ve written about this bill extensively here and elsewhere. But it’s far from the only AI bill worth discussing; in fact, it’s far from the only AI bill in California worth discussing. The state’s legislature alone has proposed several dozen AI bills. Today, I want to focus on just one of them: AB 3211, the California Provenance, Authenticity, and Watermarking Standards Act, written by Assemblymember Buffy Wicks, who represents the East Bay.

Like SB 1047, AB 3211 has passed in one of the chambers of the California legislature, so it has a real chance of becoming law. Yet in some ways, this bill is considerably more aggressive than SB 1047, mandating major requirements for all AI developers (no compute or other thresholds to protect startups), large websites and apps, and even camera makers.

Sacramento (California’s capital) is a famously insular place. The legislators do not often take kindly to public criticism of their work (my own criticisms very much included). As a result, they can—and do—regularly pass impactful bills with little public debate. And in the AI community, the conversation has been almost exclusively focused on SB 1047, ignoring the other bills that stand a serious chance of passing. I suspect that I contributed to this dynamic by also ignoring those other bills—an error I’m working to correct with today’s post. Let’s dive in.

AB 3211: The Big Picture

This bill is about precisely what it says on the tin: mandating the use of technical standards that will allow internet users to determine what content is generated by AI and what is generated by humans (like SB 1047, this bill would likely apply at least throughout the United States). How does it do this? A few ways:

• Requires generative AI providers of all kinds to “place an imperceptible and maximally indelible watermark into synthetic content produced or significantly modified by a generative AI system.”

• Requires generative AI providers to proactively identify their products as AI models and receive the user’s affirmative consent to speak to an AI model every time it is used.

• Requires generative AI providers to maintain a public database containing digital fingerprints of every model output that could plausibly be considered “deceptive,” meaning that it could plausibly be confused with human-generated content.

• Requires any website or app with more than 1 million California-based users to label synthetic and non-synthetic content, and maintain a database of “potentially deceptive content” for the California Department of Technology.

• Requires makers of any “recording device” sold in California—anything with a camera or a microphone—to offer users the option to watermark the content they record using the device.

• Requires each person or company that creates generative AI models to publish annual Risk Assessment and Mitigation Reports, to be submitted, again annually, to an auditor, who is in turn mandated to use “state-of-the-art techniques” to assess the reports.

That is a lot. And while I am sympathetic to the goals of AB 3211, the bill suffers from poor drafting and goes too far in some ways.

The bill centers on standards for watermarking synthetic content. A “watermark” can mean many different things, both in the context of this bill and in the broader AI world. It might refer to metadata attached to an AI-generated video or image that specifies the model that produced it, the date it was made, etc. In the context of text-based models, it often refers to having the model output subtle patterns in the words of its response to user prompts that can be detected by a simple algorithm; to you, it just looks like a normal chatbot response, but to the right algorithm, it is a hidden message saying, in essence “this was written by AI.”

The problem with this is that, as of writing, these watermarking standards don’t work very well. I’ve written before about the flaws of C2PA, which is a technical standard for attaching provenance metadata to images and videos. The main issue with C2PA is that it is trivial for a user to remove or change the metadata it attaches. One could remove the metadata directly, by stripping it from the file, or indirectly, by, for example, taking a screenshot of the image. The screenshot you take will be identical to the AI-generated image, yet will have none of the C2PA metadata conveying that the image was generated by AI. With text, it’s even easier, because all you need to do is, well, edit the text. As the user, you don’t know which words in the model’s output are being used to encode the watermark, so it would be easy to inadvertently remove.

So in short, this is a solved problem in the sense that watermarking standards mostly exist. It is an entirely unsolved problem in the sense that watermarking standards mostly do not work, particularly if one’s intention is to deceive people with AI-generated media.

AB 3211 mandates “maximally indelible watermarks,” which it defines as “a watermark that is designed to be as difficult to remove as possible using state-of-the-art techniques and relevant industry standards.” And it mandates that these watermarks, or other metadata attached to the AI-generated media, also contain provenance data, defined as:

…information about the history of the content, including, but not limited to, the following:

(1) The name of the generative AI provider or the camera or recording device manufacturer.

(2) The name and version number of the AI system that generated the content or the operating system, version of the operating system, or the application used to capture, create, or record the content.

(3) The time and date of the content’s creation and any additional modifications of the content.

(4) The portions of content that have been changed by a generative AI system, if applicable.

This mandate would go into effect on July 1, 2026, so there is (thankfully) time for the industry to develop better standards. But the requirement of a “maximally indelible” watermark means that complying with the bill is a constantly moving target. What if the industry converges on a standard by July 2026, and then in January 2027, some new solution is discovered? Does the entire industry have to move to the new solution, even if it has not matured into a full-blown technical standard yet? By what time do they have to transition? What if the new standard has privacy or usability implications?

Or what if someone creates a more secure watermark for AI-generated images, but it requires those images to exist purely in a web interface so that the user can never download them? Does the industry have to switch because the new standard is harder to remove, even though that now means that it is effectively illegal for users to download media generated by AI models?

AB 3211 is silent about these issues. Instead it holds the AI industry to a single, exceptionally high standard: make watermarks as difficult to modify “as possible,” regardless of other tradeoffs involved in doing so. This single-minded focus on only one objective is not generally how complex systems are engineered, and it is likely to lead to bad outcomes for AI.

Overall, the bill could make the problem of AI deepfakes and other deceptive media worse, by creating a false sense of security about what is and is not synthetically generated media. A malicious user can ultimately remove any watermark applied to AI-generated outputs if they choose to (at least as of today). They can then upload the content to social media and claim, correctly as far as the social media platform is concerned, that it is authentic media. The deceptive media would then be labeled as authentic, presumably with a pleasant green badge. In that sense, AB 3211 could actually help bad actors while burdening good actors.

Big picture musings aside, let’s explore how the bill actually works and what it requires.  

The Details

The bill applies to every single generative AI model distributed in California, regardless of size, purpose, or who created it. It does not matter if you are a grad student making a small model to predict DNA sequences or a trillion-dollar company making a generalist multimodal model. Every piece of digital content generated by an AI model will need to be watermarked.

It is possible that Assemblymember Wicks does not realize that generative models can be trained to produce DNA sequences and did not intend for her bill to apply to those modalities. But apply they do. Of course, DNA is “just text,” yet the standard way of watermarking text described above would not apply well to DNA, lest we arbitrarily modify the genome of a novel organism being created in a lab in the interest of watermarking it.

As currently written, the bill also forbids platforms like HuggingFace from hosting any model that does not have watermarking standards built into it. Here’s the relevant text:

Generative AI hosting platforms shall not make available a generative AI system that does not place maximally indelible watermarks communicating provenance data into content created or substantially modified by the system in a manner consistent with specifications set forth in paragraph (1) of subdivision (a).

And “generative AI hosting platform” is defined as “an online repository or other internet website that makes generative AI systems available for download.”

I’m not sure if it’s possible for HuggingFace to robustly validate that information in every case (if you know, reach out!), but more importantly, as written this provision would seem to apply to all existing models on HuggingFace. In other words, every single generative AI model on the internet that does not perform watermarking on its outputs would be effectively unlawful, since any website with an AI model (or perhaps the threshold is technically two, since “systems” is plural in the definition) available for download is a “generative AI hosting platform” according to AB 3211.

AB 3211 also requires generative AI model makers (again, everyone from an individual to a company—there are no thresholds here) to have their models proactively identify themselves as AI. I support laws of this kind generally, but once again AB 3211 takes this to an extreme. In addition to requiring proactive disclosure that a given system is AI-powered, it also requires affirmative consent from the user—every single time it is used. From the bill:

In all conversational interfaces of a conversational AI system, the conversational AI system shall, at the beginning of a user’s interaction with the system, obtain a user’s affirmative consent acknowledging that the user has been informed that they are interacting with a conversational AI system. A conversational AI system shall obtain a user’s affirmative consent before beginning the conversation.

This means that every time you start a new chat with ChatGPT, or ping Siri on your phone, you will have to acknowledge that you are aware that you are interacting with an AI system. I do not see how anybody benefits from this.

Finally, at least in terms of the requirements on generative AI model makers, the bill specifies that developers must keep a public record of all model outputs that could be “deceptive.” To quote again from the bill:

A generative AI system capable of producing potentially deceptive content shall generate and store, in a searchable online database in a manner that can be retrieved by a viewer of the content, a digital fingerprint of and provenance data for any piece of potentially deceptive content that they produce. This provenance shall not include personally identifiable information.

To be clear, a “digital fingerprint” means a cryptographic representation of the output rather than the output itself. Still, given that almost any text a model generates—and a great deal of other media—could be “deceptive,” (the bill defines “deceptive” content as AI-generated content that could be human-generated), this means that every AI developer will need to keep a public database of many, or perhaps all, the outputs their model has ever made. In addition to being a substantial burden on all developers, this will be impossible for open-source models to comply with, because open-source model developers have no visibility into what users are doing with their models. The bill does nothing to grapple with this fact, at least in my reading.

In addition to all this, “large online platforms,” defined as any app or website with greater than 1 million users in California, must label all user-uploaded data (including text) as synthetic, partially synthetic, nonsynthetic, nonsynthetic with minor modifications, or as having no watermark. Any producer of any recording equipment has to include an option for users to watermark content. This presumably includes audio, photo, or video recording equipment (your iPhone, and also the webcam on your laptop, your video doorbell, etc.) but since the bill does not define “recording equipment,” this could be many things: fMRI machines record data, too. I won’t go too much into these other requirements, because this post is already long enough.

AB 3211 starts with good intentions. Unlike SB 1047, it is not fundamentally wrong about anything. Instead, by focusing on one goal—protect society against synthetic media—to the exclusion of all other goals, it creates a series of headaches for everyone while only questionably improving the problem it purports to address. Sadly, that sounds about right for AI policy these days.

The bill was unanimously passed in the State Assembly and is making its way through the Senate.