A Rebuttal to Zvi Mowshowitz
Responding to some criticisms of last week’s article
Last week’s piece on California’s proposed law to regulate AI, SB 1047, got some attention, mostly thanks to a link on Marginal Revolution. Much of the feedback I’ve received has been positive, but one notable commentator, Zvi Mowshowitz, was quite critical. Zvi (I will refer to him here by his first name because that is how everyone refers to him) writes the Substack Don’t Worry About the Vase and is known in the AI world for his detailed weekly writeups of noteworthy events in the field (I recommend subscribing if you want to follow AI in detail). He’s ridiculously prolific (his weekly output on Substack is well in excess of 10,000 words) and, more importantly, consistently thoughtful and probing. Because Zvi is someone I respect, I want to take some time to respond to his critique of my piece. I’ll have another post this week that “moves the ball forward” on my project more substantively.
First, it’s important to understand that Zvi and I disagree profoundly on the issue of AI existential risk and therefore, quite likely, on the nature and limits of intelligence itself. That global disagreement animates our local disagreement on SB 1047, but I want to focus on the local issues here—I suspect we’re unlikely to persuade one another on the bigger issues.
Zvi begins (my emphasis in italics):
Ah, Tyler Cowen has a link on [SB 1047] and it’s… California’s Effort to Strange AI (my post from last week).
Because of course it is. We do this every time. People keep saying ‘this law will ban satire’ or spreadsheets or pictures of cute puppies or whatever, based on what on its best day would be a maximalist anti-realist reading of the proposal, if it were enacted straight with no changes and everyone actually enforced it to the letter.
It is correct that I am reading the text of this law as it currently stands and assuming that it will be, or at least could be, enforced more or less to its letter (you will note that I did not discuss the bill’s overbroad definition of “artificial intelligence,” because I suspect that this in particular would be narrowed down in court). I do not see a practicable alternative way of analyzing legislation; I find it odd to characterize such a reading of proposed law as “maximalist and anti-realist.” I suspect most judges and lawyers would, too, particularly for a new law on a topic that has little related case law on which to draw.
Zvi makes assertions throughout his response to me that suggest I am anti-AI regulation; I in fact wrote an article (which will be coming out in National Affairs next month) advocating for reporting requirements for frontier models—something the Biden administration did with its October 2023 Executive Order on AI (imperfectly, though the Order as written allows for evolution over time). I argued just a couple of weeks ago for mandatory safety and reliability standards for models in specific industries.
Zvi objects to my interpretation of SB 1047’s definition of a covered model. As a reminder, SB 1047 defines a “covered model” as any model that is trained using 10^26 or greater FLOPS (floating-point operations) or any model that matches such models in performance benchmarks or any model that is below benchmarks but has the same “general capability” as one that is above the benchmarks (does general here mean “general purpose” or does it mean “it can generally do X specific capability”?). I argue that this could apply to many models developed by startups in the future; he argues otherwise:
Um, no, because the open model weights models do not remotely reach the performance level of OpenAI?
Maybe some will in the future.
But this very clearly does not ‘ban all open source.’ There are zero existing open model weights models that this bans.
There are a handful of companies that might plausibly have to worry about this in the future, if OpenAI doesn’t release GPT-5 for a while, but we’re talking Mistral and Meta, not small start-ups. And we’re talking about them exactly because they would be trying to fully play with the big boys in that scenario.
I would contend that in the future, as compute becomes more available and improvements to model architectures, training practices, and training data continue, open-source models that match the performance of GPT-4 will become more common. Already, open-source models that match or exceed GPT-3.5 are common; why would we expect this to stop with GPT-4? I’d also note, for the record, that Mistral is a French startup founded less than a year ago that almost certainly has fewer than 50 employees. In what world should Mistral be mentioned in the same breath as Meta, a firm with more than 60,000 staff and a market capitalization of more than $1 trillion?
To be quite clear, my hope is that many companies can “play with the big boys,” as Zvi says. OpenAI itself was not considered “one of the big boys” just a few years ago. I do not believe we should pathologize “playing with the big boys,” I think we should encourage it, and believe that this legislation does the opposite.
Zvi continues by asserting that I am incorrect in my statement that SB 1047 imposes the precautionary principle before a model is trained. He asserts instead (italics added):
Bell (sic) is also wrong about the precautionary principle being imposed before training.
I do not see any such rule here. What I see is that if you cannot show that your model will definitely be safe before training, then you have to wait until after the training run to certify that it is safe.
Here is what SB 1047 says (my emphasis in bold):
Before initiating training of a covered model that is not a derivative model that is not the subject of a positive safety determination, and until that covered model is the subject of a positive safety determination, the developer of that covered model shall do all of the following:
(1) Implement administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, or misuse or unsafe modification of, the covered model, including to prevent theft, misappropriation, malicious use, or inadvertent release or escape of the model weights from the developer’s custody, that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.
(2) Implement the capability to promptly enact a full shutdown of the covered model.
(3) Implement all covered guidance.
(much more in subsections (4) – (9))
The bill imposes many requirements, many of which are likely to be time-consuming, on model developers who are unable to provide a “positive safety determination” before training. I think Zvi is unambiguously wrong here, and I’m not sure there’s much else to say.
Zvi notes that the bill has implicit exemptions for “derivative models,” which would presumably refer to open-source models built on top of open-source foundation models like Meta’s Llama series or Mistral’s Mixtral. I’m unsure about this: the bill’s definition of derivative model is a “modified or unmodified copy,” or a model with additional software, the former of which sounds like open source. But it makes an exception for “entirely independently trained [AI] models.” This is ambiguous; if I fine-tune Llama, I have performed model training “entirely independently” of its creator, but I have not trained an “entirely independent” model. I’m not a lawyer, so I won’t speculate, and in fact will assume Zvi is correct.
However, even if Zvi’s interpretation of the bill’s “derivative model” provisions is right, the onerous requirements the bill places upon foundation models make it unlikely (at least in my interpretation) that future foundation models will be feasible to open source. If the foundation models themselves are (practically) unlawful to open source, then there is not much use in having the freedom to make “derivative models” from them. My concern here is not “how would this law affect current players and current models today.” My concern is how this law will affect the AI field in the years to come. This is why I wrote in my introduction to last week’s piece that the legislation could effectively outlaw new open-source models. I probably should have said new open-source foundation models to be perfectly clear about what I meant. Mea culpa.
Zvi also understates the stakes this proposed bill imposes. Here's an excerpt from Zvi’s summary of the law:
So here’s what the law would actually do, as far as I can tell:
1. If your model is not projected to be state of the art level and it is not over the 10^26 limit no one has hit yet and no one except the big three are anywhere near, this law has only trivial impact upon you, it is a trivial amount of paperwork. Every other business in America and especially the state of California is jealous.
2. If your model is a derivative of an existing model, you’re fine, that’s it.
3. If your model you want to train is projected to be state of the art, but you can show it is safe before you even train it, good job, you’re golden.
Zvi fails to mention at any point in his rebuttal that being wrong about a positive safety determination is, per the bill, perjury—not a fine, not a loss of license, a felony. Any developer who claims their model is safe, if they are incorrect, risks four years in jail. Perhaps this particular provision will not stand: how can an inherently uncertain claim about the future be characterized as “knowingly false testimony,” as perjury usually requires? Still, though, I choose to read and interpret the bill as written.
Zvi concludes by comparing me to “the boy who cried wolf,” and by saying that I do not understand how regulation and lawmaking actually works.
The arguments against such rules often come from the implicit assumption that we enforce our laws as written, reliably and without discretion. Which we don’t. What would happen if, as Eliezer recently joked, the law actually worked the way critics of such regulations claim that it does? If every law was strictly enforced as written, with no common sense used, as they warn will happen? And someone our courts could handle the case loads involved? Everyone would be in jail within the week.
I am indeed worried about what would happen if we enforced this law as it is written. Lots of law is enforced as it is written; ask anyone who has tried to build a nuclear power plant in the United States in the last half century. I would also note that a major theme of American governance over the past two decades has been executive agencies stretching existing law to its breaking point in the interest of exercising as much power as they can. It seems reasonable to me to assume that the same dynamic will apply with this law.
Further, a lot of the reason that laws are not enforced to their letter as originally written is because laws are interpreted in court. These interpretations are the result of litigation, which is time-consuming and expensive—the exact sort of thing that often slows down innovation, bankrupts young companies, and kills new ideas before they have the chance to be tried.
But I would also agree that not all laws are enforced “reliably and without discretion.” Indeed, that is itself an issue that deeply concerns me, because once you realize that laws are enforced unreliably and with discretion, the issue quickly becomes whose discretion is decisive. The administrative state, the exercisers of discretion in this case, is often motivated by political currents. It’s not an ideological issue; both Republican and Democratic policymakers do this. As AI becomes more politically charged (as I suspect it will), do we believe that the same thing won’t happen here? When Eliezer Yudkowsky joked about the fact that law is not enforced evenly, I took his joke as a criticism of the current American legal system.
The status quo of overregulation, overcriminalization, and unevenly enforced law that Zvi is articulating as a defense of this legislation is one that almost nobody is happy with. One should not defend it simply because it happens to serve as a partial justification for poorly conceived legislation, or because one suspects that the regulators with the discretion will likely side with one’s own priorities. Indeed, this attitude is part of what enabled the American administrative state to become as unwieldy as it has.
Zvi finishes:
The idea that this would “spell the end of America’s leadership in AI” is laughable. If you think America’s technology industry cannot stand a whiff of regulation, I mean, do they know anything about America or California? And have they seen the other guy? Have they seen American innovation across the board, almost entirely in places with rules orders of magnitude more stringent?
Having lived in America my entire life and having devoted most of my professional life to the analysis of public policy in America generally, and in American states in particular, I like to think I do know a bit about America and California—though I am always trying to learn more. I have also seen the other guy, and I consider it a personal ambition to do everything I can to stop America’s government from becoming more like, say, the European Union’s. And I have seen “American innovation across the board”—outside of the technology industry, I have to say, my joy is less than full. I have never been more hopeful for American innovation outside of traditional Silicon Valley bounds (in things like, say, construction or chemical production or manufacturing), but a lot of that hope involves the application of AI and other advanced software to these fields. Hence why I worry about regulating AI!
The problem is not that America lacks innovative people and firms; instead, among America’s key challenges in innovating outside of software is allowing new ideas to be tried in the market and to diffuse throughout society on their own merits.That is what regulation most directly harms, and that is precisely the concern I have with SB 1047. The irony is that, at least when it comes to things like housing and nuclear regulation, I believe Zvi largely agrees with me here. I’m not sure “America’s regulatory state is already bloated, so why not throw in another sweeping set of rules” is a bumper sticker anyone wants on their car. Nor is “let’s make building AI more like building airplanes; we still sort of build airplanes, after all!” I certainly don’t anticipate putting either of those bumper stickers on my car anytime soon.