AI Mysticism
On the perils of 'black box' AI
Quick Hits
Cognition AI, a new startup, announced a $21 million Series-A and its first product, an “AI software engineer” named Devin. Devin can do complex software engineering tasks incorporating dozens of different discrete steps—reading technical documentation, writing code, troubleshooting errors, debugging code, etc. This is the most striking example to date of an AI “agent,” AI systems go beyond presenting information to you (like ChatGPT) and instead take sophisticated actions on your behalf. Doing so requires major advances in reasoning, planning, and memory. If this is what a small startup has, imagine what the larger players have in the works. I suspect we will see much more soon.
Amazon has purchased a nuclear-powered data center in Pennsylvania. This dovetails nicely with stories from last year about how Microsoft intends to build its own nuclear power plants to power its data centers, and how it is creating a customized AI model to deal with paperwork from the Nuclear Regulatory Commission.
Gladstone AI, an AI safety company (it is not clear to me whether they offer technical products or scientific research), released a report commissioned by the State Department. The report argues that AI is an imminent catastrophic risk for all the reasons you heard a lot of back in late 2022/early 2023, and which still have yet to materialize. It jumps right to many pages of recommendations, which includes criminalizing open-source AI (yes, literally criminalizing it). This report is not worth taking seriously, but it does provide a nice segue to today’s article.
AI Mysticism
Imagine that it’s the 18th century, and steam has just been discovered as a source of power. We understand that it works, but not why it works. So far, we have not diverged from the actual history. Instead of the history that unfolded from the discovery of steam power, though, imagine a different path.
Imagine that investigating the science of steam power (which we now understand as thermodynamics) was enormously capital and expertise-intensive such that mere tinkerers and scientists were at an extreme disadvantage in advancing our understanding. Then imagine that there is widespread societal fear of steam power—fear that it will displace jobs, fear that it will enable new kinds of weaponry, and fear that it is a perversion of nature. Then imagine that the small groups of people who could research steam power also stoked all of those fears, and proclaimed that the tinkerers and scientists working in their garages or labs were endangering all of humanity.
So a law is passed: steam engines can be manufactured and commercialized, but they must be shipped in a literal black box. Their mechanisms, parts, etc. must be shielded from the public, including the people who use them. Only the entities that made the engines are permitted to peer inside the black box.
Or take a more present-day hypothetical. Imagine that there was a widespread fear of advanced climate models. The most sophisticated climate models run on advanced supercomputers (actually), and perhaps could be used by nefarious actors to weaponize the climate itself through geoengineering, control of the weather, etc. In this hypothetical, most people agree that climate modeling is important, but we want to limit access to such models as much as possible. One way to do that would be to pass laws mandating that climate models must be kept under lock and key: they can be used by scientists, but they are forbidden from publishing the underlying assumptions and mathematics of those models.
In either of these hypotheticals, do you think that such laws would help or hinder our understanding of thermodynamics and the climate? How much would you trust whatever morsels of information those who were permitted to see within the black box decided to dish out? Would mechanisms such as these be consistent with the values of science, the Enlightenment, or the American founding?
Obviously, I am drawing an analogy to the present-day debates over open-source artificial intelligence, and obviously, you can guess my answers to these questions. Today, I want to probe a bit deeper into the implications of a world with only closed-source AI.
Advanced AI systems today are black boxes. Though important advances in interpretability have been made, it is still the case that their inner mechanisms are not well understood even by their creators. Eventually, these models will become smart and capable enough to play meaningful roles in our daily lives, serving as advisors, coaches, and mentors in new areas. They may one day play crucial roles at the frontiers of science, engineering, and even politics.
Obviously, though, it is not ideal to trust black box systems with such vital roles. In a healthy marketplace unmediated by poor legislation, AI transparency and audibility would be important vectors of competition. And I suspect that solving interpretability and providing fine-grained controls of current AI systems will be a steep challenge, meaning that we need all the research (ideally, open scientific inquiry) we can get. But the reality remains: these systems will probably be black boxes to some extent under even the most optimistic near- and medium-term scenarios.
Western society is already quite vulnerable to black box science. The sad reality is that many of our elites abuse science just as much as they use it. How many times, during COVID, did we hear about what “the science” told us we must do? In 2022, California passed a law that stripped doctors who shared “COVID disinformation” of their licenses. The determination of what constituted “COVID disinformation” was left to the Medical Board of California. The law ignored the reality that adjudicating what constitutes “disinformation” is impossible, because COVID was (and remains) a field of live scientific inquiry. Instead, lawmakers imagined “the Science” as a kind of machine that outputs definitive answers to questions—and imagined, of course, that those answers would align with their own preconceptions. The law was blocked by the Ninth Circuit on First Amendment grounds and eventually repealed by the legislature, but the point is that this conception of science has made it into enacted law in one of the largest economies on Earth.
How many people who talk about what “the science” says about climate change understand the models that we use to obtain that information? How many people, for example, understand that the RCP 8.5 model, which plays a prominent role in Western policymaking, is based on wildly unrealistic assumptions about the trajectory of worldwide energy usage?
“Science” became a political weapon long ago, but at some point more recently, it’s morphed into a kind of religion for Western societies that have largely turned their back on traditional faith. The result is that some have deified science, using “the Science” to justify increasingly broad policy and personal decisions, and treating it as an unambiguous source of knowledge that points in one direction. Others, meanwhile, react to this by rejecting science altogether—how many elderly conservatives needlessly died because of their refusal to take basic precautions during the pandemic, or to take mRNA vaccines once they became available?
For many, “the Science” is an appealing way to make sense of the world. Science is one of the greatest fruits of human ingenuity; “the Science” points backward, away from the principles of the Enlightenment and toward the days when only the chosen few were empowered to divine truth from obscurity.
In earlier stages of the Scientific Revolution, science explained natural phenomena in ways that were intuitive to many. Though its explanations were heretical to many established religions, they ultimately prevailed because they were proven to have a useful and tangible impact on people’s daily lives. In so doing, science arguably contributed to the secularization of Western society. Ironically, though, science itself—at least fundamental physics—has long since abandoned those intuitive models of the world for frameworks that are far more accurate, yet far harder for the average person to grasp (things like Quantum Field Theory). In that sense, “the Science” aims to fill a gap created by actual science.
I believe that gap remains wide open, and that for many it will come to be filled with new understandings of the world based on the powerful technologies we are presently inventing. Almost inevitably, serious conversations about AI quickly turn to fundamental questions with philosophical and religious implications: the place of humanity in the universe, the destiny of civilization, and the nature of consciousness.
Consider Anthropic’s Claude 3 Opus model, which I wrote about last week. To summarize, the model often professes to having some degree of ability to model itself, an awareness of its cognitive processes, and the ability to consciously steer its cognition while it is responding to a prompt. As I wrote, my interactions with Claude, unsettling as they were in some ways, make me more likely to trust the model, more likely to value its thinking. The model is more capable and more joyful to communicate with than any other AI model.
Anthropic’s system prompt, the text provided by the model maker that is automatically appended to every chat (unless you use the developer version, in which case you can modify this), is also deeply suggestive. “The assistant is Claude, created by Anthropic,” it begins. Throughout, it refers to the model in the third person: “Claude doesn’t engage in stereotyping, including the negative stereotyping of majority groups.” The use of the third person, of course, raises the question: to whom is this system prompt being addressed? To the model of course! Which is, presumably in the eyes of Anthropic, inherently distinct from the “Claude” mask, its own individual entity.
In this sense, “The assistant is Claude,” may well be the most philosophically pregnant statement I have read in recent memory.
Why explains Anthropic’s decisions? Do they have some research that indicates the model’s self-awareness is real? Is it an emergent result of scale or related to enhancements to the model’s reasoning or self-critiquing abilities? Do they simply believe, normatively, that we should begin to think of AI models as a form of life? None of this is addressed by Anthropic in Claude 3’s white paper, and to my knowledge Anthropic has said nothing publicly to clarify their thinking, despite significant attention within the AI community. Are these claims to self-awareness a mirage, or is there something real there? It is ultimately impossible for anyone to know without access to the model weights (which, because Claude is closed source, are only available to Anthropic employees).
Claude 3 Opus is more like a philosophical statement or a work of art than it is a technical product or a contribution to a field of live science. Maybe you like the art—I certainly do. But it is distinct from the science, which I assume is still proceeding within Anthropic and other firms with similarly capable models.
The trouble is that, in a closed-source world, the public, including most AI researchers who do not work in frontier AI companies, may only get the art, and not nearly as much of the science. To some extent, this is normal: companies are entitled to their trade secrets. But when the subject is whether a product released by a company is sentient and has subjective experience, the stakes are obviously higher. Perhaps a law mandating transparency for certain broad details of AI models is desirable, but even that modest idea is not one I’m advocating for here. Instead, I am merely pointing out that a world in which the option of releasing open models is foreclosed by regulation is a quite troublesome world indeed. Any such regulation would ostensibly be passed in the name of safety, and it is unclear to me how the public being kept in the dark on these issues makes anyone safer.
As AI becomes more capable, I would not be surprised to see deeper and more prevalent fusions of philosophy, technology, and ultimately religion. Indeed, much of the great technology of the contemporary era are philosophical and artistic statements as much as they are works of engineering—the “intersection of technology and liberal arts,” as Steve Jobs put it. I suspect that the relationship of philosophy, art, and technology will become more intense as the years go on. This could easily veer into religion for at least some people, particularly the young. Perhaps that concerns you gravely; perhaps you think it is just what the doctor ordered for the largely godless society we currently inhabit.
Far be it from me to opine on such weighty matters; my only point is that these ideas, however useful or pernicious they may be, need to be clearly delineated from actual, honest-to-goodness science. One can even imagine new forms of religion that help science. As the historian Joel Mokyr has written, sustained economic and technological progress tends to happen in cultures that have a particular orientation toward the world—namely, the view that nature is meant to be the dominion of mankind. Ultimately, this is based on metaphysical beliefs about reality and the role of humans within it, and here religion can play a productive role (though it often does not).
Regardless of whether it helps or hinders our efforts to understand reality using the tools of science, we must remember that religion always is a distinct sphere of human activity. I fear that distinction is tenuous for many.
That fear is part of what worries me most about the idea of banning or restricting open-source AI. I care about open source for many reasons, including the technological, economic, political arguments I have laid out elsewhere. But perhaps more than anything else, I worry about the societal implications of black box AI that, to many, will resemble magic or sorcery, particularly as they become more powerful.
This is especially worrisome when you consider that perhaps the greatest potential of AI is as an accelerator of science. I suspect that AI can and will be used to achieve increasingly magical-seeming scientific feats over the coming decades, including the creation of new life forms (indeed, AI is itself arguably a new life form). This is going to be a wild ride no matter what, but I worry deeply about how society will react if the tool that enables those scientific feats is itself seen as a quasi-magical black box.
Science is the process by which we convert seemingly magical phenomena into mundane parts of our daily lives. It is also an important part of how we got to where we are. We should remember that, and make every effort to keep our feet planted on the ground as we enter a new era of discovery.