On Algorithmic Impact Assessments

The AI Policy Trojan Horse

I’ve published two pieces in the last week or so: an essay for The New Atlantis on the future of language models, and a piece for Cointelegraph on the need for robust protocols in AI governance.

Introduction

Last week, Senator Ted Cruz added an amendment to the Senate’s Future of AI Innovation Act. The amendment began with a series of provisions that read more like a tweet thread than legislation, banning all federal agencies from promulgating any policymaking documents that “require, recommend, promote, or encourage” a variety of concepts, such as:

(1) One race or sex is inherently superior to another race or sex.

(2) The United States is fundamentally racist or sexist.

(3) An individual, by virtue of his or her race or sex, is inherently racist, sexist, or oppressive, whether consciously or unconsciously.

It goes on, and I suspect you get the idea. What is this doing in a bill about AI, though? The last few paragraphs make that clear. Keep in mind that the concepts below would be banned from federal agency rules, guidelines, and regulations with respect to AI:

(10) Artificial intelligence, algorithms, or other automated systems should be designed in an equitable way that prevents disparate impacts based on a protected class or other societal classification.

(11) Input data used by designers, developers, or deployers of artificial intelligence, algorithms, or other automated systems should be modified to prevent disparate impacts based on a protected class or other societal classification.

(12) Designers, developers, integrators, or deployers of artificial intelligence, algorithms, or other automated systems should conduct disparate impact or equity impact assessments prior to deployment or implementation of such technology to ensure inclusivity and equity in the creation, design, or development of the technology.

(13) Federal agencies should review input data used by designers, developers, or deployers of artificial intelligence, algorithms, or other automated systems to ensure the technology—

                  (A) meets the view of that Federal agency of what constitutes bias or misinformation; and

                  (B) contains no positions contrary to the position of the Federal Government

While the first nine provisions of Senator Cruz’s amendment are generalized “anti-woke” principles, the last four are aimed squarely at one of the primary emergent mechanisms of AI policymaking across America: so-called algorithmic impact assessments. Suresh Venkatasubramanian, a professor at Brown University who contributed to President Biden’s AI Bill of Rights, said the quiet part out loud on X, claiming that Senator Cruz’s amendment “guts… any future actions by the federal government on AI.”

Venkatasubramanian is probably overstating his point, but I suspect he is correct that algorithmic impact assessments could become an indelible part of American policymaking with respect to AI. Algorithmic impact assessment mandates have already made their way into proposed and enacted state laws and numerous federal legislative proposals. Regulating AI using these assessments is not only a mistake, but perhaps the worst possible way to regulate AI—and yes, that includes liability-based bills like SB 1047. Let’s dive into why.

A note on language

The phrase “impact assessment” has always bothered me. “Impact” is a word that conveys a physical strike, and physical strikes usually cause damage. Thus the phrase “impact assessment” presupposes that the thing to be assessed will have a bad effect on the world. This got us into trouble in the 1970s, when we began requiring environmental impact assessments for huge swaths of proposed infrastructure projects. The implicit idea was that any substantial construction project in the physical world would be bad for nature. Such projects are, in a sense, presumed guilty, and have to overcome the burden of proof to establish their innocence. This has made it harder to build things in America.

What are algorithmic impact assessments?

On their face, algorithmic impact assessments are meant to help government agencies and companies assess the risks of using automated decision-making tools and develop plans to mitigate those risks.

The phrase seems to have entered into widespread use in the mid-2010s, around the same time Europe General Data Protection Regulation passed. This was the heyday of “narrow” AI systems, trained to perform a discrete and highly repeatable task. For example, a bank might want to develop an automated system to approve or reject loan applications—or at least, to take a first pass at such decisions for a human to review. One way they might accomplish this is to train an AI model on the historical record of all the loan applications the bank has ever received, along with an indication of whether the application was approved or denied. And indeed, many banks did do this, and tools such as this are some of the most widely used AI systems today.

There are many questions one could ask about this: do the bank’s loan application records include time periods where the bank systematically discriminated against minorities? Do they include time periods where public policy distorted bank decision-making? Are they biased by periods of aberrant economic conditions? In broad strokes, does the distribution of historical bank application decisions reflect the decisions that the bank wishes to make going forward?

These are perfectly reasonable questions, and reflect the kind of issues that algorithmic impact assessments are at least ostensibly supposed to address.

One could also ask reasonable questions in response. For example, if there is a difference in approval or rejection rates based on some demographic characteristic, is that reflective of a systemic bias within the bank or is it reflective of facts about the world beyond the bank’s direct control? If the latter explains the difference, should banks and other large corporations be agents of social change, attempting to change these unfortunate facts about the world? Or should they just be corporations?

This is a tug-of-war that will be familiar to anyone who has paid attention to American politics for the past decade. While both sides of that conflict have veered into extremes in recent years, at root, these are issues about which reasonable people can and do disagree.

There are other, less frequently asked, questions. For example, should these automated decisions be evaluated in a vacuum, or should they be evaluated in comparison to human decisions currently made within a company? In other words, when evaluating bias in algorithmic decisions, should we ask “as opposed to what?” or should we evaluate the algorithm in isolation? It seems to me that the only analytically coherent means of evaluating these issues is to compare algorithmic decisions with purely human decisions. Otherwise, we do not know if the algorithm is an improvement over the status quo or not.

Closely related to this is the question of how one establishes evidence of discrimination. If we choose to define “discrimination” as any disparate impact on any demographic group, then discrimination will be an inevitable result of almost any decision-making process, absent a radical change to the way that the world works to enable precisely equal apportionment of all things to all demographic groups.

One can easily imagine the legitimate uses of algorithmic impact assessments. But any honest and self-reflective person who has lived in America for the past decade can probably also imagine how they might be taken too far.

Algorithmic impact for generalist AI

Algorithmic impact assessments were invented before generalist multimodal models like ChatGPT, Claude, Gemini, and Llama became mainstream. Yet the idea of using algorithmic impact assessments has only picked up steam since then. Do they translate well to our current technological reality?

On the one hand, it is indeed the case that current language models exhibit racial bias. Language models have been shown to classify speakers using African American Vernacular English as “unintelligent” or “criminals,” for example. Some research indicates that these biases remain even after post-training steps like Reinforcement Learning from Human Feedback, which is intended to mitigate issues of this kind (among many other things).

On the other hand, bias is something of a quantum phenomenon, existing in a superposition of “biased” and “not biased” until it is observed by a particular person. To use a cliche, it’s in the eye of the beholder. While the example above may be blatant, there is a galaxy of more subtle examples one could imagine. There will never be a purely unbiased language model, because it is impossible to exist in and speak intelligently about the world without exhibiting some form of bias. It is a human being’s job to exercise judgment and self-reflection about bias in any given context, and this applies when they are using AI tools. But it isn’t clear that making people fill out paperwork or otherwise legislatively beating anti-bias mantras into people is especially helpful. People don’t generally enjoy this, and in a democracy, people’s preferences do, ultimately, matter.

Because of this, it is not clear that this issue is susceptible to legislative fixes. Most laws and other forms of regulatory guidance involving algorithmic impact assessments operate by requiring companies to perform algorithmic impact assessments anytime an automated system is used to make a decision affecting consumers’ access to things like housing, education, employment, government services, and other “essential services.” This can be construed quite broadly; for instance, basic home maintenance companies like electricians or plumbers can be considered “essential services,” and thus any decision they make that affects the availability of their services—in any way—may require an algorithmic impact assessment (this was the case, for example, with Connecticut’s now-dead SB 2).  

The fundamental problem is that the uses of language models are far broader than the custom-built narrow systems of the previous decade. A business could use them for any number of customer-facing roles, from fielding inbound customer inquiries to generating proposals and invoices. Under some laws, each of these uses might require separate impact assessments, assuming they all have some unique effect on the availability of services or opportunities the state deems essential. Most businesses are ill-equipped to assess how and to what extent language models would exhibit biases in these diverse contexts. Yet most laws place the burden on the “deployer,” which often means the person or company using the system, and not the company that made it.

What’s more, many of these proposals simply focus on “algorithms,” which are just instructions for performing computations. Everything from the simplest algorithms written in a few lines of code all the way up to the most complex neural networks are covered.

Requiring businesses to go through these bureaucratic routines merely to use software is sure to hinder adoption. Just as with the environmental impact statements described above, algorithmic impact assessments implicitly presume that all algorithms are more discriminatory than human-made decisions, and seek to extend anti-discrimination measures beyond the bounds of current US law. Indeed, many algorithmic bias policies seek to eliminate all discrimination, with discrimination assumed to be taking place if any demographic group’s outcomes are different from any other’s.

For instance, here is a passage from the Biden administration’s AI Bill of Rights (emphasis added):

Any automated system should be tested to help ensure it is free from algorithmic discrimination before it can be sold or used. Protection against algorithmic discrimination should include designing to ensure equity, broadly construed.  Some algorithmic discrimination is already prohibited under existing anti-discrimination law. The expectations set out below describe proactive technical and policy steps that can be taken to not only reinforce those legal protections but extend beyond them to ensure equity for underserved communities even in circumstances where a specific legal protection may not be clearly established. These protections should be instituted throughout the design, development, and deployment process and are described below roughly in the order in which they would be instituted.

The document goes on to state that deployers of any automated decision-making system must “demonstrate that the system protects against algorithmic discrimination.” Thus, the Biden administration’s AI Bill of Rights—and nearly all other government documents like it—presumes that algorithmic discrimination is happening, putting the burden of proof on businesses seeking to use AI. This document is not formal policy; it’s more of a set of guidelines. But inasmuch as the AI Bill of Rights articulates the intended direction of policy, it is worrisome.

Indeed, the wide-ranging nature of these proposals makes them easy to view cynically. Rather than being a targeted attempt to mitigate genuine issues with modern discrimination, these efforts strike me as a power grab: an attempt by the state to seize control over the distribution of AI throughout the economy. Because the presence of any disparity in outcomes is treated as evidence for discrimination, so long as any disparities exist, there will be an avenue for the state to involve itself in decisions made by “the algorithm.” This is a convenient sleight of hand, since the algorithm’s decisions are really the business’s decisions, and algorithms can be used in service of effectively any business objective. These policies thus constitute a broad and novel method for the state to inject itself into economic activity using AI bias as a pretext.

Conclusion

If I am right, it's no wonder that Venkatasubramanian, one of the authors of the AI Bill of Rights, was so dismayed by Senator Cruz’s amendment. Yet Senator Cruz is pushing in the right direction. Algorithmic impact assessments are an extension of the same litigious, rule-laden, endlessly process-based prison that has made it impossible to build new things in America (and often, even, to repair old things), and has made new ideas in many domains an inconvenience rather than an opportunity. Policymakers should be seeking to deconstruct this prison wherever possible, not let it expand into our remaining sources of dynamism.

Unfortunately, many policy experts have grown all-too familiar with this intellectual prison. We may have criticisms of the prison—some of us believe the cafeteria should serve better food, while others think we need to fix the plumbing. But ultimately, the prison is what we know, and we are not sure there is a world outside its OSHA-notice-covered walls.

We can’t help but bring the prison to everything we think about, because our thinking is so often part of the prison—these new cells are not ADA compliant!—helping to sustain it on a daily basis whether we realize it or not. Sometimes, we are even the beneficiaries of it, in the sense that we, uniquely, understand how to navigate it. Too often, AI policy involves bringing the digital world further into the prison, adding more process, more paperwork, and more complexity of the type that kills the human creative spirit, bit by bit and day by day.

Avoiding this outcome—fighting it with all the wit and energy we can muster, and reinventing the mechanisms and methods of policymaking itself along the way—is in my view the principal task of AI policy. Radical technological change from outside the prison walls is our last hope of breaking free.