Full exemption from tort liability seems a bit too extreme -- I think there needs to be some strong incentive on *outcomes*, not just *process*. It'll be too hard for regulations to cover all the risks and mitigations. The real threat of liability from bad outcomes will force companies to think more creatively about possible risks and fully account for knightian uncertainty.
But I like the spirit of it -- liabilities should be greatly reduced for a company that's been given a high score by a validated private auditor. The amount of liability should be a function of both the damage and the negligence, and if the company gets a high score for safety, then they weren't being negligent.
Thanks John! There are definitely ways to soften the liability protections. For example, above a certain size of harm, the protection could switch from safe harbor to a rebuttable presumption that the developer met the standard of care. Another option is to make companies retroactively liable for harms that were egregious enough to trigger their certification being revoked.
As I see it, there are two problems with approaches like this. If you pursued something like the first option, you would need to find a way to estimate damages ex ante, since usually damages are determined in the litigation process. This is also a problem SB 1047 faced, but it is probably solvable.
In general, I am not sure that negligence liability as an incentive on AI lab behavior will do what the safety community wants it to do. Catastrophic harms are going to be existential financial threats for most of the relevant players (everyone except Big Tech). When liability exposure is that broad, there is empirical evidence from other industries that firms have a tendency to ignore such risks. They are so large that they are not worth doing about, a bit like what TSMC has said about how they think about planning for a Taiwan invasion--"we'd be so royally, fractally, hopelessly screwed in that situation that it is not worth us really thinking about all that much."
And then there is the question of how courts would react to tort lawsuits for lesser harms. This is a whole can of worms, but my basic conclusion is that we should not trust courts to make the right decisions here. They could make decisions that would very badly harm AI development, for not very much gain at all. Happy to explain my reasoning on this, if helpful.
Interesting proposal. I think it ends up becoming something like the equivalent of insurance against a future tort liability, which definitely has useful features and also overhead. That might even be a more direct method. The thing that I have the biggest question with is that none of the AI safety research or assessments I have seen are nearly good enough to be enough for a certification. Definitely not enough to have any views on whether it could cause particular forms of mayhem.
Agreed re: insurance. But the problem with insurance is straightforward: no insurer really wants to touch this space, because no one can quantitatively model the risks. To do that, we need real-world deployment data--lots of it. And to do that, we need to accelerate real-world deployment, which this proposal is intended to do. Ultimately, I expect that optimal governance mechanisms would converge around a price system of some kind--either insurance literally or insurance by another name. But you cannot will a market into existence when the market does not "want" to exist, which is where we currently stand.
I think that view of AI safety is still accurate, though much less accurate than it would have been a year ago. There really have been appreciable advancements, and I anticipate this to continue such that, in a year or so (the earliest possible time this system could be put into place), things will be in a more mature state. It'll still be evolving, though, because the technology moves quickly.
I'm not arguing for insurance, just noting that the fact that nobody can price the risks well enough to provide insurance is a great case and point of how we are basically dealing with unknown unknowns at best. Certification agencies will have the exact same problem.
And agreed we need more deployment data. That'll happen in due course.
>no insurer really wants to touch this space, because no one can quantitatively model the risks.
I understand there are various rich people in Silicon Valley like Marc Andreesen who claim that AI risks are low. Would they like to step in as insurers? They could put a big part of their fortune in escrow in order to cover liability.
If AI boosters aren't willing to put their money where their mouth is when it comes to safety, that seems highly suspicious.
True, most standards today only support process audits. We are likely some time away from establishing concrete, measurable technical standards that would enable certification testing, especially across safety layers (e.g. input/output classifiers, feature clamping, etc.). Developing such standards would require standard-setting bodies to have an agility and coordination with frontier labs that just does not seem to be present today.
Hi, Dean, a very timely topic, I agree that tort liability is a serious problem. I would be nervous that the standards bodies would be susceptible to government interference, much like the regional accrediting bodies for higher education, which also became memetic around certain issues. The issue of how to prevent unelected technocrats from imposing their world views on society is a broader one, and perhaps one of the most pressing of our time. P.S. Here is my own small contribution to the AI governance issue: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5095857
Agreed! But at the end of the day, if *someone* sets standards, there will always be concerns about the motivations and character of those people. The way this proposal tries to mitigate that is by allowing the governance bodies to compete with one another, but I admit that is imperfect.
A bit late to the discussion here. I think this is an interesting proposal. Oddly enough, this reminds me of my own industry: mortgage finance. This follows the private governance model in some respects, though it is still a heavily regulated area. The system tries to balance local knowledge about borrower quality, with a national interest in a stable housing system and affordable credit. It is extremely imperfect, but perhaps we can learn some useful lessons.
The basic structure is: the government-sponsored mortgage and housing entities ("GSEs," i.e. Fannie Mae, Freddie Mac, and Ginnie Mae) bear the vast majority of residential mortgage credit risk. They set minimum standards that each loan needs to meet, but they delegate final authority to private mortgage lenders, who in turn delegate to individual employees (loan officers), who are presumed to know the most about an individual borrower's financial situation and creditworthiness. If a lender fails to live up to its obligations (too many delinquent loans, too aggressive in "churning" borrowers or otherwise putting them in bad loans, too poorly capitalized, etc.) they can get kicked out of whatever programs they participate in. In extreme cases, they can be prosecuted. In this way, the regulators are able to maintain broad standards without having to get into the weeds of each individual loan.
What are some lessons?
• Regulator quality matters a lot. Fannie and Freddie are large and somewhat sophisticated institutions, and are able to set clear guidelines and monitor the behavior of mortgage originators (the people who make the loans) and servicers (the people who collect payments, foreclose on houses, and otherwise interact with borrowers). Ginnie Mae, on the other hand, has always been chronically under-staffed. The two main Ginnie Mae programs are FHA and VA, and originators that focus on this space tend to be smaller, less well capitalized, and generally more "fly by night." If we're doing private governance for AI, we need to make sure that the standards-setting organizations and the state authorizing bodies are actually competent. We can't just assume good institutions into existence.
• The border between public and private governance is always up for debate. In some market cycles, originators have lots of leeway; in others, the GSEs tighten it up. The GSEs themselves have a regulator (first the Office of Federal Housing Enterprise Oversight, then the Federal Housing Finance Agency after OFHEO manifestly failed in its duties pre-2008). The FHFA, GSEs, and mortgage market participants are always trying to push boundaries and claim more territory. If we're creating new regulatory bodies, we need to make sure they have the political ability to defend themselves.
• The consequences of bad behavior may not appear for years, even in a large-scale industry like mortgage lending. It takes time for loans to go bad. In a decent economy, questionable borrowers may be able to make their payments for years, only to default when a recession strikes.
• Liability protection can be a huge help, as we've seen in other industries. One of the major features of the 2010 Dodd-Frank Act was something called the "Qualified Mortgage Safe Harbor," where lenders were protected from certain claims, if the loan satisfies the "Qualified Mortgage" requirements. A QM loan satisfies basic requirements for documentation, debt-to-income ratio, fees, and structure (i.e. avoidance of risky features characteristic of the pre-2008 housing boom). If you make a QM loan, the lender is protected from many legal claims. Non-QM loans can still be made (e.g. "fix-and-flip" loans), and this is where a lot of innovation and entrepreneurship happens. But the lenders have to be a lot more careful.
• Financial ruin may not be a sufficient deterrent for bad behavior, especially if enforcement is inconsistent or weak. You may need stronger penalties, like the ability to ban people from the industry, or send people to jail. Many bad actors from pre-2008 got right back into the industry. If an AI company is negligent, you have to make sure the founders can't just go and do another startup.
I don't think there are any insurmountable problems here. But policymakers would do well to keep some of these lessons in mind.
There must be other industries with a similar regulatory framework. What are some examples? What else can we learn?
Great work - I love this as a plausible structure for governance that doesn't route through government policy regimes.
I'll note that we proposed something complementary to this, and narrower but similar, for using non-government standards boards for AI audit in a paper last year, https://arxiv.org/abs/2404.13060, which is now forthcoming in AI & Society.
Steven Vladic has a somewhat related proposal for self-driving cars. His primary concern is that the injured parties should be made whole promptly and not need to wait for litigation to establish fault. His proposal is that each car would be a legal person. It must carry liability insurance. If it injures someone, that person can sue the car and get a payout without having to determine which party (programmers, manufacturer, owner, safety monitor, etc.) is liable. That question is delegated to the insurance company, which, over time, is incentivized to set premiums that each party must pay based on actual experience.
Notice that Vladic's proposal to use insurance works because the number of harmed parties will be small and localized and the harms are well-scoped. Agentic AI can achieve "harm at scale", and as other commenters have noted, this is likely to bankrupt the insurers. Hence, insurance fails both to incentivize care on the part of the AI providers and to make harmed parties whole.
Is the heart of the problem that it is difficult to anticipate harms and assess their probability of occurrence? Perhaps we could design a prediction market in which forecasters predict harms and place bets that harms of a well-specified type will occur within a fixed time period. The AI providers must pay these bets if the harms occur, but conversely, the forecasters must pay the companies if the harms do not occur within the time period. This incentivizes the forecasters to be accurate and incentivizes the AI providers to make their own assessment and take action to prevent the harms that they believe would lead to a lost bet.
The main weakness of this approach is the information asymmetry between the forecasters and the companies. There is also a risk of insiders placing bets and then NOT mitigating the risk of harm in order to win the bet. I'm a novice at designs of this type, so I'm sure there are other weaknesses. But I hope this is an idea worth thinking about.
I'll check this out! Worth noting that my proposal only applies to digital goods, not hardware (self driving cars, robots etc). In those domains I suspect normal liability will work fine. It is specifically the development of AI, which I believe to be a platform technology, where I worry about the viability of tort.
I realized after writing that my forecasting proposal is similar to insurance except that the third party (the forecasters) get a payout when a harm occurs rather than when it does not (because there are no premiums). A massive harm would bankrupt the AI providers rather than the insurers. Hence, it probably fails as an incentive.
It seems to me that the Big Four are perfectly positioned to take the lead here. They don’t just have the global reach and institutional trust, they also have strong incentives to secure this role and the influence to help shape how it plays out. They’ve been steadily expanding into AI consulting and risk management and their close ties to regulators across industries make them natural candidates to become to go-to AI auditors, especially in highly regulated sectors like finance and healthcare.
As someone who is relatively new to the world of AI Governance, this was an interesting read. What would be the setup if, for example, an AI startup that was above board in all the ways you have outlined here, creates an 'agent' that has been tasked with providing financial advice and support for the elderly online, ends up actually scamming their elderly clients out of millions of dollars? Is the startup immune to any legal action or other consequences (or responsibility for reimbursing the clients) simply because they are certified and therefore proven to be mitigating risks to the best of their knowledge/ability? Or would this type of event automatically cause their certification to be revoked and legal action taken? If the former, then it has the potential to lead to all sorts of bad/unjust outcomes, if the latter, then it doesn't seem to remove much risk at all for the AI firms. I don't have a better alternative in mind that bypasses this trade-off, of course!
The outcome of the scenario you've raised would depend upon a few factors. Most importantly, did the developer itself deploy the model? Or did they sell it to other companies (eg wealth management firms) who then deployed it? In all likelihood, something as egregious as you are describing would be caught early by any firm complying in good faith with basic safety and security standards for agents. So if something like this did slip through, it would probably indicate some sort of malfeasance on the part of the developer. But as with all things in law, it is fact dependent.
It wasn't clear to me at the outset that these were the goals of the proposal. It seems like there was, in addition, at least some *implicit* goal of helping manage catastrophic AI risk. But if it's supposed to do that, I would like the argument for why it's expected to do so to be made more explicitly.
Here's a very simplified version of the proposal:
1) government licenses regulators
2) regulators can certify AI developers
3) certified AI developers have protection from tort liability.
Basically, it's regulatory markets, but instead of being mandatory, it's opt-in with the carrot of a liability shield.
Pros:
- Standard PRO of regulatory markets: regulators may have the more expertise than government.
- The proposal creates an incentive for AI developers to actually meet technical standards.
- It may indeed be politically feasible.
Cons:
- Standard CON of regulatory markets: there's a clear incentive for regulators to have low standards (since their clients are AI developers).
- It's unclear how the government is meant to ensure that private regulators have meaningful standards.
- In practice, the technical standards are unlikely to be adequate; we currently lack any means of providing meaningful assurance for advanced AI systems.
- The proposal creates a liability shield. However, liability seems like one of the best protections against risks that can't be mitigated via technical standards (e.g. unknown unknowns).
- The proposal subsizes AI development (since the government is doing the work of licensing regulators).
Overall, I believe the political feasibility of this proposal comes from the "opt-in carrot" nature of the proposal. I was already not sold on regulatory markets (as I think it's hard to imagine them not leading to regulatory capture). But this proposal also subsidizes AI development, which I think is a step in the wrong direction.
I can see why this might be appealing to people who think we can 80/20 x-risk if we get leading developers to do something like "implement sensible RSPs". I don't fall into that camp. One reason, which I think should be uncontroversial: we have no plan for how to assure ASL4-type systems won't end humanity, or coordinating to prevent their development in the absence of appropriate assurance.
Thanks Dean, I think this is great & very valuable work, and I empathize strongly with its direction. I'm a bit concerned about how this interfaces with two axes along which AI is set to get 'bigger' soon - the perception of transformative deployments by labs leading them to forego any participation in this system because they're not scared of liability; and the incentives of a securitised AI race to undercut a lot of promising features of the setup you describe.
I guess what is to stop a very cavalier private regulator setting up and providing certification with the least oversight possible?
Tort seems to imply good calibration (especially as, in the case of 1047 it only applies to very large damages). What process here makes the private regulator care 10x as much about 10x larger risks?
This would be addressed by (1) the judgment of oversight bodies in each state that authorizes this (that is a TON of oversight btw), and (2) the mechanism whereby government can retract the license of a private body and eliminate safe harbor for ALL covered companies
(2) is a really expensive thing for a state to do. There are major ramifications to doing so. So the harm would need to be quite bad. This is what incentivizes the private regulator to care especially about major harms.
Full exemption from tort liability seems a bit too extreme -- I think there needs to be some strong incentive on *outcomes*, not just *process*. It'll be too hard for regulations to cover all the risks and mitigations. The real threat of liability from bad outcomes will force companies to think more creatively about possible risks and fully account for knightian uncertainty.
But I like the spirit of it -- liabilities should be greatly reduced for a company that's been given a high score by a validated private auditor. The amount of liability should be a function of both the damage and the negligence, and if the company gets a high score for safety, then they weren't being negligent.
Thanks John! There are definitely ways to soften the liability protections. For example, above a certain size of harm, the protection could switch from safe harbor to a rebuttable presumption that the developer met the standard of care. Another option is to make companies retroactively liable for harms that were egregious enough to trigger their certification being revoked.
As I see it, there are two problems with approaches like this. If you pursued something like the first option, you would need to find a way to estimate damages ex ante, since usually damages are determined in the litigation process. This is also a problem SB 1047 faced, but it is probably solvable.
In general, I am not sure that negligence liability as an incentive on AI lab behavior will do what the safety community wants it to do. Catastrophic harms are going to be existential financial threats for most of the relevant players (everyone except Big Tech). When liability exposure is that broad, there is empirical evidence from other industries that firms have a tendency to ignore such risks. They are so large that they are not worth doing about, a bit like what TSMC has said about how they think about planning for a Taiwan invasion--"we'd be so royally, fractally, hopelessly screwed in that situation that it is not worth us really thinking about all that much."
And then there is the question of how courts would react to tort lawsuits for lesser harms. This is a whole can of worms, but my basic conclusion is that we should not trust courts to make the right decisions here. They could make decisions that would very badly harm AI development, for not very much gain at all. Happy to explain my reasoning on this, if helpful.
Interesting proposal. I think it ends up becoming something like the equivalent of insurance against a future tort liability, which definitely has useful features and also overhead. That might even be a more direct method. The thing that I have the biggest question with is that none of the AI safety research or assessments I have seen are nearly good enough to be enough for a certification. Definitely not enough to have any views on whether it could cause particular forms of mayhem.
Agreed re: insurance. But the problem with insurance is straightforward: no insurer really wants to touch this space, because no one can quantitatively model the risks. To do that, we need real-world deployment data--lots of it. And to do that, we need to accelerate real-world deployment, which this proposal is intended to do. Ultimately, I expect that optimal governance mechanisms would converge around a price system of some kind--either insurance literally or insurance by another name. But you cannot will a market into existence when the market does not "want" to exist, which is where we currently stand.
I think that view of AI safety is still accurate, though much less accurate than it would have been a year ago. There really have been appreciable advancements, and I anticipate this to continue such that, in a year or so (the earliest possible time this system could be put into place), things will be in a more mature state. It'll still be evolving, though, because the technology moves quickly.
I'm not arguing for insurance, just noting that the fact that nobody can price the risks well enough to provide insurance is a great case and point of how we are basically dealing with unknown unknowns at best. Certification agencies will have the exact same problem.
And agreed we need more deployment data. That'll happen in due course.
>no insurer really wants to touch this space, because no one can quantitatively model the risks.
I understand there are various rich people in Silicon Valley like Marc Andreesen who claim that AI risks are low. Would they like to step in as insurers? They could put a big part of their fortune in escrow in order to cover liability.
If AI boosters aren't willing to put their money where their mouth is when it comes to safety, that seems highly suspicious.
True, most standards today only support process audits. We are likely some time away from establishing concrete, measurable technical standards that would enable certification testing, especially across safety layers (e.g. input/output classifiers, feature clamping, etc.). Developing such standards would require standard-setting bodies to have an agility and coordination with frontier labs that just does not seem to be present today.
Hi, Dean, a very timely topic, I agree that tort liability is a serious problem. I would be nervous that the standards bodies would be susceptible to government interference, much like the regional accrediting bodies for higher education, which also became memetic around certain issues. The issue of how to prevent unelected technocrats from imposing their world views on society is a broader one, and perhaps one of the most pressing of our time. P.S. Here is my own small contribution to the AI governance issue: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5095857
Agreed! But at the end of the day, if *someone* sets standards, there will always be concerns about the motivations and character of those people. The way this proposal tries to mitigate that is by allowing the governance bodies to compete with one another, but I admit that is imperfect.
Will read!
Why not apply this idea to the FDA? Have the FDA only certify for safety, and have private bodies all certify efficacy.
I actually pitched something like this at a Foresight Institute event recently! Fully agreed.
A bit late to the discussion here. I think this is an interesting proposal. Oddly enough, this reminds me of my own industry: mortgage finance. This follows the private governance model in some respects, though it is still a heavily regulated area. The system tries to balance local knowledge about borrower quality, with a national interest in a stable housing system and affordable credit. It is extremely imperfect, but perhaps we can learn some useful lessons.
The basic structure is: the government-sponsored mortgage and housing entities ("GSEs," i.e. Fannie Mae, Freddie Mac, and Ginnie Mae) bear the vast majority of residential mortgage credit risk. They set minimum standards that each loan needs to meet, but they delegate final authority to private mortgage lenders, who in turn delegate to individual employees (loan officers), who are presumed to know the most about an individual borrower's financial situation and creditworthiness. If a lender fails to live up to its obligations (too many delinquent loans, too aggressive in "churning" borrowers or otherwise putting them in bad loans, too poorly capitalized, etc.) they can get kicked out of whatever programs they participate in. In extreme cases, they can be prosecuted. In this way, the regulators are able to maintain broad standards without having to get into the weeds of each individual loan.
What are some lessons?
• Regulator quality matters a lot. Fannie and Freddie are large and somewhat sophisticated institutions, and are able to set clear guidelines and monitor the behavior of mortgage originators (the people who make the loans) and servicers (the people who collect payments, foreclose on houses, and otherwise interact with borrowers). Ginnie Mae, on the other hand, has always been chronically under-staffed. The two main Ginnie Mae programs are FHA and VA, and originators that focus on this space tend to be smaller, less well capitalized, and generally more "fly by night." If we're doing private governance for AI, we need to make sure that the standards-setting organizations and the state authorizing bodies are actually competent. We can't just assume good institutions into existence.
• The border between public and private governance is always up for debate. In some market cycles, originators have lots of leeway; in others, the GSEs tighten it up. The GSEs themselves have a regulator (first the Office of Federal Housing Enterprise Oversight, then the Federal Housing Finance Agency after OFHEO manifestly failed in its duties pre-2008). The FHFA, GSEs, and mortgage market participants are always trying to push boundaries and claim more territory. If we're creating new regulatory bodies, we need to make sure they have the political ability to defend themselves.
• The consequences of bad behavior may not appear for years, even in a large-scale industry like mortgage lending. It takes time for loans to go bad. In a decent economy, questionable borrowers may be able to make their payments for years, only to default when a recession strikes.
• Liability protection can be a huge help, as we've seen in other industries. One of the major features of the 2010 Dodd-Frank Act was something called the "Qualified Mortgage Safe Harbor," where lenders were protected from certain claims, if the loan satisfies the "Qualified Mortgage" requirements. A QM loan satisfies basic requirements for documentation, debt-to-income ratio, fees, and structure (i.e. avoidance of risky features characteristic of the pre-2008 housing boom). If you make a QM loan, the lender is protected from many legal claims. Non-QM loans can still be made (e.g. "fix-and-flip" loans), and this is where a lot of innovation and entrepreneurship happens. But the lenders have to be a lot more careful.
• Financial ruin may not be a sufficient deterrent for bad behavior, especially if enforcement is inconsistent or weak. You may need stronger penalties, like the ability to ban people from the industry, or send people to jail. Many bad actors from pre-2008 got right back into the industry. If an AI company is negligent, you have to make sure the founders can't just go and do another startup.
I don't think there are any insurmountable problems here. But policymakers would do well to keep some of these lessons in mind.
There must be other industries with a similar regulatory framework. What are some examples? What else can we learn?
Lots to digest here! thank you. I sadly cannot reply in depth due to some travel and writing commitments, but I encourage others to do so.
Great work - I love this as a plausible structure for governance that doesn't route through government policy regimes.
I'll note that we proposed something complementary to this, and narrower but similar, for using non-government standards boards for AI audit in a paper last year, https://arxiv.org/abs/2404.13060, which is now forthcoming in AI & Society.
Steven Vladic has a somewhat related proposal for self-driving cars. His primary concern is that the injured parties should be made whole promptly and not need to wait for litigation to establish fault. His proposal is that each car would be a legal person. It must carry liability insurance. If it injures someone, that person can sue the car and get a payout without having to determine which party (programmers, manufacturer, owner, safety monitor, etc.) is liable. That question is delegated to the insurance company, which, over time, is incentivized to set premiums that each party must pay based on actual experience.
Notice that Vladic's proposal to use insurance works because the number of harmed parties will be small and localized and the harms are well-scoped. Agentic AI can achieve "harm at scale", and as other commenters have noted, this is likely to bankrupt the insurers. Hence, insurance fails both to incentivize care on the part of the AI providers and to make harmed parties whole.
Is the heart of the problem that it is difficult to anticipate harms and assess their probability of occurrence? Perhaps we could design a prediction market in which forecasters predict harms and place bets that harms of a well-specified type will occur within a fixed time period. The AI providers must pay these bets if the harms occur, but conversely, the forecasters must pay the companies if the harms do not occur within the time period. This incentivizes the forecasters to be accurate and incentivizes the AI providers to make their own assessment and take action to prevent the harms that they believe would lead to a lost bet.
The main weakness of this approach is the information asymmetry between the forecasters and the companies. There is also a risk of insiders placing bets and then NOT mitigating the risk of harm in order to win the bet. I'm a novice at designs of this type, so I'm sure there are other weaknesses. But I hope this is an idea worth thinking about.
I'll check this out! Worth noting that my proposal only applies to digital goods, not hardware (self driving cars, robots etc). In those domains I suspect normal liability will work fine. It is specifically the development of AI, which I believe to be a platform technology, where I worry about the viability of tort.
I realized after writing that my forecasting proposal is similar to insurance except that the third party (the forecasters) get a payout when a harm occurs rather than when it does not (because there are no premiums). A massive harm would bankrupt the AI providers rather than the insurers. Hence, it probably fails as an incentive.
It seems to me that the Big Four are perfectly positioned to take the lead here. They don’t just have the global reach and institutional trust, they also have strong incentives to secure this role and the influence to help shape how it plays out. They’ve been steadily expanding into AI consulting and risk management and their close ties to regulators across industries make them natural candidates to become to go-to AI auditors, especially in highly regulated sectors like finance and healthcare.
could be!
but are they institutional innovators? perhaps--but I'd like to see them in competition with upstarts.
As someone who is relatively new to the world of AI Governance, this was an interesting read. What would be the setup if, for example, an AI startup that was above board in all the ways you have outlined here, creates an 'agent' that has been tasked with providing financial advice and support for the elderly online, ends up actually scamming their elderly clients out of millions of dollars? Is the startup immune to any legal action or other consequences (or responsibility for reimbursing the clients) simply because they are certified and therefore proven to be mitigating risks to the best of their knowledge/ability? Or would this type of event automatically cause their certification to be revoked and legal action taken? If the former, then it has the potential to lead to all sorts of bad/unjust outcomes, if the latter, then it doesn't seem to remove much risk at all for the AI firms. I don't have a better alternative in mind that bypasses this trade-off, of course!
Thank you!
The outcome of the scenario you've raised would depend upon a few factors. Most importantly, did the developer itself deploy the model? Or did they sell it to other companies (eg wealth management firms) who then deployed it? In all likelihood, something as egregious as you are describing would be caught early by any firm complying in good faith with basic safety and security standards for agents. So if something like this did slip through, it would probably indicate some sort of malfeasance on the part of the developer. But as with all things in law, it is fact dependent.
(copying a couple of comments from Miles Brundage's google doc: https://docs.google.com/document/d/1ftDGn83gCqoc45Z_vQ0mcuPNukdb3DYBx5fsQw6PLZI/edit?tab=t.0)
It wasn't clear to me at the outset that these were the goals of the proposal. It seems like there was, in addition, at least some *implicit* goal of helping manage catastrophic AI risk. But if it's supposed to do that, I would like the argument for why it's expected to do so to be made more explicitly.
Here's a very simplified version of the proposal:
1) government licenses regulators
2) regulators can certify AI developers
3) certified AI developers have protection from tort liability.
Basically, it's regulatory markets, but instead of being mandatory, it's opt-in with the carrot of a liability shield.
Pros:
- Standard PRO of regulatory markets: regulators may have the more expertise than government.
- The proposal creates an incentive for AI developers to actually meet technical standards.
- It may indeed be politically feasible.
Cons:
- Standard CON of regulatory markets: there's a clear incentive for regulators to have low standards (since their clients are AI developers).
- It's unclear how the government is meant to ensure that private regulators have meaningful standards.
- In practice, the technical standards are unlikely to be adequate; we currently lack any means of providing meaningful assurance for advanced AI systems.
- The proposal creates a liability shield. However, liability seems like one of the best protections against risks that can't be mitigated via technical standards (e.g. unknown unknowns).
- The proposal subsizes AI development (since the government is doing the work of licensing regulators).
Overall, I believe the political feasibility of this proposal comes from the "opt-in carrot" nature of the proposal. I was already not sold on regulatory markets (as I think it's hard to imagine them not leading to regulatory capture). But this proposal also subsidizes AI development, which I think is a step in the wrong direction.
I can see why this might be appealing to people who think we can 80/20 x-risk if we get leading developers to do something like "implement sensible RSPs". I don't fall into that camp. One reason, which I think should be uncontroversial: we have no plan for how to assure ASL4-type systems won't end humanity, or coordinating to prevent their development in the absence of appropriate assurance.
Thanks Dean, I think this is great & very valuable work, and I empathize strongly with its direction. I'm a bit concerned about how this interfaces with two axes along which AI is set to get 'bigger' soon - the perception of transformative deployments by labs leading them to forego any participation in this system because they're not scared of liability; and the incentives of a securitised AI race to undercut a lot of promising features of the setup you describe.
I wrote all this down in some more detail here: https://writing.antonleicht.me/p/three-notes-on-dean-balls-private
Honestly think worth posting this on LessWrong. probably a lot of pushback but maybe some good comments
I guess what is to stop a very cavalier private regulator setting up and providing certification with the least oversight possible?
Tort seems to imply good calibration (especially as, in the case of 1047 it only applies to very large damages). What process here makes the private regulator care 10x as much about 10x larger risks?
This would be addressed by (1) the judgment of oversight bodies in each state that authorizes this (that is a TON of oversight btw), and (2) the mechanism whereby government can retract the license of a private body and eliminate safe harbor for ALL covered companies
(2) is a really expensive thing for a state to do. There are major ramifications to doing so. So the harm would need to be quite bad. This is what incentivizes the private regulator to care especially about major harms.