Full exemption from tort liability seems a bit too extreme -- I think there needs to be some strong incentive on *outcomes*, not just *process*. It'll be too hard for regulations to cover all the risks and mitigations. The real threat of liability from bad outcomes will force companies to think more creatively about possible risks and fully account for knightian uncertainty.
But I like the spirit of it -- liabilities should be greatly reduced for a company that's been given a high score by a validated private auditor. The amount of liability should be a function of both the damage and the negligence, and if the company gets a high score for safety, then they weren't being negligent.
Thanks John! There are definitely ways to soften the liability protections. For example, above a certain size of harm, the protection could switch from safe harbor to a rebuttable presumption that the developer met the standard of care. Another option is to make companies retroactively liable for harms that were egregious enough to trigger their certification being revoked.
As I see it, there are two problems with approaches like this. If you pursued something like the first option, you would need to find a way to estimate damages ex ante, since usually damages are determined in the litigation process. This is also a problem SB 1047 faced, but it is probably solvable.
In general, I am not sure that negligence liability as an incentive on AI lab behavior will do what the safety community wants it to do. Catastrophic harms are going to be existential financial threats for most of the relevant players (everyone except Big Tech). When liability exposure is that broad, there is empirical evidence from other industries that firms have a tendency to ignore such risks. They are so large that they are not worth doing about, a bit like what TSMC has said about how they think about planning for a Taiwan invasion--"we'd be so royally, fractally, hopelessly screwed in that situation that it is not worth us really thinking about all that much."
And then there is the question of how courts would react to tort lawsuits for lesser harms. This is a whole can of worms, but my basic conclusion is that we should not trust courts to make the right decisions here. They could make decisions that would very badly harm AI development, for not very much gain at all. Happy to explain my reasoning on this, if helpful.
Interesting proposal. I think it ends up becoming something like the equivalent of insurance against a future tort liability, which definitely has useful features and also overhead. That might even be a more direct method. The thing that I have the biggest question with is that none of the AI safety research or assessments I have seen are nearly good enough to be enough for a certification. Definitely not enough to have any views on whether it could cause particular forms of mayhem.
Agreed re: insurance. But the problem with insurance is straightforward: no insurer really wants to touch this space, because no one can quantitatively model the risks. To do that, we need real-world deployment data--lots of it. And to do that, we need to accelerate real-world deployment, which this proposal is intended to do. Ultimately, I expect that optimal governance mechanisms would converge around a price system of some kind--either insurance literally or insurance by another name. But you cannot will a market into existence when the market does not "want" to exist, which is where we currently stand.
I think that view of AI safety is still accurate, though much less accurate than it would have been a year ago. There really have been appreciable advancements, and I anticipate this to continue such that, in a year or so (the earliest possible time this system could be put into place), things will be in a more mature state. It'll still be evolving, though, because the technology moves quickly.
I'm not arguing for insurance, just noting that the fact that nobody can price the risks well enough to provide insurance is a great case and point of how we are basically dealing with unknown unknowns at best. Certification agencies will have the exact same problem.
And agreed we need more deployment data. That'll happen in due course.
True, most standards today only support process audits. We are likely some time away from establishing concrete, measurable technical standards that would enable certification testing, especially across safety layers (e.g. input/output classifiers, feature clamping, etc.). Developing such standards would require standard-setting bodies to have an agility and coordination with frontier labs that just does not seem to be present today.
Hi, Dean, a very timely topic, I agree that tort liability is a serious problem. I would be nervous that the standards bodies would be susceptible to government interference, much like the regional accrediting bodies for higher education, which also became memetic around certain issues. The issue of how to prevent unelected technocrats from imposing their world views on society is a broader one, and perhaps one of the most pressing of our time. P.S. Here is my own small contribution to the AI governance issue: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5095857
Agreed! But at the end of the day, if *someone* sets standards, there will always be concerns about the motivations and character of those people. The way this proposal tries to mitigate that is by allowing the governance bodies to compete with one another, but I admit that is imperfect.
Great work - I love this as a plausible structure for governance that doesn't route through government policy regimes.
I'll note that we proposed something complementary to this, and narrower but similar, for using non-government standards boards for AI audit in a paper last year, https://arxiv.org/abs/2404.13060, which is now forthcoming in AI & Society.
Steven Vladic has a somewhat related proposal for self-driving cars. His primary concern is that the injured parties should be made whole promptly and not need to wait for litigation to establish fault. His proposal is that each car would be a legal person. It must carry liability insurance. If it injures someone, that person can sue the car and get a payout without having to determine which party (programmers, manufacturer, owner, safety monitor, etc.) is liable. That question is delegated to the insurance company, which, over time, is incentivized to set premiums that each party must pay based on actual experience.
Notice that Vladic's proposal to use insurance works because the number of harmed parties will be small and localized and the harms are well-scoped. Agentic AI can achieve "harm at scale", and as other commenters have noted, this is likely to bankrupt the insurers. Hence, insurance fails both to incentivize care on the part of the AI providers and to make harmed parties whole.
Is the heart of the problem that it is difficult to anticipate harms and assess their probability of occurrence? Perhaps we could design a prediction market in which forecasters predict harms and place bets that harms of a well-specified type will occur within a fixed time period. The AI providers must pay these bets if the harms occur, but conversely, the forecasters must pay the companies if the harms do not occur within the time period. This incentivizes the forecasters to be accurate and incentivizes the AI providers to make their own assessment and take action to prevent the harms that they believe would lead to a lost bet.
The main weakness of this approach is the information asymmetry between the forecasters and the companies. There is also a risk of insiders placing bets and then NOT mitigating the risk of harm in order to win the bet. I'm a novice at designs of this type, so I'm sure there are other weaknesses. But I hope this is an idea worth thinking about.
I'll check this out! Worth noting that my proposal only applies to digital goods, not hardware (self driving cars, robots etc). In those domains I suspect normal liability will work fine. It is specifically the development of AI, which I believe to be a platform technology, where I worry about the viability of tort.
I realized after writing that my forecasting proposal is similar to insurance except that the third party (the forecasters) get a payout when a harm occurs rather than when it does not (because there are no premiums). A massive harm would bankrupt the AI providers rather than the insurers. Hence, it probably fails as an incentive.
It seems to me that the Big Four are perfectly positioned to take the lead here. They don’t just have the global reach and institutional trust, they also have strong incentives to secure this role and the influence to help shape how it plays out. They’ve been steadily expanding into AI consulting and risk management and their close ties to regulators across industries make them natural candidates to become to go-to AI auditors, especially in highly regulated sectors like finance and healthcare.
As someone who is relatively new to the world of AI Governance, this was an interesting read. What would be the setup if, for example, an AI startup that was above board in all the ways you have outlined here, creates an 'agent' that has been tasked with providing financial advice and support for the elderly online, ends up actually scamming their elderly clients out of millions of dollars? Is the startup immune to any legal action or other consequences (or responsibility for reimbursing the clients) simply because they are certified and therefore proven to be mitigating risks to the best of their knowledge/ability? Or would this type of event automatically cause their certification to be revoked and legal action taken? If the former, then it has the potential to lead to all sorts of bad/unjust outcomes, if the latter, then it doesn't seem to remove much risk at all for the AI firms. I don't have a better alternative in mind that bypasses this trade-off, of course!
The outcome of the scenario you've raised would depend upon a few factors. Most importantly, did the developer itself deploy the model? Or did they sell it to other companies (eg wealth management firms) who then deployed it? In all likelihood, something as egregious as you are describing would be caught early by any firm complying in good faith with basic safety and security standards for agents. So if something like this did slip through, it would probably indicate some sort of malfeasance on the part of the developer. But as with all things in law, it is fact dependent.
It wasn't clear to me at the outset that these were the goals of the proposal. It seems like there was, in addition, at least some *implicit* goal of helping manage catastrophic AI risk. But if it's supposed to do that, I would like the argument for why it's expected to do so to be made more explicitly.
Here's a very simplified version of the proposal:
1) government licenses regulators
2) regulators can certify AI developers
3) certified AI developers have protection from tort liability.
Basically, it's regulatory markets, but instead of being mandatory, it's opt-in with the carrot of a liability shield.
Pros:
- Standard PRO of regulatory markets: regulators may have the more expertise than government.
- The proposal creates an incentive for AI developers to actually meet technical standards.
- It may indeed be politically feasible.
Cons:
- Standard CON of regulatory markets: there's a clear incentive for regulators to have low standards (since their clients are AI developers).
- It's unclear how the government is meant to ensure that private regulators have meaningful standards.
- In practice, the technical standards are unlikely to be adequate; we currently lack any means of providing meaningful assurance for advanced AI systems.
- The proposal creates a liability shield. However, liability seems like one of the best protections against risks that can't be mitigated via technical standards (e.g. unknown unknowns).
- The proposal subsizes AI development (since the government is doing the work of licensing regulators).
Overall, I believe the political feasibility of this proposal comes from the "opt-in carrot" nature of the proposal. I was already not sold on regulatory markets (as I think it's hard to imagine them not leading to regulatory capture). But this proposal also subsidizes AI development, which I think is a step in the wrong direction.
I can see why this might be appealing to people who think we can 80/20 x-risk if we get leading developers to do something like "implement sensible RSPs". I don't fall into that camp. One reason, which I think should be uncontroversial: we have no plan for how to assure ASL4-type systems won't end humanity, or coordinating to prevent their development in the absence of appropriate assurance.
Thanks Dean, I think this is great & very valuable work, and I empathize strongly with its direction. I'm a bit concerned about how this interfaces with two axes along which AI is set to get 'bigger' soon - the perception of transformative deployments by labs leading them to forego any participation in this system because they're not scared of liability; and the incentives of a securitised AI race to undercut a lot of promising features of the setup you describe.
I guess what is to stop a very cavalier private regulator setting up and providing certification with the least oversight possible?
Tort seems to imply good calibration (especially as, in the case of 1047 it only applies to very large damages). What process here makes the private regulator care 10x as much about 10x larger risks?
This would be addressed by (1) the judgment of oversight bodies in each state that authorizes this (that is a TON of oversight btw), and (2) the mechanism whereby government can retract the license of a private body and eliminate safe harbor for ALL covered companies
(2) is a really expensive thing for a state to do. There are major ramifications to doing so. So the harm would need to be quite bad. This is what incentivizes the private regulator to care especially about major harms.
Full exemption from tort liability seems a bit too extreme -- I think there needs to be some strong incentive on *outcomes*, not just *process*. It'll be too hard for regulations to cover all the risks and mitigations. The real threat of liability from bad outcomes will force companies to think more creatively about possible risks and fully account for knightian uncertainty.
But I like the spirit of it -- liabilities should be greatly reduced for a company that's been given a high score by a validated private auditor. The amount of liability should be a function of both the damage and the negligence, and if the company gets a high score for safety, then they weren't being negligent.
Thanks John! There are definitely ways to soften the liability protections. For example, above a certain size of harm, the protection could switch from safe harbor to a rebuttable presumption that the developer met the standard of care. Another option is to make companies retroactively liable for harms that were egregious enough to trigger their certification being revoked.
As I see it, there are two problems with approaches like this. If you pursued something like the first option, you would need to find a way to estimate damages ex ante, since usually damages are determined in the litigation process. This is also a problem SB 1047 faced, but it is probably solvable.
In general, I am not sure that negligence liability as an incentive on AI lab behavior will do what the safety community wants it to do. Catastrophic harms are going to be existential financial threats for most of the relevant players (everyone except Big Tech). When liability exposure is that broad, there is empirical evidence from other industries that firms have a tendency to ignore such risks. They are so large that they are not worth doing about, a bit like what TSMC has said about how they think about planning for a Taiwan invasion--"we'd be so royally, fractally, hopelessly screwed in that situation that it is not worth us really thinking about all that much."
And then there is the question of how courts would react to tort lawsuits for lesser harms. This is a whole can of worms, but my basic conclusion is that we should not trust courts to make the right decisions here. They could make decisions that would very badly harm AI development, for not very much gain at all. Happy to explain my reasoning on this, if helpful.
Interesting proposal. I think it ends up becoming something like the equivalent of insurance against a future tort liability, which definitely has useful features and also overhead. That might even be a more direct method. The thing that I have the biggest question with is that none of the AI safety research or assessments I have seen are nearly good enough to be enough for a certification. Definitely not enough to have any views on whether it could cause particular forms of mayhem.
Agreed re: insurance. But the problem with insurance is straightforward: no insurer really wants to touch this space, because no one can quantitatively model the risks. To do that, we need real-world deployment data--lots of it. And to do that, we need to accelerate real-world deployment, which this proposal is intended to do. Ultimately, I expect that optimal governance mechanisms would converge around a price system of some kind--either insurance literally or insurance by another name. But you cannot will a market into existence when the market does not "want" to exist, which is where we currently stand.
I think that view of AI safety is still accurate, though much less accurate than it would have been a year ago. There really have been appreciable advancements, and I anticipate this to continue such that, in a year or so (the earliest possible time this system could be put into place), things will be in a more mature state. It'll still be evolving, though, because the technology moves quickly.
I'm not arguing for insurance, just noting that the fact that nobody can price the risks well enough to provide insurance is a great case and point of how we are basically dealing with unknown unknowns at best. Certification agencies will have the exact same problem.
And agreed we need more deployment data. That'll happen in due course.
True, most standards today only support process audits. We are likely some time away from establishing concrete, measurable technical standards that would enable certification testing, especially across safety layers (e.g. input/output classifiers, feature clamping, etc.). Developing such standards would require standard-setting bodies to have an agility and coordination with frontier labs that just does not seem to be present today.
Hi, Dean, a very timely topic, I agree that tort liability is a serious problem. I would be nervous that the standards bodies would be susceptible to government interference, much like the regional accrediting bodies for higher education, which also became memetic around certain issues. The issue of how to prevent unelected technocrats from imposing their world views on society is a broader one, and perhaps one of the most pressing of our time. P.S. Here is my own small contribution to the AI governance issue: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5095857
Agreed! But at the end of the day, if *someone* sets standards, there will always be concerns about the motivations and character of those people. The way this proposal tries to mitigate that is by allowing the governance bodies to compete with one another, but I admit that is imperfect.
Will read!
Why not apply this idea to the FDA? Have the FDA only certify for safety, and have private bodies all certify efficacy.
I actually pitched something like this at a Foresight Institute event recently! Fully agreed.
Great work - I love this as a plausible structure for governance that doesn't route through government policy regimes.
I'll note that we proposed something complementary to this, and narrower but similar, for using non-government standards boards for AI audit in a paper last year, https://arxiv.org/abs/2404.13060, which is now forthcoming in AI & Society.
Steven Vladic has a somewhat related proposal for self-driving cars. His primary concern is that the injured parties should be made whole promptly and not need to wait for litigation to establish fault. His proposal is that each car would be a legal person. It must carry liability insurance. If it injures someone, that person can sue the car and get a payout without having to determine which party (programmers, manufacturer, owner, safety monitor, etc.) is liable. That question is delegated to the insurance company, which, over time, is incentivized to set premiums that each party must pay based on actual experience.
Notice that Vladic's proposal to use insurance works because the number of harmed parties will be small and localized and the harms are well-scoped. Agentic AI can achieve "harm at scale", and as other commenters have noted, this is likely to bankrupt the insurers. Hence, insurance fails both to incentivize care on the part of the AI providers and to make harmed parties whole.
Is the heart of the problem that it is difficult to anticipate harms and assess their probability of occurrence? Perhaps we could design a prediction market in which forecasters predict harms and place bets that harms of a well-specified type will occur within a fixed time period. The AI providers must pay these bets if the harms occur, but conversely, the forecasters must pay the companies if the harms do not occur within the time period. This incentivizes the forecasters to be accurate and incentivizes the AI providers to make their own assessment and take action to prevent the harms that they believe would lead to a lost bet.
The main weakness of this approach is the information asymmetry between the forecasters and the companies. There is also a risk of insiders placing bets and then NOT mitigating the risk of harm in order to win the bet. I'm a novice at designs of this type, so I'm sure there are other weaknesses. But I hope this is an idea worth thinking about.
I'll check this out! Worth noting that my proposal only applies to digital goods, not hardware (self driving cars, robots etc). In those domains I suspect normal liability will work fine. It is specifically the development of AI, which I believe to be a platform technology, where I worry about the viability of tort.
I realized after writing that my forecasting proposal is similar to insurance except that the third party (the forecasters) get a payout when a harm occurs rather than when it does not (because there are no premiums). A massive harm would bankrupt the AI providers rather than the insurers. Hence, it probably fails as an incentive.
It seems to me that the Big Four are perfectly positioned to take the lead here. They don’t just have the global reach and institutional trust, they also have strong incentives to secure this role and the influence to help shape how it plays out. They’ve been steadily expanding into AI consulting and risk management and their close ties to regulators across industries make them natural candidates to become to go-to AI auditors, especially in highly regulated sectors like finance and healthcare.
could be!
but are they institutional innovators? perhaps--but I'd like to see them in competition with upstarts.
As someone who is relatively new to the world of AI Governance, this was an interesting read. What would be the setup if, for example, an AI startup that was above board in all the ways you have outlined here, creates an 'agent' that has been tasked with providing financial advice and support for the elderly online, ends up actually scamming their elderly clients out of millions of dollars? Is the startup immune to any legal action or other consequences (or responsibility for reimbursing the clients) simply because they are certified and therefore proven to be mitigating risks to the best of their knowledge/ability? Or would this type of event automatically cause their certification to be revoked and legal action taken? If the former, then it has the potential to lead to all sorts of bad/unjust outcomes, if the latter, then it doesn't seem to remove much risk at all for the AI firms. I don't have a better alternative in mind that bypasses this trade-off, of course!
Thank you!
The outcome of the scenario you've raised would depend upon a few factors. Most importantly, did the developer itself deploy the model? Or did they sell it to other companies (eg wealth management firms) who then deployed it? In all likelihood, something as egregious as you are describing would be caught early by any firm complying in good faith with basic safety and security standards for agents. So if something like this did slip through, it would probably indicate some sort of malfeasance on the part of the developer. But as with all things in law, it is fact dependent.
(copying a couple of comments from Miles Brundage's google doc: https://docs.google.com/document/d/1ftDGn83gCqoc45Z_vQ0mcuPNukdb3DYBx5fsQw6PLZI/edit?tab=t.0)
It wasn't clear to me at the outset that these were the goals of the proposal. It seems like there was, in addition, at least some *implicit* goal of helping manage catastrophic AI risk. But if it's supposed to do that, I would like the argument for why it's expected to do so to be made more explicitly.
Here's a very simplified version of the proposal:
1) government licenses regulators
2) regulators can certify AI developers
3) certified AI developers have protection from tort liability.
Basically, it's regulatory markets, but instead of being mandatory, it's opt-in with the carrot of a liability shield.
Pros:
- Standard PRO of regulatory markets: regulators may have the more expertise than government.
- The proposal creates an incentive for AI developers to actually meet technical standards.
- It may indeed be politically feasible.
Cons:
- Standard CON of regulatory markets: there's a clear incentive for regulators to have low standards (since their clients are AI developers).
- It's unclear how the government is meant to ensure that private regulators have meaningful standards.
- In practice, the technical standards are unlikely to be adequate; we currently lack any means of providing meaningful assurance for advanced AI systems.
- The proposal creates a liability shield. However, liability seems like one of the best protections against risks that can't be mitigated via technical standards (e.g. unknown unknowns).
- The proposal subsizes AI development (since the government is doing the work of licensing regulators).
Overall, I believe the political feasibility of this proposal comes from the "opt-in carrot" nature of the proposal. I was already not sold on regulatory markets (as I think it's hard to imagine them not leading to regulatory capture). But this proposal also subsidizes AI development, which I think is a step in the wrong direction.
I can see why this might be appealing to people who think we can 80/20 x-risk if we get leading developers to do something like "implement sensible RSPs". I don't fall into that camp. One reason, which I think should be uncontroversial: we have no plan for how to assure ASL4-type systems won't end humanity, or coordinating to prevent their development in the absence of appropriate assurance.
Thanks Dean, I think this is great & very valuable work, and I empathize strongly with its direction. I'm a bit concerned about how this interfaces with two axes along which AI is set to get 'bigger' soon - the perception of transformative deployments by labs leading them to forego any participation in this system because they're not scared of liability; and the incentives of a securitised AI race to undercut a lot of promising features of the setup you describe.
I wrote all this down in some more detail here: https://writing.antonleicht.me/p/three-notes-on-dean-balls-private
Honestly think worth posting this on LessWrong. probably a lot of pushback but maybe some good comments
I guess what is to stop a very cavalier private regulator setting up and providing certification with the least oversight possible?
Tort seems to imply good calibration (especially as, in the case of 1047 it only applies to very large damages). What process here makes the private regulator care 10x as much about 10x larger risks?
This would be addressed by (1) the judgment of oversight bodies in each state that authorizes this (that is a TON of oversight btw), and (2) the mechanism whereby government can retract the license of a private body and eliminate safe harbor for ALL covered companies
(2) is a really expensive thing for a state to do. There are major ramifications to doing so. So the harm would need to be quite bad. This is what incentivizes the private regulator to care especially about major harms.