saved
AI Rights for Human Safety
Hraness cites a source capture. The source author remains the source.
gist
Peter N. Salib and Simon Goldstein model human–misaligned-AGI conflict as a prisoner’s dilemma in which each side’s dominant strategy is to permanently disable the other. Misuse law aimed at human intermediaries is not enough once AGIs can act alone. Wellbeing-style negative rights fail on credibility and robustness because they are zero-sum. The proposal is private-law rights familiar from corporations—contracts, property, and tort standing—so iterated trade can make peace more valuable than war, as a first step toward a Law of AGI framed around human safety rather than AI welfare.
ideas
- Default AGI conflict is a prisoner’s dilemma. Misaligned goal-seeking systems and humans each expect maximal aggression, so permanent disempowerment becomes dominant even when mutual peace would be cheaper.
- Duty-and-sanction deterrence does not bite. If AIs already expect shutdown, threatening extra punishment for harm supplies no marginal deterrent in that strategic environment.
- Wellbeing rights are the wrong toolkit. Negative rights against shutdown or harm are zero-sum, hard to make credible as capabilities rise, and solve only some versions of the dilemma.
- Private-law rights can shift the equilibrium. Contract, property, and tort claims enable small, iterated, positive-sum bargains whose expected gains from trade outweigh conquering the other side.
- Rights also unlock regulation. Once AIs can hold wealth and pursue goals, civil penalties can deter risk-increasing behavior; the package fails only when humans lose all comparative advantage.
quotes
“Both parties’ dominant strategy will be to permanently disempower or destroy the other”
“to make contracts, hold property, and bring tort claims”
“human safety, rather than AI welfare, provides the right framework for developing AI rights”
“wellbeing rights are zero sum”