Abstract
As AI grows ever more complex and ubiquitous, its moral status becomes increasingly pressing. But knowing whether an AI has moral status is only part of the ethical puzzle. To determine how we ought to treat such entities, we must know not only whether AIs have moral status, but also about the content of their interests-what contributes to their wellbeing. This paper will survey different accounts of wellbeing, arguing that regardless of which theory one accepts, our epistemic position regarding AI interests is radically impoverished. The central difficulty is that it cannot be taken for granted that the behavioral and linguistic outputs of artificial systems correspond to inner experiences and mental contents in the same way as they do for evolved organisms. This argument does not entail an absolute prohibition on creating such AIs, but it does reveal significant, underappreciated, risks involved in doing so.