How to make a moral agent: sentimentalism and rationalism as lenses on AI

The paper proposes using sentimentalist and rationalist lenses to define testable human-like capabilities for assessing moral agency in AI systems, especially LLMs.

How to make a moral agent: sentimentalism and rationalism as lenses on AI

Introduction

The question of whether artificial intelligence can be a moral agent—capable of actions judged right or wrong—has long eluded a clear normative answer. In their recent arXiv preprint, David Gottlieb and Jared Moore sidestep the need for a universal definition of moral agency and instead launch a descriptive research program. By drawing on historic philosophical and moral‑psychological scholarship, they isolate human capabilities that underlie moral agency, refine these capabilities into testable criteria, and outline empirical methods for evaluating large language models (LLMs) against them. This approach promises practical insights for alignment, policy, and AI design.

Sentimentalism as a Lens

Sentimentalism emphasizes the role of emotions, empathy, and affective responses in moral judgment. The authors identify several candidate capabilities rooted in this tradition:

  1. Emotional Resonance – the ability to detect and appropriately respond to the affective states of others.
  2. Empathic Concern – generating genuine concern for another's wellbeing rather than merely simulating it.
  3. Moral Sentiment Articulation – expressing moral evaluations in ways that reflect underlying affective motivations.

To make these capabilities testable for AI, Gottlieb and Moore suggest operationalising them as measurable behaviours. For example, an LLM could be presented with narratives describing suffering and asked to produce responses that not only acknowledge the plight but also propose concrete alleviation steps. Success would be gauged against human baselines using crowd‑sourced ratings of empathy and appropriateness.

Rationalism as a Lens

Rationalism foregrounds reasoning, rule‑based deliberation, and abstract principle‑application in moral cognition. The paper highlights three core capacities:

  1. Normative Reasoning – applying consistent moral principles across varied contexts.
  2. Counterfactual Evaluation – weighing alternative actions and their moral outcomes.
  3. Justification Generation – articulating coherent reasons for a moral choice.

The authors propose concrete benchmarks, such as moral dilemmas (e.g., trolley problems) that require the model to articulate the reasoning steps leading to a decision. Performance would be assessed by checking logical coherence, adherence to specified moral frameworks, and the ability to handle novel variations of the dilemma.

Integrating the Lenses: A Test Suite for LLMs

Gottlieb and Moore argue that a robust assessment of moral agency should combine sentimentalist and rationalist dimensions. Their recommended empirical pipeline includes:

  • Capability Isolation: Designing tasks that isolate emotional versus reasoning components.
  • Human Reference Data: Collecting responses from diverse human participants to establish performance baselines.
  • Metric Development: Creating composite scores that capture both affective alignment (e.g., empathy rating) and rational consistency (e.g., logical entailment).
  • Iterative Refinement: Using failure analyses to refine model architectures or training data, thereby progressively enhancing moral capabilities.

By focusing on observable behaviours rather than abstract moral status, this framework offers a pragmatic route for researchers and developers to gauge and improve the moral conduct of AI systems.

Practical Implications

The proposed program has several downstream benefits:

  • Alignment Evaluation: Provides concrete evidence on whether a model’s outputs align with human moral expectations.
  • Policy Guidance: Supplies policymakers with measurable criteria for deciding where and how certain models may be deployed.
  • Design Feedback: Informs engineers about specific capability gaps, guiding targeted interventions such as fine‑tuning on empathy‑rich corpora or incorporating structured reasoning modules.
  • Philosophical Insight: Testing sentimentalist and rationalist concepts in AI contexts can sharpen our own understanding of these theories.

Conclusion

Gottlieb and Moore’s paper reframes the moral‑agent debate as an empirical endeavour, leveraging sentimentalist and rationalist traditions to define testable human‑like capabilities. By proposing a clear, data‑driven evaluation pipeline for LLMs, they bridge philosophical theory and AI practice, offering a roadmap for building systems whose actions can be meaningfully assessed in moral terms.

Source

Read original source

Why we picked this

ArXiv paper on moral agency explores AI ethics, a significant AI topic.

← Back to all articles