Not logged in.

Contribution Details

Type Conference Presentation
Scope Contributions to practice
Title Trust in human-AI interaction: an empirical exploration
Organization Unit
  • Suzanne Tolmeijer
  • Markus Kneer
  • Markus Christen
Presentation Type other
Item Subtype Original Work
Refereed No
Status Published in final form
  • English
Event Title Ethical and Legal Aspects of Autonomous Security Systems Conference 2019
Event Type conference
Event Location University of Zurich
Event Start Date May 2 - 2019
Event End Date May 3 - 2019
Abstract Text Technological advances allow progressively more autonomous systems to become part of our society. Such systems can be especially useful when time pressure and uncertainty are part of a decision-making process, e.g. in a security context. However, by using such system, there is a risk that the output of the system does not match ethical expectation, e.g. because a suboptimal solution is selected or collateral damage occurs. This has two implications. Firstly, the actual advice or action the system performs should be as we prefer it to be. Secondly, the user needs to perceive the system as an ethical and trustworthy partner in the decision-making process, to ensure the system is actually used. This project focuses on the latter, and contributes to the further elaboration of empirical issues raised by the White Paper “Evaluation Schema for the Ethical Use of Autonomous Robotic Systems in Security Applications”. While there has been research on autonomous systems and ethics, the field is still very much developing. To our knowledge, the following specific factors in this research have not been combined before: different levels of autonomy in search and rescue scenarios, uncertainty and time pressure in ethical decision-making, and trust. In order to investigate the interplay of those factors, we use a multidisciplinary and experimental approach. Compared to standard experimental ethics that is usually vignette-based, we will present morally challenging scenarios to participants in a simulation. This setting allows more immersion into the ethical scenario and adds the human interaction component, which is important to research the perception and expectations of the user. Currently, an experimental setup is designed together with a simulation prototype; the experiment is going to take place with search and rescue recruits of the Swiss army. They will participate in simulations involving the use of drones controlled by the participants in two setting: a rescue mission where a limited number can be saved and a prevention mission (bringing down a terror drone) where there will be some casualties. The system will either provide decision support for a given scenario or autonomously take a decision on what to do; the user only has a veto option. After each scenario, question will be asked on ethical acceptability, ethical responsibility and trust. At the conference, we will present results of pretesting of different scenarios and we will further outline our research program. The results of this research should ultimately shape guidelines on how to build ethically trustworthy autonomous systems.
Export BibTeX