By Karen Louise Grova Søilen (Lund University), Mia Liinason (Lund University) and Sue Anne Teo (Raoul Wallenberg Institute, Lund University)
Welcome to join the Human Rights Lunch Online seminar on Friday 16 October 2026 at 12.15 – 13:00 on Zoom: https://lu-se.zoom.us/j/65437566996 with Karen Louise Grova Søilen (Information Studies, Lund University), Mia Liinason (Gender Studies, Lund University) and Sue Anne Teo (Thematic Expert on AI and Human Rights, Raoul Wallenberg Institute, Lund University).
Their presentation will highlight ethical and legal challenges of regulating companion AI and share some initial impressions from our data collection with users of companion chatbots. Welcome to join the seminar!
In the 2024/2025 Pufendorf theme “DigiJustice: Rethinking Digital Inequalities and Human Rights in the Age of AI”, we examined impacts of AI on human rights from multiple interdisciplinary and critical perspectives. Each Thursday, two or three group members had prepared for the topic of the week. Over the year, the group focused on questions such as industrial aspects of AI and ethics, legal aspects of AI, AI and arts, AI in healthcare, AI and social robotics. In one of the sessions, the three of us had prepared the topic “Affects and Emotions of Humans in Interaction with Actual Technologies.” We had instructed the whole group to select a companion chatbot or LLM of their own choice, e.g. Nomi, ChatGPT, Replika etc. and engage in conversation before the Thursday. On site, we shared our impressions of interacting with the chatbots and focused in particular on experiences of simulated emotionality and deceptive anthropomorphism – for example, if the chatbot expressed emotions or indicated sentience – and we briefly examined the business models of the companies, their privacy policies, and terms and conditions.
Already in 2023 and 2024, serious reports had been presented about AI chatbot companions, including the tragic case of a Belgian man’s suicide and the death of an American teenager (Rissman 2024) following interactions with their AI chatbot companions. These examples brought to light increasing societal concern over the effects of anthropomorphic AI, which may create possibilities for manipulation, heightened vulnerability, and loss of control. At the same time, these models were rapidly gaining popularity around the world. For example, in a Harvard Business Review report tracking how people use generative AI, therapy and companionship emerged as the leading use case for AI chatbots in 2025 (Zao-Sanders 2025), as people increasingly turned to these applications for emotional support and to address loneliness. During the time of our Pufendorf theme, headlines like “DR host chatted with a ‘virtual girlfriend’ app. It ended with detailed advice on how to kill a friend” (Andersen Nielsen 2025) and “She is in love with ChatGPT” (Hill 2025) began to appear with more frequency. There were also reports about people invoking their companion chatbot to take on the role of a therapist, doctor or lawyer – otherwise regulated professions governed by established professional standards.
Together, these developments raise pressing questions about what happens when AI systems operating with limited ethical and safety guardrails enter increasingly intimate and sensitive areas of people’s lives. Yet, despite these growing concerns, we still know little about the actual risks and benefits of anthropomorphic AI. In particular, there is a lack of empirical research into what happens in interactions between users and AI chatbots, what kinds of vulnerabilities may emerge through these interactions, and how well existing legal frameworks protect users against potential harms. While existing laws offer protection against manipulation and the exploitation of vulnerabilities, they are not necessarily informed by how these interactions unfold in practice, nor do they fully account for new and evolving forms of vulnerability that may arise through interactions with anthropomorphic AI systems (Laestadius et al. 2024).
These reflections shaped the starting points for our ongoing project, Anthropomorphic AI and Emergent Vulnerabilities (RJ), in which we address the challenges posed by these rapidly evolving practices, the limited knowledge about the actual risks that arise from interactions with AI systems, and the shortcomings in current legal frameworks. As an interdisciplinary group of researchers with backgrounds in law and human rights, information and surveillance studies, digital cultures and gender studies, we seek to generate new empirical knowledge about how people interact with anthropomorphic AI and develop new conceptual tools for understanding the risks that may arise through these interactions. At the heart of the project is the relationship between these emerging risks and existing legal frameworks: whether and how current protections can address the potential harms introduced by anthropomorphic AI. We will also attend to possible positive effects in people’s lives of interactions with AI companions.

Karen Louise Grova Søilen Associate senior lecturer, Information Studies,
Lund University

Mia Liinason
Professor of Gender Studies,
Lund University

Sue Anne Teo
Thematic Expert on AI and Human Rights,
Raoul Wallenberg Institute, Lund University
References
Andersen Nielsen, Simon (2025) “DR-vært chattede med ‘virtuel kæreste’-app. Det endte med detaljerede råd til at slå ven ihjel” DR.dk, January 12, https://www.dr.dk/nyheder/viden/teknologi/dr-vaert-chattede-med-virtuel-kaereste-app-det-endte-med-detaljerede-raad
Hill, Kashmir Hill (2025) “She Is in Love With ChatGPT”, The New York Times, 15 January, https://www.nytimes.com/2025/01/15/technology/ai-chatgpt-boyfriend-companion.html?eafs_enabled=false
Laestadius, L.., Bishop, A., Gonzalez, M., Illenčik, D., and Campos-Castillo, C. (2024). Too Human and Not Human Enough: A Grounded Theory Analysis of Mental Health Harms from Emotional Dependence on the Social Chatbot
Replika. New Media & Society 26(10): 5923-41. doi:10.1177/14614448221142007.
Rissman, K. (2024, 24. oktober). The disturbing messages shared between AI Chatbot and teen who took his own life. The Independent. https://www.independent.co.uk/news/world/americas/crime/ai-chatbot-lawsuit-sewell-setzer-b2635090.html
Zao-Sanders, M. (2025) How People Are Really Using Generative AI Now” Harvard Business Review, https://hbr.org/2025/04/how-people-are-really-using-gen-ai-in-2025






















Comments