This new system could serve as basis to develop applications for the management of anxiety, depression and other psychological issues.
What if our smartphones could keep an eye on our mood by "observing" the objects around us? A group of British and Italian researchers developed a system that, in a picture taken with a smartphone, is able to automatically detect a number of items that are connected to our mood thanks to an AI-based method. This study was published on Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies and uses the data collected by an app that researchers specifically developed for the purposes of the study. The outcomes of this study pave the way for other smart apps based on users' emotional state and even more.
"The techniques we developed can be built in a vast array of apps", explains University of Bologna professor and research coordinator Mirco Musolesi. "For example, they could prove useful in managing the Covid-19 aftermath. Many people could find getting back to normal life very hard from a mental health point of view and be subject to high levels of anxiety and mild depression".
Several systems to infer users' mood states from real-time data of mobile devices have been studied in recent years. Some of these systems use GPS data or analyse social media feeds or study users' interactions on smartphones. This is the first time, however, that a research group attempts to exploit data drawn from people's physical environment.
MyMood is the app researchers developed to do that. This app randomly surveys users on their emotional states, asking, at the end of each survey, to take a picture of what surrounds them. In a sample of 22 users participating in the study, researchers were able to analyse 3,305 surveys and pictures. An AI-based deep learning system processed the collected data and managed to automatically identify the individual items in the pictures.
By matching the identified objects with the information about users' mood, researchers could highlight connections between physical environment and psychological state. By means of example, office-objects such as a keyboard or a computer are associated to more stress, whereas more homely items such as a bed or a TV-set are instead associated to lower levels of "physical readiness", and the proximity of a human face causes more happiness.
"This is the first time that someone carries out an observational analysis based on automatic detection of objects from users' daily lives", says Victor-Alexandru Darvariu, first author of the study. Davariu carried out this research while on an internship in professor's Musolesi research group and is now doing a Ph.D. under his supervision. "This methodology can be the starting point for a number of intelligent apps for managing anxiety, depression and other psychological issues. It could also prove useful to other fields such as architecture and design and the analysis and design of interior and outdoor spaces".
Moreover, these systems are not limited to smartphones, they could be employed in other technological devices such as smart glasses, which are able to automatically record the user's environment. Of course, all this is done in full compliance with privacy laws, which are fundamental. "Our study was carried out by fully complying with a code of ethics and with the General Data Protection Regulation (GDPR)", ensures Musolesi. "Privacy is essential in this line of research: deep learning systems allow to analyse the pictures on the user's smartphone directly. In any case, total anonymity for the user must be guaranteed".
The title of this study is “Quantifying the Relationships between Everyday Objects and Emotional States through Deep Learning Based Image Analysis Using Smartphones” and was published on Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies.
Professor Mirco Musolesi from the Department of Computer Sciences and Engineering represented the University of Bologna in this study. This paper will be also feature in the "Joint Conference on Ubiquitous and Pervasive Computing (UbiComp 2020)" next September.