Reading human emotions
To endow artificial agents with the capability of reading and interpreting human psychophysiological and emotional states represents a major issue in the field of human-machine interaction. Typically, monitoring psychophysiological and emotional states is performed through the measurements of several autonomic nervous system (ANS) parameters, like skin conductance response, hand palm temperature, heart beat and/or breath rate modulations, peripheral vascular tone and facial expression.
Conventional technology for monitoring ANS activity usually includes contact sensors or devices. These technologies are of course invasive by nature, and therefore, they can provide biased estimations, because the compliant participation of the individual is required. Thermal imaging on the other hand is increasingly regarded as a potential solution for recording thermal signatures of ANS activity non-invasively. Thermal imaging, allows the contact-less and non-invasive recording of skin temperatures through the measurement of the spontaneous body thermal heat. As the face is usually exposed to social communication and interaction, thermal imaging for psychophysiology is performed on the subject's face.
In fact, it is possible to see a wide range of psychophysiological signals in the thermal signature of a person. Breathing activity can be monitored with thermal imaging as environmental air flows via the nostrils to the lungs and back. High thermal values indicate expiration, while low values indicate inspiration/inhalation. Thermal imaging also allows the computation of the cardiac pulse through the spectral analysis of the thermal signature of the superficial vessels' blood flow pulsation. Cutaneous (skin-related) blood perfusion rate and facial perspiration activity are other examples of phenomena that can be detected, recorded and quantified by means of thermal imaging.
Emotions may have a thermal signature or may be characterized by activity of the autonomic nervous system, which in turn provides a thermal imprint through which it can be detected. In fact, thermal imaging has been indicated as a potential tool to build up an atlas of the thermal expression of emotional states, given the use of proper classification algorithms. Startling, distress, fear, sexual arousal, and even deception, all have specific thermal signatures that can be detected with thermal images cameras. One study for example suggested that temperature and cutaneous blood flow monitoring of the periorbital vessel during interrogation provides 87.2% accuracy in detecting deceptive individuals.
Assistive robots using thermal imaging
The benefit of thermal imaging for psychophysiology applications, is that it can obtain physiological information noninvasively and ecologically, which means without interfering with the spontaneous activity of a person. In that respect, thermal imaging could open the way for assistive robots, for example, for elder people or for monitoring the regular breathing activity of newborns. Automatic agents devoted to the control of environmental conditions, for example within a car or a house, could work via thermal-based monitoring of vital signs of the human user, so the system can adapt to the user. Although still theoretical, thermal imaging could be particularly effective for affective robots and automatic agents designed for improving and personalizing learning or treatment strategies on the basis of the measured user's psychophysiological feedback.
Thermal imaging for driver monitoring
In a study by Arcangelo Merla and others of the Infrared Imaging Lab, ITAB (Chieti, Italy) the possibility of real-time driver monitoring by means of thermal imaging was demonstrated. One of the most important requirements to increase the safety in both private and public motor vehicles transportation is to monitor non-invasively the driver's vigilance, attention and state.
The most sophisticated technologies so far developed use multi-sensor approaches, involving electroencephalography, contact thermistors, near infrared or visible band cameras (especially for eye blinking monitoring). Unfortunately, apart from the cameras, all the proposed approaches require to use contact sensors, which may distract the driver and interfere with the complete and safe control of the vehicle. That is why several studies have been performed only on driving simulator suites and not directly onboard of an actual vehicle.
Parameters variations, such as breathing, pulse and eye blinking rate are very effective descriptors of the physical state of individuals, the onset of drowsiness or loss of vigilance. In Merla's study, thermal imaging proved to be an efficient solution for recording almost all of these relevant vital signs.
Advanced FLIR research camera
In the study, a FLIR SC660 thermal imaging camera was used to monitor the breathing activity and rate, the blinking rate and the heart beat rate variation. The temperature fluctuations in facial regions of sympathetic interests could be monitored as well, which provided additional information, such as maintaining or loss of attention by measuring the forehead temperature, responsiveness to hazard conditions by monitoring periorbital response, and arousal by evaluating perioral and nasal temperature.
An autofocusing FLIR SC660 camera (640x480 FPA; NETD @ 30°C: 0.06 K) was installed on the vehicle windscreen, pointing at the driver's face. The thermal video was acquired at a full frame rate of 15 fps. For reference purposes, vital information (heart beat rate, breathing rate, galvanic skin response, palm temperature) was acquired by means of an ADInstruments PowerLab data acquisition system. Both the driver and the road were also filmed by means of a high-resolution video camera.
The facial regions of interest (ROI) included the orbital, periorbital, perioral, nose tip, chin and forehead areas. For each ROI, mean, standard deviation, minimum and maximum of the temperature distribution were computed. Eye blinking rate was estimated by means of the variation of the mean temperature in orbital region, as open and close eyes present a completely different temperature distribution. The proposed methodology is completely noninvasive, passive, and suitable for continuous monitoring on board, without interfering with the driving activity and without restrictions for the driver.
