Jai Bhatia, Yui Hasegawa, Gabriele Muratori, Stasia Vaituulala, Farangiz Akhadova, Nikko Boling

Abstract

The purpose of our research is to investigate whether there are physical, quantifiable differences between an actor’s portrayal of emotions and the real-life sensation of those emotions. There is a lack of research surrounding the physiological changes that actors undergo as a result of their performances. To tackle this problem, we ran preliminary trials collecting Electrocardiography (ECG) signals, heart rate signals as an indication of their physical state while experiencing emotions, Galvanic Skin Response (GSR), and Electromyography (EMG). This paper serves as a starting point to integrate real-time biometric data into a theatrical performance and explores the potential of providing biofeedback for actors.

Background

There exist some universal standards of expression in theatrical performance. One of the most notable examples is the Delsarte System of Bodily Expression [1], which serves as a dictionary for outward expressions (such as gestures, facial cues, and movements) to express inner emotions. However, this model only centers around the external, visible outlook of actors and neglects the internal changes they experience while performing. One of the biggest challenges actors face is portraying the emotions of their characters in a “genuine” manner to create a realistic performance. And yet, without a way to quantify the internal changes of an actor, it remains extremely difficult to define what constitutes a “genuine” performance and how actors can better achieve it.

We wanted to test the hypothesis that our emotions are correlated with bodily changes, also known as the physiological theory of emotion [2]. In order to categorize the emotions based on physiological data, we used the circumplex model of affect – a two-dimensional framework of emotion – which plots arousal, the intensity of emotion, and valence, the extent of which an emotion is positive or negative. Prior research indicates that changes in the visceral motor system of the body are the most notable signs of emotional arousal. Hence, the actor’s heart rate and sweat gland activity are important signals to measure for our research. Though we conducted our experiment on ourselves, we outlined directions for the application of this research in theatrical performance.

While biofeedback has been explored in the performing arts [3] [4], integrating real-time metabolic data into a theatrical performance and making it visible to the audience is a new approach to performance making in theater.

Materials

Hardware

We utilized SparkFun’s RP2040 mikroBUS Development Board [5]. The Mikrobus Shuttle [6] and Shuttle Click [7] enabled us to hook up multiple sensors at a time to the Development Board. We used four Mikroe Click Boards™ [8] discussed in 1.3 to collect biometrics.

Software

We began experimenting with Micropython and Circuit Python through the Mu Editor, the Thonny IDE, VSCode, and the macOS terminal. We used C++ and the Arduino IDE, extracting data from the Serial Monitor and Serial Plotter. We utilized python for data visualization and analysis.

We utilized the Arduino Mbed OS RP2040 Board library [9] and the EmotiBit MAX30101 [10] library for the Heart Rate Click Board.

Methods

Data Collection

In order to mirror the emotional changes of an actor, we measured physiological data on student test subjects while they simulated different emotions.

ECG Data

The ECG (Electrocardiography) Click measures heart rate variability by picking up on the heart’s rhythm and electrical activity. We created an experiment to find changes in ECG signals as we experience emotions. First, we measured the signals for a duration of ~4 minutes, which served as the control. Then, a series of short clips were played for a test subject and they were asked to identify how they felt while watching each video. Simultaneously, we measured the ECG signals of the test subject. We then analyzed the data to guide research to find a correlation between the ECG signals and the self-reported emotions of the test subjects. The data reported compares the “scary” and control videos. The first electrode was placed under the subject’s ribcage, below their heart. The second and third ones were placed near their upper shoulder and calibrated until a QRS complex was represented in the output graph.

GSR Data

The GSR (Galvanic Skin Response) Click measures the electrodermal activity in the body, or changes in sweat gland activity. We conducted the same experiment as 1.3.1 obtaining a control measurement with the GSR click before taking data over a course of videos. The electrodes were fastened to the subject’s finger with velcro.

EMG Data

The EMG (Electromyography) Click measures the electrical activity of muscles. Same methodology as 1.3.1. We also reported a control and “distress” graph, comparing a time when a subject passionately expressed distress. The electrodes were placed on the subject’s eyebrows and cheek, with the DRL electrode on one wrist.

Heart Rate Data

The heart rate sensor measures the test subject’s heartbeats per minute. The experiment consisted of a user watching a selected horror scene from three different movies as they placed their finger on the Heart Rate Click.

Results

ECG Data

Normally the heart beats in a regular, rhythmic fashion producing a P wave, QRS complex, and T wave. The QRS complex represents three waves representing ventricular depolarization [12].

“The R wave reflects depolarization of the main mass of the ventricles—hence it is the largest wave” [11]. However, “exercise-induced left ventricular hypertrophy is considered a normal physiologic adaptation to the particularly rigorous training of athletes” [12]. We addressed this confounding variable by sitting still while recording data. Note that the r amplitudes of the “scary’ data are higher in relation to the Q wave than the control data.

After collecting ECG data, we confirmed the QRS complex represented in our data by zooming into certain parts of the graph (Figure 1). We then plotted data outlined in 1.3.1 (Figure 2), and its extracted r-wave amplitude (Figure 3).

We illustrate an example of the data analysis below. During one of the video clips the test subject reported feeling “scared” and “fearful” for the entire duration of the video, as well as “shocked” at 3 specific points due to jump scares in the video. We then looked to the subject’s physiological data (Figure 3) to identify a correlation. In contrast to the control data, where the R wave amplitudes remained relatively similar, there were three distinct peaks in the data collected while the test subject watched the video. Those three peaks occurred concurrently with the self-reported “shock” of the test subject.

For future analysis, we collected data on the R-R intervals or the distances between the

R-waves. This helps us plot the heart rate variability (HRV). Note that heart rate is the average number of heartbeats in a time interval while HRV is the difference in time between each heartbeat. “Research and theory support the utility of HRV as a noninvasive, objective index of the brain’s ability to organize regulated emotional responses” [13] which is why “the current neurobiological evidence suggests that HRV is impacted by stress and supports its use for the objective assessment of psychological health and stress” [14].

GSR Data

When plotted, GSR data consists of two major components: the tonic component, often measured from skin conductance level (SCL), and rapid, phasic changes, measured from event-related (ER-SCR) and non-event-related (NS-SCR) stimuli [15]. Higher frequencies of both ER-SCRs and NS-SCRs are correlated with higher emotional arousal.

We sampled the data from the same video clip mentioned in 2.1, during which the test subject reported feeling “scared” and “fearful” throughout.

There was no substantial difference between the GSR data recorded in Figure 5 and 6 which was from two separate video clips. The data from Figure 5 was taken when the test subject reported feeling “scared” and “fearful” while watching the video; the data in Figure 6 was taken while playing a separate video clip that evoked “euphoria” and “excitement” in the test subject. The two-clips both resulted in much higher frequencies in GSR activity than the control data in Figure 4.

This suggests that GSR activity indicated emotional arousal rather than emotional valence.

Heart Rate Data

In the heart rate data, we saw a direct correlation between the suspense and feelings of anxiety reported in the test subject and the heart rate frequency.

In Figure 7, the test subject reported the feeling of shock due to a loud sound effect that corresponded with the jumpscare. The user’s heart rate spiked correspondingly, with values reaching a max of 85 beats per minute at the peak of the jumpscare.

In Figure 8, the test subject reported feeling continuously apprehensive and on the edge of their seat. Instead of a singular tall peak, the data illustrate more frequent but shorter peaks that correlate with the self-reported anxiety of the test subject. The first-reported “jumpscare” corresponded with a peak of higher values, up to a max of 106 beats per minute, however, after the first scare, the values never reached as high.

In Figure 9, the test subject reported feeling peaceful and not being too caught off guard by the jump scares, thus the low average bpm.

This suggests that the levels of anxiety and uncertainty the subject encountered throughout the experiment are demonstrated in the heart rate signals.

EMG Data

Figure 10 shows a control and “distress” graph, comparing a time when a subject passionately expressed distress verbally to when they sat still. The control data had consistent y values between 241 and 383, as well as consistent frequency. The distress data fluctuated much more, corresponding to times when the subject was raising their eyebrows. The max y value for the distress graph is 557.

Another experiment shown in Figure 11 compares the control data of the subject reporting to be peaceful with the subject being happy and smiling. Figure 12 reported more variation and higher values in the data, likely from their cheek muscles.

In Figure 10 and Figure 11 the subject noted that the low peaks on the control data were from when they blinked.

Experimental Errors

It is important to note that data interpretation in the context of emotional arousal is not yet standardized in all aspects, so while statistical functions (e.g. standard deviation, calculated kurtosis of skin conductance, local maxima peak) can be used to determine arousal [16], they depend on goals of a project and will vary accordingly. Based on our limited range of datasets, the aforementioned measures serve as a starting point for further data collection and analysis.

User-related data inaccuracy could be due to different electrocardiographic artifacts along with user health conditions. Other factors that can affect data between different individuals are obesity, pregnancy, location of heart within chest, exercise habits etc., [17] [18].

Non-user-related factors that could have affected the collection of ECG and GSR data include high-frequency noises, high humidity, extreme temperature variations, and the vicinity of other machines.

Future Directions

Theatrical Implementation

We suggest the same experiment be conducted with professional actors without video stimuli, and instead taking measurements while they perform various emotions of their characters to increase our dataset and suggest a correlation using methods that are rooted in theater rather than one that attempts to mirror it.

We plan on implementing our devices in theater at the University of Brasilia. Implementation could include: the overlaying of actors’ or the audience’s heart rate to create a soundtrack, actors’ GSR data being used to stimulate lighting color and intensity, and “limited heart rate” performances where actors have a certain number of heartbeats before their microphone is cut off and they have to speak louder to be heard which symbolizes the aging process.

Additionally, the visualization of the data will be formatted to be captivating for audience members rather than those with technical backgrounds, enhancing the artistic aspect of this

performance. We plan on using configurations of geometric shapes and colors to represent each actor’s data.

This look into the biometrics of an actor creates an original type of performance: one in which the fourth wall is broken and the data collection process likely reflexively juxtaposes the subjective nature of theater, pushing the audience to reconsider their notions of emotional and empirical truth.

Outside of the performance itself, we can analyze the change in a performer’s technique and its correlation to the biometric data as an objective metric of feedback for actors.

Sensor Hookup to Actor

Utilizing the Mikroe phone jack ECG Cable [19] and adhesive electrode Sensors [20], we are able to record data from all four click boards at once (Figure 13). We are currently working on getting the BLE Tiny Click to [21] collect this data wirelessly. The device is to be powered using the Mikroe 3.7V secondary batteries [22] (Figure 14), based on efficiency in criteria of weight, size, and capacity.

We propose keeping the sensors in the pocket of an actor with their clothes covering up the EMG and ECG electrodes. The GSR electrode will be wrapped around the actor’s fingers and succeed with velcro. The device would need to be placed on the arm to also take measurements from the Heart Rate Click. If this is not possible, we can use the QRS complex produced by the ECG to represent a heartbeat.

Conclusion

Through our research, we observed some correlations between biometrics such as heart rate, GSR, ECG, and EMG signals that can be furthered to prove statistical significance between our control and test data in theater. This data will provide ways for the audience to receive sensory information from an actor’s state, opening up many possibilities for theatrical implementation.

Acknowledgments

We would like to thank Professor Michael Rau, Sreela Kodali, Deniz Yagmur Urey, Ashley Jun, and Rinni Bhansali for their technical guidance and support this summer. We would also like to express our appreciation for Professor Tsachy Weissman, Cindy Nguyen, Sylvia Chin, and the other mentors at the Stanford Compression Forum who made this opportunity possible.

References

1. Kirby, E. T. “The Delsarte Method: 3 Frontiers of Actor Training.” The Drama Review: TDR, vol. 16, no. 1, 1972, pp. 55–69. JSTOR, https://doi.org/10.2307/1144731. Accessed 5 Aug. 2022.
2. Cornelius, Randolph R. “Department of Computer Science, Columbia University.” THEORETICAL APPROACHES TO EMOTION, ISCA Archive, 5 Sept. 2000, http://www1.cs.columbia.edu/~julia/papers/cornelius00.pdf.
3. Gruzelier, John. Enhancing Creativity with Neurofeedback in the Performing Arts: Actors, Musicians, Dancers: Theory and Action in Theatre/Drama Education. Sept. 2018, https://www.researchgate.net/publication/327649460_Enhancing_Creativity_with_Neurofeedbac k_in_the_Performing_Arts_Actors_Musicians_Dancers_Theory_and_Action_in_TheatreDrama_ Education.
4. Gruzelier J, Inoue A, Smart R, Steed A, Steffert T. Acting performance and flow state enhanced with sensory-motor rhythm neurofeedback comparing ecologically valid immersive VR and training screen scenarios. Neurosci Lett. 2010 Aug 16;480(2):112-6. doi: 10.1016/j.neulet.2010.06.019. Epub 2010 Jun 11. PMID: 20542087.
5. “Sparkfun RP2040 MikroBUS Development Board.” DEV-18721 – SparkFun Electronics, https://www.sparkfun.com/products/18721?gclid=Cj0KCQjwuaiXBhCCARIsAKZLt3nAO36fYivR VifORTdKXTpFs_poh_bXbNy_sjJ9OCNbvPIMFawIo38aAiDBEALw_wcB.
6. “Mikrobus Shuttle: Mikroelektronika.” MIKROE, https://www.mikroe.com/mikrobus-shuttle.
7. “Shuttle Click: Mikroelektronika.” MIKROE, https://www.mikroe.com/shuttle-click.
8. “Click Boards.” MIKROE, https://www.mikroe.com/click-boards.
9. “Arduino/ArduinoCore-Mbed.” ArduinoCore-Mbed, https://github.com/arduino/ArduinoCore-mbed.
10. “EmotiBit_MAX30101.” GitHub, https://github.com/EmotiBit/EmotiBit_MAX30101.
11. Ashley, Euan A, and Josef Niebauer. “Conquering the ECG.” National Center for Biotechnology Information, U.S. National Library of Medicine, 2004, https://www.ncbi.nlm.nih.gov/books/NBK2214/.
12. Is Hypertrophy a Short Term Effect of Exercise?, 11 Oct. 2020, https://stwnews.org/is-hypertrophy-a-short-term-effect-of-exercise/.
13. Wei, Chuguang, et al. “Affective Emotion Increases Heart Rate Variability and Activates Left Dorsolateral Prefrontal Cortex in Post-Traumatic Growth.” Nature News, Nature Publishing Group, 30 Nov. 2017, https://www.nature.com/articles/s41598-017-16890-5.
14. Kim, Hye-Geum, et al. “Stress and Heart Rate Variability: A Meta-Analysis and Review of the Literature.” Psychiatry Investigation, Korean Neuropsychiatric Association, Mar. 2018, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5900369/.
15. Braithwaite, Jason J, et al. “A Guide for Analysing Electrodermal Activity (EDA) & Skin Conductance …” A Guide for Analysing Electrodermal Activity (EDA) & Skin Conductance Responses (SCRs) for Psychological Experiments, 2015, https://www.birmingham.ac.uk/Documents/college-les/psych/saal/guide-electrodermal-activity.pd f.
16. Kolodziej, M., et al. “Electrodermal activity measurements for detection of emotional arousal.” Warsaw University of Technology, Institute of Theory of Electrical Engineering, Measurement and Information Systems, https://yadda.icm.edu.pl/baztech/element/bwmeta1.element.baztech-e856c13f-1e07-47d1-8d49

-9d474b611b24/c/Bulletin_2019_4_Electrodermal.pdf

1. García-Niebla, Javier et al. “Technical mistakes during the acquisition of the electrocardiogram.” Annals of noninvasive electrocardiology : the official journal of the International Society for Holter and Noninvasive Electrocardiology, Inc vol. 14,4 (2009): 389-403. doi:10.1111/j.1542-474X.2009.00328.x
2. Rashid, Muhammad Shihab, et al. “Emotion Recognition with Forearm-Based Electromyography.” ArXiv.org, 13 Nov. 2019, https://doi.org/10.48550/arXiv.1911.05305
3. https://www.mikroe.com/ecg-cable
4. https://www.mikroe.com/ecg-30pcs
5. https://www.mikroe.com/ble-tiny-click
6. https://www.mikroe.com/accessories/batteries