Portal de Periódicos da CAPES

Listening to the silent struggles of bipolar disorder through sonification of iMoodJournal data

2022; Wiley; Volume: 24; Issue: 6 Linguagem: Inglês

10.1111/bdi.13210

1399-5618

David G. Angeler, Harris A. Eyre, Michael Berk,

Digital Mental Health Interventions

This paper reports a preliminary case study for demonstrating the potential of data sonification for telling a real-life narrative of experienced mood swings through music. We obtained iMoodJournal data (2017–2020) from a voluntarily participating male who was diagnosed with type 2 bipolar disorder in 2015. The monitored period covers prolonged stretches of severe depression, particularly during fall, winter, and spring months. These "winter depressions" were usually superseded by remission during summer. These seasonal patterns were similar and recurring in the period between 2017 and 2019. In 2020, the depressions were relatively mild due to the patient spending winter in southern latitudes. However, another severe depression episode occurred during summer 2020 instead, which likely emanated from a period of medication discontinuation. The symptomatology was overall complex and highly dynamic, manifested in the combination of mood specifying tags that the user associated with determined mood scores in the iMoodJournal. This complexity was difficult to capture in the form of the numerical scores visualized in Figures S1 and S2. The patient considered the time series based on the weekly lowest scores to present the most suitably depression. These data were thus used for sonification. To facilitate the perception of mood states and trade off the underestimation of depression in the sonified data, we used different dynamics (from piano (p) [periods of remission] to triple forte (fff) [severe depression]) and crescendos and decrescendos for transitions between these states. Also, a hypomanic mood state occurred once ephemerally in the sonified data; this was highlighted with an accidental (marcato). Two complementary scores were written (Data S1). The first score is based on representing years sequentially to familiarize the audience with the sound structure inherent in the data. The second score focused on the inter-annual variability in symptoms expression which can be substantial in bipolar disorder and other mental health conditions. In this score, the individual yearly scores from the first composition were overlayed, in accordance with the visual representation in the data graph. Thus, each year became a different voice in the same score. To emphasize the melancholic nature of depression, we composed both scores in A minor. Both auditory versions were written for and are performed by violoncello, which, due to its sound quality (timbre), can be subjectively associated with sadness, an emotional state inherent in (bipolar) depression. Both sound files (in WAV and OGG format; Audio S1–S4) were created with the free software MuseScore 3.6.2. The auditory representations complement the underlying music sheets (Data S1). Two complementary spectrograms (Figures 1 and 2) that visually match the sound structure of the sound files with color patterns were created with Sonic Visualizer 4.4. The iMoodJournal data used for sonification are available in Data S2. Summarizing and briefly describing the results, the sound structure of the first composition represents periods of depression and remission, with lower and higher pitches, respectively. These sequences were also manifested in the sound volume: depression (fff) and periods of remissions (p) and transitions between these periods in the form of crescendos (relapse into depression) and decrescendos (shift to remission). These patterns were visually supported by the spectrograms (Figure 1). The depressions were particularly highlighted by the yellow areas in the melodic range spectrogram (Figure 1, upper panel) and the more smeared-out bands in the spectrogram (Figure 1, lower panel). A high mood state was only scored once in August 2019 that "sticks out" from the remainder of the sound patterns in the form of a marcato in the sound file. This moment was captured by a distinct band in the spectrogram (Figure 1, lower panel) and a lighter point amidst the blue-purple areas in the melodic range spectrogram (Figure 1, upper panel). Lower, louder pitches (depression) can be heard especially at the beginning (declining mood state from January to February/March) and more subtly toward the end of 2017, 2018, and 2019. Despite these patterns being similar across these years in the sound files (see also Figure 1), there was also between-year variability evident. This auditory variability not only emanated from the sequence of pitches but also in their relative lengths, which indicates the stability of mood states. 2020 differed from the previous years, with higher pitches superseded by lower ones at the beginning of the year (January and February), indicating more short-termed mood swings. These patterns are again supported by the spectrograms (Figure 1). A more stable emotional state followed between March and the middle of July revealed in relatively narrow pitch intervals and durations of pitches. The shift to lower and louder pitches, indicating a relapse to depression, was apparent until August. These patterns are also mirrored in the visuals (Figure 1). Afterward, the patient remitted to more functional mood states (crescendo), until the end of the year. The emerging sound structure in part 2 in which the 4 years were overplayed and performed simultaneously to emphasize the between-year variability of symptom expression in bipolar, despite seeming chaotic, carried a subjectively interpretable emergence of harmony and had esthetic appeal. None of the voices came across clearly for substantial parts across the years, despite different sound volumes. Only from October onwards were more synchronous sound patterns observed across years, evinced by the durations of pitches. Patterns in the spectrograms support these latter patterns (Figure 2). They also highlight the occasional dominance of individual years (light areas in the melodic range spectrogram; stacked bands in the spectrogram) that were less obvious in the sound file. Data sonification has been used across disparate disciplines, for instance, the social (e.g., political crises and science literacy), natural (e.g., astronomy and climate change), communication (Tweetscapes analyses), and health (e.g., seizure detection, biofeedback of facial EMG activity, and diabetes) sciences for an alternative (artistic) representation of complex scientific phenomena. Mental health comprises undoubtedly an intricate field with multidimensional and multiscale interactions between science, clinics, technology, society, economy, and policy.1 Although transdisciplinary in a sense that our approach operates at the intersection of the arts (visual and music), science (data), technology (mental health tracking app), and the clinics (patient engagement), our approach is based on data from a single patient. Our study is therefore only a very simplistic representation of this complexity and thus an initial demonstration only. Despite being preliminary, we consider data sonification, however, as a potentially useful tool for complementing traditional clinical approaches, such as, for instance, psychotherapy and cognitive behavioral therapy. Specifically, we consider the clinics as users of the approach and the arts as a channel conveying science, including monitored mental health scorings. The next step in applying data sonification may therefore be at the clinical level; for instance, in the form of prospective studies that may assess the broader potential of this approach for future research and clinical translation. More precisely, data sonification is a novel way of communicating information about data, contrasting with a visual mood chart that graphs data points. The initial steps toward likely clinical application would be to determine whether and in what ways people with bipolar disorder and healthcare providers find value in sonification beyond data presented in traditional mood charts. In essence, qualitative "market research" on the perceived value of this innovation could provide ideas for how data sonification might be integrated into routine. The clinical implementation of data sonification may be especially useful because it simultaneously operates at two interacting psychological domains that benefit the learning process: the cognitive domain which relates to logic and thinking and predominates in science, and, prevailing in the arts, the affective domain which engages emotions and feeling.2 Being able to engage emotionally with "cold" monitoring data suggests that adding data sonification to the current therapeutical tool box for managing mental health may help empower patients, not only by better understanding their mental health conditions but also to manage, predict, and maybe even communicate their mood. That the method was emotionally appealing, highlighting the power of art for the clinics, was at least evident in our patient, helping him to learn about the self in novel ways beyond traditional mental health management forms. This suggests that further work may be warranted to further explore and develop this potential for improving health outcomes for patients with mental disorders. Also, potential applications regarding other neuropathological and neurodegenerative conditions such as dementia and Alzheimer might likely be worth exploring. We acknowledge that the power of art to cause an impact and interpretations is subjective and can therefore vary with context. Specifically, subjective representations alone can be risky because societal challenges may be perceived as distant and unlinked to personal experience, due to ambiguity, leading to misunderstanding and decontextualization. This has been shown for example in the environmental sciences.3 The problem of decontextualization may be exacerbated by social, economic, and demographic factors such as deficiencies in science literacy, graphical, and numeracy understanding in the public.4 Data sonification comprises arguably an objective approach for making the silent voices of mental illness heard and can therefore complement subjective artistic representation of emotions and feelings, such as drawings, paintings, classic, and other music genres, sculpture, and photography. While we see the most immediate application of data sonification in the clinics, there may be speculatively potential to further develop the approach in subsequent steps to address the pervasive gaps in mental health awareness and public literacy operating at the societal level,5 especially those pertaining to an increasing global mental health crisis.1 However, we acknowledge that the societal scale has a significantly higher level of complexity than the patient/clinical level and other data requirements compared to those of the present study. The likely impact of data sonification as an outreach/information tool for societies is, at this stage, therefore difficult to envision. We conclude by highlighting that data sonification alone may not be a silver bullet in the clinics. However, adding it to the arsenal of therapeutical tools may allow us to give a targeted shot for refining current mental health management by striving for more holistic models not only rooted in the sciences but also in the arts. Without the generous sharing of data and the enthusiasm of our anonymous participant, this study would have been impossible. His support, interest, and engagement throughout this project are much appreciated. MB is supported by an NHMRC Senior Principal Research Fellowship (1156072). An anonymous reviewer is acknowledged for providing constructive criticism that improved the paper. The authors declare that no conflict of interest exists. The iMoodJournal data used for sonification are available in Data S2. Audio S1 Audio S2 Figure S1 Figure S2 Data S1 Data S2 Audio S3 Audio S4 Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.