Objective Measurement of Listening Effort

Listening Effort is the mental exertion listeners experience trying to solve an auditory task such as understanding speech (or detecting a target among interferers). Because listening effort is influenced by the clarity of the signal, it is therefore affected by hearing loss, as well as signal degradation caused by background noise or bad sound quality. Listening effort also increases when listening to a foreign language, a speaker with an accent, or a phone call with a bad connection. It is associated with allocation of attentional and cognitive resources, and contributes to mental fatigue and distress. An age-related decline in attention, cognitive function, and memory capacity can also lead to increased listening effort.

Subjective measurement of listening effort

Measuring listening effort has mostly been done via subjective measurements: questionnaires, rating scales, or self-reports. But these subjective measurements are – as the name implies – subjective. They are sensitive to the individual’s understanding of the situation, awareness of his or her performance on the task, and numerous other indicators of their body and mental states (energy levels, anxiety etc) not necessarily connected to the listening process itself. These confounding factors result in high risks of intra-rater variability (variability in response from the same subject in different trials), as well as inter-rater variability (variability in response from different subjects) of subjective effort measurements.

Objective measurement of listening effort

There are also a handful of objective ways to measure listening effort. For example, dual task paradigms are considered objective. This type of assessment requires participants to perform two tasks at the same time, such as listening to speech while performing a visual or memory activity; and the listener’s performance of the secondary task serves as a reflection of listening effort. This is a complex study method easily influenced by factors such as motivation, task strategy, and subject cooperation. Results from such studies are also very difficult to compare to other studies with different setups or tasks. Other objective measurement methods include: evaluating pupil dilation (pupillometry), different psychophysiological measures such as heart rate, skin conductance, skin temperature, and elecromyographic activity. All of these methods also have their drawbacks and limitations because what they measure is not the effort of auditory processing in itself, but the response of the body to the activities in the auditory system. We can listen with our eyes closed. We can listen to languages we do not understand. Each of us has a different bodily response to the same listening experience. All of these facts are examples of factors that can arguably influence the outcome of such psychophysiological measures. In general, a widely accepted method to evaluate listening effort in clinical procedures is still not available.

Oscillatory electroencephalography (EEG)

The method used to evaluate the effect of primax features on listening effort was developed and optimized in collaboration with neuroscientists over the last five years. We use an objective method for measuring listening effort based on the EEG. Electroencephalography is a method of recording electrical activity of the brain. It is typically noninvasive, with the electrodes placed along the scalp (Figure 1). EEG measures voltage fluctuations resulting from the neuronal communication within the brain. Diagnostic applications generally focus on the type of neural oscillations (popularly called “brain waves”) that can be observed in EEG signals.

An EEG cap used to position electrodes around the head

Many of us are familiar with the auditory brainstem response (ABR) test, which is an evaluation of electrophysiological events occurring in the brainstem, and is used to help determine the health of the auditory nerve and hearing thresholds. For the purpose of evaluating listening effort, we use EEGs to reveal activities in different higher processing stages of the neural pathways. Unlike ABR testing, which examines evoked responses, we record and evaluate oscillatory, or ongoing, EEGs for the purpose of measuring listening effort (Figure 2). The oscillatory EEG measurement has advantages over evoked EEG activity, because it can be analyzed during longer listening period and it is not restricted to the application of short repetitive stimuli. This way, we are able to measure listening effort in real-time while speech recognition tasks are being conducted.

EEG recordings

Measuring listening effort by analyzing EEG

ABRs are interpreted by analyzing the delay and amplitude of the averaged waveforms in response to repeated stimuli. Since ongoing EEGs are not evoked by repeating the same stimuli, waveforms cannot be simply averaged, and another method for interpreting the results is necessary. In our studies, we rely on special time-frequency transforms of the ongoing EEG activity while subjects perform speech recognition task and evaluate the instan-taneous phase of the resulting signal. In other words, we evaluate the frequency-specific instantaneous phase characteristics of the EEG waveforms during testing. Since phase is measured as an angle, i.e. in degrees, we plot these phase measurements on a so-called unit circle (Figure 3). Depending on whether or not the instantaneous phase formation of the ongoing EEG waves are uniformly distributed around the unit circle, it is possible to objectively measure listening effort expended by the subject performing the speech recognition task.1-6 If the distribution is rather uniform, it is an indication of a low effort listening task. If the distribution is not uniform, i.e., clustered on the unit circle, then the listening effort is considered high. Mathematical models were then used to translate this phase clustering to a numerical scale from 0.0 to 1.0, 0.0 being negligible listening effort, and 1.0 being very high listening effort.

Why it works

Different listening effort methods measure body reactions to sounds at different stages of sound processing. For example, pupillometry is most associated with response to stress and anxiety, whereas dual task based methods estimate complex cognitive abilities of the listener. The advantage of the EEG is that it directly reflects the activity of the central nervous system in which listening effort originates. It is not an indirect global response of the autonomic nervous system such as the other mentioned methods. In fact, the so-called frontal midline theta in the EEG is recognized for decades as a direct index of mental effort,7, 8 and has been refined in our studies for assessing listening effort using the described method. For our purposes, we focus on perceptual correlates in the EEG measurements where the effort of listening is most influenced by the individual’s hearing loss. The objective measure of listening effort obtained reflects direct consequences of impaired hearing while the subject is performing certain listening tasks, and while certain hearing aid features activated or disabled. In this way, we can directly evaluate the effect of specific hearing aid features on the wearer’s listening effort expended due to his hearing, and not confounding factors such as stress, emotions, or higher cognitive functions.

Plotting phase around the unit circle. In easy tasks requiring low listening effort, the corresponding instantaneous phase formation of the ongoing EEG waves (red tracings) is plotted as tiny blue circles distributed rather uniformly around the unit circle (bottom left). On the other hand, in EEG waves resulting from difficult tasks requiring more listening effort (blue tracings), the phase plots are not evenly distributed around the unit circle (bottom right).

References

1. Strauss, D. J., Corona-Strauss, F. I., & Froehlich, M. (2008). Objective Estimation of the Listening Effort: Towards a Neuropsychological and Neurophysical Model. Conf Proc IEEE Eng Med Biol Soc,1777-1780.

2. Strauss, D. J., Corona-Strauss, F. I., Trenado, C., Bernarding, C., Reith, W., Latzel, M., & Froehlich, M. (2010). Electrophysiological correlates of listening effort: neurodynamical modeling and measurement. Cogn Neurodyn, 4, 119-131.

3. Bernarding, C., Strauss, D. J., Latzel, M., Hannemann, R., Chalupper, J., & Corona-Strauss, F. I. (2011). Simulations of hearing loss and hearing aid: effects on electrophysiological correlates of listening effort. Conf Proc IEEE Eng Med Biol Soc, 2319-2322.

4. Bernarding, C., Strauss, D., J., Hannemann, R., Seidler, H., & Corona-Strauss, F. I. (2013). Neural Correlates of Listening Effort Related Factors: Influence of Age and Hearing Impairment. Brain Res Bull. 91, 21-30.

5. Bernarding, C., Hannemann, R., Herrmann, D., Strauss, D. J., & Corona-Strauss, F. I. (2013). Extraction of listening effort correlates in the oscillatory EEG activity: Investigation of different hearing aid configurations. Neural Engineering (NER), 2013 6th International IEEE/EMBS Conference, 1258-1261.

6. Schafer, P. J., Serman, M., Arnold, M., Corona-Strauss, F. I., Strauss, D. J., Seidler-Fallbohmer, B., Seidler, H. (2015). Evaluation of an objective listening effort measure in a selective, multi–speaker listening task using different hearing aid settings. Conf Proc IEEE Eng Med Biol Soc, 4647-50.

7. Wisniewski, M. G., Thompson, E. R., Iyer, N., Estepp, J. R., Goder-Reiser, M. N., Sullivan, S. C. (2015). Frontal midline θ power as an index of listening effort. Neuroreport, 26(2), 94-9.

8. Bruneau, N, Roux, S., Guérin, P., Garreau, B., Lelord, G. (1993). Auditory stimulus intensity responses and frontal midline theta rhythm. Electroencephalogr Clin Neurophysiol, 86(3), 213-6. Topic in brief: Objective Measurement