1. Background
One way to enhance motor imagery is to observe movement, that is, to observe the movement of the body parts relevant to the motor imagery task. Previous research has shown that mirror neurons engage in action comprehension and learning through imitation, which causes activation of the corresponding regions. Thus, action observation serves to induce stimulation of mirror neurons when a person observes another entity reflecting imagined body movements.
The event-related desynchronization (ERD) patterns of 2D and 3D movements were significantly different, with enhanced ERD in the 3D visualization group. Richer visualization and stronger ownership of the observed movements induced better ERD occurrence.
A recent research paper published in the journal IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING explored whether the rich immersion of virtual reality (VR) affects repetitive motor imagery training through the motor observation of handgrip movements. In order to investigate whether differences in display medium affect action observation when performing motor imagery, the researchers displayed the same graphical handshake action through two different displays: an immersive VR headset and a monitor. In addition, the study used graphical scenarios as stimuli to place more emphasis on the effects of illusion and concretization in immersive VR on movement observation during motor imagery training. To examine brain activity when using these two different media, the researchers used EEG and identified changes in neural signals evoked in the sensorimotor cortex. To measure the distinguishability of spatial brain activity patterns in different motor imagery tasks, the researchers applied machine learning techniques commonly used in brain-computer interfaces to learn and distinguish between different types of brain activity in motor imagery.
2. Research Procedure
The researcher conducted two experiments with each participant to investigate whether the use of an immersive VR headset to provide action observation during motor imagery training had an effect on performance:
(1) Immersive VR-based motor imagery (IVR-MI): a graphical handshake scenario using an immersive VR headset for motor imagery training The experiment.
(2) Display-based Motor Imagery (MD-MI): an experiment using a non-immersive display to show the same scene during motor imagery training.
The MD-MI results were used as a control to analyze the effect of VR on motor imagery.
2.1 Subjects
*** Twenty healthy participants between the ages of 20 and 37 participated in both experiments. All participants were also asked to use the VR headset for a longer period of time before the experiments to ensure that they did not have any problems while using the VR headset. Participants were randomized into two equal groups: group A underwent the MD-MI prior to the IVR-MI, and group B underwent the IVR-MI prior to the MD-MI.In order to reduce the likelihood of the former experiment influencing the results of the latter experiment, the latter experiment was conducted at least 7 days after the former experiment. The results of the experiments were also not revealed to the participants until the end of both experiments to avoid any feedback that might influence performance. Data collected from each participant were visually inspected, excluding data from two of the participants as they showed extensive noise, leaving a final total of 18 participants **** for analysis.
2.2 Scenario
This graphical scenario consisted of two virtual hands with arrows on a black background and was implemented using the Unity game engine. Before each experiment, the position of the virtual hands was adjusted so that the distance between the two virtual hands was roughly equal to the participant's shoulder width (Figure 1a).
(1) IVR-MI setup: participants wore the Oculus Go without the vertical strap after putting on the EEG cap with electrodes to prevent tightening of the strap on the overlapping electrodes.
(2) MD-MI setup: a monitor with a monitor arm that provided three degrees of freedom was placed on the table in front of the participant. Each participant was free to adjust the angle of the monitor arm.
Each participant was able to adjust the camera view in the Unity application to maximize virtual hand ownership. Participants were asked to place their hands on the table so that their own hands were replaced by overlapping virtual hands.
2.3 Data Acquisition
BrainProducts' actiChamp and actiCAP were used to retrieve EEG data from each participant's scalp. Data were sampled at a sampling rate of 500 Hz and active electrodes were placed according to the International 10-20 system. Throughout the experiment, EEG signals were recorded from 20 electrodes (FC5, C5, CP5, FC3, C3, CP3, FC1, C1, CP1, Cz, CPZ, FC2, C2, CP2, FC4, C4, CP4, FC6, C6, CP6) placed around the sensory-motor cortex, with the grounded electrodes and reference electrodes located at the AFz and Fz positions, respectively (Figure 1b). The EEG signals were recorded with BrainVision and the impedance of each electrode was controlled below 5 k to obtain high quality data. The data were band-pass filtered between 8 and 25 Hz. After collection, the EEG data were then re-referenced by applying an average reference over all used electrode positions. The resulting preprocessed data was used for analysis of neural activity.
2.4 Experimental design
The experiments were conducted in a dark, soundproofed room to minimize any environmental disturbances. Each motor imagery experiment consisted of 10 consecutive motor imagery experiments in six phases. Participants could rest between phases if desired. Each trial consisted of a randomized sequence of one consecutive right-handed grasping motor imagery task, one consecutive left-handed grasping motor imagery task, and a rest task (Figure 2a).
Individual tasks consisted of an initial 4-second instruction cycle and a subsequent 6-second motor imagery cycle, followed by a 2-second rest period (Figure 2b). During the instruction period, participants were given either a cross-shaped cue indicating the rest task or an arrow cue indicating a left- or right-handed grasping motor imagery task to inform the participant what the next task would be and to instruct them to gaze at the corresponding hand. Throughout the motor imagery cycle following the instruction cycle, the virtual hand corresponding to the arrow cue simulated a series of grasping movements and instructed participants to observe and imagine performing the same movements in a motor manner. Finally, during the rest period, the virtual hand remained motionless and participants were allowed to move or blink to prevent eyestrain. During both the instruction period and the motor imagery period, participants were instructed to avoid any movement, including blinking. Throughout the experiment, both virtual hands were displayed and participants were expected to visualize them as their own.
3. Research Methods
3.1 ERD Analysis
Brain regions corresponding to electrode locations C3 and C4 are associated with right- and left-handed grasping movements, respectively. To measure changes in brain activity during a single session, we first calculated the mean power spectra of the EEG data recorded from the three motor imagery tasks using the following equation:
To analyze the change in ERD amplitude induced by subjects during left- and right-handed grip motor imagery in each session over time, we calculated the ERD ratios for the two motor imagery tasks relative to the rest task using the following equation:< /p>
Thus, the ERD ratios for each phase were calculated based on the differences in brain pattern features evoked during the different motor imagery tasks at each electrode location.
To analyze the motor imagery performance of each experiment, the researchers further calculated the average ERD ratio for each experimental participant by applying the following equation:
Considering that the most active frequency band may be different for each individual, the frequency band for each participant in the two equations was determined by choosing the band with a bandwidth of 2 Hz, which leads to the maximum average ERD ratio.
The ERD results for C3 and C4 of the right-handed and left-handed grip motor imagery were analyzed separately to explore subjects' performance on the two different tasks. In order to examine the effect of using different display media on each participant, the paper conducted a two-way ANOVA on the calculated mean ERD values, where the designated group (indicating the order of the experiment) and the display media were used as two factors. To further examine the statistical enhancement of participants' ERDs in each session, the paper applied a Dunnett-type nonparametric multiple comparisons test, in which the ERD ratio of the first session was used as a control. Thus, ERD ratios for the right-handed motor imagery task and the left-handed motor imagery task were compared in two separate experiments (Figure 3).
3.2 Discriminant Analysis
Through the discriminant analysis of neural activity in the two experiments, a classical machine learning model was constructed to further evaluate performance. To compare the classification accuracy of each participant in the two experiments, 6 seconds of EEG data from each motor imagery cycle was extracted. To increase the amount of data to be learned by the model, the paper further augmented the data with each 6-second EEG data by dividing the data into 2-second-long time windows at 100-millisecond steps.
The Common*** Spatial Patterns (CSP) algorithm was applied to extract spatial features from the preprocessed EEG data and Fisher's Linear Discriminant Analysis (LDA) was used to create a classification model that predicted whether the EEG data segments were involved in a resting, left- or right-handed motor imagery task. To evaluate the motor imagery EEG data, we used two different cross-validation methods: 1) 6-fold cross-validation, in which data from individual experiments were analyzed and each fold corresponded to data retrieved from a single session of 10 motor imagery trials; and 2) 10-fold cross-validation, in which data from individual sessions were used and each fold corresponded to data retrieved from a single experiment. Cross-validation was used to test the accuracy of distinguishing between three different motor imagery tasks: left-handed grasping, right-hand grasping, and resting state. For statistical analysis, the paper performed a two-way ANOVA test on the 6-fold cross-validation results to indicate the overall performance of each experiment. To further test for statistical enhancement of neural activity discrimination, the study used a Dunnett-type nonparametric multiple comparison test on the 10-fold cross-validation results, in which the accuracy of the first session was used as a control.
4. Findings
4.1 Statistical Analysis Hypothesis Validation
Prior to performing ANOVA on the ERD results for left- and right-handed motor imagery and parametric tests on the cross-validated accuracy results, the necessary hypotheses were validated. The results of the Shapiro-Wilk normality test and Levene chi-square test are shown in Table 1. p-value results indicate that normality and homogeneity were not violated in all cases (p>0.05).
4.2 Experimental analysis of ERD performance
In order to compare the performance of participants using the two different display mediums, we analyzed ERD ratios, which are represented by the average ERD ratio of participants during motor imagery, and ERD magnitude, which represents the average of ERDs collected from each session over time.
The ERD ratios and ERD amplitudes for left- and right-handed motor imagery were compared between the two experiments, as shown in Figure 4. The ANOVA results in Figure 4a showed that the ERDs in left-handed motor imagery were greater in IVR-MI than in MD-MI (49.32 ± 12.08 and 34.75 ± 14.75 for IVR-MI and MD-MI, respectively), and the difference was highly significant (F(1, 16) = 20.182, p<0.001). The ERD values for right hand motor imagery were also greater in IVR-MI compared to MD-MI (53.29±12.57 and 41.32±15.19, respectively), and the difference was highly significant (F(1, 16)=14.693, p<0.01). On the other hand, the difference between left-handed and right-handed motor imagery in both groups of subjects was not significant (F(1, 16) = 0.131, p> 0.72; F(1, 16) = 1.034, p> 0.32).
Figure 4b shows participant ERD magnitude relative to time, which was calculated by averaging each participant's ERD magnitude across all sessions.The red and blue waveform plots for IVR-MI and MD-MI show that there was a significant difference in ERDs for both the left and right handers during the motor imagery period, and that the ERD amplitude was greater for the IVR-MI than for the MD-MI.As shown on the x-axis of the gray scale shows, the time domains of left-handed motor imagery ranged between 1.05.4 seconds and 6.27.0 seconds, and the time domains of right-handed motor imagery ranged between 1.45.8 seconds and 6.07.2 seconds, with a significant difference between the two amplitudes. There were no statistically significant differences between the two groups during the instruction period (left-hand motor imagery t<1.0s, right-hand motor imagery t<1.4s) and at the end of the resting period (left-hand motor imagery t>7.0s, right-hand motor imagery t>7.2s).
4.3 Experiment-Wise cross-validation
Figure 5 shows the results of the 6-fold object-related cross-validation accuracy for IVR-MI and MD-MI, where individual folds indicate data acquired from each session. The ANOVA results showed that the difference in accuracy between the two mediums was highly significant (F(1, 16) = 20.990, p<0.001), and that IVR-MI was more accurate than MD-MI (67.85 ± 13.50 and 57.49 ± 13.96, respectively). In contrast, the difference within the two groups was not statistically significant (F(1, 16) = 0.008, p>0.93).
4.4 Session-Wise Changes in ERD Performance
The study further analyzed how the ERD performance of left-handed and right-handed motor imagery changed with training time. As shown in Figure 6, there was a linear positive correlation between the ERD rates of both IVR-MI and MD-MI during left-handed motor imagery (IVR-MI r=0.345, p<0.001; MD-MI r=0.260, p<0.01). Similar results were found for right-handed motor imagery (IVR-MI r=0.362, p<0.001; MD-MI r=0.181, p>0.001). The r and p values for IVR-MI were statistically stronger than those for MD-MI in both left- and right-handed motor imagery.
The ERD ratio for the first session was chosen as the baseline and compared with the ERD ratios for the other sessions to analyze the improvement in ERD performance compared to the respective sessions, as shown in Figure 6 and Table 2. For left-handed motor imagery, participants in both IVR-MI and MD-MI showed significant improvement from session 5, but the improvement was stronger in IVR-MI and MD-MI (p<0.01 and p<0.05 for IVR-MI and MD-MI in session 5, p=0.014 and p=0.032 for IVR-MI and MD-MI in session 6) . For right-handed motion images, participants only showed significant differences when using the VR headset (p<0.05 and p<0.01 for 4th and 6th, respectively), whereas no significant improvement was observed in repeated tests using the monitor screen.
4.5 Cross-validated Session-Wise Changes
Figure 7 illustrates the results of using 10-fold cross-validation to discriminate between patterns of brain activity in each session, where individual folds represent data from each trial. For both IVR-MI and MD-MI, the accuracy results were positively linear (r=0.276, p<0.01 and r=0.136, p>0.05, respectively). The r and p values for cross-validation accuracy were stronger for IVR-MI compared to MD-MI.
To analyze the increase in cross-validation accuracy across time, we performed a Dunnett-type non-parametric multiple comparison test on the accuracy results for the first time period. The results in Figure 7 and Table 3 show that participants during IVR-MI were able to show significant improvements in discrimination from session 5 onwards (p<0.01 and p<0.05 for sessions 5 and 6, respectively), while no significant differences were observed during MD-MI.
4.6 Fisher ratio topography
To further investigate the spatial features obtained from the different hand-imagery tasks, we applied Fisher ratios on each electrode using the ERD results. As shown in Figure 8, electrode positions C3 and C4 were the main factors distinguishing left- and right-handed motor imagery. Compared with the Fisher ratios for MD-MI (0.544 and 0.377 for C3 and C4, respectively), the Fisher ratios for both C3 and C4 were higher in the IVR-MI group (0.997 and 0.566 for C3 and C4, respectively).
5. DISCUSSION
This study examined the effects of immersion and illusion on motor imagery training by using a VR headset and a monitor as a medium for observing left- and right-handed movements. By comparing the ERD ratios and cross-validation accuracies obtained from the two experiments, the paper provides evidence that perceiving the same movements through different media during training may lead to different motor imagery performance.
The results of the study showed that participants were able to achieve better motor imagery performance when using a VR headset. In terms of practicing motor imagery through repetitive training, not only was it confirmed that repetitive movement observation affects subjects' motor imagery performance, but it was also found that using a VR headset may improve motor imagery performance with less time cost. By showing greater improvements in both ERD rates and cross-validation accuracy results with the VR headset, the paper confirms that using a VR headset is more effective than using a monitor display in improving ERD performance and increasing spatial discrimination of brain activity.
The researchers also examined ERD amplitudes and Fisher ratios to address the concern that simply having different display media affects ERD ratios in the central motor cortex (C3 and C4). The results of this study showed a slight increase in ERD amplitude patterns with no significant difference during the instruction period, and then a statistically significant difference between the two experiments, with a large increase during motor imagery followed by a decrease during the resting period (Fig. 4b). Although the researcher expected that the slight increase with no significant difference during the instructional period was a result of the preparation and planning of the instructed movements, the fact that the ERD magnitude of the IVR-MI was significantly higher than the significant increase of the MD-MI only during motor imagery and the early resting period suggests that the statistically significant difference was caused by the motor imagery manipulation. In addition, Figure 8 shows that in both experiments, the main spatial features distinguishing the different motor imagery tasks came from the C3 and C4 electrodes, suggesting that differences in display medium alone had little effect on factors that might have influenced our results, such as spatial features from the visual cortex. These results suggest that motion observation through a VR headset is more effective than motor imagery manipulation through a monitor display.
As mentioned previously, the paper focused on whether immersion and illusion were effective for repetitive motor imagery training of action observation via a VR system. The hypotheses of the paper were validated by ERD performance and cross-validation results, which showed higher ERD ratios and more discriminative spatial brain activity during repetitive motor imagery training. The results suggest that rich immersion itself influences motor imagery (by presenting the same graphic hand movements). Thus, for any graphical scene that can be simulated, the use of an immersive VR headset may prove beneficial for motor imagery training compared to a non-immersive display.
There are some limitations and possible improvements to the study. There may be a concern that the graphical scenarios of the study may be perceived differently to some extent, as the proportions of the virtual hands may not be exactly the same for the two display mediums. To address this concern, a focus was placed on each participant's feedback prior to beginning each experiment when resizing to maximize representation. Additionally, while the researcher adjusted various environmental components to amplify embodiment in the study, the level of embodiment for each user was not directly quantified in the two experiments. Due to the considerable time gap between the two experiments, the researcher considered any possible surveys or questionnaires to be potentially unreliable and instead used the results of previous work to claim that VR enhanced concretization. Finally, the relatively small sample size was also a limitation. Although each participant underwent multiple repetitions of the experiment, the statistical power of the analysis may have been limited given the differences in each individual's performance. Therefore, the findings of the article should be interpreted carefully. Based on the findings of the article, future research will focus on using the article's metrics to compare the use of VR headsets (a fully immersive visualization tool) and stereoscopic 3D glasses (a semi-immersive virtual reality system).
6. Conclusions
Unlike previous studies that focused on comparisons between action observation and motor imagery of the visual scene itself, this study looks at the combined effects of immersive VR and concretization on motor imagery. Inspired by the ability of VR headsets to provide a more realistic experience with enhanced illusion and immersion compared to other available media, the researchers investigated whether immersive VR headsets could also be used to enhance motor imagery performance by comparing action observation of the same virtual hand movements with VR headsets and monitors.
The paper examined two different aspects of brain patterns associated with motor imagery performance in both mediums: changes in the oscillatory rhythms of signals from motor imagery-related brain regions, and the distinguishability of spatial features of the signals, which was explored using machine learning models typically used for brain-computer interfaces. The results of these two analyses suggest that the use of VR headsets may lead to greater oscillatory changes and spatial discrimination of neural signals. Therefore, the use of a VR headset that combines immersion and illusion may better present the observation of movement in motor imagery training in the fields of clinical therapy, rehabilitation and brain-computer interfaces. In the fields of clinical treatment, rehabilitation and brain-computer interfaces, the use of VR headsets can better present the observation of movements in motor imagery training.