latest research papers on haptic technology

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Front Robot AI

Applications of Haptic Technology, Virtual Reality, and Artificial Intelligence in Medical Training During the COVID-19 Pandemic

Mohammad motaharifar.

1 Advanced Robotics and Automated Systems (ARAS), Industrial Control Center of Excellence, Faculty of Electrical Engineering, K. N. Toosi University of Technology, Tehran, Iran

2 Department of Electrical Engineering, University of Isfahan, Isfahan, Iran

Alireza Norouzzadeh

Parisa abdi.

3 Translational Ophthalmology Research Center, Farabi Eye Hospital, Tehran University of Medical Sciences, Tehran, Iran

Arash Iranfar

4 School of Electrical and Computer Engineering, University College of Engineering, University of Tehran, Tehran, Iran

Faraz Lotfi

Behzad moshiri.

5 Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, ON, Canada

Alireza Lashay

Seyed farzad mohammadi, hamid d. taghirad.

Pete Culmer , University of Leeds, United Kingdom

Soroosh Shahtalebi , Montreal Institute for Learning Algorithm (MILA), Canada

Associated Data

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding authors.

This paper examines how haptic technology, virtual reality, and artificial intelligence help to reduce the physical contact in medical training during the COVID-19 Pandemic. Notably, any mistake made by the trainees during the education process might lead to undesired complications for the patient. Therefore, training of the medical skills to the trainees have always been a challenging issue for the expert surgeons, and this is even more challenging in pandemics. The current method of surgery training needs the novice surgeons to attend some courses, watch some procedure, and conduct their initial operations under the direct supervision of an expert surgeon. Owing to the requirement of physical contact in this method of medical training, the involved people including the novice and expert surgeons confront a potential risk of infection to the virus. This survey paper reviews recent technological breakthroughs along with new areas in which assistive technologies might provide a viable solution to reduce the physical contact in the medical institutes during the COVID-19 pandemic and similar crises.

1 Introduction

After the outbreak of COVID-19 virus in Wuhan, China at the end of 2019, this virus and its mutations has rapidly spread out in the world. In view of the fact that no proven treatment has been so far introduced for the COVID-19 patients, the prevention policies such as staying home, social distancing, avoiding physical contact, remote working, and travel restrictions has strongly been recommended by the governments. As a consequence of this global problem, universities have initiated policies regarding how to keep up teaching and learning without threatening their faculty members and students to the virus. Thus, the majority of traditional in-class courses have been substituted to the online courses. Notwithstanding the fact that the emergency shift of the classes have reduced the quality of education during the COVID-19 pandemics Hodges et al. (2020) , some investigators have proposed ways for rapid adaption of the university faculty and the students to the situation and improve the quality of education Zhang et al. (2020) .

Nevertheless, the case of remote learning is different in the medical universities as the learning process in the medical universities is not just rely on the in-class courses. As an illustration, the medical training in the traditional way is accomplished by a medical student through attending some training courses, watching how the procedure is performed by a trainer, performing the procedure under supervision of a trainer, and at the final stage, independently performing the procedure. In fact, the traditional method of surgery training relies on excessive presence of students in the hospital environments and the skill labs to practice the tasks on the real environments such as physical phantoms, cadavers, and patients and that is why medical students are called “residents”. Thus, the aforementioned traditional surgery training methodology requires a substantial extent of physical contact between medical students, expert surgeons, nurses, and patients, and as a result, the risk of infection is high among those people. On the other hand, the assistive technologies based on virtual reality and haptic feedback have introduced alternative surgical training tools to increase the safety and efficiency of the surgical training procedures. Nowadays, the necessity of reducing the physical contact in the hospital environments seems to make another motivation for those assistive technologies. Therefore, it is beneficial to review those technologies from COVID-19 motivation aspect.

In this paper, the existing assistive technologies for medical training are reviewed in a COVID-19 situation. While there are several motivations for those technologies such as increasing the safety, speed, and efficiency of training, the new motivations created for those technologies during the COVID-19 pandemic are the specific focus of this paper. In spite of the existing literature on COVID-19, our main focus is surgery training technologies that help to reduce physical contact during this and other similar pandemics. Notably, a number of those studies have analyzed systemic and structural challenges applicable to medical training programs with little emphasis on technological aspects of the subject Sharma and Bhaskar (2020) , Khanna et al. (2020) . On the other hand, the methods of remote diagnostics and remote treatment have received a great deal of attention after COVID-19 pandemic and a massive body of literature have covered those topics Tavakoli et al. (2020) , Feizi et al. (2021) , Akbari et al. (2021) . In contrast, less studies have given special attention on remote training and remote skill assessment which is the subject of this paper. For this reason, this paper addresses scientific methods, technologies and solutions to reduce the amount of physical contact in the medical environments that is due to training reasons.

Relevant literature was chosen from articles published by IEEE, Frontiers, Elsevier, SAGE, and Wiley with special attention to the well-known interdisciplinary journals. The search was preformed using the keywords “remote medical training,” “skill assessment in surgery,” “virtual and augmented reality for medical training,” “medical training haptic systems,” and “artificial intelligence and machine learning for medical training” until June 30, 2021. The literature was examined to systematically address key novel concepts in remote training with sufficient attention to the future direction of the subject. Finally, it is tried to review the problem in the COVID-19 context in a way that the discussed materials are distinct from similar literature in a conventional non-COVID context.

The rest of this paper is organized as follows: The clinical motivations of the training tools are discussed in Section 2 . The virtual and augmented reality and the related areas of utilization for medical training are described in Section 3 . Section 4 explains how haptic technology may be used for medical training, while Section 5 describes some data-based approaches that may be used for skill assessment. Then, the machine vision and its relevant methods used for medical training are presented in Section 6 . Finally, concluding remarks are stated in Section 7 .

2 The Clinical Motivation

The process of skill development among medical students have always been a challenging issue for the medical universities, as the lack of expertise may lead to undesired complications for the patients Kotsis and Chung (2013) . Moreover, owing to the rapid progress of minimal invasive surgeries during the past decades, the closed procedures have been becoming a method of choice over traditional open surgeries. In the minimal invasive surgery, the instruments enter the body through one or more small incisions, while this type of surgery is applicable to a variety of procedures. The foremost advantage of this technique is the minimal affection to healthy organs, which leads to less pain, fewer post-operative complications, faster recovery time, and better long-term results.

However, the closed surgery technique is more challenging from the surgeon’s point of view since the surgeon does not have a complete and direct access on the surgical site and the tiny incisions limit the surgeon’s accessibility. Owing to the limited access, some degrees of freedom are missing and surgeon’s manipulation capability is considerably reduced. Furthermore, there is fulcrum effect at the entry point of the instrument, i.e., the motion of the tip of the instrument, which is placed inside the organ, and the external part of the instrument, which is handled by the surgeon, are reversed. This results in more difficult and even awkward instrument handling and requires specific and extensive surgical training of the surgeon. As a result, the minimal invasive surgeries demands advanced expertise level, the lack of which might cause disastrous complications for the patient. These conditions are equally important in many medical interventions, especially in minimally invasive surgeries. Here a number of specific areas of surgical operation are expressed in order to address complications that might occur during the training procedures.

• • Eye surgery:

An important category of medical interventions which need a very high skill level is intraocular eye surgical procedures. Notably, the human eye is a delicate and highly complex organ and the required accuracy for the majority of intraocular surgeries is in the scale of 50–100 microns. The closed type of surgery is applicable to a number of eye surgeries such as the Cataract surgery in the anterior segment as well as the vitro-retinal surgical procedures in the posterior segment. Notably, some complications such as Posterior Capsule Rupture (PCR) for cataract surgery and retina puncture for the vitro-retinal surgical procedures are among the relatively frequent complications that might happen, due to the surgeon’s lack of surgical skills and dexterity. It is shown in a study on ophthalmic residents that the rate of complications such as retinal injuries is higher for the residents with less skills Jonas et al. (2003) .

• • Laparoscopic Cholecystectomy

Another example is Laparoscopic Cholecystectomy (LC) which is now the accepted standard procedure across the world and is one of the most common general and specialist surgical procedures. However, it can be prone to an important complication that is bile duct injury (BDI). Although BDI is uncommon but it is one of the most serious iatrogenic surgical complications. In extreme BDI cases, a liver resection or even liver transplantation becomes necessary. BDI is considered as an expensive medical treatment and its mortality rate is as high as 21% Iwashita et al. (2017) .

• • Neurosurgery

Neurosurgery is another field that deals with complex cases and requires high accuracy and ability in the surgeon’s performance. In a prospective study of 1,108 neurosurgical cases, 78.5% of errors during neurosurgery were considered preventable Stone and Bernstein (2007) . The most frequent errors reported were technical in nature. The increased use of endoscopy in neurosurgery introduces challenges and increases the potential for errors because of issues such as indirect view, elaborate surgical tools, and a confined workspace.

• • Orthopedic surgery

In the field of orthopedics, knee and shoulder arthroscopic surgeries are among the most commonly performed procedures worldwide. There is a steep learning curve associated with arthroscopic surgery for orthopaedic surgery trainees. Extensive hands-on training is typically required to develop surgical competency. The current minimum number of cases may not be sufficient to develop competency in arthroscopic surgery. It is estimated that it takes about 170 procedures before a surgeon develops consultant-level motor skills in knee arthroscopic surgery Yari et al. (2018) . With work-hour restrictions, patient safety concerns, and fellows often taking priority over residents in performing cases, it is challenging for residents to obtain high-level arthroscopic skills by the end of their residency training.

The above motivation shows the importance of skill development among the medical students. The standard process of procedural skill development in medicine and surgery is shown as a diagram in Figure 1 . In the observation stage, the medical students need to attend a clinical environment and watch how the procedure is performed by a trainee. Then, the medical students get involved in the operation as an apprentice, while the actual procedure is performed by the trainer. Later, the medical students practice the operation under direct supervision of the trainer, while the trainer assesses the skill level of the medical students. The supervised practice and skill assessment steps are repeated as long as the trainee does not have enough experience and skill to conduct the procedures without supervision of the trainer. Finally, after obtaining sufficient skill level, the trainee is able to independently perform the operation.

Process of procedural skill development in medical training and surgery.

Remarkably, a learning curve is considered for each procedure, which means that performance tends to improve with experience. This concept applies for all of the medical procedures and specialties, but complex procedures, surgery in particular, are more likely to gradual learning curves, which means that improvement and expertise is achieved after longer training time. Some of the important factors in the learning curve are manual dexterity of the surgeon, the knowledge of surgical anatomy, structured training and mentoring and the nature of the procedure. The learning curve is longer for minimally invasive procedures than that for open surgical procedures. The learning curve is also influenced by the experience of the supporting surgical team. Besides, learning curves depend on the frequency of procedures performed in a specified period. Many studies suggest that complication rates are inversely proportional to the volume of surgical workload.

Notably, the above mentioned process of skill development require a considerable extent of physical contact between the trainees, the expert surgeons, the nurses, and the patients, while this shall be reduced in the COVID-19 pandemic. In addition to the high risk of infection in the medical universities with the conventional medical training approaches, the majority of the health-care capacity is focused on fighting the COVID-19 virus and consequently, the education requirements of medical universities are failed to be entirely fulfilled. As a result, the training efficiency of medical universities will be reduced, provided that they just rely on the conventional training approaches. This will have possible side-effects on the future performance of the health-care system mainly due to the insufficient number of recently graduated students with adequate expertise level.

On the other hand, traditional education takes place in hospitals and on real patients, which face several problems during the COVID-19 pandemic: the hospital environment is contaminated with the virus, hospital staff and physicians are very busy and tired and have less training capacity, prolonged hospital stays of patients to train students put them at greater risk for exposure to the virus, especially if complication occurs by a resident who does not have gained sufficient skills during the training procedure. Therefore, training with assistive devices outside the hospital may play an effective role in this situations. The highlighted factors can significantly be improved by assisted learning, especially in minimally invasive procedures. In more complex surgeries, the complications becomes more serious, the learning curve will be longer, and the role of assisted learning becomes more prominent.

To solve the above mentioned problems, assistive training tools provide a variety of solutions through which the medical universities are able to continue their education procedures, while the risks enforced by the COVID-19 outbreak are reduced. In the following sections, the main assistive training tools including the haptic systems, virtual reality, machine vision, and data mining are reviewed and the areas in which those technologies facilitate the training process during the COVID-19 pandemic are detailed. The aim of these technologies is to have the training efficiency higher or at least equal to that of the conventional training methods without risk of infection of the involved parties to the virus.

3 Virtual and Augmented Reality

Virtual Reality is employed to create an immersive experience for various applications such as visualization, learning and education. In virtual reality, a computer generated graphical presence is visualized using a head mounted display and the user can interact with 3D objects located in the virtual world. In addition to VR, the Augmented Reality (AR) is developed to add 3D objects to the real world creating a different experience by adding digital information to the real objects in the surrounding environment. Although experiencing the 3D objects in VR scenes is far from the interaction with real objects, the VR experience is getting closer to the real world environments by the help of more realistic computer graphics and full-body haptics suits.

The virtual reality (VR) and augmented reality (AR) are getting more interest as a training technique in the medical fields, unlocking significant benefits such as safety, repeatability and efficiency Desselle et al. (2020) . Furthermore, during the COVID-19 pandemic, remote training and consulting are considered as vital advantages of VR/AR based training methods ( Singh et al., 2020 ).

Some advantages of using VR/AR in medical training are depicted in Figure 2 . Safety is the first and the most important benefit of VR/AR employment in medical education. Complex medical operations may be performed in a simulated environment based on VR with complete safety and without putting the patient’s life into danger. Repeatability is the second advantage of using VR as any simulation scenario in the field of medical training can be repeated over and over until the trainee is completely satisfied. During the COVID-19 pandemic it is vital to practice social distancing which is delivered by VR/AR employment in medical education. Medical training and surgery simulation by computer graphics in VR/AR virtual environments results in reduced training costs as no material except than a computer, a VR headset and a haptic device is required. Since medical training by VR/AR is performed using a computer, the surgery simulation is always in hand as soon the computer and VR headset are ready to be used. Therefore, the efficiency of medical training is increased as no time is required for either preparation of an operation room or getting a patient ready.

VR/AR advantages in medical training.

VR/AR techniques are employed in various applications in surgical training as it can be seen in Figure 3 . The first application of AR/VR in surgical training is surgical procedure diagnosis and planning. Using AR/VR, the real surgical operation is simulated ahead without putting the patient’s life into danger. The AR/VR is used in surgical education and training which is mentioned as the second application. Simulation based environments are developed for training of medical students by virtual human anatomy 3D models. Another application of AR/VR is robotic and tele-surgery, by which surgical consulting becomes possible even from a far distance. The last application of AR/VR in surgical training is sensor data and image visualization during the surgical operation which makes the effective usage of patient’s medical data possible.

VR/AR applications in surgical training.

It is shown that the learning curve of hip arthroscopy trainees is significantly improved using a virtual reality simulator ( Bartlett et al., 2020 ). In this study, a group of twenty five inexperienced students were chosen to perform seven arthroscopies of a healthy virtual hip joint weekly. The experimental results indicated that average total time decreased by nearly 75 % while the number of collisions between arthroscope and soft-tissues decreased almost by 90 % .

VR is also employed in orthopedic surgical training, where 37 residents participated in a study to obtain an understanding of the LISS 1 plating surgical process ( Cecil et al., 2018 ). The developed virtual surgical environment is equipped with a haptic device to perform various activities such as assembling LISS plate, placing the assembled LISS plate correctly inside the patient’s leg, and attaching the LISS plate to the fractured bone. The test was divided into pre–test where the students get familiar with the surgery process and the post–test which is devoted to the actual evaluation phase. The participants had 1 h to finish both the pre–and post–tests which resulted in improvement of learning the LISS plating surgical process.

The applicability and effectiveness of VR based training in orthopedic education is evaluated in ( Lohre et al., 2020 ), where nineteen orthopedic surgical residents cooperated in this study. The surgical residents performed a glenoid exposure module on a VR based simulator using a haptic device as the input controller. The result of training of residents using VR simulator has been compared to the conventional surgery training methods. Considering the learning time, repeating 3 to 5 VR based surgery experiments by the residents, resulted in 570 % training time reduction. Additionally, VR based surgical training helped the residents to finish glenoid exposure significantly faster than the residents trained by conventional education methods.

Orthognathic surgery is another surgery field considered for VR based training as it is one of the complex surgical procedures ( Medellin-Castillo et al., 2020 ). While conventional OSG 2 learning techniques are dependent to cadavers or models and experienced surgeons are trained after several years of experiments in operating rooms, employment of VR in surgical training can reduce the learning time and the education cost at the same time. In this study, three cases are considered for evaluation of VR in OSG, cephalometry training, osteotomy training and surgery planning to be precise. The experimental results indicated that the combination of haptics and VR is effective in skill improvement of trainees and surgery time reduction. Furthermore, the surgery errors and mistakes are reduced by using haptic feedback to recreate the sense of touch as trainees can detect landmarks more precisely in comparison to conventional techniques.

In conjunction with VR, the AR technology has also been used in various medical fields for training such as neurosurgical training ( Si et al., 2019 ). Anatomical information and other sensory information can be visualized to the surgeons more properly, and therefore, more accurate decision can be made during a surgery. Although this study is only applicable to the simulated environments because of registration problem, the experiment indicated the effectiveness of the simulator in skill improvement of surgeons.

While key features of VR/AR have led to improved training specially in surgical training, there are some limitations that should be considered ( kumar Renganayagalu et al., 2021 ). The first limitation of VR simulators is the cost of VR content production, and therefore, most of simulators are made for very specific type of simulation in a limited context. The second limitation is the immaturity of interaction devices for VR simulations, which has a great affect on the user experience. Another limitation of VR usage in medical training is the inability of using VR devices for long period of time as the VR devices are made for entertainment and not for a long training session.

It can be concluded that in spite of some limitations, VR/AR based simulators equipped with a haptic device can be used in medical surgery training in order to achieve skill improvement and training time reduction. Furthermore, during the isolation requirements due to COVID-19 pandemic, VR/AR based techniques can be well employed for medical training.

4 Teleoperated Haptic Systems

Haptic systems provide the sense of touch with remote objects without the need of actual contact. It also provides collaboration between several operators without the need of any physical contact. As depicted in Figure 4 , based on the number of the operators, the haptic systems may be classified into single user, dual-user or multi-user haptic systems. Single user haptic systems enable a single human operator to interact with a remote or virtual environment, whereas dual-user or multi-user haptic systems provide a mechanism for collaboration of two or multiple human operators. The medical training applications of those systems is presented here.

Single user vs. dual user haptic systems. (A) Single user haptic system. (B) Dual user haptic system.

4.1 Single User Haptic Systems

Single user haptic systems extend the abilities of human operators to interact with remote, virtual, and out-of-reach environment. In the field of surgery training, a number of investigations have proposed haptic training simulators for training of minimally invasive surgery (MIS) Basdogan et al. (2004) , dental procedures Wang et al. (2014) , sonography Tahmasebi et al. (2008) , and ocular therapies Spera et al. (2020) . As shown in Figure 4A , a typical single-user haptic simulator system consists of a human operator, a haptic interface, a graphical interface, and a reference model for the virtual object. Notably both the graphical interface and the haptic interface utilize the reference model to provide necessary feedback for the operator. While the graphical interface provides a visual feedback of the environment, the haptic interface provides the kinesthetic feedback of the interaction between the tool and the surgical field. Indeed, the role of haptic feedback is to recreate the sense of contact with the virtual environment for the operator. As a result, the circumstances of actual operation is provided for the medical students, while the need of physical presence in the clinical environments is eliminated. Indeed, through haptic technology, the medical students are able to practice on a virtual environment without the need of presence at the clinical environment. Thus, the risk of infection during the COVID-19 pandemic is effectively reduced.

4.2 Dual User Haptic Systems

The cooperative and joint conduction of an operation either for the purpose of collaboration or training, as a fundamental clinical task, cannot be provided by single user haptic systems. In order to make the cooperation of two surgeons possible, the system should be upgraded to a dual user haptic system by adding another haptic console. A dual user haptic system is a more recent advancement in haptic technology, and it consists of two haptic consoles, one for the trainer and one for the trainee Shahbazi et al. (2018a) . Remarkably, the traditional collaboration methods require direct physical contact of the persons conducting the operation, whereas the haptic-based collaboration approach eliminates the physical contact of the collaborators. As a result of removing the need of physical contact, the involved people are no longer in the risk of the Corona virus. A commercial dual user haptic system developed by intuitive Surgical Inc. ® is the da Vinci Si Surgical System which supports training and collaboration during minimally invasive surgery. The da Vinci Si System builds on the existing da Vinci technology, where it has a number of enabling features such as leading-edge 3D visualization, advanced motion technology, and sufficient dexterity and workspace. However, the da Vinci Si does not provide active supervision and intervention of the trainer on the trainee’s actions. As an illustration, in the case that the trainee controls the procedure, the trainer does not have the possibility to guide the trainee during the procedure.

The issue of supervision and intervention of the trainer during the operation in dual user haptic systems have been a topic of active investigation during the past years. A number of studies have utilized the concept of dominance factor to determine the task dominance of each operator Nudehi et al. (2005) , Khademian and Hashtrudi-Zaad (2012) , Shahbazi et al. (2014b) , Motaharifar et al. (2016) . In those approaches, the trainee is given a partial or full task authority by the trainer based on his/her level of expertise. Notably, the task authority provided by these control architectures is supposed to be fixed during the operation. Thus, changing the authority of the surgeons and specially blocking the trainee’s commands is not possible in the middle of the operation. This might lead to undesired operative complications specially in the case that the trainee makes a sudden unpredictable mistake.

Fortunately, a number of investigations have developed control architectures to address the above shortcoming of the previously proposed haptic architectures Motaharifar et al. (2019b) , Shahbazi et al. (2014a) , Motaharifar and Taghirad (2020) . As a case in point, an S-shaped function is proposed in Motaharifar et al. (2019b) for the adjustment of the corrective feedback in order to shape the trainee’s muscle memory. In fact, the training approach behind the presented architecture is based on allowing the trainee to freely experience the task and be corrected as needed. Nevertheless, through the above scheme, the trainee is just granted the permission to receive the trainer’s motion profile; that is, the trainee is deprived of any realistic contribution to the surgical procedure. In contrast, several investigations have proposed mechanisms for adjusting the task dominance, through which the trainee is granted partial or full contribution to the task Shahbazi et al. (2014a) , Motaharifar and Taghirad (2020) , Liu et al. (2015) , Lu et al. (2017) , Liu et al. (2020) . Remarkably, the above approaches require both the trainer and the trainee to completely perform the operation on their haptic devices, and the actual task authority is determined based on the position error between the trainer and the trainee Shahbazi et al. (2014a) , Motaharifar and Taghirad (2020) , Liu et al. (2015) , Lu et al. (2017) , Liu et al. (2020) . This constitutes an important limitation of the above architectures, since the trainer is enforced to be involved in every detail of each operation and even the trivial ones. Notably, the trainer’s obligation to precisely perform every part of the surgical procedure has little compatibility with the trainer’s responsibilities in terms of supervisory assistance and interference. In fact, by grabbing the idea from the conventional training programs of the medical universities, the haptic architecture should be developed in such a manner that the trainer is able to intervene only in order to prevent a complication to the patient due to the trainee’s mistake. The issue of trainer’s supervisory assistance and interference is addressed in Motaharifar et al. (2019a) by adjusting the task authority based on the trainer’s hand force Motaharifar et al. (2019a) . That is, the trainer is able to grant the task authority to the trainer by holding the haptic device loosely or overrule the trainee’s action by grasping the haptic device tightly. Therefore, the active supervision and interference of the trainer is possible without the need of any physical contact between the trainer and the trainee.

Although the above investigations address the essential theoretical aspects regarding dual user haptic systems, the commercialization of collaborative haptic system needs more attention. In the past years, some research groups have developed pilot setups of dual user haptic system with the primal clinical evaluation that have the potential of commercialization. For instance, the ARASH-ASiST system provides training and collaboration of two surgeons and it is preliminary designed for Vitreoretinal eye surgical procedures ARASH-ASiST (2019) . It is expected that the commercialization and widespread utilization of those assistive surgery training tools is considerably beneficial to the health-care systems in order to decrease the physical contact during the COVID-19 pandemic, and to increase the safety and efficiency of training programs during and after this crisis.

Notwithstanding the fact that teleoperated haptic systems provide key benefits for remote training during COVID-19 pandemic, they face a number of challenges that inspire perspectives of future investigations. First, the haptic modality is not sufficient to recreate the full sense of actual presence at the surgical room near an expert surgeon. To overcome this challenge and increase the operators telepresence, the haptic, visual, and auditory components are augmented to achieve a multi–modal telepresence and teleaction architecture in Buss et al. (2010) . The choice of control structure and clinical investigation of the above multi–modal architecture is still an area of active research Shahbazi et al. (2018b) , Caccianiga et al. (2021) . On the other hand, the on-line communication system creates another challenge for the haptic training systems. Notably, owing to the high-bandwith requirement for an appropriate on-line haptic system, the majority of existing haptic architectures in applications such as collaborative teleopertion, handwriting and rehabilitation cover off-line communication Babushkin et al. (2021) . However, due to the complexity, uncertainty, and diversity of the surgical procedures, the online feedback from the expert surgeon is necessary for a safe and efficient training. The advent of 5G technology with faster and more robust communication network may provide enough bandwidth for an effective real-time remote surgery training.

5 Data Driven Scoring

A vital element of a training program is how to evaluate the effectiveness of exercises by introducing a grading system based on participants’ performance. The conventional qualitative skill assessment methods require physical contact between the trainer and the trainee since they are based on direct supervision of the trainer. On the other hand, the systematic approaches for skill assessment are based on collecting the required data using appropriate instruments and analyzing the obtained data, while they eliminate the requirement of physical contact between the trainer and the trainee. Thus, reviewing the systematic data-based methods is of utmost importance, as they can be utilized to reduce the physical contact during the COVID-19 Pandemic. In this section, some of the state of the art methods in surgical skill evaluation are reviewed. Following the trend of similar research in the context of surgical skill evaluation, we categorize the reviewed methods by two criteria. The first is the type of data, and the method uses for grading the participant. The second criterion is the features extraction techniques that are used during the evaluation stage.

Generally speaking, two types of data may be available in Robotic-Assisted surgery; kinematic and video data. Kinematic data is available when a robot or haptic device is involved. The most common form of capturing kinematic information is using IMUs, encoders, force sensors, magnetic field positioning sensors, etc. The video is generally recorded in all minimally invasive surgeries using endoscopy procedures.

Kinematic data are more comfortable to analyze because the dimensionality of kinematic data is lower than video data. Moreover, Kinematic information is superior to video in measuring the actual 3D trajectories, and 3D velocities Zappella et al. (2013) . On the other hand, video data is more convenient to capture since no additional equipment and sophisticated sensors are needed to be attached to the surgical tool. Additionally, video data reflects the contextual semantic information such as the presence or absence of another surgical instrument, which can not be derived from the kinematic data Zappella et al. (2013) . To use the video data effectively, one should overcome some common obstacles like occlusion and clutter. Using multiple cameras, if possible, can greatly assist in this procedure Abdelaal et al. (2020) . In conclusion, it can be said that each type of data has its own merits and limitations, and using kinematic data as well as the video may result in a richer dataset.

Other than the kinematic and video data, another source of information is often disregarded in the literature. The expert surgeon who conducts the training program can evaluate the trainee’s performance and provide useful feedback regarding his/her performance. This type of information, which is at another semantic level compared to the sensory data, is called soft data. The hard and soft information fusion methods can merge the expert’s opinion with the kinematic and video data (hard data) to accomplish a better grading system.

Most surgical skill evaluation methods utilize a feature extraction technique to classify the participant’s skill level after acquiring the data, like expert, intermediate, and novice. The classification problem can be solved by employing some hand-engineered features or features that are automatically extracted from the data. Hand-engineered features are interpretable and easy to obtain. However, hand-engineered features are hard to define. Specifically, defining a feature that represents the skill level regardless of the task is not trivial. Therefore, the states of the art methods are commonly based on automatic feature extraction techniques. An end-to-end deep neural network is used to unfold the input data’s spatial and temporal features and classify the participant in one of the mentioned skill levels in an automated feature extraction procedure. While, Table 1 summarizes the topic of different data types and feature extraction techniques, we are going to cover some of the reviewed methods in the next sections.

Summery of different sources of data and different feature extraction techniques.

The most convenient hand-engineered features are those introduced by descriptive statistics Anh et al. (2020) . In a skill rating system proposed by Brown et al. (2016) , eight values of mean, standard deviation, minimum, maximum, range, root-mean-square (RMS), total sum-of-squares (TSS), and time integral of force and acceleration signals are calculated. Together with time features like task completion time, these values are used as inputs for a random forest classifier to rate the peg transfer score of 38 different participants. In Javaux et al. (2018) , metrics like mean/maximum velocity and acceleration, tool path length, depth perception, maximum and integral of planar/vertical force, and task completion time are considered as a baseline for skill assessment Lefor et al. (2020) . Another commonly used method in the literature is to use statistical tests such as Mann-Whitney test Moody et al. (2008) , Kruskal–Wallis test Javaux et al. (2018) , Pearson or Spearman correlation Zendejas et al. (2017) , etc. These tests are utilized to classify the participants directly Moody et al. (2008) or automatically calculate some of the well-known skill assessment scores like GOALS and FLS Zendejas et al. (2017) .

Since many surgical tasks are periodic by nature, the data frequency domain analysis proves to be effective Zia et al. (2015) . For periodic functions like knot tying and suturing Zia et al. (2015) suggests that transforming the data into time series and performing a Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT) on the data extracts features, will assist the skill level classification task. The results show that such an approach outperforms many machine-learning-based methods like Bag of Words (BoW) and Sequential Motion Texture (SMT). In another work by the same author, symbolic features, texture features, and frequency features are employed for the classification. A Sequential Forward Selection (SFS) algorithm is then utilized to reduce the number of elements in the feature vector and remove the irrelevant data Zia et al. (2016) . Hojati et al. (2019) suggests that since Discrete Wavelet Transform (DWT) is superior to DFT and DCT in a sense that it offers simultaneous localization in time and frequency domain, DWT is a better choice for feature extraction in surgical skill assessment tasks.

As it is mentioned before, hand-engineered features are task-specific. For example, the frequency domain analysis discussed in the previous section is only viable when the task is periodic. Otherwise, the frequency domain features should be concatenated with other features. Moreover, perceiving the correct features that reflect participants’ skill levels in different surgical tasks requires an intensive knowledge of the field. As a result, developing a method in which the essential features are identified automatically is advantageous.

With the recent success of Convolutional Neural Networks (CNN) in classification problems like image classification, action recognition, and segmentation, it is safe to assume that CNN can be used in skill assessment problems. However, unlike image classification, improvement brought by end-to-end deep CNN remains limited compared to hand-engineered features for action recognition Wang et al. (2018) . Similarly, using conventional CNN does not contribute too much to the result in surgical skill evaluation problems. For example, Fawaz et al. (2018) proposed a CNN-based approach for dry-lab skill evaluation tasks such as needle passing, suturing, and knot-tying. However, a hand-engineered-based method with a set of features introduced as holistic features (SMT, DFT, DCT, and Approximate Entropy (ApEn)) suggested by Zia and Essa (2018) reaches the same accuracy as the CNN-based method in the needle passing and suturing tasks and outperforms the CNN-based method in the knot-tying task.

Wang et al. (2018) suggests that conventional CNN falls short compared to traditional hand-crafted feature extraction techniques because it only considers the appearances (spatial features) and ignores the data’s temporal dynamics. In Wang and Fey (2018) , a parallel deep learning architecture is proposed to recognize the surgical training activity and assess trainee expertise. A Gated recurrent unit (GRU) is used for temporal feature extraction, and a CNN network is used to extract the spatial features. The overall accuracy calculated for the needle passing, suturing, and knot tying tasks is 96% using video data. The problem of extracting spatiotemporal features is addressed with 3D ConvNets in Funke et al. (2019) . In this method, inflated convolutional layers are responsible for processing the video snippets and unfolding the classifier’s input data.

To the best of our knowledge, all of the proposed methods in the literature have used single classifier techniques in their work. However, methods like classifier fusion have proved to be useful in the case of medical-related data. In Kazemian et al. (2005) an OWA-based fusion technique is used to combine multiple classifiers and improve the accuracy. For a more advanced classifier fusion technique, one can refer to the proposed method in Kazemian et al. (2010) where more advanced methods such as Dempster’s Rule of Combination (DCR) and Choquet integral are compared with more basic techniques. Activity recognition and movement classification is another efficient way to calculate metrics representing the surgical skill automatically Khan et al. (2020) . Moreover, instrument detection in a video and drawing centroid based on the orientation and movement of the instruments can reflect the focus and ability to plan moves in a surgeon. Utilizing these centroids and calculating the radius, distance, and relative orientation can aid with the classification based on skill level Lavanchy et al. (2021) .

In conclusion, the general framework illustrated in Figure 5 can summarize the reviewed techniques. The input data, either kinematic and video, is fed to a feature extraction block. A fusion block Naeini et al. (2014) can enrich the semantic of the data using expert surgeon feedback. Finally, a regression technique or a classifier can be employed to calculate a participant’s score based on his/her skill level or represent a label following his/her performance.

A general framework for surgical skill assessment.

6 Machine Vision

The introduction of new hardware capable of running deep learning methods with acceptable performance led artificial intelligence to play a more significant role in any intelligent system Han (2017) . It is undeniable that there is a huge potential in employing deep learning methods in a wide range of various applications Weng et al. (2019) , Antoniades et al. (2016) , Lotfi et al. (2018) , Lotfi et al. (2020) . In particular, utilizing a camera along with a deep learning algorithm, machines may precisely identify and classify objects by which either performing a proper reaction or monitoring a process may be realized automatically. For instance, considering a person in a coma, any tiny reaction is crucial to be detected, and since it is not possible to assign a person for each patient, a camera can solve the problem satisfactorily. Regarding the COVID-19 pandemic situation, artificial intelligence may be used to reduce both physical interactions and the risk of a probable infection especially when it comes to a medical training process. Considering eye surgery as an instance, not only should the novice surgeon closely track how the expert performs but also the expert should be notified of a probable mistake made by the novice surgeon during surgery. In this regard, utilizing computer vision approaches as an interface, the level of close interactions may be minimized effectively. To clarify, during the training process, the computer vision algorithm may act as both the novice surgeon looking over the expert’s hand and the expert monitoring and evaluating how the novice performs. This kind of application in a medical training process may easily extend to other cases. By this means, the demand for keeping in close contact is met properly.

Not needing a special preprocessing, deep convolutional neural networks (CNNs) are commonly used for classifying images into various distinct categories. For instance, in medical images, this may include probable lesions Farooq et al. (2017) , Chitra and Seenivasagam (2013) . Moreover, they can detect intended objects in the images which can be adopted not only to find and localize specific features but also to recognize them if needed. Since most of the medical training tasks require on-line and long-term monitoring, by utilizing a camera along with these powerful approaches, an expert may always keep an eye on the task assigned to a trainee. Besides, methods based on CNNs are capable of being implemented on graphics processor units (GPUs) to process the images with an applicable performance in terms of both speed and accuracy Chetlur et al. (2014) , Bahrampour et al. (2015) . This will reduce the probable latency and makes it possible for the trainer to be notified on time and correct the trainee remotely.

There are numerous researches carried out in the field of image processing based on CNNs. These methods are mainly divided into two single-stage and two-stage detectors. The former is known to be fast while the latter results in higher accuracy. In Figure 6 the difference between a two-stage and a single-stage detector is illustrated. Considering single-stage detectors and starting with the LeCun et al. (1998) as one of the earliest networks, plenty of different approaches have been presented in the literature among which single-shot multi-box detector (SSD) Liu et al. (2016) , RetinaNet Lin et al. (2017) , and you only look once (YOLO) Redmon and Farhadi (2018) may be counted as nominated ones. Some of these approaches have been proposed with several structures including simpler and more complex structures to be employed depending on whether the speed is of high importance or accuracy. Mainly, training and the test are two phases when utilizing these methods. While it is crucial to define a proper optimization problem in the first phase, it is indispensable to implement the trained CNN optimally. Coming up with various solutions, methods like Krizhevsky et al. (2012) , Simonyan and Zisserman (2015) , Szegedy et al. (2015) , and Szegedy et al. (2016) suggest utilizing specific CNN models to obtain better outcomes. On the other hand, to further improve the accuracy, in two-stage detectors like Girshick et al. (2014) , it is suggested to first determine a region of interest (ROI) then identify probable objects in the related area. As a representative, Uijlings et al. (2013) , which is known as selective search, is designed to propose 2 k proposal regions, while a classifier may be employed for the later stage. Dealing with some challenging problems in these detectors, He et al. (2015) , Girshick (2015) , and Ren et al. (2015) are proposed to enhance the results in terms of both accuracy and speed.

Example of two-stage and single-stage detectors Kathuria (2021) . (A) Two-stage detector (RCNN). (B) Single-stage detector (YOLO).

To put all in a nutshell, when dealing with critical situations such as the current COVID-19 epidemic, it is highly recommended to employ artificial intelligence techniques in image processing namely deep CNNs for medical training tasks. By this means, neither is a close physical interaction between the expert and novice necessary, nor the quality of the training is reduced adversely due to the limitations. In fact, the computer vision approach acts as an interface making it possible both to learn from the expert and to evaluate the novice, remotely.

7 Conclusion and Future Prospects

The faculty members and the students of the medical universities are classified in the high-risk category due to the potential exposure to coronavirus through direct contact and aerosol-generating procedures. As a result, many medical schools have suspended their clinical programs or implemented social distancing in their laboratory practices. Furthermore, the current fight against the COVID-19 virus have used nearly all capacity of health-care systems, and some less urgent and less emergent medical services including the education issues are limited or even paused. Therefore, unless some assistive training tools are utilized to support the educational procedures, the training efficiency of medical universities will be reduced and it have future consequences for the world health-care system.

Practicing medical tasks with current lock-down policies can be solved utilizing state of the art techniques in haptics, virtual reality, machine vision, and machine learning. Notably, utilization of the above technologies in medical education has been researched actively within the past years in order to increase the safety and efficiency of the surgical training procedures. Nowadays, another motivation is created for those assistive technologies owing to the COVID-19 pandemic. In this paper, the existing assistive technologies for medical training are reviewed in the COVID-19 context and a summary of them is presented in Table 2 .

The main tools and approaches that help to reduce physical contact in medical training.

It is reviewed that a surgical simulator system including a VR/AR based graphical interface and a haptic interface is able to provide the circumstances of actual surgical operation for the medical students, without the necessity of attending the hospital environments. Furthermore, through augmenting the system with another haptic console and having a dual user haptic system, the opportunity of collaboration with and receiving guidance cues from an expert surgeon in a systematic manner is given to the trainees. In contrast to the traditional collaboration methodologies, the haptic-based collaboration does not require the physical contact between the involved people and the risk of infection is reduced. Assessment of the expertise level of the medical students is another element of each training program. The necessity of reducing physical contact during the COVID-19 pandemic have also affected the skill assessment methodologies as the traditional ways of skill assessment are based on direct observation by a trainer. In contrast, data-based analysis may be utilized as a systematic approach for skill assessment without any need of physical contact. In this paper, some of the ongoing methods in surgical skill evaluation have been reviewed.

Biomedical engineering technology has progressed by leaps and bounds during the past several decades and advancements in remote diagnostics and remote treatment have been considered as a leading edge in this field. For instance, the tele-surgery robotic-assisted da Vinci system have received a great deal of attention in the healthcare marketplace with more than 5 million surgeries in the last 2 decades DaVinci (2021) . However, the rate of advancement in medical training, which usually follows traditional methods, has been considerably less than the other aspects of medical field, and modern training technologies have received fewer attention during the past several decades. While remote training and remote skill assessment technologies make relatively lower risk to the patient than remote diagnostics and remote treatment, the reason behind fewer attention to the former is the lack of sufficient motivations. It is hoped that the motivations created for those advanced medical training methods during the COVID-19 crisis are strong enough to continuously increase their utilization among the medical universities. Although wide utilization of those technologies needs a considerable extent of time, effort, and investment, immediate and emergent decisions and actions are required to widely utilize those potential techniques. Notably, all of the presented approaches and techniques are targeted to be utilized in the normal situations without any pandemic in order to provide safer and more efficient medical training. Therefore, even after the world recovers from this crisis, these techniques, tools, and approaches deserve more attention, recognition, investigation, and utilization. There needs to be a global awareness among the medical universities that haptic technology and virtual reality integrated with machine learning and machine vision provides an excellent systematic medical training apparatus that ensures the requirements of health-care systems to enhance the safety, efficiency, and robustness of medical training.

1 Less invasive stabilization system

2 Orthoganthic surgery

Data Availability Statement

Author contributions.

Conceptualization, HT, SFM, and MM; original draft preparation MM, AN, PA, AI, and FL; review and editing, HT, SFM, BM, and AL.

This work was supported in part by the National Institute for Medical Research Development (NIMAD) under Grant No. 942314, in part by Tehran University of Medical Sciences, Tehran, Iran under Grant No. 35949-43-01-97, and in part by K. N. Toosi University of Technology, Tehran, Iran Research Grant.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

• Abdelaal A. E., Avinash A., Kalia M., Hager G. D., Salcudean S. E. (2020). A Multi-Camera, Multi-View System for Training and Skill Assessment for Robot-Assisted Surgery . Int. J. CARS 15 , 1369–1377. 10.1007/s11548-020-02176-1 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Akbari M., Carriere J., Meyer T., Sloboda R., Husain S., Usmani N., et al. (2021). Robotic Ultrasound Scanning with Real-Time Image-Based Force Adjustment: Quick Response for Enabling Physical Distancing during the Covid-19 Pandemic . Front. Robotics AI 8 , 62. 10.3389/frobt.2021.645424 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Anh N. X., Nataraja R. M., Chauhan S. (2020). Towards Near Real-Time Assessment of Surgical Skills: A Comparison of Feature Extraction Techniques . Comput. Methods Programs Biomed. 187 , 105234. 10.1016/j.cmpb.2019.105234 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Antoniades A., Spyrou L., Took C. C., Sanei S. (2016). “ Deep Learning for Epileptic Intracranial Eeg Data ,” in 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP), Vietri sul Mare, Salerno, Italy, September 13–16, 2016 (IEEE; ), 1–6. 10.1109/mlsp.2016.7738824 [ CrossRef ] [ Google Scholar ]
• ARASH-ASiST (2019). Dataset. Aras Haptics: A System for EYE Surgery Training . Available at: https://aras.kntu.ac.ir/arash-asist// . (Accessed 08 05, 2020).
• Babushkin V., Jamil M. H., Park W., Eid M. (2021). Sensorimotor Skill Communication: A Literature Review . IEEE Access 9 , 75132–75149. 10.1109/access.2021.3081449 [ CrossRef ] [ Google Scholar ]
• Bahrampour S., Ramakrishnan N., Schott L., Shah M. (2015). Comparative Study of Caffe, Neon, Theano, and Torch for Deep Learning . CoRR . arXiv:1511.06435. Available at: http://arxiv.org/abs/1511.06435 . [ Google Scholar ]
• Bartlett J. D., Lawrence J. E., Yan M., Guevel B., Stewart M. E., Audenaert E., et al.fnm (2020). The Learning Curves of a Validated Virtual Reality Hip Arthroscopy Simulator . Arch. Orthopaedic Trauma Surg. 140 ( 6 ), 761–767. 10.1007/s00402-020-03352-3 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Basdogan C., De S., Kim J., Muniyandi M., Kim H., Srinivasan M. A. (2004). Haptics in Minimally Invasive Surgical Simulation and Training . IEEE Comput. Graphics Appl. 24 , 56–64. 10.1109/mcg.2004.1274062 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Brown J. D., O’Brien C. E., Leung S. C., Dumon K. R., Lee D. I., Kuchenbecker K. J. (2016). Using Contact Forces and Robot Arm Accelerations to Automatically Rate Surgeon Skill at Peg Transfer . IEEE Trans. Biomed. Eng. 64 , 2263–2275. 10.1109/TBME.2016.2634861 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Buss M., Peer A., Schauß T., Stefanov N., Unterhinninghofen U., Behrendt S., et al. (2010). Development of a Multi-Modal Multi-User Telepresence and Teleaction System . Int. J. Robot. Res. 29 , 1298–1316. 10.1177/0278364909351756 [ CrossRef ] [ Google Scholar ]
• Caccianiga G., Mariani A., de Paratesi C. G., Menciassi A., De Momi E. (2021). Multi-Sensory Guidance and Feedback for Simulation-Based Training in Robot Assisted Surgery: A Preliminary Comparison of Visual, Haptic, and Visuo-Haptic . IEEE Robot. Autom. Lett. 6 , 3801–3808. 10.1109/lra.2021.3063967 [ CrossRef ] [ Google Scholar ]
• Cecil J., Gupta A., Pirela-Cruz M. (2018). An Advanced Simulator for Orthopedic Surgical Training . Int. J. Comput. Assist. Radiol. Surg. 13 , 305–319. 10.1007/s11548-017-1688-0 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Chetlur S., Woolley C., Vandermersch P., Cohen J., Tran J., Catanzaro B., et al. (2014). cuDNN: Efficient Primitives for Deep Learning . CoRR . arXiv: 1410.0759. Available at: http://arxiv.org/abs/1410.0759 . [ Google Scholar ]
• Chitra R., Seenivasagam V. (2013). Heart Disease Prediction System Using Supervised Learning Classifier . Bonfring Int. J. Softw. Eng. Soft Comput. 3 , 01–07. 10.9756/bijsesc.4336 [ CrossRef ] [ Google Scholar ]
• DaVinci (2021). Dataset. Enabling Surgical Care to Get Patients Back to what Matters . Available at: https://www.intuitive.com/en-us/products-and-services/da-vinci . (Accessed 202107 04.
• Desselle M. R., Brown R. A., James A. R., Midwinter M. J., Powell S. K., Woodruff M. A. (2020). Augmented and Virtual Reality in Surgery . Comput. Sci. Eng. 22 , 18–26. 10.1109/mcse.2020.2972822 [ CrossRef ] [ Google Scholar ]
• Farooq A., Anwar S., Awais M., Rehman S. (2017). “ A Deep Cnn Based Multi-Class Classification of Alzheimer’s Disease Using Mri ,” in 2017 IEEE International Conference on Imaging systems and techniques (IST) (IEEE), Beijing, China, October 18–20, 2017, 1–6. 10.1109/ist.2017.8261460 [ CrossRef ] [ Google Scholar ]
• Fawaz H. I., Forestier G., Weber J., Idoumghar L., Muller P.-A. (2018). “ Evaluating Surgical Skills from Kinematic Data Using Convolutional Neural Networks ,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, Granada, Spain, September 16–20, 2018 (Springer; ), 214–221. 10.1007/978-3-030-00937-3_25 [ CrossRef ] [ Google Scholar ]
• Feizi N., Tavakoli M., Patel R. V., Atashzar S. F. (2021). Robotics and Ai for Teleoperation, Tele-Assessment, and Tele-Training for Surgery in the Era of Covid-19: Existing Challenges, and Future Vision . Front. Robot. AI 8 , 610677. 10.3389/frobt.2021.610677 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Funke I., Mees S. T., Weitz J., Speidel S. (2019). Video-based Surgical Skill Assessment Using 3d Convolutional Neural Networks . Int. J. Comput. Assist. Radiol. Surg. 14 , 1217–1225. 10.1007/s11548-019-01995-1 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Girshick R., Donahue J., Darrell T., Malik J. (2014). “ Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation ,” in Proceedings of the IEEE conference on computer vision and pattern recognition, Columbus, OH, June 23–28, 2014, 580–587. 10.1109/cvpr.2014.81 [ CrossRef ] [ Google Scholar ]
• Girshick R. (2015). “ Fast R-Cnn ,” in Proceedings of the IEEE international conference on computer vision, Boston, MA, June 7–12, 2015, 1440–1448. 10.1109/iccv.2015.169 [ CrossRef ] [ Google Scholar ]
• Han S. (2017). Efficient Methods and Hardware for Deep Learning . Stanford University. [ Google Scholar ]
• He K., Zhang X., Ren S., Sun J. (2015). Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition . IEEE Trans. pattern Anal. machine intell. 37 , 1904–1916. 10.1109/tpami.2015.2389824 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Hodges C., Moore S., Lockee B., Trust T., Bond A. (2020). The Difference between Emergency Remote Teaching and Online Learning. Boulder, CO . Educause Rev. 27 ( 1 ), 1–9. [ Google Scholar ]
• Hojati N., Motaharifar M., Taghirad H., Malekzadeh A. (2019). “ Skill Assessment Using Kinematic Signatures: Geomagic Touch Haptic Device ,” in 2019 7th International Conference on Robotics and Mechatronics (ICRoM), Tehran, Iran, November 20–22, 2019 (IEEE; ), 186–191. 10.1109/icrom48714.2019.9071892 [ CrossRef ] [ Google Scholar ]
• Iwashita Y., Hibi T., Ohyama T., Umezawa A., Takada T., Strasberg S. M., et al. (2017). Delphi Consensus on Bile Duct Injuries during Laparoscopic Cholecystectomy: an Evolutionary Cul-De-Sac or the Birth Pangs of a New Technical Framework? J. Hepato-Biliary-Pancreatic Sci. 24 , 591–602. 10.1002/jhbp.503 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Javaux A., Joyeux L., Deprest J., Denis K., Vander Poorten E. (2018). Motion-based Skill Analysis in a Fetoscopic Spina-Bifida Repair Training Model . In CRAS , Date: 2018/09/10-2018/09/11, London, United Kingdom. [ Google Scholar ]
• Jonas J. B., Rabethge S., Bender H.-J. (2003). Computer-assisted Training System for Pars Plana Vitrectomy . Acta Ophthalmol. Scand. 81 , 600–604. 10.1046/j.1395-3907.2003.0078.x [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Kathuria A. (2021). Dataset. Tutorial on Implementing yolo V3 from Scratch in Pytorch . Available at: https://blog.paperspace.com/how-to-implement-a-yolo-object-detector-in-pytorch/ . (Accessed on 01 07, 2021). [ Google Scholar ]
• Kazemian M., Moshiri B., Nikbakht H., Lucas C. (2005). “ Protein Secondary Structure Classifiers Fusion Using Owa ,” in International Symposium on Biological and Medical Data Analysis, Aveiro, Portugal, November 10–11, 2005 (Springer; ), 338–345. 10.1007/11573067_34 [ CrossRef ] [ Google Scholar ]
• Kazemian M., Moshiri B., Palade V., Nikbakht H., Lucas C. (2010). Using Classifier Fusion Techniques for Protein Secondary Structure Prediction . Int. J. Comput. Intell. Bioinf. Syst. Biol. 1 , 418–434. 10.1504/ijcibsb.2010.038225 [ CrossRef ] [ Google Scholar ]
• Khademian B., Hashtrudi-Zaad K. (2012). Dual-user Teleoperation Systems: New Multilateral Shared Control Architecture and Kinesthetic Performance Measures . Ieee/asme Trans. Mechatron. 17 , 895–906. 10.1109/tmech.2011.2141673 [ CrossRef ] [ Google Scholar ]
• Khan A., Mellor S., King R., Janko B., Harwin W., Sherratt R. S., et al. (2020). Generalized and Efficient Skill Assessment from Imu Data with Applications in Gymnastics and Medical Training . New York, NY, ACM Trans. Comput. Healthc. 2 ( 1 ), 1–21. [ Google Scholar ]
• Khanna R. C., Honavar S. G., Metla A. L., Bhattacharya A., Maulik P. K. (2020). Psychological Impact of Covid-19 on Ophthalmologists-In-Training and Practising Ophthalmologists in india . Indian J. Ophthalmol. 68 , 994. 10.4103/ijo.ijo_1458_20 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Kotsis S. V., Chung K. C. (2013). Application of See One, Do One, Teach One Concept in Surgical Training . Plast. Reconstr. Surg. 131 , 1194. 10.1097/prs.0b013e318287a0b3 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Krizhevsky A., Sutskever I., Hinton G. E. (2012). “ Imagenet Classification with Deep Convolutional Neural Networks ,” in Advances in neural information processing systems, Lake Tahoe, NV, December 3–6, 2012, 1097–1105. [ Google Scholar ]
• kumar Renganayagalu S., Mallam S. C., Nazir S. (2021). Effectiveness of Vr Head Mounted Displays in Professional Training: A Systematic Review . Technol. Knowl. Learn . (Springer), 1–43. 10.1007/s10758-020-09489-9 [ CrossRef ] [ Google Scholar ]
• Lavanchy J. L., Zindel J., Kirtac K., Twick I., Hosgor E., Candinas D., et al. (2021). Automation of Surgical Skill Assessment Using a Three-Stage Machine Learning Algorithm . Scientific Rep. 11 , 1–9. 10.1038/s41598-021-88175-x [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• LeCun Y., Bottou L., Bengio Y., Haffner P. (1998). Gradient-based Learning Applied to Document Recognition . Proc. IEEE 86 , 2278–2324. 10.1109/5.726791 [ CrossRef ] [ Google Scholar ]
• Lefor A. K., Harada K., Dosis A., Mitsuishi M. (2020). Motion Analysis of the Jhu-Isi Gesture and Skill Assessment Working Set Using Robotics Video and Motion Assessment Software . Int. J. Comput. Assist. Radiol. Surg. 15 , 2017–2025. 10.1007/s11548-020-02259-z [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Lin T.-Y., Goyal P., Girshick R., He K., Dollár P. (2017). “ Focal Loss for Dense Object Detection ,” in Proceedings of the IEEE international conference on computer vision, 2980–2988. 10.1109/iccv.2017.324 [ CrossRef ] [ Google Scholar ]
• Liu F., Lelevé A., Eberard D., Redarce T. (2015). “ A Dual-User Teleoperation System with Online Authority Adjustment for Haptic Training ,” in 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy, August 25–29, 2015, 1168–1171. 10.1109/embc.2015.7318574 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Liu W., Anguelov D., Erhan D., Szegedy C., Reed S., Fu C.-Y., et al. (2016). “ Ssd: Single Shot Multibox Detector ,” in European conference on computer vision (Springer; ), 21–37. 10.1007/978-3-319-46448-0_2 [ CrossRef ] [ Google Scholar ]
• Liu F., Licona A. R., Lelevé A., Eberard D., Pham M. T., Redarce T. (2020). An Energy-Based Approach for N-Dof Passive Dual-User Haptic Training Systems . Robotica 38 , 1155–1175. 10.1017/s0263574719001309 [ CrossRef ] [ Google Scholar ]
• Lohre R., Bois A. J., Athwal G. S., Goel D. P. (2020). Improved Complex Skill Acquisition by Immersive Virtual Reality Training: a Randomized Controlled Trial . JBJS 102 , e26. 10.2106/jbjs.19.00982 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Lotfi F., Ajallooeian V., Taghirad H. D. (2018). “ Robust Object Tracking Based on Recurrent Neural Networks ,” in 2018 6th RSI International Conference on Robotics and Mechatronics (IcRoM), 507–511. 10.1109/icrom.2018.8657608 [ CrossRef ] [ Google Scholar ]
• Lotfi F., Hasani P., Faraji F., Motaharifar M., Taghirad H., Mohammadi S. (2020). “ Surgical Instrument Tracking for Vitreo-Retinal Eye Surgical Procedures Using Aras-Eye Dataset ,” in 2020 28th Iranian Conference on Electrical Engineering (ICEE) (IEEE; ), 1–6. 10.1109/icee50131.2020.9260679 [ CrossRef ] [ Google Scholar ]
• Lu Z., Huang P., Dai P., Liu Z., Meng Z. (2017). Enhanced Transparency Dual-User Shared Control Teleoperation Architecture with Multiple Adaptive Dominance Factors . Int. J. Control. Autom. Syst. 15 , 2301–2312. 10.1007/s12555-016-0467-y [ CrossRef ] [ Google Scholar ]
• Medellin-Castillo H. I., Zaragoza-Siqueiros J., Govea-Valladares E. H., de la Garza-Camargo H., Lim T., Ritchie J. M. (2020). Haptic-enabled Virtual Training in Orthognathic Surgery . Virtual Reality 25 , 53–67. 10.1007/s10055-020-00438-6 [ CrossRef ] [ Google Scholar ]
• Moody L., Waterworth A., McCarthy A. D., Harley P. J., Smallwood R. H. (2008). The Feasibility of a Mixed Reality Surgical Training Environment . Virtual Reality 12 , 77–86. 10.1007/s10055-007-0080-8 [ CrossRef ] [ Google Scholar ]
• Motaharifar M., Taghirad H. D. (2020). A Force Reflection Robust Control Scheme with Online Authority Adjustment for Dual User Haptic System . Mech. Syst. Signal Process. 135 , 106368. 10.1016/j.ymssp.2019.106368 [ CrossRef ] [ Google Scholar ]
• Motaharifar M., Bataleblu A., Taghirad H. (2016). “ Adaptive Control of Dual User Teleoperation with Time Delay and Dynamic Uncertainty ,” in 2016 24th Iranian conference on electrical engineering (ICEE), Shiraz, Iran, May 10–12, 2016 (IEEE; ), 1318–1323. 10.1109/iraniancee.2016.7585725 [ CrossRef ] [ Google Scholar ]
• Motaharifar M., Taghirad H. D., Hashtrudi-Zaad K., Mohammadi S. F. (2019a). Control of Dual-User Haptic Training System with Online Authority Adjustment: An Observer-Based Adaptive Robust Scheme . IEEE Trans. Control. Syst. Technol. 28 ( 6 ), 2404–2415. 10.1109/tcst.2019.2946943 [ CrossRef ] [ Google Scholar ]
• Motaharifar M., Taghirad H. D., Hashtrudi-Zaad K., Mohammadi S.-F. (2019b). Control Synthesis and ISS Stability Analysis of Dual-User Haptic Training System Based on S-Shaped Function . IEEE/ASME Trans. Mechatron. 24 ( 4 ), 1553–1564. 10.1109/tmech.2019.2917448 [ CrossRef ] [ Google Scholar ]
• Naeini M. P., Moshiri B., Araabi B. N., Sadeghi M. (2014). Learning by Abstraction: Hierarchical Classification Model Using Evidential Theoretic Approach and Bayesian Ensemble Model . Neurocomputing 130 , 73–82. 10.1016/j.neucom.2012.03.041 [ CrossRef ] [ Google Scholar ]
• Nudehi S. S., Mukherjee R., Ghodoussi M. (2005). A Shared-Control Approach to Haptic Interface Design for Minimally Invasive Telesurgical Training . IEEE Trans. Control. Syst. Technol. 13 , 588–592. 10.1109/tcst.2004.843131 [ CrossRef ] [ Google Scholar ]
• Redmon J., Farhadi A. (2018). Yolov3: An Incremental Improvement . CoRR abs/1804.02767 . Available at: http://arxiv.org/abs/1804.02767 . [ Google Scholar ]
• Ren S., He K., Girshick R., Sun J. (2015). “ Faster R-Cnn: Towards Real-Time Object Detection with Region Proposal Networks ,” in Advances in neural information processing systems, Montreal, Quebec, Canada, December 7–12, 2015, 91–99. [ Google Scholar ]
• Shahbazi M., Atashzar S. F., Talebi H. A., Patel R. V. (2014a). An Expertise-Oriented Training Framework for Robotics-Assisted Surgery . Proc. IEEE Int. Conf. Rob. Autom. , 5902–5907. 10.1109/icra.2014.6907728 [ CrossRef ] [ Google Scholar ]
• Shahbazi M., Atashzar S. F., Talebi H. A., Patel R. V. (2014b). Novel Cooperative Teleoperation Framework: Multi-Master/single-Slave System . IEEE/ASME Trans. Mechatron. 20 , 1668–1679. 10.1109/tmech.2014.2347034 [ CrossRef ] [ Google Scholar ]
• Shahbazi M., Atashzar S. F., Patel R. V. (2018a). A Systematic Review of Multilateral Teleoperation Systems . IEEE Trans. Haptics 11 , 338–356. 10.1109/toh.2018.2818134 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Shahbazi M., Atashzar S. F., Ward C., Talebi H. A., Patel R. V. (2018b). Multimodal Sensorimotor Integration for Expert-In-The-Loop Telerobotic Surgical Training . IEEE Trans. Robot. 34 , 1549–1564. 10.1109/tro.2018.2861916 [ CrossRef ] [ Google Scholar ]
• Sharma D., Bhaskar S. (2020). Addressing the Covid-19 burden on Medical Education and Training: the Role of Telemedicine and Tele-Education during and beyond the Pandemic . Front. Public Health 8 , 838. 10.3389/fpubh.2020.589669 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Si W.-X., Liao X.-Y., Qian Y.-L., Sun H.-T., Chen X.-D., Wang Q., et al. (2019). Assessing Performance of Augmented Reality-Based Neurosurgical Training . Vis. Comput. Industry, Biomed. Art 2 , 6. 10.1186/s42492-019-0015-8 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Simonyan K., Zisserman A. (2015). “ Very Deep Convolutional Networks for Large-Scale Image Recognition ,” in International Conference on Learning Representations, San Diego, CA, May 7–9, 2015. [ Google Scholar ]
• Singh R. P., Javaid M., Kataria R., Tyagi M., Haleem A., Suman R. (2020). Significant Applications of Virtual Reality for Covid-19 Pandemic . Diabetes Metab. Syndr. Clin. Res. Rev. 14 ( 4 ), 661–664. 10.1016/j.dsx.2020.05.011 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Spera C., Somerville A., Caniff S., Keenan J., Fischer M. D. (2020). Virtual Reality Haptic Surgical Simulation for Sub-retinal Administration of an Ocular Gene Therapy . Invest. Ophthalmol. Vis. Sci. 61 , 4503. 10.1039/d0ay90130j [ CrossRef ] [ Google Scholar ]
• Stone S., Bernstein M. (2007). Prospective Error Recording in Surgery: an Analysis of 1108 Elective Neurosurgical Cases . Neurosurgery 60 , 1075–1082. 10.1227/01.neu.0000255466.22387.15 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Szegedy C., Liu W., Jia Y., Sermanet P., Reed S., Anguelov D., et al. (2015). “ Going Deeper with Convolutions ,” in Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, MA, June 7–12, 2015, 1–9. 10.1109/cvpr.2015.7298594 [ CrossRef ] [ Google Scholar ]
• Szegedy C., Vanhoucke V., Ioffe S., Shlens J., Wojna Z. (2016). “ Rethinking the Inception Architecture for Computer Vision ,” in Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, June 27–30, 2016, 2818–2826. 10.1109/cvpr.2016.308 [ CrossRef ] [ Google Scholar ]
• Tahmasebi A. M., Hashtrudi-Zaad K., Thompson D., Abolmaesumi P. (2008). A Framework for the Design of a Novel Haptic-Based Medical Training Simulator . IEEE Trans. Inf. Technol. Biomed. 12 , 658–666. 10.1109/titb.2008.926496 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Tavakoli M., Carriere J., Torabi A. (2020). Robotics, Smart Wearable Technologies, and Autonomous Intelligent Systems for Healthcare during the Covid-19 Pandemic: An Analysis of the State of the Art and Future Vision . Adv. Intell. Syst. 2 , 2000071. 10.1002/aisy.202000071 [ CrossRef ] [ Google Scholar ]
• Uijlings J. R., Van De Sande K. E., Gevers T., Smeulders A. W. (2013). Selective Search for Object Recognition . Int. J. Comput. Vis. 104 , 154–171. 10.1007/s11263-013-0620-5 [ CrossRef ] [ Google Scholar ]
• Wang Z., Fey A. M. (2018). “ Satr-dl: Improving Surgical Skill Assessment and Task Recognition in Robot-Assisted Surgery with Deep Neural Networks ,” in In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, July 17–21, 2018 (IEEE; ), 1793–1796. 10.1109/EMBC.2018.8512575 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Wang D., Shi Y., Liu S., Zhang Y., Xiao J. (2014). Haptic Simulation of Organ Deformation and Hybrid Contacts in Dental Operations . IEEE Trans. Haptics 7 , 48–60. 10.1109/toh.2014.2304734 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Wang L., Xiong Y., Wang Z., Qiao Y., Lin D., Tang X., et al. (2018). Temporal Segment Networks for Action Recognition in Videos . IEEE Trans. pattern Anal. machine intell. 41 , 2740–2755. 10.1109/TPAMI.2018.2868668 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Weng J., Weng J., Zhang J., Li M., Zhang Y., Luo W. (2019). “ Deepchain: Auditable and Privacy-Preserving Deep Learning with Blockchain-Based Incentive ,” in IEEE Transactions on Dependable and Secure Computing. 10.1109/tdsc.2019.2952332 [ CrossRef ] [ Google Scholar ]
• Yari S. S., Jandhyala C. K., Sharareh B., Athiviraham A., Shybut T. B. (2018). Efficacy of a Virtual Arthroscopic Simulator for Orthopaedic Surgery Residents by Year in Training . Orthopaedic J. Sports Med. 6 , 2325967118810176. 10.1177/2325967118810176 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Zappella L., Béjar B., Hager G., Vidal R. (2013). Surgical Gesture Classification from Video and Kinematic Data . Med. image Anal. 17 , 732–745. 10.1016/j.media.2013.04.007 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Zendejas B., Jakub J. W., Terando A. M., Sarnaik A., Ariyan C. E., Faries M. B., et al. (2017). Laparoscopic Skill Assessment of Practicing Surgeons Prior to Enrollment in a Surgical Trial of a New Laparoscopic Procedure . Surg. Endosc. 31 , 3313–3319. 10.1007/s00464-016-5364-1 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Zhang W., Wang Y., Yang L., Wang C. (2020). Suspending Classes Without Stopping Learning: China’s Education Emergency Management Policy in the Covid-19 Outbreak . Multidisciplinary digital publishing institute, J. Risk Finan. Manag. 13 ( 3 ), 1–6. [ Google Scholar ]
• Zia A., Essa I. (2018). Automated Surgical Skill Assessment in Rmis Training . Int. J. Comput. Assist. Radiol. Surg. 13 , 731–739. 10.1007/s11548-018-1735-5 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Zia A., Sharma Y., Bettadapura V., Sarin E. L., Clements M. A., Essa I. (2015). “ Automated Assessment of Surgical Skills Using Frequency Analysis ,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, October 5–9, 2015 (Springer; ), 430–438. 10.1007/978-3-319-24553-9_53 [ CrossRef ] [ Google Scholar ]
• Zia A., Sharma Y., Bettadapura V., Sarin E. L., Ploetz T., Clements M. A., et al. (2016). Automated Video-Based Assessment of Surgical Skills for Training and Evaluation in Medical Schools . Int. J. Comput. Assist. Radiol. Surg. 11 , 1623–1636. 10.1007/s11548-016-1468-2 [ PubMed ] [ CrossRef ] [ Google Scholar ]

More From Forbes

The whole world in your hand: major advances in haptic technology.

• Share to Facebook
• Share to Twitter
• Share to Linkedin

The Whole World in Your Hand: Major Advances in Haptic Technology

Recent advances in scientific research may allow robotic prosthetics and virtual reality simulations to be even more effective than before. Researchers in Hong Kong have developed a new, glove-like technology that not only allows users to experience sensations in their hands when interacting with virtual objects but customizes the intensity of sensations according to the sensitivity levels of a person’s nerves.

Haptic technology is defined as technology that relies on computer-induced forces, vibrations, or motions to provide people with an artificial sense of touch. This technology, along with virtual reality, has become more relevant in the medical field within the past decade. Not only do surgeons rely on virtual reality to perform surgeries, but haptic technology can vastly improve the lives of amputees with robotic prosthetics. In recent years, researchers have found that robotic prosthetics with the ability to provide an artificial sense of touch to patients significantly decrease the mental effort required to operate the prosthetic. An artificial sense of touch can also improve patients’ overall ability to control their prosthetics.

While scientists have developed haptic technology for clinical application in the past, previous attempts have often been bulky, inconvenient to use, and have not been customizable for each user.

Now, researchers have developed a new iteration of haptic technology that involves an ultrathin, glove-like technology called WeTac. WeTac contains several electrodes throughout the glove structure and provides electrical feedback to users to induce sensations of touch throughout their hands. This technology not only has the potential to improve the outcomes of robotic surgeries but is a significant development in haptic technology that could also be applied to those who are disabled and using robotic prosthetics.

WeTac is an ultrathin, glove-like haptic technology.

Best Travel Insurance Companies

Best covid-19 travel insurance plans.

The first challenge of creating the WeTac was to come up with a design that could emulate the dynamic and variable sensations that people feel when using their hands.

Consider the experience of shaking a person’s hand. You may only feel the handshake in certain areas of your palm or fingers. These areas of contact may change as you go through the movements of shaking their hand. The pressure of your grip or their grip may alter as well. Beyond this, some regions of our hands are innately more sensitive to touch than others. The feelings we experience in our hands are very dynamic even for something as basic as shaking somebody else’s hand.

The goal of Yao et al. was to design a haptic glove that could capture these dynamic sensations when users interacted with virtual objects. To do so, the WeTac was designed with 32 electrodes spanning the surface of the palm and the fingers. This would allow the researchers to adjust the intensity of electrical signals at 32 different points in the hand and allow them to induce more accurate sensations of touch throughout the hand.

The use of electrodes to create artificial sensations also enabled Yao et al. to create the WeTac in an incredibly lightweight form. Electrodes have previously been used in thin, wearable devices and can sit directly on the skin without causing any irritation. This makes them optimal for creating a lightweight and convenient device.

Electrodes are dispersed throughout the WeTac glove.

By using the electrodes, Yao et al. could induce electrical currents throughout the hand. The idea was that these electrical currents would activate the nerves in a person’s hand, effectively inducing sensations of touch that a person might feel from interacting with a physical object. Yao et al. designed the WeTac so that electrical currents could be produced by a blue control unit that would attach to the user’s wrist. This control unit would have wireless capabilities and could be controlled with a phone or computer. This would allow WeTac users to move freely.

Electrodes throughout the WeTac induce sensations in the hand wirelessly.

To test the device, the first step for Yao et al. was to optimize WeTac’s electrical stimulation settings according to each participant. The sensitivity levels of people’s hands can differ across populations. For example, men typically display decreased sensitivity to touch compared to women. Older people also display decreased sensitivity compared to younger people. To customize the device, Yao et al. measured the average electrical stimulation threshold for each participant and across each of the 32 electrodes in their hands.

As expected, on average, women had lower thresholds for electrical stimulation than men. Younger individuals also displayed lower thresholds. The exception to this pattern was that women who exhibited a greater number of calluses on their hands due to their jobs had higher thresholds. In other words, Yao et al. found that beyond gender and age, hand sensitivity can also differ according to a person’s job or daily activities.

After calibrating the WeTac according to each volunteer’s sensitivity levels, Yao et al. were ready to test the WeTac in virtual reality simulations. The first simulation involved participants slowly grabbing a virtual tennis ball and a virtual cactus. This simulation would allow the researchers to determine that the WeTac could produce different sensations according to the texture of a stationary virtual object. After running the simulations, the team found that the tennis ball could induce gentle touch sensations, while the cactus would induce a spike sensation that was slightly painful or uncomfortable.

The researchers tested the haptic technology with multiple simulations. Two of which, involved ... [+] interacting with a tennis ball or cactus.

The researchers also tested a simulation where a virtual mouse and pieces of cheese appeared on the participants' hands. The participant would then report the sensations they felt as the virtual mouse traveled across their hand to eat each piece of cheese. This allowed the researchers to determine that the WeTac could also effectively induce sensations of touch for a moving, dynamic object.

Overall, this study demonstrates significant progress in haptic technology. As the WeTac and other lightweight haptic feedback devices continue to be developed, we may begin to see more complex virtual reality technology and robotic prosthetics that utilize haptic feedback and can improve the outcomes of remote/robotic surgeries as well as the lives of amputees.

• Editorial Standards
• Reprints & Permissions

Join The Conversation

One Community. Many Voices. Create a free account to share your thoughts. 

Forbes Community Guidelines

Our community is about connecting people through open and thoughtful conversations. We want our readers to share their views and exchange ideas and facts in a safe space.

In order to do so, please follow the posting rules in our site's  Terms of Service.   We've summarized some of those key rules below. Simply put, keep it civil.

Your post will be rejected if we notice that it seems to contain:

• False or intentionally out-of-context or misleading information
• Insults, profanity, incoherent, obscene or inflammatory language or threats of any kind
• Attacks on the identity of other commenters or the article's author
• Content that otherwise violates our site's  terms.

User accounts will be blocked if we notice or believe that users are engaged in:

• Continuous attempts to re-post comments that have been previously moderated/rejected
• Racist, sexist, homophobic or other discriminatory comments
• Attempts or tactics that put the site security at risk
• Actions that otherwise violate our site's  terms.

So, how can you be a power user?

• Stay on topic and share your insights
• Feel free to be clear and thoughtful to get your point across
• ‘Like’ or ‘Dislike’ to show your point of view.
• Protect your community.
• Use the report tool to alert us when someone breaks the rules.

Thanks for reading our community guidelines. Please read the full list of posting rules found in our site's  Terms of Service.

• Corpus ID: 11668405

Haptic Technology :-A Comprehensive Review of its Applications and Future Prospects

• Anupam Alur , Pratik Shrivastav , A. Jumde
• Published 2014
• Computer Science, Psychology

Figures from this paper

54 Citations

Electronics haptics, it used to be all about resonant frequency, design and rapid construction of a cost-effective virtual haptic device, the effect of person and thing orientation on the experience of haptics, a review of virtual touch interaction systems, users' perspectives on haptic technology use in hand rehabilitation, developing a novel haptic device for non-rigid computer graphic objects utilizing unmanned aerial vehicles, a low-cost multi-modal auditory-visual-tactile framework for remote touch, real-time feedback for children's pre-writing activities using haptic technology.

• Highly Influenced

“Going on a sensory adventure, a touchy subject?”: investigating haptic technology and consumer adventure orientation

Inviting robot touch (by design), 13 references, design of a haptic device for teleoperation and virtual reality systems, haptibasic: learning basic concepts of a haptic technology through edutainment games, algorithms for haptic rendering of 3d objects, wireless haptic rendering for mobile platforms, the feeling of color: a haptic feedback device for the visually disabled, fuzzy haptic augmentation for telerobotic stair climbing, a tangible goal for 3d modeling, simulation of patient specific cervical hip fracture surgery with a volume haptic interface, haptic feedback for medical applications, a survey, related papers.

Showing 1 through 3 of 0 Related Papers

Showing papers on "Haptic technology published in 2023"

Citation Count

6  citations

5  citations

4  citations

3  citations

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Journal Menu

• Applied Sciences Home
• Aims & Scope
• Editorial Board
• Reviewer Board
• Topical Advisory Panel
• Instructions for Authors
• Special Issues
• Sections & Collections
• Article Processing Charge
• Indexing & Archiving
• Editor’s Choice Articles
• Most Cited & Viewed
• Journal Statistics
• Journal History
• Journal Awards
• Society Collaborations
• Conferences
• Editorial Office

Journal Browser

• arrow_forward_ios Forthcoming issue arrow_forward_ios Current issue
• Vol. 14 (2024)
• Vol. 13 (2023)
• Vol. 12 (2022)
• Vol. 11 (2021)
• Vol. 10 (2020)
• Vol. 9 (2019)
• Vol. 8 (2018)
• Vol. 7 (2017)
• Vol. 6 (2016)
• Vol. 5 (2015)
• Vol. 4 (2014)
• Vol. 3 (2013)
• Vol. 2 (2012)
• Vol. 1 (2011)

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

Haptics: Technology and Applications — 2021

• Print Special Issue Flyer
• Special Issue Editors

Special Issue Information

• Published Papers

A special issue of Applied Sciences (ISSN 2076-3417). This special issue belongs to the section " Mechanical Engineering ".

Deadline for manuscript submissions: closed (31 October 2021) | Viewed by 38823 The Deadline for manuscript submissions is 31 October 2021, but we still accept papers in case of new submissions. Related Special Issue: Haptics: Technology and Applications

Share This Special Issue

Special issue editor.

Dear Colleagues,

This Special Issue seeks papers which examine some of the latest advances with respect to haptic actuators, haptic rendering, haptic applications in virtual reality/augmented reality, haptic applications in virtual education/training, and all aspects of haptics, including neuroscience, psychophysics, perception, and interactions. This Special Issue also welcomes papers related to medical and surgical simulations, skills training, rehabilitation robotics, collaborative human–robot interactions, communication, and haptic feedback for design and the arts.

Prof. Dr. Sang-Youn Kim Guest Editor

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website . Once you are registered, click here to go to the submission form . Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Applied Sciences is an international peer-reviewed open access semimonthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2400 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

• haptic/vibrotactile actuator
• psychophysics and perception
• multimodal interaction
• virtual reality
• haptic interfaces design
• haptic rendering and modeling

Published Papers (14 papers)

Jump to: Review

Jump to: Research

Further Information

Mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

Suggestions or feedback?

MIT News | Massachusetts Institute of Technology

• Machine learning
• Social justice
• Black holes
• Classes and programs

Departments

• Aeronautics and Astronautics
• Brain and Cognitive Sciences
• Architecture
• Political Science
• Mechanical Engineering

Centers, Labs, & Programs

• Abdul Latif Jameel Poverty Action Lab (J-PAL)
• Picower Institute for Learning and Memory
• Lincoln Laboratory
• School of Architecture + Planning
• School of Engineering
• School of Humanities, Arts, and Social Sciences
• Sloan School of Management
• School of Science
• MIT Schwarzman College of Computing

Study models how ketamine’s molecular action leads to its effects on the brain

Press contact :.

Previous image Next image

Ketamine, a World Health Organization Essential Medicine, is widely used at varying doses for sedation, pain control, general anesthesia, and as a therapy for treatment-resistant depression. While scientists know its target in brain cells and have observed how it affects brain-wide activity, they haven’t known entirely how the two are connected. A new study by a research team spanning four Boston-area institutions uses computational modeling of previously unappreciated physiological details to fill that gap and offer new insights into how ketamine works.

“This modeling work has helped decipher likely mechanisms through which ketamine produces altered arousal states as well as its therapeutic benefits for treating depression,” says co-senior author Emery N. Brown , the Edward Hood Taplin Professor of Computational Neuroscience and Medical Engineering at The Picower Institute for Learning and Memory at MIT, as well as an anesthesiologist at Massachusetts General Hospital and a professor at Harvard Medical School.

The researchers from MIT, Boston University (BU), MGH, and Harvard University say the predictions of their model, published May 20 in Proceedings of the National Academy of Sciences , could help physicians make better use of the drug.

“When physicians understand what's mechanistically happening when they administer a drug, they can possibly leverage that mechanism and manipulate it,” says study lead author Elie Adam , a research scientist at MIT who will soon join the Harvard Medical School faculty and launch a lab at MGH. “They gain a sense of how to enhance the good effects of the drug and how to mitigate the bad ones.”

Blocking the door

The core advance of the study involved biophysically modeling what happens when ketamine blocks the “NMDA” receptors in the brain’s cortex — the outer layer where key functions such as sensory processing and cognition take place. Blocking the NMDA receptors modulates the release of excitatory neurotransmitter glutamate.

When the neuronal channels (or doorways) regulated by the NMDA receptors open, they typically close slowly (like a doorway with a hydraulic closer that keeps it from slamming), allowing ions to go in and out of neurons, thereby regulating their electrical properties, Adam says. But, the channels of the receptor can be blocked by a molecule. Blocking by magnesium helps to naturally regulate ion flow. Ketamine, however, is an especially effective blocker.

Blocking slows the voltage build-up across the neuron’s membrane that eventually leads a neuron to “spike,” or send an electrochemical message to other neurons. The NMDA doorway becomes unblocked when the voltage gets high. This interdependence between voltage, spiking, and blocking can equip NMDA receptors with faster activity than its slow closing speed might suggest. The team’s model goes further than ones before by representing how ketamine’s blocking and unblocking affect neural activity.

“Physiological details that are usually ignored can sometimes be central to understanding cognitive phenomena,” says co-corresponding author Nancy Kopell , a professor of mathematics at BU. “The dynamics of NMDA receptors have more impact on network dynamics than has previously been appreciated.”

With their model, the scientists simulated how different doses of ketamine affecting NMDA receptors would alter the activity of a model brain network. The simulated network included key neuron types found in the cortex: one excitatory type and two inhibitory types. It distinguishes between “tonic” interneurons that tamp down network activity and “phasic” interneurons that react more to excitatory neurons.

The team’s simulations successfully recapitulated the real brain waves that have been measured via EEG electrodes on the scalp of a human volunteer who received various ketamine doses and the neural spiking that has been measured in similarly treated animals that had implanted electrode arrays. At low doses, ketamine increased brain wave power in the fast gamma frequency range (30-40 Hz). At the higher doses that cause unconsciousness, those gamma waves became periodically interrupted by “down” states where only very slow frequency delta waves occur. This repeated disruption of the higher frequency waves is what can disrupt communication across the cortex enough to disrupt consciousness.

Previous item Next item

But how? Key findings

Importantly, through simulations, they explained several key mechanisms in the network that would produce exactly these dynamics.

The first prediction is that ketamine can disinhibit network activity by shutting down certain inhibitory interneurons. The modeling shows that natural blocking and unblocking kinetics of NMDA-receptors can let in a small current when neurons are not spiking. Many neurons in the network that are at the right level of excitation would rely on this current to spontaneously spike. But when ketamine impairs the kinetics of the NMDA receptors, it quenches that current, leaving these neurons suppressed. In the model, while ketamine equally impairs all neurons, it is the tonic inhibitory neurons that get shut down because they happen to be at that level of excitation. This releases other neurons, excitatory or inhibitory, from their inhibition allowing them to spike vigorously and leading to ketamine’s excited brain state. The network’s increased excitation can then enable quick unblocking (and reblocking) of the neurons’ NMDA receptors, causing bursts of spiking.

Another prediction is that these bursts become synchronized into the gamma frequency waves seen with ketamine. How? The team found that the phasic inhibitory interneurons become stimulated by lots of input of the neurotransmitter glutamate from the excitatory neurons and vigorously spike, or fire. When they do, they send an inhibitory signal of the neurotransmitter GABA to the excitatory neurons that squelches the excitatory firing, almost like a kindergarten teacher calming down a whole classroom of excited children. That stop signal, which reaches all the excitatory neurons simultaneously, only lasts so long, ends up synchronizing their activity, producing a coordinated gamma brain wave.

“The finding that an individual synaptic receptor (NMDA) can produce gamma oscillations and that these gamma oscillations can influence network-level gamma was unexpected,” says co-corresponding author Michelle McCarthy , a research assistant professor of math at BU. “This was found only by using a detailed physiological model of the NMDA receptor. This level of physiological detail revealed a gamma time scale not usually associated with an NMDA receptor.”

So what about the periodic down states that emerge at higher, unconsciousness-inducing ketamine doses? In the simulation, the gamma-frequency activity of the excitatory neurons can’t be sustained for too long by the impaired NMDA-receptor kinetics. The excitatory neurons essentially become exhausted under GABA inhibition from the phasic interneurons. That produces the down state. But then, after they have stopped sending glutamate to the phasic interneurons, those cells stop producing their inhibitory GABA signals. That enables the excitatory neurons to recover, starting a cycle anew.

Antidepressant connection?

The model makes another prediction that might help explain how ketamine exerts its antidepressant effects. It suggests that the increased gamma activity of ketamine could entrain gamma activity among neurons expressing a peptide called VIP. This peptide has been found to have health-promoting effects, such as reducing inflammation, that last much longer than ketamine’s effects on NMDA receptors. The research team proposes that the entrainment of these neurons under ketamine could increase the release of the beneficial peptide, as observed when these cells are stimulated in experiments. This also hints at therapeutic features of ketamine that may go beyond antidepressant effects. The research team acknowledges, however, that this connection is speculative and awaits specific experimental validation.

“The understanding that the subcellular details of the NMDA receptor can lead to increased gamma oscillations was the basis for a new theory about how ketamine may work for treating depression,” Kopell says.

Additional co-authors of the study are Marek Kowalski, Oluwaseun Akeju, and Earl K. Miller.

The work was supported by the JPB Foundation; The Picower Institute for Learning and Memory; The Simons Center for The Social Brain; the National Institutes of Health; George J. Elbaum ’59, SM ’63, PhD ’67; Mimi Jensen; Diane B. Greene SM ’78; Mendel Rosenblum; Bill Swanson; and annual donors to the Anesthesia Initiative Fund.

Share this news article on:

Related links.

• Institute for Medical Engineering and Science
• The Picower Institute for Learning and Memory
• Department of Brain and Cognitive Sciences

Related Topics

• Pharmaceuticals
• Neuroscience
• Brain and cognitive sciences
• Health sciences and technology
• Health care
• Picower Institute
• Institute for Medical Engineering and Science (IMES)

Related Articles

Anesthesia technology precisely controls unconsciousness in animal tests

Study finds tracking brain waves could reduce post-op complications

3 Questions: Emery Brown on improving anesthesia with neuroscience

Statistical model defines ketamine anesthesia’s effects on the brain

More mit news.

New computer vision method helps speed up screening of electronic materials

Read full story →

MIT Faculty Founder Initiative announces three winners of entrepreneurship awards

How a quantum scientist, a nurse, and an economist are joining the fight against global poverty

Catalyst Symposium helps lower “activation barriers” for rising biology researchers

Protein study could help researchers develop new antibiotics

Through econometrics, Isaiah Andrews is making research more robust

• More news on MIT News homepage →

Massachusetts Institute of Technology 77 Massachusetts Avenue, Cambridge, MA, USA

• Map (opens in new window)
• Events (opens in new window)
• People (opens in new window)
• Careers (opens in new window)
• Accessibility
• Social Media Hub
• MIT on Facebook
• MIT on YouTube
• MIT on Instagram

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

To revisit this article, visit My Profile, then View saved stories .

• Backchannel
• Newsletters
• WIRED Insider
• WIRED Consulting

Will Knight

OpenAI Offers a Peek Inside the Guts of ChatGPT

ChatGPT developer OpenAI’s approach to building artificial intelligence came under fire this week from former employees who accuse the company of taking unnecessary risks with technology that could become harmful.

Today, OpenAI released a new research paper apparently aimed at showing it is serious about tackling AI risk by making its models more explainable. In the paper , researchers from the company lay out a way to peer inside the AI model that powers ChatGPT. They devise a method of identifying how the model stores certain concepts—including those that might cause an AI system to misbehave.

Although the research makes OpenAI’s work on keeping AI in check more visible, it also highlights recent turmoil at the company. The new research was performed by the recently disbanded “superalignment” team at OpenAI that was dedicated to studying the technology’s long-term risks.

The former group’s coleads, Ilya Sutskever and Jan Leike—both of whom have left OpenAI —are named as coauthors. Sutskever, a cofounder of OpenAI and formerly chief scientist, was among the board members who voted to fire CEO Sam Altman last November, triggering a chaotic few days that culminated in Altman’s return as leader.

ChatGPT is powered by a family of so-called large language models called GPT, based on an approach to machine learning known as artificial neural networks. These mathematical networks have shown great power to learn useful tasks by analyzing example data, but their workings cannot be easily scrutinized as conventional computer programs can. The complex interplay between the layers of “neurons” within an artificial neural network makes reverse engineering why a system like ChatGPT came up with a particular response hugely challenging.

“Unlike with most human creations, we don’t really understand the inner workings of neural networks,” the researchers behind the work wrote in an accompanying blog post . Some prominent AI researchers believe that the most powerful AI models, including ChatGPT, could perhaps be used to design chemical or biological weapons and coordinate cyberattacks. A longer-term concern is that AI models may choose to hide information or act in harmful ways in order to achieve their goals.

OpenAI’s new paper outlines a technique that lessens the mystery a little, by identifying patterns that represent specific concepts inside a machine learning system with help from an additional machine learning model. The key innovation is in refining the network used to peer inside the system of interest by identifying concepts, to make it more efficient.

OpenAI proved out the approach by identifying patterns that represent concepts inside GPT-4, one of its largest AI models. The company released code related to the interpretability work, as well as a visualization tool that can be used to see how words in different sentences activate concepts, including profanity and erotic content, in GPT-4 and another model. Knowing how a model represents certain concepts could be a step toward being able to dial down those associated with unwanted behavior, to keep an AI system on the rails. It could also make it possible to tune an AI system to favor certain topics or ideas.

By Matt Burgess

By David Robson

By Joseph Cox

By Boone Ashworth

Even though LLMs defy easy interrogation, a growing body of research suggests they can be poked and prodded in ways that reveal useful information. Anthropic, an OpenAI competitor backed by Amazon and Google, published similar work on AI interpretability last month. To demonstrate how the behavior of AI systems might be tuned, the company's researchers created a chatbot obsessed with San Francisco's Golden Gate Bridge . And simply asking an LLM to explain its reasoning can sometimes yield insights .

“It’s exciting progress,” says David Bau , a professor at Northeastern University who works on AI explainability, of the new OpenAI research. “As a field, we need to be learning how to understand and scrutinize these large models much better.”

Bau says the OpenAI team’s main innovation is in showing a more efficient way to configure a small neural network that can be used to understand the components of a larger one. But he also notes that the technique needs to be refined to make it more reliable. “There’s still a lot of work ahead in using these methods to create fully understandable explanations,” Bau says.

Bau is part of a US government-funded effort called the National Deep Inference Fabric , which will make cloud computing resources available to academic researchers so that they too can probe especially powerful AI models. “We need to figure out how we can enable scientists to do this work even if they are not working at these large companies,” he says.

OpenAI’s researchers acknowledge in their paper that further work needs to be done to improve their method, but also say they hope it will lead to practical ways to control AI models. “We hope that one day, interpretability can provide us with new ways to reason about model safety and robustness, and significantly increase our trust in powerful AI models by giving strong assurances about their behavior,” they write.

You Might Also Like …

Navigate election season with our WIRED Politics Lab newsletter and podcast

Don’t think breakdancing is an Olympic sport ? The world champ agrees (kinda)

How researchers cracked an 11-year-old password to a $3M crypto wallet

The uncanny rise of the world’s first AI beauty pageant

Give your back a break: Here are the best office chairs we’ve tested

Steven Levy

Paresh Dave